Tools→Queue Capacity Designer

Queue Capacity Designer

Design YARN queue hierarchies with real-time resource distribution and capacity-scheduler config export

Total Cluster Memory (GB)

Total Cluster Vcores

Capacity %

Max Capacity %

User Limit Factor

parent

Capacity %

Max Capacity %

User Limit Factor

Capacity %

Max Capacity %

User Limit Factor

Capacity %

Max Capacity %

User Limit Factor

Capacity %

Max Capacity %

User Limit Factor

Root Queue Distribution100%

default

production

development

default (40%)

production (40%) [2 sub-queues]

development (20%)

Configuration Output

# Capacity Scheduler Config

yarn.scheduler.capacity.root.queues = default,production,development

yarn.scheduler.capacity.root.default.capacity = 40

yarn.scheduler.capacity.root.default.maximum-capacity = 100

yarn.scheduler.capacity.root.default.user-limit-factor = 1

yarn.scheduler.capacity.root.default.state = RUNNING

yarn.scheduler.capacity.root.production.capacity = 40

yarn.scheduler.capacity.root.production.maximum-capacity = 100

yarn.scheduler.capacity.root.production.user-limit-factor = 1

yarn.scheduler.capacity.root.production.state = RUNNING

yarn.scheduler.capacity.root.production.queues = batch,realtime

yarn.scheduler.capacity.root.production.batch.capacity = 60

yarn.scheduler.capacity.root.production.batch.maximum-capacity = 100

yarn.scheduler.capacity.root.production.batch.user-limit-factor = 1

yarn.scheduler.capacity.root.production.batch.state = RUNNING

yarn.scheduler.capacity.root.production.realtime.capacity = 40

yarn.scheduler.capacity.root.production.realtime.maximum-capacity = 100

yarn.scheduler.capacity.root.production.realtime.user-limit-factor = 1

yarn.scheduler.capacity.root.production.realtime.state = RUNNING

yarn.scheduler.capacity.root.development.capacity = 20

yarn.scheduler.capacity.root.development.maximum-capacity = 50

yarn.scheduler.capacity.root.development.user-limit-factor = 1

yarn.scheduler.capacity.root.development.state = RUNNING

# Resource Allocation

root.default = 960 GB / 176 vcores

root.production = (parent) 960 GB / 176 vcores

root.production.batch = 576 GB / 106 vcores

root.production.realtime = 384 GB / 70 vcores

root.development = 480 GB / 88 vcores

Sanity Checks

7 pass

root children sum to 100%

Total: 100% = 100%

root.default: max >= min

Max 100% >= Min 40%

root.production: max >= min

Max 100% >= Min 40%

root.production children sum to 100%

Total: 100% = 100%

root.production.batch: max >= min

Max 100% >= Min 60%

root.production.realtime: max >= min

Max 100% >= Min 40%

root.development: max >= min

Max 50% >= Min 20%

Master YARN queue management in the CDP Administration course

How YARN Queue Capacity Design Works

The YARN Capacity Scheduler organizes cluster resources into a hierarchical queue tree. Each queue under root receives a guaranteed percentage of the cluster, and can elastically burst up to its maximum capacity when sibling queues are idle.

Queues can have sub-queues to create multi-level hierarchies. A parent queue distributes its allocated resources among its children. For example, a "production" queue with 40% cluster capacity containing "batch" (60%) and "realtime" (40%) sub-queues gives batch 24% and realtime 16% of the total cluster.

Sanity checks validate that sibling capacities sum to 100% at every level, that maximum capacity is never less than guaranteed capacity, and that user-limit-factor values are positive. The generated configuration maps directly to capacity-scheduler.xml properties.