Some basic problems of resource Scheduler

1 Scheduling algorithm

Capacity based, DRF (dominant recourse fairness), label based, etc.

Polymorphic, plug-in, can work with multiple strategies, corresponding to different jobs (priority, job characteristics, service or batch job)

2 Fault tolerance, HA

The scheduler is generally in a central location with SPF issues. can use a similar hdfs,jobtracker ha scenario

1) Active-standby structure, (standby is generally one, can also be multiple)

2) Stateless (state-of-the-storage), using a reliable external shared storage state (Nfs,bookeeper,zookeeper)

3) The form of the state is generally snapshot + read ahead log (image + edits)

4) Fault Detect, the specific plan has many:

A) Active to standby heartbeat, timeout standby take over

b) Active to the database heartbeat, standby timed check active last heart beat time.

c) based on the global lock, the lock can be obtained is active

d) membership list notifications based on

5) switch. Standby restore status from shared storage, VIP switch (or not based on vip,client liberary own retry judgment)

3 The schema of the scheduler and the corresponding scalability

The scheduler is faced with the entire cluster resource and usage, which is a global composite state. The safest architecture is serialization, and the job is a schedule. The general architecture is single-threaded.

1) Monolithic Scheduler

Just one scheduer, for the entire resource.

2) Static partitioning of the scheduler

static resource partition, each scheduler responsible for a partition resource scheduling, so that each scheduler parallel work, no conflict with each other.

3) Two-tier scheduling architecture

A central recourse allocator dynamically assigns resources to each scheduler, and the dynamic of scheme two. With a resource lock implementation, a scheduler has a resource that is the lock that owns the resource.

Disadvantages:

A) deadlock problem.

b) Each scheduler can see only a subset of resources, affecting the effectiveness of the scheduling algorithm

c) A lower concurrency level

Note: yarn, although formally like a double-layered architecture, is inherently monolithic, because the second-tier applicationmaster scheduling is essentially job management, not resource management.

4) Share State Scheduler

Each scheduler is oriented towards the entire resource, based on an optimistic concurrency mechanism (post-concurrency). Applies to situations where there is less conflict.