design scheduling mechanism

[zen-xu]

Currently, there will be two components, a controller and a scheduler.
First of all, we need to list the responsibilities of each component.

Controller

• Manage the lifecycle of habitat crds.
• Update the status of the job.
• Create/reclaim pods based on the status of the job.
• Handle events from jobs/workflows.

Scheduler

• Assign the pod to a suitable node.
• Evict specific pods when preemption is required.
• Reserve resources.
• Balance cluster utilization and equity.
• Provide rich observability mechanisms for acquiring the current state of the scheduler, Jobs, and Pods (sending events, Prometheus, logs).
• High error tolerance, avoiding task failures(due to non-user factors) that requires users to resubmit the job.