Anatoly Nikiforov, 2023
The basic unit of concurrency around the Universe is known as Job. The Job — is an MQ message containing Job’s GUID, the Unified Models of CIRs it meant to work on, logs and so on. Jobs are executed concurrently. To handle Jobs and track down their state, the Job Pool introduced.
The Job Pool is a structure that collects all the jobs in two different states. There are running and waiting jobs in the Job Pool.
The main responsibility of Job Pool is to decide whether to instantly execute a particular job coming from MQ or enqueue it keeping it in the local memory (another option is to send it back to MQ, but this requires additional MQ calls over time). An enqueued job then waits for its execution in the Job Pool.
Every time a new job arrives from an MQ into the main process, it being handled two different ways, depending on some conditions known as job execution criteria (JEC). The result of handling a job is either executing it immediately or delaying it by putting into the pool as a job, waiting for conditions are met for execution.
One of the first criteria implemented in Universe is an intersection criterion. It is implemented to solve a problem of simultaneous updates of the same CIRs when it’s hard to reason about the behaviour of an end-cloud infrastructure. In such cases, Universe does not rely on end-cloud infrastructure, instead it uses a job pooling mechanism to resolve it locally. The condition of solving it this: if there is a job already updating some CIR and there is another incoming job from MQ containing the same CIR, then an incoming one is blocked. The blocking of job means that there is an intersection criterion occurred between these jobs, and an incoming one should be placed in the pool, instead of executing it. The blocked jobs are then waiting their execution in the pool until the running jobs matched their intersection criterion are executed.
The figure 1 below demonstrates an intersection criterion.
https://miro.com/app/board/uXjVMhuRI70=/?share_link_id=273664990535
The JEC itself is a boolean function, so it’s easy to extend that logic with additional criteria.
Along with JEC, Job Event Messaging Protocol is a part of the job pooling mechanism. JEMP allows the pool to track the state of jobs it runs concurrently. The main communication channel for JEMP is knows as Event Bus. The communication over Event Bus occurs between the pool and the job executed. The main reason for JEMP is to provide the pool with a knowledge of job execution results.
JEMP currently implements three type of messages known as Events.
Events are reported from the jobs back to the pool through the event bus. If there is no any event reported by some running job for a long time, then such job considered lost.