"Inside the Worker: Building a Resilient AI Execution Platform"
Inside the brain of a under-rated worker
This is Part 5 of the Beyond FastAPI series
In the previous article, The Listener Loop: Turning Jobs into Work, we built the scheduling layer responsible for monitoring the queue, enforcing concurrency limits, and spawning isolated worker processes.
In this article we'll step inside one of those workers and explore what happens after the Queue Manager decides a job should run.
Previously...
So far our architecture looks like this:
Client
│
▼
FastAPI
│
▼
MongoDB Job Queue
│
▼
Queue Manager
│
▼
Spawn Worker Process
The Queue Manager continuously polls the database for pending jobs, claims work atomically, checks whether there's available capacity, and finally launches a completely new operating system process for every accepted job.
At this point, the API request has already completed.
The client has received a Job ID.
The Queue Manager has finished scheduling.
Everything from here happens independently inside a worker process.
The obvious question is...
What exactly happens inside that worker?
The Worker is More Than "Some Code"
The real reasons to go for worker based design are architectural.
We want every job to execute inside its own isolated runtime.
That gives us:
independent memory
independent event loop
isolated failures
individual lifecycle management
safe termination
clean resource ownership
Instead of thinking of a worker as a function running elsewhere, think of it as a miniature application dedicated to executing a single job.
Crossing the Process Boundary
The Queue Manager creates workers using Python's multiprocessing module.
proc = Process(
target=worker_entry,
args=(
job_id,
payload,
DBFactory,
PIPELINE,
logger,
),
daemon=False,
)
proc.start()
When proc.start() is called.
A completely new Python interpreter is launched.
That new interpreter starts with its own memory space, its own global variables, its own event loop, and its own execution context.
Conceptually, this is what happens:
Queue Manager
│
▼
Create Process
│
▼
New Python Interpreter
│
▼
Import Modules
│
▼
Run worker_entry()
│
▼
Execute Job
Notice that the worker shares nothing with the Queue Manager except the arguments passed during creation.
There is no shared application state.
No shared database objects.
No shared event loop.
Everything begins from a clean slate.
Why Processes Instead of Threads?
This is one of the most important architectural decisions in the entire platform.
Many engineers immediately ask:
Why not simply create a thread?
The answer has very little to do with performance.
It has everything to do with isolation.
| Threads | Processes |
|---|---|
| Share memory | Independent memory |
| Shared global state | Completely isolated |
| Harder to terminate safely | Can terminate individually |
| Errors may affect application state | Failures remain isolated |
| Good for lightweight concurrent tasks | Ideal for long-running AI workloads |
Imagine one analysis pipeline suddenly starts consuming several gigabytes of memory.
Or enters an infinite loop.
Or launches a subprocess that never exits.
With threads, those failures happen inside your application's own process.
With worker processes, the Queue Manager simply terminates the offending worker without affecting any other running jobs.
That's an enormous operational advantage.
💡 Engineering Insight
The worker process is not a performance optimisation.
It's an isolation boundary.
Every job becomes independently manageable, observable and disposable.
Every Worker Starts at the Same Entry Point
Every worker begins execution inside a single function.
def worker_entry(
job_id,
payload,
DBFactory,
PIPELINE,
logger=None,
):
You can think of worker_entry() as the worker's own main() function.
Every worker follows exactly the same lifecycle.
Worker Starts
↓
Configure Logging
↓
Create Event Loop
↓
Connect Database
↓
Load Pipeline
↓
Run Analysis
↓
Report Progress
↓
Cleanup
↓
Exit
Having a single entry point makes every worker predictable.
Regardless of what type of analysis is executed in the future, the execution lifecycle remains identical.
Every Worker Has Its Own Event Loop
Inside worker_entry() we eventually reach this line:
asyncio.run(main())
This creates a completely new asyncio event loop inside the worker process.
The FastAPI server already has its own event loop.
The Queue Manager also runs inside that application's event loop.
Once the worker starts, it creates an entirely separate event loop dedicated to executing exactly one job.
That means asynchronous operations inside the worker never interfere with the web server.
The API can continue accepting new requests while workers independently perform long-running AI workloads.
Loading the Analysis Pipeline Dynamically
One of my favourite design decisions in this project is that the worker doesn't know anything about thematic analysis.
Instead, it receives the pipeline to execute as configuration.
analysis = _load_analysis(
PIPELINE,
constructor_args
)
The helper responsible for loading the pipeline looks roughly like this:
module = importlib.import_module(module_path)
cls = getattr(module, class_name)
return cls(**constructor_args)
Rather than importing a concrete implementation directly, the worker loads whatever class is configured at runtime.
Conceptually:
Configuration
↓
Pipeline Class Name
↓
importlib
↓
Instantiate Class
↓
Run Pipeline
This makes the worker completely reusable.
Today it executes a thematic analysis pipeline.
Tomorrow the exact same worker could execute:
document processing
OCR pipelines
image classification
audio transcription
video analysis
ETL jobs
without changing a single line inside the worker itself.
💡 Engineering Insight
The worker knows how to execute work, but it deliberately knows nothing about what the work actually is.
This is a classic plugin architecture.
The execution platform remains stable while individual analysis pipelines evolve independently.
Why Dynamic Loading Matters
Imagine if the worker looked like this:
analysis = ThematicAnalysisPipeline(...)
Every time a new pipeline is introduced, we'd need to modify the worker itself.
Instead, our worker depends only on an interface.
PipeLineInterface
Any implementation satisfying that interface can be loaded dynamically.
That means the orchestration layer remains generic while new AI workflows can be introduced simply by changing configuration.
This separation of concerns is one of the key reasons the platform is extensible.
Architecture So Far
At this point, our execution platform looks like this:
Client
│
▼
FastAPI
│
▼
MongoDB Queue
│
▼
Queue Manager
│
▼
Spawn Worker
│
▼
worker_entry()
│
▼
Dynamic Pipeline Loader
│
▼
Analysis Pipeline
The Queue Manager doesn't know what kind of work is being executed.
The Worker doesn't know which specific pipeline it will load.
And the Pipeline doesn't know anything about FastAPI, queues, or scheduling.
Each component has exactly one responsibility.
Coming Up Next
So far we've focused on how a worker starts.
In the next section we'll explore what happens after the pipeline has been loaded.
