Building the Orchestration Layer: Designing a Thin FastAPI API for Long-Running AI Workloads
Accepting requests elegantly
Part 2 of the "Beyond FastAPI" series.
In the previous article, we explored why long-running AI workloads don't belong inside a traditional request-response cycle. We introduced an architecture that decouples request handling from job execution, allowing our API to remain responsive while computationally intensive analysis runs asynchronously in the background.
In this article, we'll build the orchestration layer of that architecture.
Rather than focusing on FastAPI itself, we'll look at how the application is composed, how dependencies are injected, and why the API layer should remain intentionally thin. We'll also see how background services such as our queue manager are integrated into the application's lifecycle without leaking infrastructure concerns into the web layer.
The Role of the Orchestration Layer
When developers first begin building APIs, it's common to let the web framework become the centre of the application.
A request arrives.
The endpoint validates input.
The endpoint queries the database.
The endpoint performs business logic.
The endpoint calls external APIs.
The endpoint returns the response.
This works remarkably well—until the business logic becomes expensive.
Our thematic analysis pipeline isn't a simple CRUD operation. A single request can involve fetching thousands of survey responses, cleaning and preprocessing text, generating embeddings, orchestrating multiple LLM calls, aggregating intermediate results, and finally persisting the generated insights.
That's not the responsibility of an HTTP endpoint.
Instead, our FastAPI application has a much simpler responsibility:
Accept work. Persist work. Return immediately.
Everything else belongs elsewhere.
Key Design Principle
HTTP APIs should be responsible for communication, not computation.
Overall Architecture
Notice something important.
The API is no longer responsible for executing the analysis.
Instead, it acts as the entry point into a much larger execution platform.
This separation of concerns has several advantages:
API requests complete in milliseconds instead of minutes.
Long-running analyses survive independently of HTTP connections.
Workers can scale independently of the API.
Failures are isolated to individual jobs.
Progress can be tracked asynchronously.
FastAPI simply becomes the gateway into the system rather than the system itself.
The Composition Root
One of the most overlooked aspects of application architecture is where objects are created.
Every application has a point where its major components are instantiated and connected together. This is commonly referred to as the Composition Root.
In our project, that responsibility belongs to the application's entry point.
def run():
myDB = _make_db_impl()
DBFactory = _make_db_impl
app = create_app(
DBFactory
)
uvicorn.run(app, host="0.0.0.0", port=8000)
At first glance this looks almost trivial.
In reality, this function defines the entire dependency graph of the application.
Before FastAPI ever starts accepting requests, we create:
the database implementation
the worker database factory
the queue manager
the web server
the FastAPI application
Only once every component has been assembled do we hand control over to Uvicorn.
This may seem like a small design choice, but it has significant architectural benefits.
The application has exactly one place responsible for wiring dependencies together. No class reaches into another module to construct its own dependencies. Nothing relies on global state. Every component receives exactly what it needs through its constructor.
This makes the application significantly easier to understand, maintain, and test.
Why Dependency Injection Matters
Let's imagine we wrote our queue manager like this:
class QueueManager:
def __init__(self):
self.db = MongoJobsDB(...)
This works.
But now the queue manager is permanently coupled to MongoDB.
Testing becomes difficult.
Switching databases becomes difficult.
Replacing MongoDB with Redis, PostgreSQL, or a mock implementation becomes impossible without modifying the class itself.
Instead, our implementation receives its dependencies from the outside.
class QueueManager:
def __init__(
self,
DBFactory,
):
self.db = DBFactory()
This small change fundamentally alters the architecture.
The queue manager no longer knows or cares where the database comes from. It simply depends on the behaviour defined by the interface. This is an example of the Dependency Inversion Principle, one of the SOLID design principles.
Rather than depending on concrete implementations, high-level components depend on abstractions.
As a result:
business logic becomes independent of infrastructure
testing becomes straightforward
implementations become interchangeable
components become easier to reuse
QueueManager
│▼DBInterface▲│┌─────────────┴──────────────┐▼ ▼MongoJobsDB MockDatabase
Although our current implementation only uses MongoDB, the queue manager itself has no knowledge of that decision.
That's exactly where we want to be.
Building a Thin API Layer
One of the easiest ways for an application to become difficult to maintain is to allow the web framework to absorb business logic. Over time, the web layer becomes tightly coupled to every other component in the system, making it increasingly difficult to test, extend, or replace.
Instead, the FastAPI application in this project acts purely as a transport layer. Its responsibilities are deliberately limited to:
Receiving HTTP requests
Validating incoming data
Delegating work to the orchestration layer
Returning an appropriate HTTP response
Everything else belongs elsewhere.
This philosophy is reflected in the WebServer class.
class WebServer:
def __init__(self, db: DBInterface, queue_mgr: QueueManager):
self.db = db
self.queue_mgr = queue_mgr
self.app = FastAPI(
title="Async Queue Server",
lifespan=self._lifespan,
)
self.app.add_api_route(
"/enqueue",
self.enqueue_post,
methods=["POST"],
response_model=EnqueueResponse,
)
self.app.add_api_route(
"/jobs/{job_id}",
self.get_job,
methods=["GET"],
)
There are a couple of interesting design decisions here.
First, notice that the WebServer doesn't construct its own dependencies.
Both the database implementation and the queue manager are injected into the constructor, continuing the dependency injection pattern introduced earlier.
Second, the class doesn't contain any scheduling logic.
It doesn't spawn workers.
It doesn't communicate with LLMs.
It doesn't poll queues.
It doesn't even know how jobs are executed.
Its only responsibility is exposing a clean HTTP interface over the orchestration platform.
Why Register Routes Programmatically?
If you've used FastAPI before, you're probably familiar with decorators.
@app.post("/enqueue")
async def enqueue():
...
Decorators are concise and work well for smaller applications.
However, once the application begins to grow, I generally prefer encapsulating related endpoints inside classes and registering routes programmatically.
self.app.add_api_route(
"/enqueue",
self.enqueue_post,
methods=["POST"],
)
This provides several advantages:
The web server becomes a self-contained component.
Dependencies are injected through the constructor.
Routes are registered in one place.
And the application structure more closely resembles the architecture itself.
Instead of scattered decorators across multiple modules, the routing layer becomes an object with a clearly defined responsibility. This is largely a matter of preference, but I've found it scales better as applications grow beyond a handful of endpoints.
Managing Background Services with FastAPI Lifespan
One of the more elegant features introduced in modern versions of FastAPI is the lifespan protocol.
Rather than relying on startup events or background threads, FastAPI allows applications to define asynchronous startup and shutdown logic using an asynchronous context manager.
@asynccontextmanager
async def _lifespan(self, app: FastAPI):
await self.queue_mgr.start()
try:
yield
finally:
await self.queue_mgr.stop()
It defines the lifecycle of the entire orchestration platform. When the application starts:
FastAPI Starts
│
▼
Queue Manager Starts
│
▼
Listener Loop Starts
│
▼
Watchdog Starts
│
▼
API Ready
Likewise, during shutdown:
SIGTERM
│
▼
FastAPI Shutdown
│
▼
QueueManager.stop()
│
▼
Cancel Async Tasks
│
▼
Terminate Workers
│
▼
Exit
Every long-running background component starts together.
Every component stops together.
This avoids one of the most common problems in asynchronous applications—background tasks that continue running after the web server has begun shutting down.
Instead, the entire application behaves as a single coordinated unit.
Here is the comparison illustrated:
Accepting Work Instead of Executing Work
The most important endpoint in the application is /enqueue.
Its implementation is surprisingly small.
async def enqueue_post(self, req: EnqueueRequest):
job_id = await self.db.insert_pending_job(req.payload)
return EnqueueResponse(
id=job_id,
status="pending",
)
That’s all it does: no worker creation, no scheduling, no analysis, no LLM calls, and no expensive computation. The endpoint simply records a pending job and immediately returns a response. It may seem almost too simple until you compare it with the alternative.
Synchronous Design
Client
│
▼
POST /analyse
│
▼
FastAPI
│
▼
Fetch Data
│
▼
Generate Embeddings
│
▼
Call LLM
│
▼
Store Results
│
▼
HTTP Response
The client waits for every stage of the pipeline to complete.
If the analysis takes five minutes, the HTTP request remains open for five minutes.
Now compare that to the asynchronous design.
Asynchronous Design
Client
│
▼
POST /enqueue
│
▼
Insert Pending Job
│
▼
Return Job ID
Milliseconds later, the response has already been returned.
Meanwhile, independently of the HTTP request:
Queue Manager
│
▼
Claim Job
│
▼
Spawn Worker
│
▼
Run Analysis
│
▼
Store Results
The API remains completely free to serve additional clients.
The orchestration platform performs the expensive work independently.
This decoupling is one of the defining characteristics of asynchronous systems.
Tracking Progress Instead of Holding Connections Open
Returning a Job ID fundamentally changes the interaction model between the client and the server, the client receives an acknowledgement that work has been accepted.
POST /enqueue
HTTP/1.1 200 OK
{
"id": "0f9d4b8f",
"status": "pending"
}
The client can now continue using the application while the analysis executes in the background.
Whenever progress is needed, it simply queries another endpoint.
GET /jobs/0f9d4b8f
This pattern has several advantages.
The API never holds long-lived HTTP connections.
Progress can be reported incrementally.
Users can safely refresh the page without interrupting processing.
Clients can poll or subscribe for updates.
Jobs become durable entities rather than transient HTTP requests.
From a user's perspective, the experience is actually better.
Instead of staring at a loading spinner for several minutes, they receive immediate feedback that their request has been accepted and can monitor its progress asynchronously.
Let's continue to design reliable worker processes and queue manager in part 3!
