# Building the Orchestration Layer: Designing a Thin FastAPI API for Long-Running AI Workloads * * * > *Part 2 of the "Beyond FastAPI" series.* In the [previous article](https://hashnode.com/edit/cmrm4rj0c00000akjac41alas), we explored why long-running AI workloads don't belong inside a traditional request-response cycle. We introduced an architecture that decouples request handling from job execution, allowing our API to remain responsive while computationally intensive analysis runs asynchronously in the background. In this article, we'll build the **orchestration layer** of that architecture. Rather than focusing on FastAPI itself, we'll look at how the application is composed, how dependencies are injected, and why the API layer should remain intentionally thin. We'll also see how background services such as our queue manager are integrated into the application's lifecycle without leaking infrastructure concerns into the web layer. * * * ## The Role of the Orchestration Layer When developers first begin building APIs, it's common to let the web framework become the centre of the application. A request arrives. The endpoint validates input. The endpoint queries the database. The endpoint performs business logic. The endpoint calls external APIs. The endpoint returns the response. This works remarkably well—until the business logic becomes expensive. Our thematic analysis pipeline isn't a simple CRUD operation. A single request can involve fetching thousands of survey responses, cleaning and preprocessing text, generating embeddings, orchestrating multiple LLM calls, aggregating intermediate results, and finally persisting the generated insights. That's not the responsibility of an HTTP endpoint. Instead, our FastAPI application has a much simpler responsibility: > **Accept work. Persist work. Return immediately.** Everything else belongs elsewhere. * * * > **Key Design Principle** > > HTTP APIs should be responsible for communication, not computation. * * * ## Overall Architecture ![](https://cdn.hashnode.com/uploads/covers/6a51387ca8cf56c623690316/4616d43b-a15a-44cf-bdfa-d58885cf2966.png align="center") Notice something important. The API is no longer responsible for executing the analysis. Instead, it acts as the entry point into a much larger execution platform. This separation of concerns has several advantages: * API requests complete in milliseconds instead of minutes. * Long-running analyses survive independently of HTTP connections. * Workers can scale independently of the API. * Failures are isolated to individual jobs. * Progress can be tracked asynchronously. FastAPI simply becomes the gateway into the system rather than the system itself. * * * ### The Composition Root One of the most overlooked aspects of application architecture is **where objects are created**. Every application has a point where its major components are instantiated and connected together. This is commonly referred to as the **Composition Root**. In our project, that responsibility belongs to the application's entry point. ```python def run(): myDB = _make_db_impl() DBFactory = _make_db_impl app = create_app( DBFactory ) uvicorn.run(app, host="0.0.0.0", port=8000) ``` At first glance this looks almost trivial. In reality, this function defines the entire dependency graph of the application. Before FastAPI ever starts accepting requests, we create: * the database implementation * the worker database factory * the queue manager * the web server * the FastAPI application Only once every component has been assembled do we hand control over to Uvicorn. This may seem like a small design choice, but it has significant architectural benefits. The application has exactly one place responsible for wiring dependencies together. No class reaches into another module to construct its own dependencies. Nothing relies on global state. Every component receives exactly what it needs through its constructor. This makes the application significantly easier to understand, maintain, and test. ### Why Dependency Injection Matters Let's imagine we wrote our queue manager like this: ```python class QueueManager: def __init__(self): self.db = MongoJobsDB(...) ``` This works. But now the queue manager is permanently coupled to MongoDB. Testing becomes difficult. Switching databases becomes difficult. Replacing MongoDB with Redis, PostgreSQL, or a mock implementation becomes impossible without modifying the class itself. Instead, our implementation receives its dependencies from the outside. ```python class QueueManager: def __init__( self, DBFactory, ): self.db = DBFactory() ``` This small change fundamentally alters the architecture. The queue manager no longer knows or cares where the database comes from. It simply depends on the behaviour defined by the interface. This is an example of the **Dependency Inversion Principle**, one of the SOLID design principles. Rather than depending on concrete implementations, high-level components depend on abstractions. As a result: * business logic becomes independent of infrastructure * testing becomes straightforward * implementations become interchangeable * components become easier to reuse * QueueManager `│` `▼` `DBInterface` `▲` `│` `┌─────────────┴──────────────┐` `▼ ▼` `MongoJobsDB MockDatabase` Although our current implementation only uses MongoDB, the queue manager itself has no knowledge of that decision. That's exactly where we want to be. ### Building a Thin API Layer One of the easiest ways for an application to become difficult to maintain is to allow the web framework to absorb business logic. Over time, the web layer becomes tightly coupled to every other component in the system, making it increasingly difficult to test, extend, or replace. Instead, the FastAPI application in this project acts purely as a **transport layer**. Its responsibilities are deliberately limited to: * Receiving HTTP requests * Validating incoming data * Delegating work to the orchestration layer * Returning an appropriate HTTP response Everything else belongs elsewhere. This philosophy is reflected in the `WebServer` class. ```python class WebServer: def __init__(self, db: DBInterface, queue_mgr: QueueManager): self.db = db self.queue_mgr = queue_mgr self.app = FastAPI( title="Async Queue Server", lifespan=self._lifespan, ) self.app.add_api_route( "/enqueue", self.enqueue_post, methods=["POST"], response_model=EnqueueResponse, ) self.app.add_api_route( "/jobs/{job_id}", self.get_job, methods=["GET"], ) ``` There are a couple of interesting design decisions here. First, notice that the `WebServer` doesn't construct its own dependencies. Both the database implementation and the queue manager are injected into the constructor, continuing the dependency injection pattern introduced earlier. Second, the class doesn't contain any scheduling logic. It doesn't spawn workers. It doesn't communicate with LLMs. It doesn't poll queues. It doesn't even know how jobs are executed. Its only responsibility is exposing a clean HTTP interface over the orchestration platform. * * * ### Why Register Routes Programmatically? If you've used FastAPI before, you're probably familiar with decorators. ```plaintext @app.post("/enqueue") async def enqueue(): ... ``` Decorators are concise and work well for smaller applications. However, once the application begins to grow, I generally prefer encapsulating related endpoints inside classes and registering routes programmatically. ```python self.app.add_api_route( "/enqueue", self.enqueue_post, methods=["POST"], ) ``` This provides several advantages: * The web server becomes a self-contained component. * Dependencies are injected through the constructor. * Routes are registered in one place. * And the application structure more closely resembles the architecture itself. Instead of scattered decorators across multiple modules, the routing layer becomes an object with a clearly defined responsibility. This is largely a matter of preference, but I've found it scales better as applications grow beyond a handful of endpoints. * * * ### Managing Background Services with FastAPI Lifespan One of the more elegant features introduced in modern versions of FastAPI is the **lifespan protocol**. Rather than relying on startup events or background threads, FastAPI allows applications to define asynchronous startup and shutdown logic using an asynchronous context manager. ```python @asynccontextmanager async def _lifespan(self, app: FastAPI): await self.queue_mgr.start() try: yield finally: await self.queue_mgr.stop() ``` It defines the lifecycle of the entire orchestration platform. When the application starts: ```plaintext FastAPI Starts │ ▼ Queue Manager Starts │ ▼ Listener Loop Starts │ ▼ Watchdog Starts │ ▼ API Ready ``` Likewise, during shutdown: ```plaintext SIGTERM │ ▼ FastAPI Shutdown │ ▼ QueueManager.stop() │ ▼ Cancel Async Tasks │ ▼ Terminate Workers │ ▼ Exit ``` Every long-running background component starts together. Every component stops together. This avoids one of the most common problems in asynchronous applications—background tasks that continue running after the web server has begun shutting down. Instead, the entire application behaves as a single coordinated unit. Here is the comparison illustrated: ![](https://cdn.hashnode.com/uploads/covers/6a51387ca8cf56c623690316/8411b7c9-ee79-402c-979b-d3d9ebc37e12.jpg align="center") * * * ### Accepting Work Instead of Executing Work The most important endpoint in the application is `/enqueue`. Its implementation is surprisingly small. ```python async def enqueue_post(self, req: EnqueueRequest): job_id = await self.db.insert_pending_job(req.payload) return EnqueueResponse( id=job_id, status="pending", ) ``` That’s all it does: no worker creation, no scheduling, no analysis, no LLM calls, and no expensive computation. The endpoint simply records a pending job and immediately returns a response. It may seem almost too simple until you compare it with the alternative. ## Synchronous Design ```plaintext Client │ ▼ POST /analyse │ ▼ FastAPI │ ▼ Fetch Data │ ▼ Generate Embeddings │ ▼ Call LLM │ ▼ Store Results │ ▼ HTTP Response ``` The client waits for every stage of the pipeline to complete. If the analysis takes five minutes, the HTTP request remains open for five minutes. Now compare that to the asynchronous design. ## Asynchronous Design ```plaintext Client │ ▼ POST /enqueue │ ▼ Insert Pending Job │ ▼ Return Job ID ``` Milliseconds later, the response has already been returned. Meanwhile, independently of the HTTP request: ```plaintext Queue Manager │ ▼ Claim Job │ ▼ Spawn Worker │ ▼ Run Analysis │ ▼ Store Results ``` The API remains completely free to serve additional clients. The orchestration platform performs the expensive work independently. This decoupling is one of the defining characteristics of asynchronous systems. * * * ### Tracking Progress Instead of Holding Connections Open Returning a Job ID fundamentally changes the interaction model between the client and the server, the client receives an acknowledgement that work has been accepted. ```http POST /enqueue HTTP/1.1 200 OK { "id": "0f9d4b8f", "status": "pending" } ``` The client can now continue using the application while the analysis executes in the background. Whenever progress is needed, it simply queries another endpoint. ```http GET /jobs/0f9d4b8f ``` This pattern has several advantages. * The API never holds long-lived HTTP connections. * Progress can be reported incrementally. * Users can safely refresh the page without interrupting processing. * Clients can poll or subscribe for updates. * Jobs become durable entities rather than transient HTTP requests. From a user's perspective, the experience is actually better. Instead of staring at a loading spinner for several minutes, they receive immediate feedback that their request has been accepted and can monitor its progress asynchronously. * * * *Let's continue to design reliable worker processes and queue manager in part 3!*