Part 3 of the "Beyond FastAPI" series.

In the previous article, we built a thin orchestration layer using FastAPI. The API validates incoming requests, persists them as pending jobs, and immediately returns a Job ID to the client. At this point, our application is responsive, but it's also incomplete.

Although jobs are now safely stored in MongoDB, nothing is actually executing them.

Something still needs to answer a number of important questions.

• Which job should run next?

• How many jobs can execute concurrently?

• How do we prevent two workers from processing the same job?

• What happens when a worker crashes?

• How do we detect jobs that become stuck?

• How do we clean up finished processes?

These responsibilities don't belong inside the API. They belong to a dedicated scheduling component. In our architecture, that component is the Queue Manager.

The Queue Manager Is Not the Queue

One misconception developers often have when reading architectures like this is assuming that the Queue Manager is the queue.

It isn't. The queue already exists. It's our MongoDB collection. Although it can be any other SQL based database as following.

┌──────────────────────────┐
│ MongoDB Jobs Collection  │
├──────────────────────────┤
│ pending                  │.            │
│ pending                  │.            │ 
│ running                  │.            │
│ completed                │.            │
│ failed                   │.            │
└──────────────────────────┘

Every analysis request submitted through FastAPI is stored as a document (or row) inside this collection. Each document represents the current state of a job.

The Queue Manager doesn't own this data. Instead, it continuously observes it and makes scheduling decisions based on the current state of the system. This distinction is subtle but extremely important.

So MongoDB as the source of truth. The Queue Manager is simply the scheduler responsible for moving jobs through their lifecycle.

Separating Scheduling from Execution

One of the design goals of this platform was ensuring that no single component performs too many responsibilities.

Let's compare two possible implementations.

Option 1

FastAPI

↓

Find Pending Job

↓

Execute Analysis

↓

Store Result

This is simple.

Unfortunately, it's also tightly coupled.

The API now owns:

• HTTP communication

• Scheduling

• Concurrency

• Worker lifecycle

• Analysis execution

Every new feature increases the complexity of the web application.

Instead, our architecture separates these concerns.

              FastAPI
                  │
                  ▼
        MongoDB Job Queue
                  │
                  ▼
           Queue Manager
          ┌───────┴────────┐
          ▼                ▼
     Listener Loop   Watchdog Loop
          │
          ▼
     Worker Process

Each component now has a single responsibility.

FastAPI accepts work.

MongoDB stores work.

Queue Manager schedules work.

Workers execute work.

Because each component focuses on one job, the overall system becomes much easier to understand, test and evolve.

Anatomy of the Queue Manager

Let's look at the class itself.

class QueueManager:

    def __init__(
        self,
        DBFactory
    ):
        self.db = DBFactory()
        self._db_impl_factory_for_worker = DBFactory

        self._listener_task = None
        self._watchdog_task = None

        self._running_processes = {}

Although the implementation is relatively small, every member variable exists for a specific reason.

Database Interface

self.db

The scheduler never communicates directly with MongoDB. Instead, it depends on the same abstraction used throughout the application. This keeps scheduling logic independent of the underlying persistence implementation.

Worker Database Factory

self._db_impl_factory_for_worker

This is one of the more interesting design decisions in the project.

Instead of sharing a single database instance with worker processes, we inject a factory function capable of creating fresh database implementations.

Why?

Because worker processes execute in separate operating system processes. Sharing an already connected database client across processes can lead to unpredictable behaviour depending on the driver and operating system.

Instead, every worker creates its own independent database connection.

Parent Process
      │
      ▼
Queue Manager
      │
      ▼
Factory Function
      │
 ┌────┴────┐
 ▼         ▼
Worker 1  Worker 2
 │         │
 ▼         ▼
MongoDB   MongoDB
Connection Connection

This keeps process boundaries clean while avoiding shared state between workers.

A Note on Database Connections

At first glance, creating a separate database client for each worker process might seem expensive. However, workers are not spawned without bounds. The Queue Manager enforces a configurable concurrency limit, ensuring that only a fixed number of worker processes can run simultaneously. Each worker establishes its own process-local MongoDB client; following MongoDB's recommendation of one client per process and exits once its job completes. Because the number of concurrent workers is tightly controlled, the maximum number of database clients is also bounded and predictable. We'll explore how this concurrency limit is enforced in the next section when we dive into the Listener Loop and scheduling logic.

Background Tasks

The Queue Manager owns two asynchronous tasks.

_listener_task

_watchdog_task

These are long-running background coroutines that execute independently of incoming HTTP requests. Each solves a different problem:

The Listener Loop continuously searches for new work.

The Watchdog Loop monitors already running work.

Keeping these responsibilities separate makes the scheduling logic significantly easier to reason about. We'll dive into both loops next.

Tracking Running Processes

Finally, the Queue Manager maintains an in-memory registry.

_running_processes

Conceptually, it looks like this.

{
    "job-123": Process(pid=4128),

    "job-456": Process(pid=4132),

    "job-789": Process(pid=4141)
}

Notice what this dictionary is not.

It isn't the source of truth for job state. MongoDB already provides that. Instead, this registry exists purely so the Queue Manager can interact with operating system processes.

For example:

• Determining whether a worker is still alive

• Terminating workers during shutdown

• Reclaiming finished processes

• Detecting crashed workers

Keeping process metadata separate from persisted job state keeps the architecture both simple and resilient.

Starting the Execution Engine

Once FastAPI starts, it invokes the Queue Manager's startup routine.

async def start(self):


    self._listener_task = asyncio.create_task(
        self._listener_loop()
    )

    self._watchdog_task = asyncio.create_task(
        self._watchdog_loop()
    )

Notice something interesting. Neither loop blocks the application. Instead, both are started using:

asyncio.create_task(...)

This allows them to execute concurrently alongside the FastAPI event loop.

                Event Loop

         ┌─────────┬──────────┐
         ▼         ▼          ▼

    HTTP Server  Listener   Watchdog

Although all three components share the same event loop, they remain completely independent.

• The HTTP server continues accepting requests.

• The Listener discovers new jobs.

• The Watchdog monitors existing workers.

No component blocks another. This is precisely the type of workload asyncio was designed for—multiple long-running, I/O-bound tasks cooperating within a single thread.

Why asyncio Instead of Threads?

A common question is why the scheduler itself isn't implemented using threads. The answer lies in understanding what the Queue Manager actually does. It isn't performing CPU-intensive computation. Instead, it's mostly waiting.

• Waiting for database operations.

• Waiting for polling intervals.

• Waiting for workers to finish.

• Waiting for timeout checks.

These are classic I/O-bound operations.

Using asyncio allows a single thread to efficiently coordinate hundreds or even thousands of these waiting operations without the overhead of managing multiple operating system threads.

The workers, on the other hand, are a completely different story.

They execute expensive LLM pipelines, consume significant CPU and memory resources, and need strong isolation from one another.

That's why the workers are implemented using separate processes, not asynchronous coroutines.

This separation allows each technology to solve the problem it is best suited for:

• asyncio coordinates work.

• Processes execute work.

As we'll see next, the Listener Loop brings these two worlds together by continuously transforming pending jobs into isolated worker processes.

Part 2 of the "Beyond FastAPI" series.

In the previous article, we explored why long-running AI workloads don't belong inside a traditional request-response cycle. We introduced an architecture that decouples request handling from job execution, allowing our API to remain responsive while computationally intensive analysis runs asynchronously in the background.

In this article, we'll build the orchestration layer of that architecture.

Rather than focusing on FastAPI itself, we'll look at how the application is composed, how dependencies are injected, and why the API layer should remain intentionally thin. We'll also see how background services such as our queue manager are integrated into the application's lifecycle without leaking infrastructure concerns into the web layer.

The Role of the Orchestration Layer

When developers first begin building APIs, it's common to let the web framework become the centre of the application.

A request arrives.

The endpoint validates input.

The endpoint queries the database.

The endpoint performs business logic.

The endpoint calls external APIs.

The endpoint returns the response.

This works remarkably well—until the business logic becomes expensive.

Our thematic analysis pipeline isn't a simple CRUD operation. A single request can involve fetching thousands of survey responses, cleaning and preprocessing text, generating embeddings, orchestrating multiple LLM calls, aggregating intermediate results, and finally persisting the generated insights.

That's not the responsibility of an HTTP endpoint.

Instead, our FastAPI application has a much simpler responsibility:

Accept work. Persist work. Return immediately.

Everything else belongs elsewhere.

Key Design Principle

HTTP APIs should be responsible for communication, not computation.

Overall Architecture

Notice something important.

The API is no longer responsible for executing the analysis.

Instead, it acts as the entry point into a much larger execution platform.

This separation of concerns has several advantages:

• API requests complete in milliseconds instead of minutes.

• Long-running analyses survive independently of HTTP connections.

• Workers can scale independently of the API.

• Failures are isolated to individual jobs.

• Progress can be tracked asynchronously.

FastAPI simply becomes the gateway into the system rather than the system itself.

The Composition Root

One of the most overlooked aspects of application architecture is where objects are created.

Every application has a point where its major components are instantiated and connected together. This is commonly referred to as the Composition Root.

In our project, that responsibility belongs to the application's entry point.

def run():
    myDB = _make_db_impl()
    DBFactory = _make_db_impl

    app = create_app(
       DBFactory
    )

    uvicorn.run(app, host="0.0.0.0", port=8000)

At first glance this looks almost trivial.

In reality, this function defines the entire dependency graph of the application.

Before FastAPI ever starts accepting requests, we create:

• the database implementation

• the worker database factory

• the queue manager

• the web server

• the FastAPI application

Only once every component has been assembled do we hand control over to Uvicorn.

This may seem like a small design choice, but it has significant architectural benefits.

The application has exactly one place responsible for wiring dependencies together. No class reaches into another module to construct its own dependencies. Nothing relies on global state. Every component receives exactly what it needs through its constructor.

This makes the application significantly easier to understand, maintain, and test.

Why Dependency Injection Matters

Let's imagine we wrote our queue manager like this:

class QueueManager:

    def __init__(self):
        self.db = MongoJobsDB(...)

This works.

But now the queue manager is permanently coupled to MongoDB.

Testing becomes difficult.

Switching databases becomes difficult.

Replacing MongoDB with Redis, PostgreSQL, or a mock implementation becomes impossible without modifying the class itself.

Instead, our implementation receives its dependencies from the outside.

class QueueManager:

    def __init__(
        self,
        DBFactory,
    ):
        self.db = DBFactory()

This small change fundamentally alters the architecture.

The queue manager no longer knows or cares where the database comes from. It simply depends on the behaviour defined by the interface. This is an example of the Dependency Inversion Principle, one of the SOLID design principles.

Rather than depending on concrete implementations, high-level components depend on abstractions.

As a result:

• business logic becomes independent of infrastructure

• testing becomes straightforward

• implementations become interchangeable

• components become easier to reuse

• QueueManager
  │
▼
DBInterface
▲
│
┌─────────────┴──────────────┐
▼ ▼
MongoJobsDB MockDatabase

Although our current implementation only uses MongoDB, the queue manager itself has no knowledge of that decision.

That's exactly where we want to be.

Building a Thin API Layer

One of the easiest ways for an application to become difficult to maintain is to allow the web framework to absorb business logic. Over time, the web layer becomes tightly coupled to every other component in the system, making it increasingly difficult to test, extend, or replace.

Instead, the FastAPI application in this project acts purely as a transport layer. Its responsibilities are deliberately limited to:

• Receiving HTTP requests

• Validating incoming data

• Delegating work to the orchestration layer

• Returning an appropriate HTTP response

Everything else belongs elsewhere.

This philosophy is reflected in the WebServer class.

class WebServer:
    def __init__(self, db: DBInterface, queue_mgr: QueueManager):
        self.db = db
        self.queue_mgr = queue_mgr

        self.app = FastAPI(
            title="Async Queue Server",
            lifespan=self._lifespan,
        )

        self.app.add_api_route(
            "/enqueue",
            self.enqueue_post,
            methods=["POST"],
            response_model=EnqueueResponse,
        )

        self.app.add_api_route(
            "/jobs/{job_id}",
            self.get_job,
            methods=["GET"],
        )

There are a couple of interesting design decisions here.

First, notice that the WebServer doesn't construct its own dependencies.

Both the database implementation and the queue manager are injected into the constructor, continuing the dependency injection pattern introduced earlier.

Second, the class doesn't contain any scheduling logic.

It doesn't spawn workers.

It doesn't communicate with LLMs.

It doesn't poll queues.

It doesn't even know how jobs are executed.

Its only responsibility is exposing a clean HTTP interface over the orchestration platform.

Why Register Routes Programmatically?

If you've used FastAPI before, you're probably familiar with decorators.

@app.post("/enqueue")
async def enqueue():
    ...

Decorators are concise and work well for smaller applications.

However, once the application begins to grow, I generally prefer encapsulating related endpoints inside classes and registering routes programmatically.

self.app.add_api_route(
    "/enqueue",
    self.enqueue_post,
    methods=["POST"],
)

This provides several advantages:

• The web server becomes a self-contained component.

• Dependencies are injected through the constructor.

• Routes are registered in one place.

• And the application structure more closely resembles the architecture itself.

Instead of scattered decorators across multiple modules, the routing layer becomes an object with a clearly defined responsibility. This is largely a matter of preference, but I've found it scales better as applications grow beyond a handful of endpoints.

Managing Background Services with FastAPI Lifespan

One of the more elegant features introduced in modern versions of FastAPI is the lifespan protocol.

Rather than relying on startup events or background threads, FastAPI allows applications to define asynchronous startup and shutdown logic using an asynchronous context manager.

@asynccontextmanager
async def _lifespan(self, app: FastAPI):
    await self.queue_mgr.start()

    try:
        yield
    finally:
        await self.queue_mgr.stop()

It defines the lifecycle of the entire orchestration platform. When the application starts:

FastAPI Starts
      │
      ▼
Queue Manager Starts
      │
      ▼
Listener Loop Starts
      │
      ▼
Watchdog Starts
      │
      ▼
API Ready

Likewise, during shutdown:

SIGTERM
      │
      ▼
FastAPI Shutdown
      │
      ▼
QueueManager.stop()
      │
      ▼
Cancel Async Tasks
      │
      ▼
Terminate Workers
      │
      ▼
Exit

Every long-running background component starts together.

Every component stops together.

This avoids one of the most common problems in asynchronous applications—background tasks that continue running after the web server has begun shutting down.

Instead, the entire application behaves as a single coordinated unit.
Here is the comparison illustrated:

Accepting Work Instead of Executing Work

The most important endpoint in the application is /enqueue.

Its implementation is surprisingly small.

async def enqueue_post(self, req: EnqueueRequest):
    job_id = await self.db.insert_pending_job(req.payload)

    return EnqueueResponse(
        id=job_id,
        status="pending",
    )

That’s all it does: no worker creation, no scheduling, no analysis, no LLM calls, and no expensive computation. The endpoint simply records a pending job and immediately returns a response. It may seem almost too simple until you compare it with the alternative.

Synchronous Design

Client
   │
   ▼
POST /analyse
   │
   ▼
FastAPI
   │
   ▼
Fetch Data
   │
   ▼
Generate Embeddings
   │
   ▼
Call LLM
   │
   ▼
Store Results
   │
   ▼
HTTP Response

The client waits for every stage of the pipeline to complete.

If the analysis takes five minutes, the HTTP request remains open for five minutes.

Now compare that to the asynchronous design.

Asynchronous Design

Client
   │
   ▼
POST /enqueue
   │
   ▼
Insert Pending Job
   │
   ▼
Return Job ID

Milliseconds later, the response has already been returned.

Meanwhile, independently of the HTTP request:

Queue Manager
      │
      ▼
Claim Job
      │
      ▼
Spawn Worker
      │
      ▼
Run Analysis
      │
      ▼
Store Results

The API remains completely free to serve additional clients.

The orchestration platform performs the expensive work independently.

This decoupling is one of the defining characteristics of asynchronous systems.

Tracking Progress Instead of Holding Connections Open

Returning a Job ID fundamentally changes the interaction model between the client and the server, the client receives an acknowledgement that work has been accepted.

POST /enqueue

HTTP/1.1 200 OK

{
    "id": "0f9d4b8f",
    "status": "pending"
}

The client can now continue using the application while the analysis executes in the background.

Whenever progress is needed, it simply queries another endpoint.

GET /jobs/0f9d4b8f

This pattern has several advantages.

• The API never holds long-lived HTTP connections.

• Progress can be reported incrementally.

• Users can safely refresh the page without interrupting processing.

• Clients can poll or subscribe for updates.

• Jobs become durable entities rather than transient HTTP requests.

From a user's perspective, the experience is actually better.

Instead of staring at a loading spinner for several minutes, they receive immediate feedback that their request has been accepted and can monitor its progress asynchronously.

Let's continue to design reliable worker processes and queue manager in part 3!

Modern organizations collect an enormous amount of unstructured textual data through customer surveys, support tickets, employee feedback, product reviews, and research responses. While Large Language Models (LLMs) have made it possible to automate qualitative analysis, applying them to production-scale datasets introduces challenges that go far beyond simply making an API call. Processing thousands of responses can take several minutes, involve multiple stages of analysis, require retries, and depend on external services that are inherently slow or unreliable.

The core feature I wanted to build was an LLM-powered thematic analysis engine capable of processing large collections of textual responses and automatically identifying recurring themes, topics, and insights. The objective was to provide users with a simple interface where they could submit an analysis request, continue using the application, and retrieve the results once processing had completed. Rather than treating each request as a traditional synchronous API call, the system needed to support long-running, resource-intensive workloads while remaining responsive to other users.

This immediately shifted the problem from being purely about AI to being about backend systems engineering. The challenge was no longer just generating prompts or orchestrating LLM interactions—it was designing an execution platform that could reliably schedule analysis jobs, execute them in isolation, track progress, recover from failures, and scale as demand increased. In other words, the thematic analysis pipeline became just one component of a much larger asynchronous processing architecture.

In this series, I'll walk through the design and implementation of that execution platform. We'll look at how FastAPI, asyncio, multiprocessing, and MongoDB can be combined to build a production-style asynchronous job orchestration system capable of supporting long-running AI workloads without compromising the responsiveness or reliability of the API layer.

Part 1 - Why FastAPI Alone Isn't Enough for Long-Running Workloads

Introduction

Imagine exposing an API endpoint that performs thematic analysis on thousands of customer comments. Behind a single request lies a complex, multi-stage workflow that includes data retrieval, text preprocessing, embedding generation, multiple interactions with LLM services, aggregation of intermediate results, and persistence of the final analysis. Since each stage may involve expensive computation or external network calls, the total execution time can range from several seconds to many minutes. Executing such workloads synchronously would tie up server resources, increase the likelihood of request timeouts, and degrade the responsiveness of the API for other users..

That also creates several other problems:

• Reverse proxy timeouts

• No progress tracking

• Poor scalability

• Inability to recover if the server crashes

In other words, the API is trying to fulfil two very different responsibilities: serving HTTP requests and executing computationally expensive background jobs. These concerns have different scalability and reliability requirements, and coupling them together makes the system harder to scale, monitor, and recover. A better approach is to keep the API lightweight by accepting the request, persisting the job, and delegating the long-running work to a dedicated orchestration and worker layer.

A Better Architecture

Instead of coupling the API directly to the analysis pipeline, we can decouple request handling from job execution by introducing an asynchronous orchestration layer. In this architecture, the API is responsible only for validating incoming requests, creating a job, and returning a unique job identifier to the client. The actual analysis is performed independently in the background, allowing the API to remain responsive regardless of how long the analysis takes.

The journey begins when a client submits an analysis request to the FastAPI application. Rather than executing the analysis immediately, FastAPI persists the request as a pending job in a MongoDB-backed job queue and immediately responds with a job ID. This completes the HTTP request within milliseconds while ensuring that the analysis request has been durably stored.

A dedicated Queue Manager continuously monitors the job queue for pending work. Based on configurable concurrency limits, it claims jobs atomically and dispatches them to a pool of isolated worker processes. This orchestration layer acts as the scheduler for the system, ensuring that only a safe number of analyses execute concurrently while preventing multiple workers from processing the same job.

Each worker process provides an isolated execution environment for a single analysis job. Rather than containing the business logic itself, the worker dynamically loads the appropriate LLM analysis pipeline through a plugin architecture. This keeps the orchestration layer generic and allows new analysis pipelines to be introduced without changing the worker infrastructure. During execution, workers periodically report progress milestones, making it possible for clients to monitor long-running jobs instead of waiting for a single HTTP response.

Finally, once the analysis completes, the generated themes, insights, and metadata are persisted to MongoDB, where they can be retrieved through a separate API endpoint. If a worker crashes or becomes unresponsive, the orchestration layer can detect the failure, mark the job appropriately, and recover gracefully without affecting the API server or other running analyses.

By separating request handling, job scheduling, and analysis execution into distinct components, each part of the system can scale independently. The API remains lightweight and responsive, the queue manager efficiently coordinates workloads, and the worker pool focuses exclusively on executing computationally expensive AI tasks. This separation of concerns results in a more scalable, resilient, and maintainable architecture for long-running LLM-powered applications.

Here is the high level break down of the components involved in the new architecture along with their primary function

Key Design Principles

Although the technologies used to build this platform are important, the architecture is ultimately driven by a handful of fundamental design principles. These principles allow the system to remain responsive, resilient, and scalable as the complexity and volume of analysis jobs grow.

Decoupling

The API layer and the analysis pipeline serve very different purposes and therefore should not depend on each other. The API is responsible for accepting requests, validating input, and creating jobs, while the worker infrastructure is responsible for executing long-running analyses. By introducing a job queue between these components, requests can be accepted and acknowledged immediately without waiting for the analysis to complete. This separation also allows each component to evolve independently without impacting the others.

Fault Isolation

Long-running workloads inevitably fail. External APIs may become unavailable, network requests may time out, or unexpected errors may occur during processing. By executing each analysis in an isolated worker process, failures are contained to a single job rather than affecting the API server or other running analyses. A crashed worker simply results in a failed job, while the rest of the platform continues operating normally.

Horizontal Scalability

As demand increases, the solution should be to add more execution capacity rather than making a single server more powerful. Since jobs are persisted in a shared queue and workers operate independently, additional worker processes—or even workers running on different machines—can be introduced to increase throughput. The API layer, queue manager, and worker pool can each be scaled independently according to their individual workloads.

Asynchronous Processing

The request-response lifecycle should be measured in milliseconds, even if the underlying analysis takes several minutes. By treating analysis as an asynchronous background task, the API remains responsive and clients are free to continue using the application while their jobs execute. Progress can be tracked independently, and results can be retrieved once processing has completed.

Worker Isolation

Each analysis executes within its own operating system process, providing complete isolation from every other job. This prevents memory leaks, crashes, or resource-intensive analyses from impacting other workers. It also enables graceful shutdown, independent process lifecycle management, and more predictable resource utilisation, all of which are essential for building reliable long-running processing systems.

Lets continue to work on FastAPI in part 2