# Beyond FastAPI: Building a Production-Ready Asynchronous Job Orchestrator in Python ## Background Modern organizations collect an enormous amount of unstructured textual data through customer surveys, support tickets, employee feedback, product reviews, and research responses. While Large Language Models (LLMs) have made it possible to automate qualitative analysis, applying them to production-scale datasets introduces challenges that go far beyond simply making an API call. Processing thousands of responses can take several minutes, involve multiple stages of analysis, require retries, and depend on external services that are inherently slow or unreliable. The core feature I wanted to build was an **LLM-powered thematic analysis engine** capable of processing large collections of textual responses and automatically identifying recurring themes, topics, and insights. The objective was to provide users with a simple interface where they could submit an analysis request, continue using the application, and retrieve the results once processing had completed. Rather than treating each request as a traditional synchronous API call, the system needed to support long-running, resource-intensive workloads while remaining responsive to other users. This immediately shifted the problem from being purely about AI to being about **backend systems engineering**. The challenge was no longer just generating prompts or orchestrating LLM interactions—it was designing an execution platform that could reliably schedule analysis jobs, execute them in isolation, track progress, recover from failures, and scale as demand increased. In other words, the thematic analysis pipeline became just one component of a much larger asynchronous processing architecture. In this series, I'll walk through the design and implementation of that execution platform. We'll look at how **FastAPI**, **asyncio**, **multiprocessing**, and **MongoDB** can be combined to build a production-style asynchronous job orchestration system capable of supporting long-running AI workloads without compromising the responsiveness or reliability of the API layer. ## Part 1 - Why FastAPI Alone Isn't Enough for Long-Running Workloads ### Introduction Imagine exposing an API endpoint that performs thematic analysis on thousands of customer comments. Behind a single request lies a complex, multi-stage workflow that includes data retrieval, text preprocessing, embedding generation, multiple interactions with LLM services, aggregation of intermediate results, and persistence of the final analysis. Since each stage may involve expensive computation or external network calls, the total execution time can range from several seconds to many minutes. Executing such workloads synchronously would tie up server resources, increase the likelihood of request timeouts, and degrade the responsiveness of the API for other users.. That also creates several other problems: * Reverse proxy timeouts * No progress tracking * Poor scalability * Inability to recover if the server crashes ![](https://cdn.hashnode.com/uploads/covers/6a51387ca8cf56c623690316/3e644a60-1708-4c95-a833-3e071c09534e.jpg align="center") In other words, the API is trying to fulfil two very different responsibilities: serving HTTP requests and executing computationally expensive background jobs. These concerns have different scalability and reliability requirements, and coupling them together makes the system harder to scale, monitor, and recover. A better approach is to keep the API lightweight by accepting the request, persisting the job, and delegating the long-running work to a dedicated orchestration and worker layer. ## A Better Architecture ![](https://cdn.hashnode.com/uploads/covers/6a51387ca8cf56c623690316/f1dc7309-8aab-4fbc-8d24-83c7230f08c7.jpg align="center") Instead of coupling the API directly to the analysis pipeline, we can decouple request handling from job execution by introducing an asynchronous orchestration layer. In this architecture, the API is responsible only for validating incoming requests, creating a job, and returning a unique job identifier to the client. The actual analysis is performed independently in the background, allowing the API to remain responsive regardless of how long the analysis takes. The journey begins when a client submits an analysis request to the **FastAPI** application. Rather than executing the analysis immediately, FastAPI persists the request as a **pending job** in a MongoDB-backed job queue and immediately responds with a job ID. This completes the HTTP request within milliseconds while ensuring that the analysis request has been durably stored. A dedicated **Queue Manager** continuously monitors the job queue for pending work. Based on configurable concurrency limits, it claims jobs atomically and dispatches them to a pool of isolated worker processes. This orchestration layer acts as the scheduler for the system, ensuring that only a safe number of analyses execute concurrently while preventing multiple workers from processing the same job. Each worker process provides an isolated execution environment for a single analysis job. Rather than containing the business logic itself, the worker dynamically loads the appropriate **LLM analysis pipeline** through a plugin architecture. This keeps the orchestration layer generic and allows new analysis pipelines to be introduced without changing the worker infrastructure. During execution, workers periodically report progress milestones, making it possible for clients to monitor long-running jobs instead of waiting for a single HTTP response. Finally, once the analysis completes, the generated themes, insights, and metadata are persisted to **MongoDB**, where they can be retrieved through a separate API endpoint. If a worker crashes or becomes unresponsive, the orchestration layer can detect the failure, mark the job appropriately, and recover gracefully without affecting the API server or other running analyses. By separating **request handling**, **job scheduling**, and **analysis execution** into distinct components, each part of the system can scale independently. The API remains lightweight and responsive, the queue manager efficiently coordinates workloads, and the worker pool focuses exclusively on executing computationally expensive AI tasks. This separation of concerns results in a more scalable, resilient, and maintainable architecture for long-running LLM-powered applications. Here is the high level break down of the components involved in the new architecture along with their primary function | Component | Responsibility | | --- | --- |

FastAPI

Accepts Requests for thematic analysis

MongoDb

Persists jobs, intermediate and final results of analsyis, also stores milestone progress of analysis process

Queue Manager

Schedules jobs

Workers

Execute analysis

LLM pipeline

Core logic of LLM based analysis

## Key Design Principles Although the technologies used to build this platform are important, the architecture is ultimately driven by a handful of fundamental design principles. These principles allow the system to remain responsive, resilient, and scalable as the complexity and volume of analysis jobs grow. ### Decoupling The API layer and the analysis pipeline serve very different purposes and therefore should not depend on each other. The API is responsible for accepting requests, validating input, and creating jobs, while the worker infrastructure is responsible for executing long-running analyses. By introducing a job queue between these components, requests can be accepted and acknowledged immediately without waiting for the analysis to complete. This separation also allows each component to evolve independently without impacting the others. ### Fault Isolation Long-running workloads inevitably fail. External APIs may become unavailable, network requests may time out, or unexpected errors may occur during processing. By executing each analysis in an isolated worker process, failures are contained to a single job rather than affecting the API server or other running analyses. A crashed worker simply results in a failed job, while the rest of the platform continues operating normally. ### Horizontal Scalability As demand increases, the solution should be to add more execution capacity rather than making a single server more powerful. Since jobs are persisted in a shared queue and workers operate independently, additional worker processes—or even workers running on different machines—can be introduced to increase throughput. The API layer, queue manager, and worker pool can each be scaled independently according to their individual workloads. ### Asynchronous Processing The request-response lifecycle should be measured in milliseconds, even if the underlying analysis takes several minutes. By treating analysis as an asynchronous background task, the API remains responsive and clients are free to continue using the application while their jobs execute. Progress can be tracked independently, and results can be retrieved once processing has completed. ### Worker Isolation Each analysis executes within its own operating system process, providing complete isolation from every other job. This prevents memory leaks, crashes, or resource-intensive analyses from impacting other workers. It also enables graceful shutdown, independent process lifecycle management, and more predictable resource utilisation, all of which are essential for building reliable long-running processing systems. * * * *Lets continue to work on FastAPI in part 2*