The Inevitable Need for Background Jobs

Background jobs... You can put them off for a while, but they seem to always find a home in a maturing codebase. Without them, scheduled, or slow running operations at the API layer quickly hit a scale wall, and these types of operations also just generally promote bad UX patterns.

So what are some ways to approach this?

There are pleeeenty of tools out there, varying from the raw DIY building block of messaging queues, serverless compute, schedulers, to full-fledged SaaS services offering an opinionated, out-of-the-box job processing experience.

We found it a bit of a challenge to sort through these, recognizing that while we could DIY something purpose-built to our immediate needs, our time would likely not be well spent stitching things together. We also recognized that a highly structured job processing tool could require refactoring into a more brittle approach and potentially lock us into a solution.Key Learnings

1. Embrace the API Layer at the MVP/Prototype Stage

Embrace the API layer at the MVP/prototype stage. This is where you learn the details of what you need, where the bottlenecks are, etc. You probably do not need background jobs processing immediately; embrace the wait spinner!

• Some features we determined we needed from a background jobs tool at this stage:
  • triggered jobs (async, pollable, batches)
  • parent/child relationships for nested jobs
  • scheduled jobs (cron)
  • queue + configurable retries etc.
  • the task/configuration code lives in our codebase (this is really big for dev ops)
  • github actions integration for prod deployment
  • vercel integration
  • Local server for dev
  • can be self hosted
  • typescript
  • dashboard, task level debugging, and logs/tracing
  • slack/email alerts
  • versioning
  • db integration
  • hosted option with auto scaling
  • pricing based on worker runtime

2. Filter Options by Reading Documentation

Filter options by reading documentation, search for features you know you will need and make sure the contracts and approaches to those features make sense for your project.

3. Build Extreme MVPs

Build extreme MVPs using those features on promising solutions. It is important to build these out comprehensively across the feature set. We found several tools with the features we needed on paper, like a resumable step-based architecture, that had serious timeout issues with our Vercel backend.

4. Ask Questions Early

Use available support channels early. This gives a good sense of responsiveness and community engagement.

5. Think Through Team Workflows

1. Can multiple developers work on the same job at the same time?
2. Will runs in development or staging branches overlap?
   1. Some tools have a great dev environment that struggle in staging/production due to having multiple branches pointing to the same backend. Others only have a hosted dev environment, which requires a network connection, and can be an annoying extra step/configuration especially across a development team.

6. Understand Deployment Thoroughly

Really understand deployment. The existence of a "Production" tab in the docs is not enough! Understand how development, staging (multiple branches?), and production environments are configured. Then actually integrate it, even at a small scale. It is really painful to figure out a blocker with some aspect of your deploy pipeline after doing all the above work. Ask us how we know!

7. Evaluate Versioning and Backwards Compatibility

Versioning, how does this work in the tool? Are there ways to maintain backwards compatibility? Do you have any long-running processes that can be impacted by a new deployment? Think through what happens when you deploy a new version, and how this impacts different parts of your running application.

‍

Our Decision: Why We Chose Inngest

After running through the above with a few different options, our team settled on Inngest, which has been a great experience so far and checks all our boxes.

Great docs, responsive support, and active community, and provides an excellent development experience. It is easy to deploy in multiple staging branches and production, and has great visibility into metrics and logs at the service, job, and task levels. It has all the features we need, and has made background jobs a consistent and comfortable part of our codebase.

Knowing we can easily spin up a new job for something long-running has also made it easier for us to build a smooth UX, as we can push slower processes (even just 2-3 seconds) to a background job and keep our front end snappy.

‍

Key Takeaways

• Start with API layer implementation before moving to background jobs - this helps identify actual requirements and bottlenecks
• Essential features needed: triggered jobs, job relationships, scheduling, configurable queues, and integration with development tools
• Thoroughly test potential solutions with comprehensive MVPs across the full feature set
• Consider development ecosystem factors like multi-developer support and branch management
• Carefully evaluate deployment processes and versioning support across all environments
• Inngest emerged as the chosen solution, due to strong documentation, support, and development experience, while meeting all required features

‍