Five users worked. Ten broke us.
Google Calendar webhooks rate-limited us at ten beta users — until we fanned jobs out through Inngest and gated concurrency per user.
6 min read
Build log
We started adding our first customers, our first active users. Fractional.tools worked fine for five or six test users. At ten active beta testers it started failing — and that absolutely destroyed our expectations of how we could go live.
The problem was Google Calendar watches. Whenever someone changed their calendar, we got notified via a dedicated watch function. That watch function URL would pull down the data, determine what changed, update their calendar, and do the matching. If three or four of these were running simultaneously, we were getting rate limited. If we were getting rate limited on just five clients concurrent, there was no way we could build this to scale up to several hundred.
We stopped and looked for new ways to handle this.
Our first instinct: CRON and retry
Our first instinct when we looked at this was we needed to build our own cron process and we needed to figure out our own retry.
Inngest on signup — the first mind-blow
When we tried Inngest, we first started by saying let’s just give it responsibility. When a user signs up, we need to do three or four things. Let’s break each one of those things into a task because they can kind of run concurrently.
That works stunningly well. When a user gets signed up, we set up their defaults, we set up their Stripe customer ID, we go over and run a test against their OAuth account, and then we send them two or three emails to get them signed up. These all ran now simultaneously, but if one of them failed, Inngest would try again for a certain retry count and it would go through. This was kind of mind-blowing.
The beta wall — Google rate limits
Then our real big unblock was when we went to our first 10 active beta testers. We started noticing that some of the functions that were running wrong were actually getting us rate limited by Google.
Per-user gating and fan-out
We decided to do something a little bit unexpected, which was use the event process and use the retry process and use Inngest’s ability to gate or control the amount of concurrency per user ID.
We labeled every one of the job functions coming in, broke them down, fanned them out, and let them run concurrently. This gave us huge throughput because we sometimes have to update several hundred calendar entries. If we did them one by one, then we were going to run long. But if we took it, figured out the priority, fanned them out to their own individual jobs, and let multiple jobs spin up, run, and then shut themselves down, Inngest would do all of it.
A lot of the scaling things that we thought we had to build on our own, we could simply structure our code base to be very simple and very efficient at the job level and let Inngest handle the fan out, the logic, the gating, and the rate limiting per user.
Once we broke it down into smaller chunks, optimized them for speed, and then let Inngest handle the orchestration, we could just follow standard job practices and we could let the AI help us structure a process that would be hyperscalable. We could do something that would require a senior Google engineer to do — and we could do it just by breaking it into semantic ideas.
What small teams should decide differently
Small teams should look at this and think something that was formerly very dev-heavy can now be broken down into some simple components — components that can be managed, monitored, and if need be tracked via a web interface. The people who are tracking to make sure it stays operational don’t have to be coders.
Your coders can stay focused on operations, throughput, speed, but the operations side of the team immediately gets visibility in what the application is doing. Very often they notice patterns that can lead to optimization that a developer might not know because the developer is focused on the code and not focused on the actual use case.
What I’d regret skipping
Historic job runs. It may not seem like a big deal, but once your application starts going live, it’s kind of nice to know what resources are being burned. If things are failing but recovering, you kind of want to know — but you don’t need to know on the fly. I would highly regret if I didn’t have the ability to fix things later on once I got more free time and not let a production issue just slam its way into my workflow.
Steps and restart. The ability to have tasks run either sequentially or restart. If one of these jobs fails, Inngest will keep track of the state and retry it at a later date at that current state. You don’t get duplicate emails sent or duplicate records created. Your code can be actually much, much simpler.
Default on every project now
We were so happy with this that now I use this on every project because every web project has some tasks that have to run live in line with what the user is doing and some of them need to run in the background either at night or off hours. In order to make the user experience live we don’t want to have to drive everything through the API through user requests. We need to have the system responsive.
Whenever we do vectorization, we can farm it out, let it run slowly in the background, let it bleed down like a tank of jobs, and if one of them fails, it will pick up and run again.
Investor frame
Investors don’t really care how you build things, but they do care about how a team is focused. If we were talking to our investors about what we had done in the last six months, we would be able to say we are able now to scale from five to ten to fifty to a hundred users and we’re not worried about our current architecture having to be rebuilt and structured. Your investment goes towards scaling and growth, not for architecture and building.
Try the prompts
If you’re hitting the same wall — webhooks, calendar sync, rate limits — start with the Inngest playbook on this site. The prompt cards there are what we used to wire edge functions and Supabase without weeks of manual config: