A schema change, a 30-second alarm, and a billion-row night

Part 1The incident

The short version

On 2–3 September 2026, Lucimark production entered a Durable Object SQLite storage storm.

We had just shipped a Proof of Play schema change: storing player location as a geohash trail on 15-minute rollups instead of baking geography into each row's unique key. The data model was sound. The migration strategy was not sized for our largest tenant — hundreds of thousands of rollup rows inside a single TenantDO.

Two failure modes, in sequence.

Day one — writes. A synchronous rebuild during Durable Object startup produced roughly 625 million billed SQLite writes within a few hours. At Cloudflare's Durable Objects SQLite list prices (docs last updated 25 August 2026), Workers Paid writes cost $1.00 / million rows and reads $0.001 / million — about 1,000×. That is a gross list-price equivalent before included allowances, not the invoiced amount.

Night and next morning — reads. We moved the remaining work into an alarm-driven asynchronous backfill — a batched replay of what boot could not finish. That stopped the startup failures, but with the UNIQUE index down, the state machine repeatedly executed O(n) SQL — time proportional to the number of rows in the table — at roughly 30-second cadence. Hourly read peaks reached approximately 0.3–1.4 billion billed rows. Reads were much cheaper per row. The danger was different: the migration had become an unbounded loop. Nothing in the platform would naturally stop it.

Enlarge

This was not a GraphQL artifact, a billing glitch, or runaway Proof of Play ingest. Ingest that day was only thousands of rollup inserts. Worker HTTP traffic had increased modestly.

Once we understood the loop, containment was direct: park alarm-driven housekeeping, deploy cooldown and a helper index, stop re-arming heavy states every 30 seconds, remove disposable historical Proof of Play rollups from the affected tenant, and mark the migration complete. Reads fell from tens of millions per hour toward baseline the same morning.

The storm was in the data plane — cost and saturation, not a fleet-wide playback outage.

Why this architecture made the incident possible

Lucimark is a DOOH platform for operating screen networks from the browser.

Our API runs on Cloudflare Workers. Each customer tenant is represented by a Durable Object — a TenantDO — backed by its own SQLite database.

Proof of Play rollups live there too: 15-minute and daily summaries describing what played, where, and when.

Enlarge

That architecture gives us useful properties:

• tenant isolation;
• data locality;
• a clear blast radius;
• no shared relational database becoming the bottleneck for every customer.

But it also creates a billing surface.

Durable Object SQLite charges for rows read and rows written. Writes are dramatically more expensive per row than reads, but either can become operationally significant when an algorithm touches an entire large table repeatedly.

A high-volume tenant in this architecture is not simply "more HTTP."

It is more rows inside one SQLite database, potentially being revisited by one state machine every time that Durable Object wakes.

Median tenants crossed this migration without incident.

One large rollup table did not.

Cloudflare provides budget notifications, but they do not stop runtime consumption. The system itself still needs engineering bounds.

What we were trying to ship

Our Proof of Play 15-minute rows originally carried geography inside their natural key — the business key that identifies the rollup, distinct from the internal primary key.

That meant geography was effectively part of record identity: changing location information could fork what should otherwise represent the same logical rollup.

The new model stored a geohash trail separately and removed geography from the unique key.

Enlarge

That required the migration to:

• introduce the new data representation;
• update existing rollup rows;
• change the natural key;
• deduplicate collisions;
• rebuild uniqueness.

For a small tenant, this is a migration.

For our largest affected tenant — roughly 540k 15-minute rollups plus 280k daily rows — it was effectively a rewrite of one of the hottest historical tables in the object.

We underestimated that table.

Timeline

Times are BRT (UTC−3), the timezone used in the operational post-mortem.

Enlarge

1. 2 Sep ~14:16 BRT

   geohash-trail migration deploy

   Heavy work on Durable Object startup.

2. 2 Sep 14:00–15:00 BRT

   Write burst

   Two consecutive hourly buckets, ~333M and ~290M billed rows written/h — we do not assign which hour to which total. 503s and resets on the large tenant.

3. 2 Sep late afternoon

   Migration leaves boot

   Alarm-driven async backfill. The 503s stop; the amplifier only moves.

4. 2 Sep overnight

   Read storm

   Writes normalize. Billed reads ~0.3–1.4B/h while UNIQUE is down.

5. 3 Sep morning

   Alarms parked (~09:20) and hotfix

   The scan loop stops. Residual still ~57M–74M reads/h.

6. 3 Sep ~12:01 BRT

   Historical PoP removed

   Non-authoritative operational dataset; forward ingest continues. Migration marked done.

Part 2The recovery

The hourly read curve after that belongs in the recovery section — not here.

Root cause: four things that became dangerous together

1. Heavy work on a large table

The migration required operations equivalent to:

• temporarily remove uniqueness;
• update existing rows;
• deduplicate;
• recreate the UNIQUE index.

While the UNIQUE structure was unavailable, operations such as deduplication and index reconstruction had to touch large portions of the table.

On hundreds of thousands of rows, that matters.

Repeatedly, it becomes the incident.

2. Moving sync work to async did not make it safe

Our first implementation did too much work during Durable Object startup.

That caused resets and 503s.

Moving the work to an alarm-driven state machine was the correct response to the startup problem.

Async ≠ safe.

The synchronous version had a natural failure boundary: startup could time out or the object could reset.

The asynchronous version could keep waking indefinitely.

Moving expensive work off the request path removed one operational constraint without adding another.

3. Alarm cadence became a cost multiplier

During the migration, states roughly looked like:

pending → updating → deduping → indexing

The batched row updates themselves were manageable.

The dangerous states were the ones that could require O(n) work — scanning or rebuilding the whole table.

The backfill could re-arm itself at roughly 30-second cadence while still incomplete.

On a table containing more than half a million rollup rows, that meant a theoretically expensive operation could be revisited around 120 times per hour.

This is where a slow migration becomes a runaway one.

Enlarge

Alarm cadence is a cost multiplier.

Scheduling frequency cannot be treated as an implementation detail when the scheduled operation scales with table size.

4. Failed UNIQUE creation had no breaker

Duplicates remained.

The UNIQUE index rebuild failed.

The error was logged.

Then the state machine tried again.

And again.

We found roughly 2,800 migration tick failures, all associated with the same large tenant.

The worst possible shape is straightforward:

scan → fail → wait 30 seconds → scan again

A failed DDL operation in a recurring state machine is not merely an error message.

It is a loop.

And loops need breakers.

What we ruled out

Several plausible explanations did not survive the data.

GraphQL or billing artifact

Namespace-scoped Durable Object metrics matched the log behavior and the sharp recovery after removing the historical dataset.

Proof of Play ingest explosion

Ingest volume was only thousands of rollup inserts that day.

That could not explain hundreds of millions or billions of rows being touched.

Account-wide Worker traffic

HTTP traffic increased roughly 2–3×.

The SQLite movement was orders of magnitude larger.

Every tenant failing

Other production tenants did not show the same backfill-failure signature.

The problem was concentrated in one large table.

Frontend outage

The 503s on day one came from TenantDO migration resets — wakes that needed the object during the synchronous rebuild. After the migration moved async, the dominant issue became housekeeping churn, not a blank screen across the fleet.

Product impact

On the affected tenant, during the synchronous migration, some operations that needed to wake its TenantDO — including parts of the Screens experience — could fail with 503s. Other tenants did not show the same signature.

For that high-volume tenant, we deliberately removed historical Proof of Play rollups involved in the migration. This was a dataset-specific operational decision: those rollups were not authoritative contractual evidence and forward ingest continued normally.

Deleting data is not a generic migration strategy. In this incident, continuing an unbounded rewrite of replaceable history was worse than rebuilding history forward from a clean state.

How we mitigated it

First: stop the multiplier

The migration was driven by TenantDO alarms.

So the first useful lever was not shutting down Proof of Play ingest.

It was parking alarm-driven housekeeping.

That stopped the repeating scan loop.

This distinction is now part of our operational model: the correct kill switch must correspond to the amplifier causing the incident.

Then: fix the state machine

The production hotfix introduced:

• approximately 15-minute cooldowns around heavy dedupe/index states;
• a non-unique helper index while uniqueness is temporarily unavailable;
• short-circuiting when the desired UNIQUE index is already present;
• no immediate UNIQUE retry while duplicates still exist;
• failed index creation returning to dedupe instead of retrying the same full-table operation;
• migration progress decoupled from the normal 30-second active alarm cadence.

The key rule is simple:

Never treat O(n) SQL like a heartbeat.

Then: eliminate unnecessary migration work

Even with cooldown and helper indexing, rebuilding the historical dataset in place would still have been expensive.

The durable fix for this tenant was to eliminate the need to migrate those disposable historical rows at all.

Once the historical rollups were removed and the migration state marked complete, the scan surface disappeared.

Finally: verify recovery

We did not call the incident over because a deployment succeeded.

We looked for the actual signals:

• rows read per hour fell sharply;
• migration tick failures went to zero;
• TenantDO behavior returned to baseline;
• normal housekeeping could be restored.

The recovery curve mattered more than the deploy status.

Enlarge

Part 3What changed

The storm radar

We already had Cost Sense, a production job running every 15 minutes to estimate Cloudflare spend and compare it with recent baselines.

Before this incident, it was primarily designed to catch slower account-level cost drift.

That was not enough.

This failure did not look like a classic application outage.

It looked like one quiet Durable Object repeatedly reading its own database.

A generic alert saying:

"Cloudflare spend looks high"

would have been directionally useful.

But it would not tell the operator what to do next.

So we added a TenantDO storm radar.

Its goal is simple:

Detect an unbounded storage pattern early and identify the lever that can bound it.

Enlarge

1. MetricsReads · writes · active time
2. DetectPace · rate · hourly floor
3. ClassifyRead storm or write storm
4. AlertSlack
5. MitigatePark alarms · pause ingest

What it watches

Every production tick now samples the Lucimark TenantDO namespace for:

• SQLite rows read;
• SQLite rows written;
• Durable Object active time;
• recent hourly storage buckets;
• Worker and Durable Object request volume as supporting signals;
• comparison against recent production baselines.

The previous sample is stored so the monitor can calculate deltas rather than relying only on day-to-date totals.

We also guard against missing samples after deployment. Treating an absent previous value as zero could manufacture a fake billion-row spike.

Three ways a storage storm can trip

Pace

If day-to-date consumption is already far ahead of where historical baseline suggests it should be, raise a finding.

This catches slower burns early in the day.

Rate

Extrapolate the latest short-window delta.

If the current slope would produce an abnormal full-day result, raise a finding even if the cumulative total still looks harmless.

This catches sudden accelerations.

Absolute hourly floors

Relative comparisons are not enough for cliffs.

So the radar has absolute floors calibrated from this incident.

Read storm

≥ 10 million billed rows read/hour, sustained across two consecutive hourly buckets.

Healthy TenantDO activity is normally around the low hundreds of thousands to roughly one million reads per hour.

The incident produced 50–70M/hour even during its quieter morning phase and as much as 0.3–1.4B/hour overnight.

The point of 10M is to alert far below the level we experienced.

Write storm

≥ 2 million billed rows written/hour, also sustained.

Writes are the expensive meter.

A healthy Lucimark day produces only a fraction of that number in total. Millions of writes packed into individual hours therefore deserve immediate attention even before a relative baseline catches up.

Detection should name the lever

When the radar fires, the alert is built for someone who has not spent the previous twelve hours reading migration code.

It contains:

• current Durable Object storage metrics;
• the abnormal finding;
• directional cost information;
• recent baseline;
• the appropriate first-response control.

A read storm points first to the alarm/housekeeping kill switch.

A write storm points first to the ingest/write-amplification controls.

This is the difference between observability and mitigation.

"Spend is high" is an observation. "Park the alarms" is a runbook.

On a loop that does not naturally stop itself, the second is what bounds the incident.

Backstops

A 15-minute production monitor is useful, but no single monitoring path should be trusted as the only line of defense.

We added two additional mechanisms.

Daily overnight check

The daily operational snapshot independently checks for abnormal Durable Object read rates and can raise a separate notification.

The goal is to catch the class of problem most likely to get a long head start while nobody is watching.

Backfill-state inventory

Namespace metrics tell us that the TenantDO fleet is abnormal.

They do not necessarily identify which tenant is responsible.

So the snapshot also inspects Proof of Play migration state across active tenants and reports incomplete or unknown migrations.

The states are surfaced worst-first:

indexing → deduping → updating → pending → done

Timeouts are classified as unknown rather than healthy.

That is the inventory we did not have on 2 September.

At the time, thousands of identical error lines had to be traced back to one tenant manually.

After every deploy

Production API, app, and player deployments now arm a temporary higher-sensitivity Cost Sense watch.

During that window:

• pace thresholds tighten;
• rate thresholds tighten;
• abnormal Durable Object behavior becomes easier to trip;
• any alert includes deploy context.

This migration entered production as a routine API deploy.

A future storage regression should not receive an overnight head start simply because HTTP error rates look acceptable.

What changed in our engineering rules

What already holds on the platform, in brief:

• TenantDO boot stays cheap schema only — no large-table DML or expensive uniqueness rebuilds;
• heavy backfill is asynchronous, batched, resumable, marked done, and safe to re-run after a reset;
• cooldowns on O(n) states;
• test the tail: the largest real dataset, not the median;
• Cost Sense storm radar, alarm kill switch, post-deploy watch, and backfill inventory.

A circuit breaker here is a limit that stops retry after N failures — pause, alert, and require a state change — instead of letting the tick repeat forever. It is not yet on the UNIQUE path; it sits in the right-hand column.

The radar does not make a bad migration inexpensive. It reduces how long a bad migration can remain unbounded.

What we would tell another team running SQLite on Durable Objects

Most of that is already in the table. What does not fit a two-column grid:

Split observability by failure surface. HTTP charts will not tell you SQLite is scanning itself. Watch rows read / rows written directly.

Failed DDL is a state-machine transition. If CREATE UNIQUE INDEX can fail inside a recurring worker, design that transition before shipping.

A useful alert names the lever. Not “spend is high.” Yes: “this pattern matches an alarm-driven scan loop — park the alarms.”

Hope is not a circuit breaker.

Closing

We shipped a Proof of Play data model change whose migration strategy underestimated one large TenantDO.

The first failure mode was expensive writes.

The second was more revealing: a 30-second alarm had turned O(n) migration work into a loop with no natural stopping condition.

We stopped the multiplier, fixed the state machine, removed historical data that did not justify an open-ended rewrite, and restored the system to its normal range.

Then we changed the platform so a similar failure should be visible much earlier.

The most important lesson was not the number of rows.

It was the shape of the failure.

A recurring operation whose cost scales with table size needs a bound before it reaches production.

For Lucimark, that now means stricter migration mechanics, cooldowns, kill switches, realistic high-volume testing, and a radar that does more than report that cost is rising.

It tells us where the amplifier is — and which lever stops it.

Lucimark's job is to let operators run screen networks with confidence.

That includes the parts nobody sees on the wall: migrations, rollups, storage behavior, and the cost of being wrong about a loop.