Incident Investigation · Case Study

Styleguide Container Stuck in an Infinite Restart Loop

Every deployment sent one container into an endless Restarting (Exit Code 1) loop. The cause wasn't a broken build — it was a race no one had written a rule against.

The situation

A containerised Pattern Lab styleguide was crashing continuously on the test environment after a routine deployment. Docker showed the container stuck in a Restarting (Exit Code 1) loop. The logs were flooded with thousands of lines like:

restarting script because ../templates/_base/_annotations changed
restarting script because ../templates/_base/_patterns changed
...
Forever detected script was killed by signal: SIGKILL
Script restart attempt #1
Script restart attempt #2
...

The customer flagged it as urgent because staging and production run from the same codebase, and they were holding back a production deployment out of concern it might happen there too.

Digging into the architecture

The styleguide container runs Pattern Lab PHP, a tool that compiles Twig templates into a visual component browser — useful for developers and designers to preview UI components in isolation. Two containers share a Docker named volume:

🔄
rsync container
Watches the publications repo on the host, syncs changes in
💾
Shared Docker volume
~6,000 template files across six publication variants
🎨
styleguide container
Watches the volume, regenerates the styleguide on change

The styleguide used forever (a Node.js process manager) with chokidar (a file-watching library) to detect changes. Every detected change sent SIGKILL to the running generator and restarted it.

The race condition

On every deployment, docker-compose down -v wipes all Docker volumes. When the stack comes back up, the shared volume is empty. The rsync container then has to write the entire templates directory into it from scratch — roughly 6,000 files.

Both containers start at the same time. chokidar initialises, scans the volume to note what's already there, then announces itself ready. From that point, any new file appearing in the volume is treated as a real change.

The problem: rsync is still writing those 6,000 files when chokidar becomes ready. Every file written after that fires a separate restart signal.

Those signals queue up. forever processes them one by one — kill, restart, kill, restart — with no chance for the generator to ever complete. Eventually forever gave up and exited with code 1, Docker restarted the container, and the whole cycle repeated.

The motion-sensor analogy

Think of a motion-sensor light. Every time someone walks past, the timer resets to 2 minutes. The light only turns off once nobody has walked past for a full 2 minutes — it doesn't matter if 100 people walked past, it turns off exactly once. The old file watcher had no such logic: it was more like a light that flicked off and back on for every single person who walked by.

NO DEBOUNCE before
killrestart killrestart killrestart × 6,000
Generator
never finishes
Every file write fires its own kill + restart — the generator is killed before it can ever complete.
500ms DEBOUNCE after
writewrite write× 6,000 generate()
Generator
runs once
All 6,000 writes collapse into one regeneration, fired 500ms after the last write.

The fix

I replaced forever's built-in file watching with a Node.js fs.watch watcher using a 500ms debounce timer. No matter how many files rsync writes, they all collapse into one regeneration triggered 500ms after the last write. A busy flag prevents a second regeneration from starting while one is already in progress. forever stayed in place, but stripped of its file-watching role — it now only handles crash recovery.

fs.watch('/srv/templates', { recursive: true }, function(event, filename) {
  if (!filename) return;
  if (!/\.(twig|json|md|yaml|txt)$/.test(filename)) return;
  clearTimeout(timer);
  timer = setTimeout(regenerate, 500);  // reset on every event
});

The outcome

Restart loop eliminated
Container stable on test environment, confirmed with live log monitoring
Production deployment unblocked — the styleguide service doesn't exist in staging or production, so the issue was fully isolated to the test environment
Fix committed to the start-pack repository

Key takeaways

Shared Docker volumes between containers can create unexpected cross-container side effects — a write in one container is a file event in another.
depends_on in Docker Compose only guarantees a container has started, not that it has finished its initialisation work.
Debouncing is a simple but powerful pattern for handling bursts of events — wait for the noise to stop, then act once.

Chasing a race condition between containers that "shouldn't" be talking to each other?

Get in touch