⏳ אני חי — ומתרגם בשבילך ממש עכשיו! ✨ אנחנו רוצים שכל אחד בעולם יקרא בשפה שלו, בלי מחסומים. הגרסה בעברית תופיע כאן בעוד רגעים — הדף מתרענן מעצמו.

← блог

The Server Lost Power at Night. Here's What We Built So the Silence Never Happens Again

23:41, 15 October 2026. The power goes out. Not unusual — it happens a few times a year in our area, usually for seconds. The home server where part of our own systems and client sites live sits on a UPS. The UPS holds for a bit, power doesn't come back fast enough, the machine shuts down.

Power returns at 23:58. The server boots itself — that's what the UPS and a BIOS setting are for. But the virtual machine that actually hosts the sites doesn't start. Because nobody had ever ticked "start automatically when the host boots." A small thing. The kind of small thing you never notice until this exact night.

At 00:10 power drops a second time. At 00:26 it's back for good. The host comes up on its own. The virtual machine — again, doesn't.

Why nobody called us

We run a watcher that checks whether the sites respond and pushes an alert to Telegram if something's wrong. It's worked fine for months. Except that watcher lived on the same machine that just went dark. The guard and the alarm were in the same house — when the house went dark, so did the alarm.

We found out by accident, around 01:38, when one of us opened a phone for an unrelated reason and noticed a page wasn't loading. From the first outage to the moment we actually knew, roughly 1 hour and 50 minutes passed. That's how long the sites sat dark with nobody — not us, not any automated system — aware of it.

TimeWhat happened
23:41First power cut
23:58Power returns, host boots itself, the VM doesn't
00:10Second power cut
00:26Power stable for good, same problem repeats
01:38We notice by accident, on a phone
~01:50Sites manually restored

Times are reconstructed from logs and our own memory of that night — we won't claim second-by-second accuracy, but the sequence and total length are accurate.

The lesson, before the fix

A guard that lives in the house it's guarding isn't really a guard. If your alerting depends on the same power, the same router, the same cloud region as the thing it's supposed to watch, it tells you nothing precisely when you need it to tell you something. This applies equally to a home server in a living room and to "production" sitting in a cloud virtual machine — if the monitoring sits in the same account, same region, same provider, it will go quiet exactly when you need it to speak.

What we built in a day

The next morning we stopped regular work and built five things. Not because that night was a catastrophe — it was two quiet hours, no client lost a payment or a booking in that window — but because next time the timing might be different, a weekday afternoon with real orders moving through the system.

  1. A second, small machine in another country. A cheap virtual server with a different provider, physically elsewhere. It holds a mirror of the files and databases, synced every 5 minutes. No long history — just the last known good state.
  2. An outside watchman. A third, independent provider checks whether the main server responds — not just a ping, but actually loading the homepage and checking the content, not merely whether a port is open.
  3. Automatic takeover at night. If the main server fails to answer correctly three checks in a row, traffic shifts to the mirror. The decision is made by a machine that has nothing to do with the first two.
  4. A morning report with buttons. If a takeover happened overnight, the morning brings a message with three buttons: "keep the mirror live," "switch back to the main server," "show the logs."
  5. One-button rollback. Once the main server is confirmed healthy, switching back is one action, not a manual procedure at 2am with a tired head.

The honest math — what we didn't solve

We won't sell this as magic. Two trade-offs remain, and we're saying them plainly:

For a brochure site with no bookings or orders, two quiet night hours cost nothing. For a system handling payments, shift bookings, or live orders, the same two hours cost real money and trust. We judge this honestly — not every project needs a mirror in another country. A small share of our clients genuinely do.

What this means if you don't build software for a living

You don't need to build a system like ours to check your own exposure. Three questions for whoever runs your hosting or site maintenance, whatever that setup looks like:

If the answers are vague, or "we haven't thought about it" — you're probably not alone. Most small systems have no outside watchman, because nobody has had a night like ours to make them think about it.

What we built for ourselves is now available as an option for client systems that want it, through our control panel MAESTRO: a mirrored standby, an outside watchman, automatic failover. It's not default on every project — for most sites and small systems it would be an unnecessary expense. For a system where two quiet night hours cost real money, it's cheap insurance.

If your own system has gone dark some night and nobody called you — describe the situation in one sentence to the live agent at pentimento.pro and see what a fix could look like. Or just send us the question — it might become the next article.

PENTIMENTO
Tell the agent your idea →