Backups Aren't Simple: One Engineer's Journey from Family Photo Loss to a 3-2-1 Backup
Aleksandar Filipovski recounts how his family lost all their photos when his dad reformatted a drive for a TV set-top box. That early lesson leads him through the escalating complexity of real backups: snapshots, rotation, deduplication, databases, offsite storage, and finally the 3-2-1 rule. He concludes that rolling your own is a trap—use Borg or Restic, and always test your restores.
There are two types of people: those who have suffered a catastrophic loss of data, and those who will.
- cuvert
For my small library photos, keepass passwords, etc around 30gb, I use syncthing, which will backup on my pc, on my parents pc (they live in another country), on my sisters pc and on my phone. They also use this triangular mode to backup their stuff. Syncthing is setup to only sync through tailscale or home routers network.
- dirkc
> “There are two types of people: those who have suffered a catastrophic loss of data, and those who will.”
When I was a teenager, I was the reason for data loss for my dad, twice. Both times it was because I was re-partitioning a hard drive to install linux.
You would think that taught me a lesson about backups, instead it reminds me every now and again to be grateful for an awesome dad and aspire to handle situations with my kid similarly :)
- publlus_enigma
There are four times in my life I have suffered regrettable data loss incidents.
The first was when the telephone pole outside our house was struck directly by lightning. Not only was it the loudest thing I have ever heard, the current surged through the telephone line, into the internal fax modem, and fries everything within its vicinity. I was 10. I did have backuos, but only only floppy and they didn't cover everything.
The second was storing data in OneDrive - a change to their terms surrounding "lifetime" unlikely noted storage, combined with a client that was unusably slow to download and a deadline for data retrieval meant that I lost most of my files.
The third was SD card failure in digital camera on holiday, the controller chip died catastrophically, leaving the card completely unrecognised. It was a brand new Sony 128GB card, manufactured by Toshiba, and it seemed to be a common issue. I now shoot to two cards simultaneously.
And the fourth time was ... Performing a backup. An errant script deleted the source content, but I'd also deleted the existing backup to free up space for the new backup. I've been weary of using rewritable media for some time now as a consequence, but I think backups themselves are high risk activities.
- abejora
I agree with this sentiment! For our Abejora timesheet SaaS, setting up backups was one of the harder parts. We had to change direction a few times while implementing it.
We finally got a nice setup with CloudNativePG + Barman. This allows for point-in-time restores, but there were a lot of lessons to learn along the way.
- The various types of (database) backups (logical, binary, onsite, offsite, snapshots, write-ahead log...) in combination with the various types of data (database, files, cluster configuration...)
- In our earlier approaches, we tried to preserve the old database volume if it was not corrupt, and use that in our restore. This caused so many complications, because you are fighting the recommended approach. So now, when we need to restore, we always restore from backups and the 'live volume' is dropped.
- For a restore, we just spin up a completely new Kubernetes cluster, instead of trying to restore in-cluster. This is a lot easier.
- Many object stores allow for retention periods, which you can put to good use to prevent malicious or accidental removal of backups. HOWEVER, not all of them are really 'locked'. In some services, you can still override the lock with a forced delete; in others, you can still remove the project holding the storage buckets, which will delete the buckets, and so on... so test those things, instead of just blindly depending on a 'retention period' claim.
- We now automatically run a scheduled restore with verifications on a weekly […]
- AdieuToLogic
A friend of mine used to work at Veritas[0] making enterprise data retention solutions. When I spoke about their product as being "making backups", he corrected me by saying:
We are not in the backup business. We are in the restoration
business.
- Helmut10001
I really like ZFS snapshots with offsite pull-mode sync using
Jim Salter's sanoid/syncoid [1]. ZFS is the base for all OS/filesystems on top of it. If you have a good system for organizing ZFS datasets, and separating ephemeral from persistent data (e.g. [2]), then this is 90% of the backup requirements already fullfilled.
[1]: https://github.com/jimsalterjrs/sanoid
[2]: https://du.nkel.dev/blog/2026-05-16_rootless_docker_virtiofs_proxmox/
- ebrahimh
I’m setting up 3-2-1-ish backups for my infra of 3 hosts, and definitely leaning towards Restic + Backrest.
All my hosts run the same CoreOS setup (https://github.com/ebrahim37/infra-template), where container volumes are placed in one central volumes/ folder and that is the only thing I have to backup.
I plan to implement it like this:
vps1:
- restic container with custom sh entrypoint that will backup volumes/ to homelab every 24 hours
homelab:
- backrest container, to back up volumes/, do prune/check, replicate repo to offsite
- rest-server container, will store backups from vps1, homelab, offsite
offsite:
- restic container, backs up volumes/ to homelab every 24 hours
- rest-server container, store copy of backups from homelab
Only caveat is backing up databases, will either have to do: stop container, backup volume/database-data, start container; or use pg dump etc.
The deduplication is nice, you can have a snapshot for each week of the past year without crazy storage cost
- bloomingeek
"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)