Table of contents
Data loss #
Firstly, a premise on my troubled experience with storage, hard drives and data management.
I messed up several times in handling my few (three) nevertheless too many hard disks. I lost all of my data two times.
The first time it happened it was summer 2015 and I was at Giffoni Film Festival; I was importing on my MacBook some pictures I just shot with my camera. After completing the import, I needed to free the SD card where the images were, and I mindlessly formatted it from MacOS’s Disk Utility without remembering I had my hard disk plugged, too. What I did was resetting the hard disk instead of the SD Card. Luckily, without overwriting anything, restoring data from a formatted device is not too complicated, and I managed to recover most of what I had on that memory device.
I gotta admit that my first data loss did not teach me anything about care and meticolousness in handling external hard drives, especially when there’s only one single copy of your data.
Sure enough, it happened again.
In 2018 I was in a hurry and I had to quickly go from my bedroom to the living room to connect through LAN my MacBook and send some images ASAP. Those pictures were on an hard disk which while I was running from one room to another fell and broke.
As it’s easy to imagine, this damage was irreparable. I was shocked, angry with myself and profoundly dejected. The only opportunity I had to somewhat recover something from that device was giving it to a data recovery company in Milan which asked me 990€ for the process. I rejected the offer, but if I could go back I would have payed. Memories have no price, and I no find myself with no data whatsoever about my three arguably most exciting years of my life, up to now; no pictures, no documents, no tickets of concerts I went to, no school homework; nothing.
up to 2020 #
After the second data loss, I eventually learned my lesson. It’s worth spending a couple hundred bucks more and a few hours to backup everything twice than risking to lose data. For my 18th birthday, one of the gifts my lovely friends made me was composed of two identical WD hard disks of 2TB of memory.
Up to the end of 2020, everything I had fitted on the two devices, which contained two identical copies of the same data, so that if for any reason one of the two broke, I still wouldn’t lose anything. Even all of the 500ish GB of raw footage of Everyone did not fill the 2TB.
Now (time of writing: January 2021), space is running out. I need to find a valuable alternative. I will try to solve my main issues here, publicly. In this way, I could easily receive suggestions (if any, you may comment below) and anyone who shares my same doubts may get in touch and discuss some solutions to address the huge problem of storage with me.
Let’s point out my needs, my questions and doubts on the topic.
- Durability: I want to be able to effortlessly access all of my data now as 30 years from now.
- Reliability: my storage system has to be absolutely fail-proof (of course, no system is. I need to achieve the best reliability I can afford). I can’t lose any of my data again.
- Backward compatibility: the solution I’ll find has to be compatible with the current one.
- Affordability: I should not spend too much on this. In the end, shooting pictures is not my job (yet).
- Accessibility: if possible, I’d like to be able of accessing all of my pictures (mainly) from anywhere anytime
- Can I rely on Piwigo alone to host all of my JPGs, even my private ones?
- is it reliable? How much do I risk of losing my data?
- is it secure? Are my private pictures somehow server-side encrypted?
- Does it make sense to use my Cubbit Cell’s 4TB for long-term picture storage?
- Should I use portable Hard Disks, as the ones I already have, or internal hard disks to plug to Linuxplosion?
- This lecture on Backups, from MIT’s Missing Semester is useful to understand the reasoning behind safe backups. Note that what The Missing Semester evolves around is code, hence small if not tiny plain text files, while our interest is in huge amounts of data, such as high quality videos and images.
- 💽 How to Setup Software RAID with MDADM Comand on Linux Ubuntu in 2021 💾, a YouTube video of September 2020
- Ubuntu 20.04 Mdadm Setup Guide a YouTube video