Computer backups: after recommendations for offsite

[SpeedyChix]

Mac household, CarbonCopyCloner to SSD for regular incremental backups. Looking for recommendations for offsite addition. Apple cloud or other? Doesn't need to be a daily or constant update but most likely an encrypted upload. Something that we'd do monthly? TIA.

[snotrockets]

Apple cloud is very problematic for backups:

* There isn't any good tooling to automate interactions with it.
* Last I checked, Apple doesn't provide any durability guarantees.
* While it provides encryption in-transit and at-rest, Apple can decrypt the data at-will (look at the section titled "end-to-end encrypted data" here. Anything not listed there, Apple can unlock).

I'm not yet paranoid enough to worry about protecting myself from state level entities, but the first two are a deal breaker.

The best in business online backups solution I know of is tarsnap, but the tooling requires some learning curve if you're not used to admining UNIX.
That being said, I haven't found anything close, and neither are any of the professionals I've asked.

BTW, considering consumer HDs have annual failure rate of about 10% (depending on brand and model,) and that such failure rates are catastrophic, that broadband is highly available, and that cloud object stores have 11 9s annual reliability per object (and not per the whole account) I stopped bothering with on site storage, and don't think anyone should, except for highly specialized cases (shitty connectivity, large amount of new data, etc.)

[sk_tle]

I was about to recommend rsync.net which is similar to tarsnap. It is supposedly very crude, i.e. they don't provide a specific proprietary software but show guidance to use rsync/rclone. But their support is top notch and will actually set it up for you. Pricing is simple and easy to understand without hidden costs.

[NYCfixie]

Mac household, CarbonCopyCloner to SSD for regular incremental backups. Looking for recommendations for offsite addition. Apple cloud or other? Doesn't need to be a daily or constant update but most likely an encrypted upload. Something that we'd do monthly? TIA.

What you are taking about really is disaster recovery more than backup.

Find someone you know a few thousands miles away and send them hard drive "A" every other month. And then have them send hard drove "B" back to you every other month. If you encrypt the drives, and add up the cost of shipping, and the cost of the drives, it should be cheaper and easier than any backup service. In an emergency, you can have you drive back in 24 hours. With backup services, it could take several days to restore you data over even the fastest consumer internet connection. If the other person wants to do the same, then the shipping costs half as much for both.

If you want offsite "storage" more than once a month, this plan falls apart.

The other option is to find (or use the same) buddy and send each-other encrypted files. Send them to each-other's computers overnight so it does not eat bandwidth when you really need it. But again, restoration time may take several days.

[snotrockets]

Find someone you know a few thousands miles away and send them hard drive "A" every month. And then have them send hard drove "B" back to you. If you encrypt the drives, and add up the cost of shipping, and the drives themselves, it should be cheaper and easier than any backup service. In an emergency, you can have you drive back in 24 hours. With backup services, it could take several day to restore you data over even the fastest consumer internet connection. If the other person wants to do the same, than the shipping costs half as much
for both.

If you want offsite backup more than once a month, this plan falls apart.

Or even if you just do backup. Huge part of my job is estimating cloud costs, and the math doesn't work out, even with that atrocious SLA:

* 1TB HDD is ~$50 + tax. You need at least three of those (one offsite, one onsite, one backup) = $150. Annual failure rate is 10%, but that's amortized - old drives are more likely to break than new ones, and ones constantly shipped around are more likely to fail. You can buy more resilient HDs, but the costs for those are sky high. To make the calculations easier, we'd put the drives cost at $150 for the first year, and $50 for each year after. Note that you'd need bigger drives now and then as you incur more data (even if you don't add, just change. A good backup has old copies of the file, in case you overwrite them with a bad version, and only realize it after you recycle the old backups), and that's an upfront costs.

* A small priority mail flat rate box is ~$7.5 You need to ship one drive each way every month, so 24 times a year, so $180. We assume a restore would happen on schedule, so we save the extra mail.

That's assuming everything works. From experience, I can assure you it won't.
You want to try to restore every your files every now and then (if you never tried to restore your data, I promise you it won't work when you need to). You'd need to buy that friend a good dinner now and then for the effort. But let's assume software is perfect, labour is free, and the above costs are all you need.

1st year would be $330, and every year after would be $230, assuming the data set never increases in size.

Now, consider the cost for online storage. I'd do Tarsnap, but that would mean taking labour into account (where you get for free there), so to try and even the playing field, we'll do AWS S3 Standard Tiers. No cutting costs! (though you can probably can go to lower tiers & save a bit. Or use another cloud object store). First 50 TB are at $0.023/GB/month, so for your TB, that's $276 a year (you also scale linearly w/ storage size). That's the equivalent to the HD cost, but notice that it isn't a step-function like there, but scale linearly.

Now the transfer costs: transferring data into AWS from the internet is free, so backups are only the initial API requests cost. I'll put that at 10K/month, so $0.05/month, even though it's probably won't even be close to that with efficient software.
And let's say we need to restore one backup a year, so we have a month where we retrieve 0.5 TB, and we pay for the data transfer. That's $0.09 for each of the 499GB (the first is free!), so $44.91 for that restore.

In total, we got $275.51 every year. We have a durability of 11 9s, scale linearly with size, get almost immediate backups & restores (if we're ready to wait for 24 for a restore, you can cut price to about 1/5 of the above), and don't need to buy that friend dinner.

With a good system in place, it'd be much less – because you do incremental, compressed backups (in practice, I've seen actual Tarsnap costs at about ~$60/year for 1TB of source data – a good amount savings comes form that Tarsnap only charges for the post-deduplication & compression, which reduces the actual set data by a lot).

Sending HD by mail only makes sense if you have a truck full of them.

[j44ke]

What is the goal for the online backup? Security in case of fire, theft or loss of the device?

[snotrockets]

What is the goal for the online backup? Security in case of fire, theft or loss of the device?

tbh, I see no reason for local backups anymore, unless you are legally required to, or have so much data you need an actual fedex truck full with drives to haul it away.
Bandwidth is cheap, highly available online service also (offline backups are different, but I'm assuming my online backup provider backups his data offline as well, so no need for me to do it. I only have local and remote offline backups for the encryption keys).

[SpeedyChix]

Goal: Offsite backup would be in case of fire. So it's not really ongoing backup but periodic full data stored in another location. When I worked out of home, I'd simply swap SSDs and could do similar with an out of town family member. On rare occasion I've needed to restore a few files but the only time it's been a complete drive is when I switch to a new computer (data transfer) and Carbon Copy Cloner backups make that simple and fast.

Thanks for help.

[snotrockets]

I was about to recommend rsync.net which is similar to tarsnap. It is supposedly very crude, i.e. they don't provide a specific proprietary software but show guidance to use rsync/rclone. But their support is top notch and will actually set it up for you. Pricing is simple and easy to understand without hidden costs.

Not so similar: Tarsnap is a frontend to AWS S3, so you mostly get their guarantees (11 9s, world wide redundancy, elasticity).
You could think about Tarsnap's cost as one itemized line for both AWS and the backup software.

rsync.net are just selling shared hosting, with weaker SLA (for personal backups, it's probably enough.)

[j44ke]

Maybe. Assuming the financial health of the service provider doesn't suddenly fail. That never happens in tech?

Anyway, I was just wondering what Speedy's goal was for an online backup versus a local backup.

[SpeedyChix]

Maybe. Assuming the financial health of the service provider doesn't suddenly fail. That never happens in tech?

Anyway, I was just wondering what Speedy's goal was for an online backup versus a local backup.

Mostly a fire issue. Prior to travel with laptop it gets backed up so theft wouldn't = loss of all data. Without travel, the backup gets run at unspecified intervals. Tend to run it when I am in the midst of a project or have just completed one. O'wise it's 2-3x/month.

[sk_tle]

What is the goal for the online backup? Security in case of fire, theft or loss of the device?

tbh, I see no reason for local backups anymore, unless you are legally required to, or have so much data you need an actual fedex truck full with drives to haul it away.
Bandwidth is cheap, highly available online service also (offline backups are different, but I'm assuming my online backup provider backups his data offline as well, so no need for me to do it. I only have local and remote offline backups for the encryption keys).

offsite backups can save your ass in case of fire or theft. Local are still nice for quick recovery.

Some online backup services have an option to ship you a hard drive for quickee recovery in case of total loss. At a cost obviously.

[sk_tle]

Not so similar: Tarsnap is a frontend to AWS S3, so you mostly get their guarantees (11 9s, world wide redundancy, elasticity).
You could think about Tarsnap's cost as one itemized line for both AWS and the backup software.

rsync.net are just selling shared hosting, with weaker SLA (for personal backups, it's probably enough.)

I personnally use wasabi which uses s3 protocol but outside of aws and a pricing that felt more convenient/easy to understand to me. It supports snapshot whise management is accessible only with a different api key so I should be able to restore the previous data even if a ransomware started crypting my remote bucket.

With restic as the software to handle the backups/encryption.

One thing to keep in mind is keeping a copy of your encryption keys in a safe but reachable place. There is nothing worse than knowing you have a complete backup but you not being able to decrypt and recover it. I am personnally storing a copy in an sdcard in my company's office (I have a calendar reminder to test the cheksum of the file every few months) + in my password manager that is also sync to my phone.

[j44ke]

Mostly a fire issue. Prior to travel with laptop it gets backed up so theft wouldn't = loss of all data. Without travel, the backup gets run at unspecified intervals. Tend to run it when I am in the midst of a project or have just completed one. O'wise it's 2-3x/month.

Makes sense.

My wife is considering a career move where she would be responsible for attorney-client document management. I have no idea how we are going to handle that, but she seems pretty confident about it. Her clients are very often in disputes with other entities who may or may not be countries. I almost think that she should put everything on a computer that never gets connected to the Internet.

[sk_tle]

[

Makes sense.

My wife is considering a career move where she would be responsible for attorney-client document management. I have no idea how we are going to handle that, but she seems pretty confident about it. Her clients are very often in disputes with other entities who may or may not be countries. I almost think that she should put everything on a computer that never gets connected to the Internet.

Some still use tapes for that. I used to work for a bank and a copy of every night backup was transported in tape form in a brief case by a security company to a safe in another bank.

[NYCfixie]

As to your AWS (and Tarsnap ) plan, it falls apart because the average user has no idea what AWS is, let alone how to use it. And, once you tell them that if they lose the Admin password their data is gone for good, you have lost the few that may have even been willing to try it.

...Now, consider the cost for online storage. I'd do Tarsnap, but that would mean taking labour into account (where you get for free there), so to try and even the playing field, we'll do AWS S3 Standard Tiers. No cutting costs! (though you can probably can go to lower tiers & save a bit. Or use another cloud object store). First 50 TB are at $0.023/GB/month, so for your TB, that's $276 a year (you also scale linearly w/ storage size). That's the equivalent to the HD cost, but notice that it isn't a step-function like there, but scale linearly.

Now the transfer costs: transferring data into AWS from the internet is free, so backups are only the initial API requests cost. I'll put that at 10K/month, so $0.05/month, even though it's probably won't even be close to that with efficient software.
And let's say we need to restore one backup a year, so we have a month where we retrieve 0.5 TB, and we pay for the data transfer. That's $0.09 for each of the 499GB (the first is free!), so $44.91 for that restore.

In total, we got $275.51 every year. We have a durability of 11 9s, scale linearly with size, get almost immediate backups & restores (if we're ready to wait for 24 for a restore, you can cut price to about 1/5 of the above), and don't need to buy that friend dinner.

With a good system in place, it'd be much less – because you do incremental, compressed backups (in practice, I've seen actual Tarsnap costs at about ~$60/year for 1TB of source data – a good amount savings comes form that Tarsnap only charges for the post-deduplication & compression, which reduces the actual set data by a lot).

I also gave this option which is almost zero cost and more user friendly.

...The other option is to find (or use the same) buddy and send each-other encrypted files. Send them to each-other's computers overnight so it does not eat bandwidth when you really need it. But again, restoration time may take several days.

[snotrockets]

Makes sense.

My wife is considering a career move where she would be responsible for attorney-client document management. I have no idea how we are going to handle that, but she seems pretty confident about it. Her clients are very often in disputes with other entities who may or may not be countries. I almost think that she should put everything on a computer that never gets connected to the Internet.

If you're worried about state-level entities, you need to contact some really paranoid people. Knowing even the quasi-paranoid, they would go and epoxy the network ports on the computer, and would declare even that as not enough (if you need refs for such people, I can ask around. But such people are an acquired taste)

[snotrockets]

As to your AWS (and Tarsnap ) plan, it falls apart because the average user has no idea what AWS is, let alone how to use it. And, once you tell them that if they lose the Admin password their data is gone for good, you have lost the few that may have even been willing to try it.

You don't need to do what AWS is. You need to know what their SLA is, what the SLA for HD-over-the-mail is, and who can provide support (and you can get one ay additional cost).

And as for safeguarding your keys, this is true for every system – either you put the key in escrow (like Apple disk encryption does,) and trust the escrow service, or guard it with your life. Consider a password or unencrypted backups in a safe deposit box – if you can open it, you have the key. If it's secure enough only you can open it, and you loose the ability to…

And tbh, I wouldn't send even my mom an unencrypted HD. Not because I don't trust her, as much as danger from theft and misdeliveries.

I also gave this option which is almost zero cost and more user friendly.

Same issues as shipping hard drives, minus the USPS part.

The reason you go to a commercial service (AWS, Wasabi, Tarsnap, rsync.net) is because they have SLA. You can DIY, but then you have no guarantees but your own. So it's a sliding scale of effort/$/reliability. Have at it.

[snotrockets]

Maybe. Assuming the financial health of the service provider doesn't suddenly fail. That never happens in tech?

That's why I usually recommend AWS S3 for data you care about. My assumption, following what a Netflix VP once told me, is that if Amazon/AWS suddenly fail completely, the market is probably in such a bad shape, Netflix/your data doesn't matter anymore.

With backups, not as big problem as Netflix would be at: I'll just start backing up my data elsewhere. The problem is if a perfect storm kills both Amazon and my drive at the same time. And under such a storm, I don’t know if I'd care about my bytes anyway. Would probably be more concerned about not being eaten by Zombies.

[snotrockets]

One important thing, regardless of how and where you keep your backup: every once in a while, actively RESTORE some part of your backup. Make sure it is valid.

More than once some <big co> found out there was fault in backups for a <very long period>, and they didn't know about it because they never bothered with retries.
<big tech co I can't mention by name in a public forum> actively restores 2% of their backups (they have a few exabytes of those,) to make sure the backups system are all performing like they should.