[nylug-talk] Recovery is not about media, but availability and storage -- WAS: backup media

[Bryan J. Smith]

TOP-POST PORTION ...

I noticed you varied on context.  When you vary from the context, you
start comparing apples-to-oranges.  For example ...

- Commodity for cost v. reliability (e.g., SCSI/SAS won't apply)
- System interface v. media interface (e.g., SCSI v. cartridge)
- External v. Network Attached (e.g., USB/FW/eSATA v. network)

These are important considerations that you have to look at in detail to
understand my points.

Also, I will make these clarifications ...

- Over 99% of your operations are near-line, less than 1% off-line
- Near-line is disc, anyone who uses tape is 7+ years outta date
- Near-line is network-attached, not local/external (for many reasons)
- Off-line is your disaster recovery and long-term archiving
- Off-line should be high-G / closet storage tolerant media

Lastly ...

- 100% of the issues with off-line operations are people backing up
directly to (or restoring from) off-line media from end-systems, when
they should off-line from near-line, not end-systems.

100% of the complaints I've seen of tape is that last comment.  Some
people still "just don't get" the fact that you should _never_ backup
directly from end-systems to your "off-line" media.  There's no reason
for that at all, and it can be done cheaply (especially with 2.5" drives
coming down below $0.50/GiB).


On Wed, 2008-03-12 at 23:55 -0400, Alex Pilosov wrote:
> A very good question. Can you read 15-year-old tape today? I've done some
> research, it isn't as simple as you think.  You generally have
> compatibility with 2-3, sometimes 4 generations of tapes, for "mainline"  
> formats (some formats are just declared obsolete and nothing is backward
> compatible).  Example, current "best" DLT-S4 drive can read SDLT-1 tapes.
> Drive was released in '07, media was released in '98. That's only 10 years
> of backwards compatibility.

I have restored data from not just proprietary IBM as well as more
commodity DLT drives (I could throw 8mm in there, but that's really past
its time when I used it) after 15 years.  Commodity tape technologies
have a lifetime of around 2 dozen years.  There is also about a dozen
years in transition, where a newer, commodity technology shows up to
replace it.

DLT was the 1984-2004 format, but is being phased out (officially, as of
2007).  DLT typically has write compatibility back 2-3 generations (5+
years), read compatibility back 4-5 generations (10+ years) with the
"latest model."  But the previous 2-3 models were still sold, so you
could buy a _new_ drive that could read back at least 15 years.

LTO is the 2000-'25 format.  It also has write compatibility back 2-3
generations (we're possibly going to see 4), with read compatibility at
least 4 (and likely 6).  Again, figure the same 5/10/15 rule -- at least
5 years write with the latest model, at least 10 years read with the
latest model, and a model still made, new, that could read 15 years.

> Compare this with hard drives. Technology for the *interface* between 
> drive and computer changes far less frequently than the tape media. Your 
> 20 year old SCSI-1 drive going to work just fine today. Similarly for ATA. 
> Lifespan of a interface standard is ~20 years.

Not true!  Not true at all in the case of ATA!

Do not compare "commodity" ATA as if it's the same as SCSI SE or LVD.
ATA has a very _poor_ history with compatibility and, more importantly,
"recovery."  Don't fool yourself by "switching" a non-commodity
technology to fit the commodity sell.  ;)

It wasn't until a dozen years ago that ATA even added checking for DMA,
and it's still a major compatibility issue.  Also, plugging in a
pre-2000 ATA drives into today's ATA is a major PITA.  Many pre-ATA, as
well as ATA-1/2/3 spec'd drives have issues in newer ATA-5/6
controllers.

ATA devices also have re-mapping issues that are vendor-centric.
Sectors do go bad.  SCSI-2 protocols let you remap easily in any OS.
ATA does not.  Hopefully ATA standards will get there, and be
implemented in libata, but so far that has not been the case -- not even
for "disconnect" (which SCSI-2 has handled for a long time).

The "disconnect" issue with ATA is holding up eSATA.  It's why most
enterprises are "paying out" for SAS instead of eSATA, despite the
non-commodity pricing.

> As far as off-line storage, I don't know.

That is 100% what I'm talking about, "off-line."  I would _never_ argue
to use tape for "near-line" backup.  99% of your operations are going to
be "near-line."  In the majority of my integrations, I _rarely_
implement off-line backup more than once per week, and one off-line
"verify" once per month.

Disc is the way over 99% of backup/restore operations should happen.
Understand that's pretty much universal now, except for people who "just
don't get" the whole on-line -> near-line -> off-line reality.

> I frankly doubt that drives would be much worse than tapes,
> based on empirical experience of dusting off some old drives,
> but I don't know if anyone has done more specific research.

First off, we've all had our "experiences."  Some of us have been in
large environments to get some great samples.  For example, when I was
at one major client, we experienced a 10% commodity drive failure rate
in one 6 month period in 2005.  The majority were 24x7 usage or regular
off-line periods more than a week.

Ever "knock" a powered hard drive to get it to "spin up"?  I always get
a laugh when that works, and people think I'm joking.  The lubricants in
the drive have solidified or otherwise lost their viscosity.  That's
just one problem with the materials used in "commodity" disk
technologies.  I'm not talking enterprise or mobile, but "commodity"
disk technologies.

The whole "near-line" concept that evolved into backup storage, commonly
marketed as "virtual tape libraries," were basically out of such varied
research from the late '90s to early 2000.  It was basically that
commodity 3.5" disks should regularly be exercised, but not spun 24x7,
but in so "managing" them in such a way could produce very affordable,
very reliable, thermally distributed (e.g., only 1/3-1/4th of the discs
would be spinning in a unit at any time).  That made them ideal for the
"near-line" store, at 1/2-1/5th the cost.

One of the best whitepapers I ever read was from IBM's Storage Group
just before they sold the unit to Hitachi.  IBM, at the time, was still
reeling from the 5 x 15GB platter in 1" fiasco, and took a long-hard
look at drive failure rates.  It caused IBM to start invalidating both
current QA (e.g., vibration tolerances), as well warranties for various
things, including greater than 14x5 operation or over 1 month downtime.

It also lead to the (now) industry-wide labeling of "enterprise
commodity disk," where vendors test the vibration, thermal and other
tolerances of their disks.  E.g., Western Digital Caviar SE (Standard
Edition) as the Caviar RE (RAID Edition), the Seagate Barracuda 7200 to
the original NL35 (Near-line 3.5") and now Barracuda ES (Enterprise
Storage), etc...  They are fabbed 0 different than standard, commodity
3.5" disks.  They are just tested to better vibration and other
tolerances.

Enterprise fabbed capacities as well as 2.5" mobile designs are far more
tolerant of G forces, off-line periods, etc... because more costly
materials are used.  That's why the ... tada ... cost more!  ;)
Although we're now starting to see a phase-out of traditional
Enterprised fabbed capacities as 2.5" Enterprise 10-15Krpm designs are
becoming popular, are much smaller, use less power, etc...  They are
quickly becoming the preferred SAS drive in the data-center as a result.

> Only takes someone inventing a hdd tray with integrated shock mount. :)

But then that makes it like ... tada ... "cartridge"!  ;)  There is no
"standards" to that either.  And there is shock involved in mounting and
unmounting it from tray.

> Sounds like bs, sorry. Reference, please.

Assume it's BS.  Assume all you want.  That's fine.  Or you could hit
the engineering data sheets on products, and they will point out things
like shock and operational assumptions.  IBM used to be very explicit on
theirs, before selling to Hitachi, especially on the 5-platter
"Deskstar" -- which has re-appeared from Hitachi in various, newer
5-platter products (with similar issues).

> <lots snipped> It's not about the name. It is about pricing.

No, it's about the concept, "virtual tape library" (VTL).  That's the
marketing name attached to many commercial products, because they offer
that "legacy software interface" to what is, otherwise, a "software
switched," near-line disc-based storage array.

You don't have to buy a labeled "VTL" to get one.  I've implemented many
with rsync and other tools in scripts.

> Saying word 'storagetek' makes you fail, immediately.

Amazing.  One of the key architects of StorageTek left the company to
focus entirely on near-line disc solutions.  Didn't you catch that
point?  Not long afterwards, StorageTek sold out to Sun (because their
products weren't as viable anymore). 

I thought you'd catch that.

Like everything, things start with the "enterprise size," such as Copan
System's MAID implementation.  MAID is about distributed (25%) disc
operation, thermals, rotation and exercise.  But the concept of the 8 x
14-drive "can" (112 drive) in a 4U "shelf", 8 x 112 drive shelves in a
32U rackmount, doesn't have to be that big.

E.g., it could be a 3 x 14-drive (42 drive) a 2U "shelf", if you arrange
the "cans" on their side, and that still leaves room for intelligence in
the 2U unit.  The operational principles of MAID are pretty universal.
Again, things start "big" to afford initial R&D investments, then
"trickle down" into commodity designs and pricing.

> Currently, price of the enclosure is still far higher than price of
> drives.  When it stops being the case, then you can say its "MAID"
> or whatever.

Sigh.  From lack of hearing of VTL (let alone thing I'm talking about
"tape") to dismissing the future of MAID (MAID complements blade for
storage, and there are now 2U "blade" designs coming out too), I don't
know what to tell you.  I was trying to point out _exactly_ what you
were hoping for, that it's becoming reality.

> Reference, please. 

If it makes you happy, assume I'm pulling it right out of my rectum.  ;)

The commodity, 3.5" fixed disc makes an outstanding, near-line storage
technology.  Managed in groups for power, thermals and moderate
exercise, it lasts very well.  That's why it's ideal for network
attached storage, including for VTL and related near-line recovery
solutions.  Those solutions are then network-attached, providing a
completely independent near-line implementation from the desktops and
servers they efficiently back-up via a nightly (or even hourly) "delta."

External, commodity 3.5" fixed disc has largely suffered due to the lack
of complementary technologies being done well.  USB was always a joke
for block interface (never designed for it).  FireWire promised much,
but even Apple conceded varying disconnect and other reliability issues
several years ago.  eSATA has bit me really hard just recently, and even
my discussions with Seagate has turned up disc vendors who just don't
want to certify any cabling or other, physical interface details -- let
alone the ATA command implementations are just lacking (as always).

That's why I cannot recommend external, commodity 3.5" fixed disc.  I
have been bitten way too many times.  Non-commodity SCSI or SAS is not
an option for small businesses either.

For a small company with only a couple hundred GBs to off-line every
month, I always deploy two (2) servers:  An on-line data server with
near-line recovery server (which also backs up desktops).  For off-line,
I use a 4 x 2.5" drives (160GB is the "best bang for the buck" now) on
trays in a 5.25" half-high bay in the near-line recovery server,
typically connected to an inexpensive 3Ware 8506-4LP card.  For larger
companies with several TB of data they need to off-line every month, I
deploy an LTO drive attached to the near-line recovery server.



-- 
Bryan J  Smith              Professional, Technical Annoyance
mailto:b.j.smith at ieee.org  http://www.linkedin.com/in/bjsmith
-------------------------------------------------------------
           Fission Power:  An Inconvenient Solution