It shouldn’t be this difficult.
NOTE: This blog grew out of my webinar last week on how to archive Final Cut Pro projects. If you are interested in archiving, click here to learn more.
The process of creating films and videos generates LOTS of data. From getting the footage shot, through the editing process, we are dealing with dozens, if not hundreds, of gigabytes of data.
And we solve this problem by buying ever more and ever larger hard disks and RAIDS. So far, a trifle expensive, but not really a problem.
However, once a project is done, we can’t just throw all these files away. The images and interviews are irreplaceable and have a value long after a project is complete.
We need to save these for the long-term, a process called “archiving.” And here, we are caught between a rock and a hard place.
In the camera industry’s head-long rush to convert to cameras that shoot tapeless video, they seem to have overlooked the fact that we need to save these video files after editing is complete.
I have spoken with executives at Sony, Canon, JVC, Panasonic, and the rest of the camera industry, not one offers any kind of long-term archiving solution.
NOTE: I define archiving as the ability to save all video project and source files for a period of 10 – 25 years. Backup is what we do to protect ourselves during a project from the risk of data loss.
One camera vendor told me, dismissively, that tapeless media should just be handled as part of an “IT workflow.” Which is great if you have an IT department and the budget to support it. But, I looked in my checkbook, and the ability to write a $100,000 check for an automated tape archive library is not possible this week.
For me, the long-term solution to archiving tapeless video is LTO-tape. This has the flexibility, storage, and longevity to meet all our archiving needs. But, while this seems obvious to me, manufacturers of LTO gear seem to be doing everything they can to avoid selling us the products we need.
HOW YOU CONNECT IS CRITICAL
Currently, most LTO-tape drives connect via FibreChannel, SAS, or mini-SAS. Which makes sense if you are selling to the corporate market, where every server supports plug-in cards, speed is everything, and budgets are always expressed in units of ten thousand dollars.
But that isn’t the reality of video editing.
More than 70% of all video editors work in companies of five employees or less. Generally, the video editor provides all their own tech support, without access to an IT department.
Additionally, I would guess that close to half of all video editing is done on a laptop with a FireWire-attached RAID or hard disk. Portability is more important to a producer shooting a documentary than sheer horsepower.
Or, take an independent filmmaker on a budget. They have a collection of multi-terabyte FireWire drives, containing all the elements of their film, scattered across their desktop while editing on an iMac to take advantage of the large screen. Saving money yet getting a large screen is more important than buying the fastest possible computer.
I don’t have any research here, but I would guess that two-thirds of video editors are working on a system that isn’t a MacPro. And every one of those editors is juggling terabytes of data and praying they can figure out a way to store it for the long-term.
NOTE: The newly announced Thunderbolt provides a very fast data bus that tape vendors can tap into that would give them access to every new Mac that Apple ships. However, it doesn’t solve the archiving problems of the millions of Mac users that already have gear.
SPEED ISN’T EVERYTHING
When backing up corporate servers, speed is important, because many of the files on a server change daily.
But this isn’t typical for video. Video and film shoot a lot of files in a very short period of time, then spend a while — sometimes a long while — editing them. In other words, once the initial backup is complete, the only files that change are relatively small project files, not the massive media files.
This means that a workflow that features easy-to-connect and easy-to-use is far superior to one that emphasizes blinding speed. We are making archives here to last for decades. Taking an extra few hours to back everything up is a non-issue.
My large media files don’t change that often. My small project files change daily. And I only need to archive my project once – when the entire project is complete.
A FireWire-attached LTO drive would not be a speed demon, but I’d bet you’d never be able to keep them in stock. Security of my assets is FAR more important than transfer speed.
THE SIMPLE TASK OF GETTING AN HP LTO-5 TAPE DRIVE TO WORK ON A MAC
Here’s a specific example.
HP is a leading manufacturer of LTO drives. About a month ago, I contacted them for a loaner unit to review so that I could use it to illustrate a webinar I just did on how to archive Final Cut Pro projects.
HP agreed and shipped me an HP StorageWorks Ultrium 3000 SAS LTO-5 tape drive. It uses an ATTO ExpressSAS H680 interface card, which, because it is a plug-in PCIe card, only works with a MacPro.
The drive arrived a month ago and I am STILL trying to figure out how to connect it to my MacPro!
NOTE: Installation on Windows and Linux is close to click-one-button easy. Not so on the Mac.
I teamed up with two software engineers to help me get this to work. The first took one look at the installation instructions and gave up.
The second spent many hours over the last four weeks trying to configure the system. Here’s what he learned:
1) The HP unit didn’t come with firmware compatible with the software
2) Updating the firmware is seriously non-trivial.
3) It’s loud. The drive is about two feet from his ear and measures 57dB in operation.
Here is a BRIEF summary of the steps necessary to install the HP StorageWorks Ultrium 3000 SAS unit on a MacPro. (Keep in mind this required a Unix-trained hardware engineer to accomplish — this is only a summary, the actual installation process took more than 30 steps and two weeks to figure out!)
1. Install the ATTO SAS card into PCIe slot #1 of the MacPro.
2. Go to Google and download MacFuse, which is special utility software that allows the drive to operate on a Mac)
3. Install MacFuse
4. Go to HP’s website — www.hp.com/go/ltfs — to download the latest drive firmware.
HINT: To get this to work, you need to know the version you need. To make this easier, HP doesn’t tell you which version that is.
5. Download and install ICUFramework.pkg
6. Download and install LinearTapeFileSystem.pkg
7. Format the tape cartridge – this is LOUD! 57 dB.
HINT: Formatting a tape can only be done from the Terminal window. You ARE familiar with entering Unix commands via Terminal, aren’t you?)
8. Update Firmware on your drive.
HINT: This involves a return to the Terminal.
9. Prior to installation move the downloaded firmware into the /VAR/ROOT/ folder.
HINT: You will be glad to know that this folder is invisible on the Mac.
10. Make yourself a SuperUser in order to install the software.
HINT: You should be familiar with how to become a root user.
And this is only the START! There are still about 20 steps left to go involving terminal commands, more downloads, accessing invisible folders, and, if I read the fine print correctly, sacrificing a small goat.
This is ridiculous!
THERE ARE OTHER SOLUTIONS
Yes, there are existing Mac solutions out there.
The Tolis Group — www.tolisgroup.com — has BRU Producer, a hardware/software system for archiving that runs on Mac, Windows, and Linux systems. I’ve spent the last 30 minutes on their website, I still have no idea what products I need or how they are priced. The purchase they product, they refer me to a dealer. When I go to a dealer’s site, for information or pricing, they refer me back to the Tolis Group site.
The Tolis Group has hardware bundles, software bundles, and add-ons, which, apparently, are neither hardware nor software, but assume you already have the hardware you need, whatever that is.
Cache-A — www.cache-a.com — is another company that makes archiving solutions for the Mac. They only have two products, which makes deciding a lot simpler. However, nowhere on their website do they mention a price.
OK, that means I need to buy it from a dealer. So, I went to four of the dealers listed on their website:
* AbleCineTech — doesn’t list Cache-A in their vendor list, and Cache-A can’t be found using a search
* Digital Video Group — doesn’t list the products they sell and their site is not searchable
* B & H Studios — Cache-A not listed, site not searchable
* TekServe — Cache-A can’t be found using a search
And these are not small dealers here. It seems I can only buy Cache-A through dealers and the dealers are too embarrassed to mention they carry the product!
Is selling archiving solutions REALLY that difficult??? Is this much secrecy about your pricing, your products, or your service necessary?
Apparently, archiving IS rocket science!
If you are an manufacturer that has an LTO-tape-based archiving solution that:
* Can be installed easily on the Mac OS
* Is willing to list the product and its price on their website
* Has an announced shipping date
* Has an announced price of between $2,000 and $4,000
* Can connect to Macs via Thunderbolt or Firewire
* Supports the LTO-5 standard, which makes file backup/restore VERY easy
* And includes archiving software that can be run by mere mortals and not IT gods…
Contact me. I will happily share the news of your product to the MILLIONS of video editors that are desperately searching for ways to safeguard their media.
Otherwise, we run the very real possibility that every movie we are watching today will slowly disappear over the next few years as our current archive solutions fail.
34 Responses to An Exercise in Frustration← Older Comments
My company shoot mostly on P2 and XDCAM. I have been using the BRU PE back up system for the last year and it has performed very well. The BRU software is relatively easy to set up and use-the toughest part of the install was getting the ATTO card to work on my MacPro-it took a couple of tries to install the drivers but once that is done everything works easily. I use the Edit Bay Production Desktop LTO-4 version which costs about $3000 and is worth every penny. Each $30 LTO-4 tape holds 800 GB (I would not recommend using compression) . It is ironic to me that the ultimate end stage of “tapeless” workflow is a tape back up… I’d be happy to help you with the install if you need help Larry!
Thanks for the offer — and the information.
Well, there are two things at work here:
LTO5 is a tape media type. LTFS is a feature that came about when the LTO5 standard was being developed. So, using software like BRU, PresStore or Rertrospect is using LTO5, but not using LTFS.
LTFS is a nice standard, but has some limitations. Because it is a filesystem it can be read by any OS that supports the file system. Unfortunately, there aren’t too many OSes that support it out of the box. But the company to talk to would be Apple, since they could most easily add it. Until they do, Fuse is the only option. But if Apple were to add it, they could do it as a read/write media, making it very similar to a finder copy. One of the limitations though is tape size: while the above backup software packages can span multiple tapes easily, that isn’t possible with LTFS without some background work happening. It makes using an autoloader not that valuable, since there is no software to control things and write it out to the tapes in order-the user would need to manually move tapes, defeating the purpose of an autoloader.
However, there is a third option to look into. Quantum makes an LTO drive that sits directly on the network (hey iMac and MBP users), and uses FTP to transfer files to and from the tape. It looks like this only supports LTO3 currently, but this directly addresses most of your complaints. All you need is a copy of Fetch for drag-and-drop ease.
Surprised no one mentioned Object Matrix. Their MatrixStore software is integrated with FCP server. They tell me they’ve sold a petabyte of it and have some very large customers.
Not much footprint in the US today, but check them out.
Larry, we’ve been using BRU and HP LTO 4 on an Xserve for a couple of years. Best cheap archive solution out there IMHO.
I’ve been googling “thunderbolt lto” and “thunderbolt usb 3.0” every month or so for the past two years. I’m amazed that no one has stepped up to the plate. I completely agree.. there’s a great need waiting for a convenient solution.
With video generating massive amounts of data, there needs to be an easier way to back up and archive it all. I’ve spoken to many people who work in the industry and I’m surprised how many people backup using hard drives! When I dig, they all admit knowing that it’s not the best practice.
I’ve been holding out for two years.. I hope someone delivers!
I’ve been looking into tape backup on and off for a year or so. Very recently, I losy an entire year’s work when a Drobo-FS ate my data, so I am very intent on figuring out a decent archive solution. The Cache-A system looks great, but my business simply isn’t big enough revenue-wise to afford the cost ($7995).
I think I could justify the cost of an HP LTO-5 drive, but note the difficulty of using it on my Pac Pro. I am wondering if the software would install on Windows running under Parallels Desktop on the Mac. Has anyone tried that?
You can also get internal LTO-5 drives for MacPros – but you’ll need to also get archiving software and the software installation for the drive is not easy.
As for installation under Windows, that’s outside my expertise.
Admirіng the commіtmеnt you рut intο уour blοg
anԁ detailed informatiοn yοu
оffeг. It’s nice to come across a blog every once in a while that isn’t the same unωаntеԁ rehashed іnfoгmatіon.
Fаntaѕtic read! I’ve bookmarked your site and I’m inсludіng
уouг RЅS feedѕ tο my Gοogle accοunt.