An Exercise in Frustration

Posted on by Larry

It shouldn’t be this difficult.

NOTE: This blog grew out of my webinar last week on how to archive Final Cut Pro projects. If you are interested in archiving, click here to learn more.

The process of creating films and videos generates LOTS of data. From getting the footage shot, through the editing process, we are dealing with dozens, if not hundreds, of gigabytes of data.

And we solve this problem by buying ever more and ever larger hard disks and RAIDS. So far, a trifle expensive, but not really a problem.

However, once a project is done, we can’t just throw all these files away. The images and interviews are irreplaceable and have a value long after a project is complete.

We need to save these for the long-term, a process called “archiving.” And here, we are caught between a rock and a hard place.

In the camera industry’s head-long rush to convert to cameras that shoot tapeless video, they seem to have overlooked the fact that we need to save these video files after editing is complete.

I have spoken with executives at Sony, Canon, JVC, Panasonic, and the rest of the camera industry, not one offers any kind of long-term archiving solution.

NOTE: I define archiving as the ability to save all video project and source files for a period of 10 – 25 years. Backup is what we do to protect ourselves during a project from the risk of data loss.

One camera vendor told me, dismissively, that tapeless media should just be handled as part of an “IT workflow.” Which is great if you have an IT department and the budget to support it. But, I looked in my checkbook, and the ability to write a $100,000 check for an automated tape archive library is not possible this week.

For me, the long-term solution to archiving tapeless video is LTO-tape. This has the flexibility, storage, and longevity to meet all our archiving needs. But, while this seems obvious to me, manufacturers of LTO gear seem to be doing everything they can to avoid selling us the products we need.

HOW YOU CONNECT IS CRITICAL

Currently, most LTO-tape drives connect via FibreChannel, SAS, or mini-SAS. Which makes sense if you are selling to the corporate market, where every server supports plug-in cards, speed is everything, and budgets are always expressed in units of ten thousand dollars.

But that isn’t the reality of video editing.

More than 70% of all video editors work in companies of five employees or less. Generally, the video editor provides all their own tech support, without access to an IT department.

Additionally, I would guess that close to half of all video editing is done on a laptop with a FireWire-attached RAID or hard disk. Portability is more important to a producer shooting a documentary than sheer horsepower.

Or, take an independent filmmaker on a budget. They have a collection of multi-terabyte FireWire drives, containing all the elements of their film, scattered across their desktop while editing on an iMac to take advantage of the large screen. Saving money yet getting a large screen is more important than buying the fastest possible computer.

I don’t have any research here, but I would guess that two-thirds of video editors are working on a system that isn’t a MacPro. And every one of those editors is juggling terabytes of data and praying they can figure out a way to store it for the long-term.

NOTE: The newly announced Thunderbolt provides a very fast data bus that tape vendors can tap into that would give them access to every new Mac that Apple ships. However, it doesn’t solve the archiving problems of the millions of Mac users that already have gear.

SPEED ISN’T EVERYTHING

When backing up corporate servers, speed is important, because many of the files on a server change daily.

But this isn’t typical for video. Video and film shoot a lot of files in a very short period of time, then spend a while — sometimes a long while — editing them. In other words, once the initial backup is complete, the only files that change are relatively small project files, not the massive media files.

This means that a workflow that features easy-to-connect and easy-to-use is far superior to one that emphasizes blinding speed. We are making archives here to last for decades. Taking an extra few hours to back everything up is a non-issue.

My large media files don’t change that often. My small project files change daily. And I only need to archive my project once – when the entire project is complete.

A FireWire-attached LTO drive would not be a speed demon, but I’d bet you’d never be able to keep them in stock. Security of my assets is FAR more important than transfer speed.

THE SIMPLE TASK OF GETTING AN HP LTO-5 TAPE DRIVE TO WORK ON A MAC

Here’s a specific example.

HP is a leading manufacturer of LTO drives. About a month ago, I contacted them for a loaner unit to review so that I could use it to illustrate a webinar I just did on how to archive Final Cut Pro projects.

HP agreed and shipped me an HP StorageWorks Ultrium 3000 SAS LTO-5 tape drive. It uses an ATTO ExpressSAS H680 interface card, which, because it is a plug-in PCIe card, only works with a MacPro.

The drive arrived a month ago and I am STILL trying to figure out how to connect it to my MacPro!

NOTE: Installation on Windows and Linux is close to click-one-button easy. Not so on the Mac.

I teamed up with two software engineers to help me get this to work. The first took one look at the installation instructions and gave up.

The second spent many hours over the last four weeks trying to configure the system. Here’s what he learned:

1) The HP unit didn’t come with firmware compatible with the software
2) Updating the firmware is seriously non-trivial.
3) It’s loud. The drive is about two feet from his ear and measures 57dB in operation.

Here is a BRIEF summary of the steps necessary to install the HP StorageWorks Ultrium 3000 SAS unit on a MacPro. (Keep in mind this required a Unix-trained hardware engineer to accomplish — this is only a summary, the actual installation process took more than 30 steps and two weeks to figure out!)

1. Install the ATTO SAS card into PCIe slot #1 of the MacPro.

2. Go to Google and download MacFuse, which is special utility software that allows the drive to operate on a Mac)

3. Install MacFuse

4. Go to HP’s website — www.hp.com/go/ltfs — to download the latest drive firmware.
HINT: To get this to work, you need to know the version you need. To make this easier, HP doesn’t tell you which version that is.

5. Download and install ICUFramework.pkg

6. Download and install LinearTapeFileSystem.pkg

7. Format the tape cartridge – this is LOUD! 57 dB.
HINT: Formatting a tape can only be done from the Terminal window. You ARE familiar with entering Unix commands via Terminal, aren’t you?)

8. Update Firmware on your drive.
HINT: This involves a return to the Terminal.

9. Prior to installation move the downloaded firmware into the /VAR/ROOT/ folder.
HINT: You will be glad to know that this folder is invisible on the Mac.

10. Make yourself a SuperUser in order to install the software.
HINT: You should be familiar with how to become a root user.

And this is only the START! There are still about 20 steps left to go involving terminal commands, more downloads, accessing invisible folders, and, if I read the fine print correctly, sacrificing a small goat.

This is ridiculous!

THERE ARE OTHER SOLUTIONS

Yes, there are existing Mac solutions out there.

The Tolis Groupwww.tolisgroup.com — has BRU Producer, a hardware/software system for archiving that runs on Mac, Windows, and Linux systems. I’ve spent the last 30 minutes on their website, I still have no idea what products I need or how they are priced. The purchase they product, they refer me to a dealer. When I go to a dealer’s site, for information or pricing, they refer me back to the Tolis Group site.

The Tolis Group has hardware bundles, software bundles, and add-ons, which, apparently, are neither hardware nor software, but assume you already have the hardware you need, whatever that is.

Cache-Awww.cache-a.com — is another company that makes archiving solutions for the Mac. They only have two products, which makes deciding a lot simpler. However, nowhere on their website do they mention a price.

OK, that means I need to buy it from a dealer. So, I went to four of the dealers listed on their website:

* AbleCineTech — doesn’t list Cache-A in their vendor list, and Cache-A can’t be found using a search
* Digital Video Group — doesn’t list the products they sell and their site is not searchable
* B & H Studios — Cache-A not listed, site not searchable
* TekServe — Cache-A can’t be found using a search

And these are not small dealers here. It seems I can only buy Cache-A through dealers and the dealers are too embarrassed to mention they carry the product!

Is selling archiving solutions REALLY that difficult??? Is this much secrecy about your pricing, your products, or your service necessary?

Apparently, archiving IS rocket science!

CONCLUSION

If you are an manufacturer that has an LTO-tape-based archiving solution that:

* Can be installed easily on the Mac OS
* Is willing to list the product and its price on their website
* Has an announced shipping date
* Has an announced price of between $2,000 and $4,000
* Can connect to Macs via Thunderbolt or Firewire
* Supports the LTO-5 standard, which makes file backup/restore VERY easy
* And includes archiving software that can be run by mere mortals and not IT gods…

Contact me. I will happily share the news of your product to the MILLIONS of video editors that are desperately searching for ways to safeguard their media.

Otherwise, we run the very real possibility that every movie we are watching today will slowly disappear over the next few years as our current archive solutions fail.

Larry


34 Responses to An Exercise in Frustration

Newer Comments →
  1. Russ says:

    Larry,
    Does not surprise me about HP!!
    As a footnote to your article working as a “root user” is only for experienced admins and not recommended by Apple themselves.
    It is ridiculous HP can market such a product, and this is NOT trivial!! As a root user it is incredibly easy to TOTALLY DESTROY your working system with a couple of “wrong moves”. When I am working in a terminal session I give it my 100% concentration at all times.
    Larry great article and yes a definite possible solution for archiving if some forward thinking company can come up with a viable solution.

  2. Guy McLoughlin says:

    One other less expensive solution might be Delkin “200 year” archival Blu-ray discs that can store 25 GB of data. Yes, not a huge amount of storage, but it’s a start.

    http://delkin.com/c-155604-archive-archival-blu-ray-disc.html

  3. Paul Heneghan says:

    Great points all around, Larry. I am “comfortable” working in the Terminal, but you’re absolutely right, it shouldn’t be necessary to do so. These are Macs! Couldn’t the engineering team at HP put it all together in a scripted bundle for Vise installer or the like?

    I have had nothing but great interactions with the Tolis Group. I originally met them at MacWorld SF back in 2002/3(?), looking for a way to back up a FreeBSD server to my Exabyte VXA rotating storage drive Firewired to my Xserve. Great guys, very knowledgeable, very friendly and super helpful. I’d call them and tell them your story, they’ll commiserate and probably give you some great pointers.

    I spoke with one of their honchos (sorry can’t remember his name) this year at the SF Supermeet, where they had a table for the vendors showcase. They’re all about LTO now.

    But your points are right on the money, regardless of who and what is out there. How will people know what to use, and then how will they make it work if the available tools are so inscrutably difficult to install, let alone operate.

  4. Marc Lamphear says:

    Check out the following web site for info on hardware/software bundles for BRU Producers Edition. It seems to answer one of your concerns about packages for various users and the associated costs. Go to this page on the BRU website. Hope that helps.

  5. Mark says:

    um, Sony PDW-U1 – take your pick:

    1) media manage project to a 23 or 50gb disc to the “user” folder i.e. all Mac centric for direct access to FCP editing later

    OR

    2) write out an industry standard HD video master in the form of XDCAM HD i.e. playback and/or re-ingest from any XDCAM deck

    • Larry says:

      Mark:

      Thanks for the ideas.

      The problem I have with archiving in XDCAM is that this is a highly compressed format (it uses MEPG-2) as it’s base. It would be better if the files from an existing project didn’t have to be transcoded into another compressed format.

      And, while this works for the final movie, individual assets – such as Photoshop files – can’t be converted to XDCAM for storage.

      Larry

  6. Simon Morice says:

    I had one conversation at London’s recent BVE about this. I spoke to someone who has developed a >£5000 solution with all sorts of internal drives and multiple redundancy doodads.

    I told him this was just too expensive. I told him of the large and global market needing simple and cheap. He told me there was no demand. I gave up.

  7. Bryan Peel says:

    We have been archiving to LTO-4 for about three years now and love it. The only bump has been at the software side. Retrospect became an issue on REV8 so we switched to BRU PE….

    Roxio took over Retrospect, fixed the issues and we have now completely rebuilt our archive and back to Retrospect….and VERY happy.

  8. Matt W says:

    We shoot our jobs on Sony Z7’s record to tape and CF, our tape gets shelved and CF gets captured. Most of our corporate jobs are under 2 hours, so once its done the whole project folder and footage are archived to hard disk and shelved. We spin them up every few months, its nice if a client needs a re-cut or footage for another project we just slap the bare drive into a dock and its all ready to go.
    I know its not super long term storage but it works till other solutions are more affordable.

  9. Larry: Great article. I recently saw Promax’s ProjectSTOR Archive series of devices. It’s a little bit of a strange workflow, but probably more accessible than many. It is still a SAS conneted device and I believe you would have to use backup software of some kind. It is pricey but they bundle a RAID with the LTO device. Kind of a weird added value concept. Could this be a step in the right direction?

  10. stu aull says:

    RIGHT ON LARRY!!
    Have followed your laments in the area of Archiving this past year, and thanks to you have come to appreciate its importance as our company morphs from (*gasp*!) Beta SP to R3D Red/Scarlet sometime this year. My thanks for your continued harrasement of the Archive industry to provide real-world usable tools for mac editors! Will continue to follow your lead with great interest….
    kind regards

    Stu Aull
    Moving Images, Alaska

Newer Comments →

Leave a Reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX 10.5 Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.