An Exercise in Frustration

Posted on by Larry

It shouldn’t be this difficult.

NOTE: This blog grew out of my webinar last week on how to archive Final Cut Pro projects. If you are interested in archiving, click here to learn more.

The process of creating films and videos generates LOTS of data. From getting the footage shot, through the editing process, we are dealing with dozens, if not hundreds, of gigabytes of data.

And we solve this problem by buying ever more and ever larger hard disks and RAIDS. So far, a trifle expensive, but not really a problem.

However, once a project is done, we can’t just throw all these files away. The images and interviews are irreplaceable and have a value long after a project is complete.

We need to save these for the long-term, a process called “archiving.” And here, we are caught between a rock and a hard place.

In the camera industry’s head-long rush to convert to cameras that shoot tapeless video, they seem to have overlooked the fact that we need to save these video files after editing is complete.

I have spoken with executives at Sony, Canon, JVC, Panasonic, and the rest of the camera industry, not one offers any kind of long-term archiving solution.

NOTE: I define archiving as the ability to save all video project and source files for a period of 10 – 25 years. Backup is what we do to protect ourselves during a project from the risk of data loss.

One camera vendor told me, dismissively, that tapeless media should just be handled as part of an “IT workflow.” Which is great if you have an IT department and the budget to support it. But, I looked in my checkbook, and the ability to write a $100,000 check for an automated tape archive library is not possible this week.

For me, the long-term solution to archiving tapeless video is LTO-tape. This has the flexibility, storage, and longevity to meet all our archiving needs. But, while this seems obvious to me, manufacturers of LTO gear seem to be doing everything they can to avoid selling us the products we need.

HOW YOU CONNECT IS CRITICAL

Currently, most LTO-tape drives connect via FibreChannel, SAS, or mini-SAS. Which makes sense if you are selling to the corporate market, where every server supports plug-in cards, speed is everything, and budgets are always expressed in units of ten thousand dollars.

But that isn’t the reality of video editing.

More than 70% of all video editors work in companies of five employees or less. Generally, the video editor provides all their own tech support, without access to an IT department.

Additionally, I would guess that close to half of all video editing is done on a laptop with a FireWire-attached RAID or hard disk. Portability is more important to a producer shooting a documentary than sheer horsepower.

Or, take an independent filmmaker on a budget. They have a collection of multi-terabyte FireWire drives, containing all the elements of their film, scattered across their desktop while editing on an iMac to take advantage of the large screen. Saving money yet getting a large screen is more important than buying the fastest possible computer.

I don’t have any research here, but I would guess that two-thirds of video editors are working on a system that isn’t a MacPro. And every one of those editors is juggling terabytes of data and praying they can figure out a way to store it for the long-term.

NOTE: The newly announced Thunderbolt provides a very fast data bus that tape vendors can tap into that would give them access to every new Mac that Apple ships. However, it doesn’t solve the archiving problems of the millions of Mac users that already have gear.

SPEED ISN’T EVERYTHING

When backing up corporate servers, speed is important, because many of the files on a server change daily.

But this isn’t typical for video. Video and film shoot a lot of files in a very short period of time, then spend a while — sometimes a long while — editing them. In other words, once the initial backup is complete, the only files that change are relatively small project files, not the massive media files.

This means that a workflow that features easy-to-connect and easy-to-use is far superior to one that emphasizes blinding speed. We are making archives here to last for decades. Taking an extra few hours to back everything up is a non-issue.

My large media files don’t change that often. My small project files change daily. And I only need to archive my project once – when the entire project is complete.

A FireWire-attached LTO drive would not be a speed demon, but I’d bet you’d never be able to keep them in stock. Security of my assets is FAR more important than transfer speed.

THE SIMPLE TASK OF GETTING AN HP LTO-5 TAPE DRIVE TO WORK ON A MAC

Here’s a specific example.

HP is a leading manufacturer of LTO drives. About a month ago, I contacted them for a loaner unit to review so that I could use it to illustrate a webinar I just did on how to archive Final Cut Pro projects.

HP agreed and shipped me an HP StorageWorks Ultrium 3000 SAS LTO-5 tape drive. It uses an ATTO ExpressSAS H680 interface card, which, because it is a plug-in PCIe card, only works with a MacPro.

The drive arrived a month ago and I am STILL trying to figure out how to connect it to my MacPro!

NOTE: Installation on Windows and Linux is close to click-one-button easy. Not so on the Mac.

I teamed up with two software engineers to help me get this to work. The first took one look at the installation instructions and gave up.

The second spent many hours over the last four weeks trying to configure the system. Here’s what he learned:

1) The HP unit didn’t come with firmware compatible with the software
2) Updating the firmware is seriously non-trivial.
3) It’s loud. The drive is about two feet from his ear and measures 57dB in operation.

Here is a BRIEF summary of the steps necessary to install the HP StorageWorks Ultrium 3000 SAS unit on a MacPro. (Keep in mind this required a Unix-trained hardware engineer to accomplish — this is only a summary, the actual installation process took more than 30 steps and two weeks to figure out!)

1. Install the ATTO SAS card into PCIe slot #1 of the MacPro.

2. Go to Google and download MacFuse, which is special utility software that allows the drive to operate on a Mac)

3. Install MacFuse

4. Go to HP’s website — www.hp.com/go/ltfs — to download the latest drive firmware.
HINT: To get this to work, you need to know the version you need. To make this easier, HP doesn’t tell you which version that is.

5. Download and install ICUFramework.pkg

6. Download and install LinearTapeFileSystem.pkg

7. Format the tape cartridge – this is LOUD! 57 dB.
HINT: Formatting a tape can only be done from the Terminal window. You ARE familiar with entering Unix commands via Terminal, aren’t you?)

8. Update Firmware on your drive.
HINT: This involves a return to the Terminal.

9. Prior to installation move the downloaded firmware into the /VAR/ROOT/ folder.
HINT: You will be glad to know that this folder is invisible on the Mac.

10. Make yourself a SuperUser in order to install the software.
HINT: You should be familiar with how to become a root user.

And this is only the START! There are still about 20 steps left to go involving terminal commands, more downloads, accessing invisible folders, and, if I read the fine print correctly, sacrificing a small goat.

This is ridiculous!

THERE ARE OTHER SOLUTIONS

Yes, there are existing Mac solutions out there.

The Tolis Groupwww.tolisgroup.com — has BRU Producer, a hardware/software system for archiving that runs on Mac, Windows, and Linux systems. I’ve spent the last 30 minutes on their website, I still have no idea what products I need or how they are priced. The purchase they product, they refer me to a dealer. When I go to a dealer’s site, for information or pricing, they refer me back to the Tolis Group site.

The Tolis Group has hardware bundles, software bundles, and add-ons, which, apparently, are neither hardware nor software, but assume you already have the hardware you need, whatever that is.

Cache-Awww.cache-a.com — is another company that makes archiving solutions for the Mac. They only have two products, which makes deciding a lot simpler. However, nowhere on their website do they mention a price.

OK, that means I need to buy it from a dealer. So, I went to four of the dealers listed on their website:

* AbleCineTech — doesn’t list Cache-A in their vendor list, and Cache-A can’t be found using a search
* Digital Video Group — doesn’t list the products they sell and their site is not searchable
* B & H Studios — Cache-A not listed, site not searchable
* TekServe — Cache-A can’t be found using a search

And these are not small dealers here. It seems I can only buy Cache-A through dealers and the dealers are too embarrassed to mention they carry the product!

Is selling archiving solutions REALLY that difficult??? Is this much secrecy about your pricing, your products, or your service necessary?

Apparently, archiving IS rocket science!

CONCLUSION

If you are an manufacturer that has an LTO-tape-based archiving solution that:

* Can be installed easily on the Mac OS
* Is willing to list the product and its price on their website
* Has an announced shipping date
* Has an announced price of between $2,000 and $4,000
* Can connect to Macs via Thunderbolt or Firewire
* Supports the LTO-5 standard, which makes file backup/restore VERY easy
* And includes archiving software that can be run by mere mortals and not IT gods…

Contact me. I will happily share the news of your product to the MILLIONS of video editors that are desperately searching for ways to safeguard their media.

Otherwise, we run the very real possibility that every movie we are watching today will slowly disappear over the next few years as our current archive solutions fail.

Larry


34 Responses to An Exercise in Frustration

← Older Comments Newer Comments →
  1. Mark says:

    Larry just to clarify:

    Appreciate HDCAM is compressed – for the moment I keep all my BPAV on a 8GB RAID Smarstor while I figure out if I need to archive the whole lot.

    To clarify on the PDW-U1 you can copy anything to the “USER” folder hence I media manage the entire FCP project (QT.mov, Project file, PSD files, audio etc). In effect I have 23 or 50Gb rewritable media with claimed 50yr shelf-life.

    One can even combine standard HDCAM HD footage with anything you like in the USER folder, the HDCAM HD video will playback in Sony decks, while the USER folder contents accessible from a drive hooked to Mac. Limiting factor being 50Gb capacity of DL media.

  2. Josh Petok says:

    There’s an interesting thread (32 pages long!) on this very subject. Tim Jones from BRU has a ton of interesting observations:

    http://reduser.net/forum/showthread.php?t=24548

  3. Chris says:

    Hi Larry,

    We’ve been using a Quantum LTO-3A deck for the last two years and it’s been great. Connects over ethernet with an FTP client, is cross platform, and no propriety software.

    Really happy with it. Recovered a media managed FCP project quickly with no problems.

    http://www.quantum.com/Products/TapeDrives/LTOUltrium/LTO-3A/Index.aspx

  4. Hey Larry-

    I smell a buzz segment?

    Storage DNA has been proudly using LTFS since it’s inception – incorporating archiving of FCP and Avid projects & segments. Add in WAN/LAN acceleration, delta differencing, snapshotting – and you’ve got an excellent archiving solution – tape based and disk, at a decent price. It also alleviates the multi step process you outline for installation and configuration.

    Key Code Media (full disclosure: I work there! has been one the largest (and proud of it!) Cache-A resellers in the country. We actively discuss it, held seminars and events around it… and have even produced a video about it:

    http://www.youtube.com/watch?v=HbZzH2cnR7c

    (slightly dated, January 2010)

    Other angles include:

    Insurance: insurance is expensive for a firewire drive containing the days shoots for big budget films. Use a Tape backup, and insurance rates drop. A Cache-A, since it;s on the network, can be used both on set and in post. 10 cents a GB compared to 75 cents – 1 dollar a GB = no brainer.

    I’d love to have a mind meld with you on this, or point some archiving people in this realm your way.

    ~Michael

  5. Filip Vandoorne says:

    perhaps also have a look at the software of ArchiWare, Presstore 4
    which also works with the HP LTO drive.
    http://www.archiware.com/index.php?hp=126
    the archiving version of the software i not as cheap as BRU, but with more and different functionality.

  6. Mario Vivanco says:

    “The problem I have with archiving in XDCAM is that this is a highly compressed format (it uses MEPG-2) as it’s base. It would be better if the files from an existing project didn’t have to be transcoded into another compressed format.

    And, while this works for the final movie, individual assets – such as Photoshop files – can’t be converted to XDCAM for storage.”

    So, it sounds like is better to work with an HDV camera (like HVR Z7 and similars) rather than XDCAM even when its the final movies offer better image quality, if we think about long term archiving movies. I understand that if one day we have to “retrieve” or restore the archived video MPEG-2 files (gotten from XDCAM) they will no go back to the original quality?

    What about archiving HDV footage?

    I care very much for the long time archiving and I’d like to know if it’s better to get an XDCAM or and HDV (like the Z7). Which one offer better conditions for long time archiving?

    Thx for your patience.

    • Larry says:

      Mario:

      Both HDV and XDCAM compress their video using MPEG-2. From an archiving point of view, the two formats are the same. However, given other differences between the two formats, shooting XDCAM will yield higher-quality images and be easier to edit than HDV. (I am not a fan of HDV as a format.)

      Whether you shoot XDCAM or HDV, your archiving issues remain the same.

      Larry

  7. Bryan says:

    Larry

    I’ve had the BRU system on my target list for a while now. The link that Marc posted earlier is the one that I have set as a bookmark for when I’m ready to purchase (www.productionbackup.com/info/bundles.php).

    It looks like it provides a complete LTO-4 or LTO-5 solution that includes the choice of drives (LTO-4 or LTO-5), a media starter kit, a SAS card, cables and a license for BRU PE. $2979 for LTO-4 and $3929 for LTO-5).

    These are available from their dealers and when I checked one of their websites (www.advantagevideosystems.com) it’s listed there under POST-PRODUCTION, ARCHIVING/BACKUP, TOLIS GROUP.

    I’ve always hoped that this would be a very easy way for me to get into an LTO system. Have you found problems with this???

    THanks,

    -Bryan

    • Larry says:

      I have never worked with it, so I have no opinion.

      I have, however, talked with their CEO and I believe that they are a reputable company that is worth consideration.

      Larry

  8. Tim Allison says:

    Larry,

    You are missing an important point about archiving using Sony XDCAM optical disks and the PDW-U1 drive. By using the “User” folder on a formatted disk, you can store ANY digital file. The video does NOT have to utilize any of the XDCAM codecs. Text files, still images, editing program files….it doesn’t matter. When using the “User” folder, the disk just becomes a storage container. Each disk cost approximately $20-$25. The PDW-U1 cost somewhere around $2000. Everyone needs to do their own math to decide if this works for their specific operation.

    • Larry says:

      Tim:

      This is a great suggestion. Since I don’t know the PDW reader, I was not properly aware of this.

      Thanks for letting me know.

      Larry

  9. We just bought a 10 TB NAS for all stations to work off of. And we also bought an external Blu Ray burner. When a project is done, we back it up to BD-R and get it off the NAS. And a backup program runs at night to back up all work done that day onto drives attached to, but separate from, the NAS (you can attach up to 4 TB). I am absolutely loving it and am selling my 30-40 hard drives on ebay to offset the cost of the NAS. You might think that it would be hard for several computers to work off an NAS, and it would be if we were all working in uncompressed HD. But we are shooting mainly XDCAM 720p and there are no hang ups in editing.

  10. Cameron Tingley says:

    Just discovered this blog ….. I’ve got a solution. Don’t shoot solid state. Shoot on a “thing”, be it film, tape or optical disc. No fuss, no muss. And your long-term archiving is automatically covered. XDCAM optical discs are a great format; you get the benefit of file-based media plus built in archiving.

← Older Comments Newer Comments →

Leave a Reply to Bryan Cancel reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX 10.5 Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.