The Basics of Media Management

Posted on by Larry

Whether you use Apple or Adobe, Macintosh or Windows, the basics of media management for video and audio editing remain the same: always store media on a drive separate from your boot drive. The reason is simple: the boot drive is busy doing everything EXCEPT playing media reliably.

The first priority for the boot drive – that’s the drive that has the Applications folder on it – is to meet the needs of the operating system. This could be transferring data from RAM memory to the hard disk, loading preference files, running OS applications … whenever the OS needs something, the boot drive drops everything else to meet it.

The second priority for the boot drive is meeting the needs of the active (foreground) application. Applications today are no longer a single file. Many are hundreds, even thousands of files that need to be loaded and managed by the boot drive. Add to that all the temporary files an application creates while it is running and the boot drive is starting to get really busy.

The third priority for the boot drive are all the background applications. These are programs that run whenever the OS or the foreground application isn’t busy doing something else. Video compression, rendering, and file backups are examples of background applications.

Then, there’s media playback; with a priority near, um, dirt. The problem is that media files are huge and require a significant amount of bandwidth second after second, minute after minute. Except that the boot drive is way too busy working with the OS and the applications you are running.

For this reason, always store media to a second drive. However, while that is easy to say, there are always tons of questions. So, let me answer the most common ones below.

WHAT IF… I PARTITION THE BOOT DRIVE TO CREATE TWO DRIVES

Partitioning takes a single drive and divides it into two or more logical sections. Each section acts as its own drive.

The good news is that this is a excellent technique when you need to run two different operating systems, such as Mac and Windows, or OS X 10.9 with an earlier version OS.

The bad news is that a hard disk has only one set of platters (the silver discs that spin) and one set of heads (the magnetic sensors that read or write the data stored on the hard disk. This means that if one side of the partition is used for the OS and the other partition has media on it, your hard disk needs to work EXTRA hard bouncing back and forth between partitions.

In fact, when you partition a drive and expect the heads to access both partitions, the system slows down.

So, don’t use a partition on the boot drive for storing media. (However, here’s an article that describes how to create a dual-partition boot disk to run two different versions of the operating system.)

WHAT IF… I DON’T CARE ABOUT PERFORMANCE?

While you can always copy media from the internal drive to an external drive if you need to move it somewhere, by storing your media and projects on an external drive, it makes it much easier to move files from one computer to the next.

Also, if you need to add more storage, it is easy to expand an external drive by adding more media, or replacing a smaller external drive with a larger one. Expanding internal storage is not as easy.

Plus, if an external drive needs to go into the shop, you don’t lose access to your computer. (For more answers to this question, see the next section.)

WHAT IF… I USE THE BOOT DRIVE FOR MEDIA?

First, nothing breaks, nor will you do any damage to your system. However, you may get “dropped frame” errors. In general, a dropped frame error means that your hard disk is too slow to support the media playback your project requires. A dropped frame error only affects playback, it doesn’t have any effect on the quality of your exported master file.

I often use the internal (boot) drive when I don’t need to playback a project. For instance, I just need to add an open or close to an existing project. I know the open is OK, I know the project is OK. So, I don’t need to play them, I just need to combine them and export the file. Since exporting does not require a fast hard drive, though it does benefit if the media drive is fast, I will often use the internal drive for these type of assembly projects.

Another reason to use the boot drive is that you don’t have any other choice. You only have a laptop and the project needs to be done NOW! Any dropped frame errors will be fixed when you export.

Another reason to use the boot drive is when you are working with very small codecs, such as AVCHD. These require very limited bandwidth, which can be easily supported by the boot drive. However, in the case of Final Cut Pro X, if you optimize these AVCHD files to ProRes, you may have playback problems, because ProRes requires much faster hard drives than AVCHD.

Also, for multicam playback, you may have better success using the internal SSD drive on the new MacPro; provided your multicam clips are short. I’m of two minds about this, but mention here so that you can try it for yourself. I don’t recommend this, but I also don’t advise against it.

WHAT IF… I HAVE A SECOND HARD DRIVE INSIDE MY COMPUTER?

This is perfectly fine. Many Macs don’t allow adding a second hard drive inside the computer, so I long ago got into the habit of saying “external hard drive,” when what I REALLY meant was a separate hard drive from the boot drive.

A second hard drive, installed inside an older Mac Pro, or inside a Windows tower, is perfectly acceptable and should provide reasonable performance for most video editing except high-resolution media or multicam editing.

While multiple hard drives are common in Windows systems, they are becoming increasingly rare on Mac systems. For this reason, I always tend to think of a separate box for storage.

WHAT IF… I HAVE AN SSD DRIVE?

SSD drives are really, REALLY fast. And, when used as your boot drive, can make your computer fly. But they have some limitations when used for media.

First, SSD drives don’t hold as much media as standard hard disks (what the techies call “spinning media,” because the silver platters inside a hard disk rotate at high speed). So, for larger projects, an SSD drive will fill up more quickly than a spinning media drive.

Second, the SSD drive still needs to meet the needs of the OS and applications. When I was testing the new Mac Pro earlier this year, I noticed that the speed of the internal SSD drive would drop by 70% when the OS suddenly needed something. An SSD drive that is used as the boot drive does not deliver the same speed consistently; it can fluctuate wildly.

Third, unlike spinning media, SSD drives are composed of “cells” that have a fixed number of writes (recording). When that number is exceeded, the reliability of an SSD drive starts to deteriorate. The exact number of IO operations (input/output) varies by the density of the SSD drive. Older 50nm systems handle about 10,000 writes, while newer 25nm systems can only handle about 3,000 (source). The more you use the drive, the shorter the lifespan it will have.

Personally, I like SSD drives to replace the boot drive, but use spinning media for all media storage. As SSD technology continues to improve, I may change my mind on this for the future by adding an external SSD drive. But I still would keep media separate from the boot drive.

WHAT IF… I USE A CAMERA CARD?

Camera cards record media in the camera. But, in general, you should copy the card to your computer before importing it into your editing system.

But!! (And this is a KEY point) Always copy the entire contents of the camera card into a folder on your hard disk. Camera cards have a very specific folder structure and name. If you just copy the contents of the card to your hard disk, the editing software will see folders that it expects to see on a camera card, so it will just assume your entire hard disk is a camera. And this will REALLY confuse things.

Always create a new folder on your hard disk and copy all the folders from your camera card into that folder. Then, import media into your editing system from the folder on your hard disk. This will be faster, more reliable, and makes sure you have have a backup of the media on your camera card.

WHAT IF… I COPY MEDIA AND PROJECTS/LIBRARIES TO SEPARATE DRIVES?

You can do this, but it won’t help a whole lot.

In general, project files are loaded into RAM memory, while media (including render files) should play off a separate hard disk. This means that:

The wild card with this is render files. Render files have the same playback requirements as media files. For this reason, render files, in fact all scratch disks, should be stored on separate hard disks.

WHAT IF… I USE A USB2 DRIVE?

USB2 devices tend to be very slow, though Apple improve USB2 data transfer rates (the speed which the hard drive transfers data between the hard drive and the computer) with the 10.6.4 update to Mac OS X.

In general, I recommend against using USB2 for video editing unless you are editing AVCHD, DV, or H.264 files natively. And, even then, I would be cautious.

On the other hand, if you are simply copying files from one computer to another, a USB2 drive is perfect. USB2 drives are fine drives, they are just slow.

WHAT IF… I USE A USB3 OR THUNDERBOLT DRIVE?

These drives are very fast and should be suitable for most applications. However, in both the case of USB3 or Thunderbolt, the speed you get from the system is based upon the number of drives in the USB3 or Thunderbolt enclosure.

The general rule of thumb is that you get 120 MB/second for each hard drive in the RAID attached vis USB3 or Thunderbolt. So, a two-drive system provides about 240 MB/second of data. For most editing – except multicam and high-resolution media, this is plenty fast enough.

Here’s some examples of how to maximize the data rates between three different protocols?

In general, don’t mix and match Thunderbolt 1 and 2 devices. Pick one or the other. Unless you are working with very high-resolution media (2K or above) either protocol is fine for editing. Ultimately, Thunderbolt 1 will be replaced by Thunderbolt 2.

WHAT IF… I FOLLOW ALL THESE SUGGESTIONS AND STILL HAVE PROBLEMS?

Video editing is about the hardest thing we can do on a computer. Far more challenging than the biggest spreadsheet, audio editing, or, even, 3D modeling. Everything on your computer needs to be working right to edit video successfully.

If you follow these suggestions and still have problems, its time to contact the vendor that made your computer and start trouble-shooting. The problem is most likely somewhere other than your hard disk.


Bookmark the permalink.

14 Responses to The Basics of Media Management

  1. Michael Fiala says:

    Hi Larry –
    I’ve installed an extra 2 TB internal 7200 rpm HD to hold camera card avchd and an extra 4 TB internal 7200 rpm HD that I only put libraries, events and projects in an early 2009 Mac Pro.

    I just finished a 90 minute 2 camera angle multicam figure skating show and had very few stutters and no dropped frame warnings.

    I did experience several reboots as rendering would completely lock up the machine even with 28 GB of RAM installed in the machine.

    • Larry Jordan says:

      Michael:

      The answer depends upon what software you are using.

      FCP 7 only uses 4 GB of RAM and doesn’t use your graphics card for rendering.

      Premiere and FCP X both use the graphics card for rendering, not RAM or CPU (as much). So if your system is “locking up” during rendering consider upgrading your GPU.

      Larry

  2. Hi Larry, thank you for this valuable article. I still have a doubt. I’m working with FCPX. How should I label the folder to storage my Media files and, should I use several folders for each project or only one for all the projects?
    Thank you
    Jorge

    • Larry Jordan says:

      Jorge:

      FCP doesn’t care how you name your media folder, nor whether you use one or multiple folders. The KEY is not to change the folder name once you’ve started importing media from it.

      Whatever system works best for you, FCP can work with.

      Larry

  3. Lars-Lennart Nielsen says:

    Thank you Larry for a better understanding of how the boot drive is used by the OS. Still, I am not sure where to put my media content. I recently got myself an iMac 27″ with Fusion Drive; an SSD 128GB of flash storage combined with a 3TB HDD. Can I regard the HDD as a (internal) second HDD? I became doubtful because of your comment “…multiple hard drives…are becoming increasingly rare on Mac systems.” My option is to use my 1TB WD Passport by a USB 3.0 connection.

    Thank you
    Lars-Lennart

    • Larry Jordan says:

      Lars-Linnert:

      No. The Fusion Drive is a single unit. You can’t access the SSD separate from the HDD.

      In other words, use the Fusion Drive as your boot drive and get a second drive for media. (Though, on my Fusion Drive here, I will often export to the desktop because it is fast and easy, then move the file to permanent storage on a second drive.)

      Larry

  4. Fernando L Mangino says:

    One more thing, missing in your article Larry,
    NEVER, NEVER do connect in Dasy Chain mode Thunderbolt 1 and 2 or FW400 and FW800 to your computer, because your transfer rate will be the slowest devices transfer rate.
    Fernando

  5. Barbara says:

    HI LARRY
    I AM A LATECOMER TO FCP7 AND IT’S NOW APRIL 2016!
    Just a question..
    Am i right in setting up a new project under System Settings I should be putting the Autosave vault on an external drive and Capture Scratch too? Do I put the Video and Audio Capture on a different external drive?

    I want to be able to take the drives across town to another facility to share with a director who can’t come to me.

    Many thanks for all your output over the years ..you have been a real asset to editor friends of mine…I’m semi retired now from another part of it but know how well you are respected in the industry.

  6. davor says:

    Hello guys,

    Could you please recommend external ssd disk? I would like to work on a project inside Premiere Pro on two computers…

  7. The issue point for my practice is about transferring files from camera cards to a hard drive.

    The standard suggestions start here: “In general, you should copy the card to your computer before importing it into your editing system.” And add a couple admonitions about file structure, entire folders, etc.

    Most tutorials skip over this process and go on to talk about how NlE software can help; DITs and pros on shoots say to use special software to do this that will compare checksums.

    I’ve found it’s possible to copy files with the Mac Finder, but learned that total file sizes may be slightly different because of differences between file structure and organization of Mac hard drives and non-Mac camera cards. I wonder what it’s like to transfer files to Windows machines.

    I’d like to know more about how this part of media management works — the short answers I keep getting are “Use XXX software.” Maybe this could become larger a part of a media management tutorial.

    • Larry says:

      Robert:

      From what I’ve been told, the Finder does not do a bit-by-bit comparison of the old and new file when transferring. If that’s true, then what the other software tools do is compare the two files to make sure they are bit-for-bit accurate.

      This is the underlying reason techs recommend using specialized – but not expensive – software for the file copies. In thinking about this, though, I’ve never had this point confirmed by Apple. So, I will look into this to see if it is still true.

      Larry

Leave a Reply to Larry Jordan Cancel reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.