Thoughts on Hard Drive Speed

I got an email from Tiziano this morning with an increasingly common question:

I’m looking forward [to] buying the new Mac Pro later this year! My question is about the external disk to use for editing all formats from DV to 1080p 50 f/s and in the future maybe 4k (I use Final Cut Pro X). I’m planning to buy one of LaCie with Thunderbolt connection. I read your article on RAID, Is it still necessary a Raid system with Thunderbolt? I’m looking more on speed than security, I do the back up in other storage disks every day. What do you suggest?

THE SHORT ANSWER

RAIDs are always faster and store more than single drive. If you can afford it, always buy a RAID to store media for video editing.

However, there’s more to the story than that, which requires a longer explanation.

A DEFINITION

Before we start, though, let’s clarify a term: “hard drive speed.” The “speed” of a hard drive could be defined as the speed at which is rotates. While a useful measure in some engineering environments, this isn’t really helpful in the real world.

NOTE: For technical reasons, the faster a hard drive spins, the less data it stores.  7200 RPM drives store more data in the same space than 10,000 RPM drives.

Rather, let’s define “hard disk speed” as the speed with which the hard drive sends data to and from the drive to the computer. This is more accurately called the “data transfer rate,” or “transfer rate.”

Recording information sent FROM the computer to the hard disk is called “writing.” Sending information from the hard disk TO the computer is called “reading.” While the data transfer rate of these two operations is similar, writing (recording) information on a hard disk is almost always faster than reading (playing) information. (This speed difference is in the neighborhood of 5%.)

THE LONGER ANSWER

The data transfer rate of a hard drive is, essentially, determined by three factors:

• The rotational speed of the disks themselves
• The chips used by the drive to manage information processing
• The protocol used to connect the drive to the computer (FireWire, USB, Thunderbolt)

NOTE: Recently, drive manufacturers have started accelerating drives by integrating solid state drive (SSD) mechanisms into the drive hardware. This can accelerate operations where the same file is accessed more than once, but has limited effect when accessing multiple different files.

In the past, the speed of hard drives was limited by the speed of the protocol that connected the drive to the computer. For example, in practical terms, USB 2 limited hard drive speeds on Mac to around 18 MB/second. FireWire 800 limited hard drive speeds to around 85 MB/second. It was because of these differences in the protocol that video editors on a Mac were strongly encouraged to use FireWire 800 drives for all media storage and editing. By comparison, a hard drive attached internally to a MacPro (using a SATA protocol) could transfer data around 120MB/second.

NOTE: There are variations in drives and drive speeds. These numbers should be considered ranges, I’ve seen single drives both faster and slower than these numbers.

The reason speed is important is that a single stream of ProRes 422, which is the video format Final Cut Pro X uses for optimization and render files, is about 20 MB/second. (It’s actually a bit less, but the math is easier using a round number.)

So, if you are using a USB 2 drive, the drive’s transfer speed is too slow to even play one stream of ProRes 422.

When connecting a drive using FireWire 800, the drive will, generally, play about four streams of ProRes 422 before the FireWire connection is fully saturated.

Clearly, as multicam editing, video image sizes and frame rates increased, something faster was called for. Enter USB 3 and Thunderbolt.

IT’S IN THE PIPES

USB 3 and Thunderbolt are very high-speed connection protocols. Which means they are a “pipe,” like a pipe that carries water. Both of these pipes are really, really big – built to carry lots and lots of water all at once.

USB 3 is built to transfer data around 460 MB/second, while Thunderbolt is built to transfer data at about 1,100 MB/second! Suddenly, the speed restrictions we faced with FireWire 800 or USB 2, which were very narrow pipes by comparison, are gone. But – and this is a big BUT! – a pipe like USB 3 or Thunderbolt can only carry the data that’s fed into it.

If I connect a single hard drive to a computer using a Thunderbolt connection, the hard drive is connected using a very large pipe, but the total amount of data (water) that the pipe can carry is based, not on the size of the pipe, but how fast that single hard drive can transmit data. As we saw above, that limit is about 120 MB/second.

This means that if the data transfer rate of the pipe is bigger than the hard drive, the speed of the hard drive determines how much data gets sent down the pipe to the computer.

NOTE: Both USB 3 and Thunderbolt are excellent protocols. However, from what I’ve been told, Thunderbolt is more efficient at handling large media files. So, if your computer allows you the option, Thunderbolt devices should be chosen over USB 3. Also, USB 2 ports can not be converted to USB 3.

IT’S TIME FOR A RAID!

The way we fill the huge data transfer pipes provided by USB 3 and Thunderbolt is by combining multiple hard drives into a single unit called a “RAID” (which stands for “Redundant Array of Inexpensive Drives”).

If one hard drive transfers data at 120 MB/second, then combining two hard drives can provide double that: 240 MB/second. If we gang ten drives together, we can totally fill the immense size of a Thunderbolt pipe. But… it takes TEN drives to fill it. Not one.

NOTE: In reality, there are a variety of different flavors of RAIDs, called RAID levels, with differing levels of performance. While the analogy I’m building is essentially correct, reality provides a bit more variety. This article explains more.

DECIDING WHAT YOU NEED

So, let’s boil this down into something a bit simpler.

If you are only shooting and editing single camera narrative, a single hard drive connected via Thunderbolt will be a good choice. (While FireWire is supported on older Mac systems, it is not supported on newer systems. If your computer only supports FireWire 800, then that determines the type of hard drive you need to buy.)

However, if you are doing multicam, stereoscopic 3D, 2K, 4K or high frame rate video, a single drive, no matter how you connect it, will not be fast enough. You need a RAID that contains at least 2 drives.

To determine how many drives you need, take the number of video files you are playing at the same time and multiply that by 20 MB/second. (In other words, an eight camera multicam shoot would be 8 * 20 = 160 MB/second.) Assume that you get 100 MB/second per hard drive in your RAID (this allows for necessary overhead in the RAID). This means a 2 drive RAID will transfer about 200 MB/second of data.

For media with resolutions higher than 1080p HD (for example, 2K, 4K, or Ultra HD), you absolutely need a RAID with a minimum of 4 hard drives. More drives are ALWAYS better. As soon as you go beyond 2 drives, you need to look for a RAID 5. They are out there, but there are not a lot of vendor choices at the moment. Currently, I know of two: Promise Technology and Areca.

THREE WILD CARDS

There are three wild cards further clouding this mix: Drobo, RAID controller chips, and SSD drives.

Drobo makes drives that I like, but they are not fast. In order to get the speeds I was referring to earlier, the RAID needs to use a hardware controller chip. However, Drobo, in order to get both the expandability and flexibility that it is famous for, uses a software RAID controller. This software RAID controller is far slower than a hardware RAID controller. While I recommend using Drobo for expandable backup storage, their current performance is not sufficient for any task demanding high performance. This includes the new Drobo 5D and Drobo Mini.

The second wild card is those hardware RAID controller chips. For whatever reason, and I’ve heard LOTS of theories, RAID level 5 chips that also support Thunderbolt are very, very scarce. Promise Technology invented their own (which is a VERY expensive option), as has Areca with their CineRAID drives. Everyone else is waiting for a final chipset they can integrate into their units.

This wait for a certified RAID 5 controller chipset that supports Thunderbolt has taken two years and, I hope, ends really soon. Until it does, however, choices for RAID 5 systems will be very limited.

WHAT ABOUT SSD DRIVES?

SSD (Solid State Drives) are blazingly fast. However, they only achieve that speed when you are accessing the same files over and over. That performance advantage falls off when you are constantly accessing different files. Also, SSD drives can only store a fraction of what traditional hard drives can store.

The trend for the future is blending the speed of SSD drives with the vast storage of spinning media. Which means, over time, transfer speeds for both single hard drives and RAIDs will increase.

The Apple Fusion drive, available in some of the new iMacs, is a blending of SSD with traditional hard disks. I really like these for boot drives and applications. However, I prefer the vast storage and greater speed of RAID 5 systems for media storage and editing.

SUMMARY

With new protocols, like Thunderbolt, the speed of our storage depends upon the number of hard drives that are working together to feed the pipe. One hard drive transfers data at about 100 MB/sec. When you combine multiple hard drives together into a RAID, you can increase the total speed until you get all the speed you need.

ADDITIONAL READING

Here’s a collection of articles I’ve written that expands on the subject of hard drives.