RAIDs, SSDs, iCLOUD and Performance

Posted on by Larry

A question I am getting more and more these days revolves around something Jerry Thompson asks:

While I am interested in performance and speed between [Thunderbolt and USB 3], I find I am not completely understanding all I need to regarding RAID technology.

Or, as Craig McKenna writes:

[I recently bought] a 120 GB external SSD with Thunderbolt, I’m wondering how you would go about organizing my media.

I’ve spent a lot of time reviewing specific storage products. In this article, I want to take a step back and discuss storage performance in general.

RAID

A RAID (Redundant Array of Inexpensive Drives) is a collection of hard drives that create a pool of storage this is both very large and very fast. To the computer, and on your desktop, it looks like a single very big, very fast hard drive. A RAID builds all the drives into a single box with a single connection to the computer. (Yes, you can custom-build a RAID using stand-alone disks, but let’s just keep this simple.)

RAIDs are categorized into “levels,” which describe a combination of speed, redundancy, and price.

NOTE: “Redundancy” is defined as the ability to recover data in the event one, or more, hard drives dies.

For the purposes of this example, let’s assume each of the RAIDs below contains 1 TB drives which transfer data at 100 MB/second. (For comparison, the fastest a single drive can transfer data is around 120 MB/sec assuming it is connected internally to a recent-issue Mac.)

There are other RAID levels – 3, 10, 11, 50 and 60 – but these four are the most relevant to individual video editors. From my point of view, while RAID 6 provides more security, I am happy with the balance of speed vs security of RAID 5, which is what I recommend most.

NOTE: Drobo is a special case. In general, all RAIDS must use drives of the same size and speed. As well, all drives need to be installed at the time you first power up the system. Drobo, on the other hand, has invented a new technology which allows you to add drives, or mix and match drives of different sizes, even after you’ve put the RAID into operation. While Drobo does not provide the fastest RAIDs, this flexibility can be a huge benefit.

SIDEBAR: HOW DATA REDUNDANCY WORKS

This is so cool… This works because all digital data is stored as either a 1 or a 0.

Imagine a 3D checkerboard — let’s make it 5 stories high. Look down on the top left square and count the number of checkers on that square for each of the top four layers.

If they total an odd number, put a checker on the same square on the bottom layer. If they total an even number, don’t put a checker on the same square on the bottom layer.

Now, remove the second layer and all it’s checkers, and put in an empty new checkerboard to take its place. By counting the number of checkers on the remaining top three layers and comparing the total to the indicator on the bottom layer, you can exactly rebuild all the missing checkers on the second layer. For example, if the total of the other three layers is even, and there’s a checker on the bottom layer, add a checker to the new layer. If the total of the other three layers is odd, and there’s a checker on the bottom layer, don’t add a checker to the new layer.

This is exactly how RAID redundancy works. Except each checkerboard represents a hard drive in the RAID. The bottom layer, which provides data redundancy, doesn’t need to know which drive failed, it only needs to compare the totals on all the different hard disks with the total stored on the redundancy disk in the RAID. This technique works whether you have three drives – the minimum – or twenty drives. The only difference is that more drives take longer to count and only one drive can fail at a time.

SSD DRIVES

An SSD (Solid State Drive) drive is essentially RAM that has been configured to act like a regular hard disk. You copy and move files around in it the same as a hard disk. And, unlike RAM, it remembers your data when the power is turned off. Depending upon which version of the operating system you are using, an SSD drive ranges from “so-so” performance to blinding. Later versions of the Mac operating system do a much better job supporting SSD drives.

The big benefit an SSD provides is speed. Its two big limitations are cost and limited storage size.

While you can put an SSD drive anywhere you can put a “normal” hard disk – which we often call “spinning media,” the best place to put an SSD drive is inside your computer as a replacement for your boot drive.

Attaching an SSD drive externally via FireWire 800 will severely limit its performance and is not recommended. You won’t see any significant speed improvement because FireWire 800 is too slow.

NOTE: There is a limitation of SSD, however, in that it only allows a certain number of read/writes before the unit starts to fail. While the overall longevity of SSD is still being determined, for now, assume that you will need to replace an SSD drive sooner than a spinning media drive – probably after 3-4 years of normal use.

iCLOUD

iCLOUD, and other Internet services like DropBox and YouSendIt, are essentially file servers that store your files outside of your computer.

If we ignore issues like file security, these services are excellent for backing up data, sharing files between devices, and moving files between computer systems. However, they are not good for storing source media files for editing. It isn’t because they don’t store enough. Just the opposite, these services can store a vast amount of data. The problem is that the connection speed – called the “data transfer rate” – between your computer and the iCloud is too slow. Video editing requires data transfer rates far beyond anything supplied by even the fastest DSL or cable modem.

Use the Cloud for sharing, but not for storing or editing source files.

WHAT IS THUNDERBOLT?

Thunderbolt is a method for connecting monitors and hard disks to your system. In this regard it is just like FireWire or USB – its a cable and communication protocol that move data to and from your computer and storage.

The big benefit to Thunderbolt is that it is REALLY fast! More than 1 GB/sec of data transfer speed! However, in order for that speed to be realized, you need a REALLY fast RAID. A two-drive RAID 0 won’t begin to fill a Thunderbolt “pipe.”

Thunderbolt is how you connect your drive to your computer. The speed you get will depend upon the speed of the RAID you have attached. Here are some very general expectations for data transfer:

NOTE: A single drive connected via Thunderbolt will be only marginally faster than the same drive connected via Firewire. In order to see significant performance improvement, you’ll need to use a RAID that contains at least four hard disks.

GETTING THE BEST PERFORMANCE

For best performance, I recommend replacing the spinning media hard drive inside your computer – this is also called the boot drive – with an SSD drive.

In general, media should not be stored on your boot drive. This means that only applications and the operating system are stored on the boot drive – along with other files that tend to be small, like email or word processing documents. If you have a large iTunes collection, or large iPhoto library, moving them to an external drive may allow better performance.

If I were setting up a new system, which I am doing next month, I would get a Mac with a SSD drive as the boot drive, and a Thunderbolt RAID 5 drive for media and project files.

My current boot drive uses 148 GB to store all applications and operating system files. I have hundreds of apps which don’t take a lot of storage. So, you don’t need to get a gigantic SSD drive – 250 – 500 GB is more than sufficient.

My media RAID, though, can’t be big enough. I haven’t decided exactly what I’m getting, but I’m looking for 6-10 TB (that’s Terabytes) of storage. I’ve discovered that hard drives have two states: empty or full. I want this one to remain as empty as possible for as long as possible.

This configuration provides a huge speed boost for the operating system and applications, while providing extremely fast access to huge amounts of media, with full redundancy in case of drive failure. This set up also offers a good balance between price and performance.


12 Responses to RAIDs, SSDs, iCLOUD and Performance

  1. Caesar Darias says:

    Thanks for the informative post, Larry.

    I recently went to buy a Thunderbolt drive and noticed is was 5400rpm. It seemed to me to defeat the purpose of Thunderbolt so I didn’t get it.

    I’m still not sure if I was correct.

    http://www.bhphotovideo.com/c/product/824996-REG/LaCie_9000107_2TB_Little_Big_Disk.html

    • Larry says:

      Probably not a good choice.

      RPM is one factor, but not the biggest factor, in choosing a hard disk. When your files are large, as they are with video, rotational speed plays less of a role. When you have lots of smaller files, rotational speed makes a bigger impact.

      I’m not saying rush out and buy this hard drive, but I am saying don’t make a decision solely on RPM.

      Larry

  2. Ron Robinson says:

    Great article Larry! I particularly appreciated your suggestion of an SSD drive for applications and system. Makes a lot of sense. I hope you will share publicly the configuration of your new system — that will be of great interest, as I’m looking at doing the same in the next few months.

    Your response regarding the 5400 RPM question very much surprised me! I thought that it was always better to use a 7200 RPM drive. Could you expand on this please? Should I even consider slower drives in a five drive RAID 5 configuration?

    • Larry says:

      A RAID is the sum of its parts. So, a larger number of slower drives will always be faster than a single faster drive. I’m not saying 7200 RPM is not important, but when you are working with multiple drives – which is what a RAID is – it is less important than other factors.

  3. Emerson says:

    Hi Larry
    I didn´t understand “A single drive connected via Thunderbolt will be only marginally faster than the same drive connected via Firewire” it means that a single drive will not benefit with this tecnology? I mean when you started bought a single hard disk with USB1 you change to USB2 for higher speed then Firewire 400 and 800 then USB3 and so on…so single hard disk with thunderbolt is not the next step for higher speed?

    • Larry says:

      Correct. The maximum speed we can get from a single hard drive today is around 120 MB/second. FireWire 800 delivers about 85-90 MB/sec from both single drives and RAIDs. So, attaching a single hard drive via Thunderbolt is not that much faster than FireWire 800. The KEY is when you use Thunderbolt with a RAID. Then you see seriously faster performance. Larry

  4. Tony says:

    Larry, I just bought a 15″ Retina and plan to connect my 4TB CalDigit VR (250 MB/s in RAID0) enclosure via the LaCie HUB (which is a eSATA to Thunderbolt adapter).

    What read/write speeds should I be expecting with this setup?

    • Larry says:

      I haven’t tested this unit. But, a RAID 0 consists of 2 drives, so, assuming some overhead from the HUB, I would expect somewhere around 180 – 210 MB/second.

      Larry

  5. Jeffrey says:

    Hi Larry,
    Great explanation of RAID. It cleared everything up for me.

    On a semi-related topic, I have a question about USB 3.0 and video. I know that USB sends data out in spurts and that USB 2.0 was never an option for media drives. With USB 3.0 in a lot of Mac portables now, is it an option for a media drive? Could I get 120 MB/sec? Thank you so much.

    • Larry says:

      Jeffrey:

      Thanks for the kind words.

      Theoretically, USB 3 would work for media drives – while a single drive will only deliver about 100 MB/sec of data, a full-blown five drive RAID should be able to deliver the theoretical maximum for that protocol of around 480 MB/sec.

      However, I haven’t tested this and am not sure if there is an internal gate that would prevent speeds from attaining their maximum potential. Still, USB 3 is a much better choice for video than USB 2, but both protocols are totally smoked by Thunderbolt.

      Larry

  6. Tony says:

    Larry, yet again to the rescue.

    Your prediction is right on the money. I just bought the LaCie eSATA Hub, connected the whole thing up, and am averaging 175 MB/s Read & Write speeds. Keeping in mind the drive does only have 1.9TB left of the 4TB available.

    So I’m guessing if this drive was empty, I’d probably be pushing closer to the 200 MB/s mark?

    Just out of curiosity, with the drive being advertised at 250 MB/s, how exactly can one make it reach these speeds? When I spoke to CalDigit, their advice was those speeds are all dependant on the card it’s connected to.

    Looking at the specifications, the hub esata port is rated at a max. bus speed of up to 3Gb/s. Had this been a 6Gb/s bus speed, would the CalDigit read/write faster?

    • Larry says:

      Tony:

      The speed of a RAID is limited by the sum of the speeds of the drives it contains. So, two drives would max out around 200 MB/second, regardless of whether you were using 3 GB/s or 6 GB/s eSATA.

      Larry

Leave a Reply to Larry Cancel reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX 10.5 Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.