Caution: Storage Speeds Can Vary Dramatically and Cause Problems

Posted on by Larry

workflowAs I was writing an article on compression speed using the 4.3.1 version of Apple Compressor, I needed to transfer some files from one hard disk to another. More specifically from a single, spinning media hard disk to an external SSD connected via USB3.

Things started out well – then they got weird. Then, they got REALLY weird.

BACKGROUND

All storage devices have maximum speeds which can vary based on a number of factors:

It has long been reported that the Finder is not particularly efficient at copying files between drives, though moving files within the same hard disk is almost instantaneous.

compressor007

The computer doing the transferring was an iMac running OS X 10.12.3.

The spinning media drive transferred data around 75 MB/sec, while the SSD transfered data around 115 MB/sec; as measured by the Blackmagic Disk Speed Test.

NOTE: The specific brands of drives are not the issue, directly, so just stay with me a for a bit.

I was transferring 143 GB of files, the smallest of which was 2.5 GB. Due to the way the Finder works, small files transfer more slowly than large files. These are really big files, so they should transfer faster than “normal.”

WHAT HAPPENED?

Um, I don’t know, really, but here’s what happened in the first two minutes. The spinning media drive (in blue) started playing back data at 45 MB/sec, while the SSD started recording it at 150 MB/sec.

NOTE: The key numbers are Data read/sec (playback) and Data written/sec (recording). Both are measured using Activity Monitor, which is stored in your Utilities folder, under the Disk tab.

This transfer shouldn’t be asymmetrical – you can only record data at the speed it is played back. The SSD was recording data three times faster than it was playing back. There are always measurement issues when a transfer first starts, but this lasted almost two minutes.

Then, pulsing started. The fastest sections (the fat ones) pushed data at close to 120 MB/sec, while the slower sections dragged at 17 MB/sec. Notice also that the “saw-tooth edge” of the spinning media drive indicates significant speed variations, while the SSD drive records at a more steady pace.

Then, things just got insulting – speeds slowed to 8 MB/sec. This data rate is far too slow to support even SD video!

NOTE: I consider 75 MB/sec the minimum speed necessary to support reasonable HD video editing. Faster is, obviously, better; but steady and constant speeds are essential, too.

ANALYSIS

During the transfer, Time Machine was turned off and all other applications were quit. Nothing should have interfered with this transfer running as fast as possible.

I’ve known for a long time that spinning media drive speeds will vary slightly when transferring data, depending upon how long it takes the heads to find the right spot on the disk. But, I’ve never measured anything like this.

Worse, if you were editing and this happened, you’d be dropping frames left and right, and never think that the hard disks you were using were slowing down so severely. Still, I’m not sure where the problem is:

One thing I noticed was that when I transferred a large file from the internal Fusion drive to the SSD, I got consistent speeds with much less fluctuation. So, this may point the finger toward the spinning media drive being the cause of the problem.

SUMMARY

I don’t have a solution, but I do have some recommendations:

There are two free utilities that are excellent for testing hard disk speeds:

In all cases, when dropped frames or slow data transfers start occurring, ignorance is not bliss. And, as always, let me know what you think.


Bookmark the permalink.

6 Responses to Caution: Storage Speeds Can Vary Dramatically and Cause Problems

  1. Tom Thornton says:

    A thousand thank yous Larry for this one 🙂
    I installed and ran BlackMagic ( wow its free) and was relieved to find my speeds are adequate for all my needs.
    Had I not read your article I would never have known I was courting diaster.

    tomt

  2. stu aul says:

    Hi Larry-
    forgive me here if I am missing something, but I need some clarity. The tests you conducted, as I understand it, are TRANSFER speeds (like:dubbing) between drives?
    Am trying to understand how that relates to editing app playback, when, at least for me, I am viewing (editing) in FCPX, I am playing material off a single source/drive. In other words, my understanding is there is NO transfer going on in this environment. ?? Are your observations valid in this situation?
    This is not to say you may be making the same valid point as to disc performance. I just need some clarification about what this info is telling me…
    ?
    Thanks as always!!
    Stu

    • Larry says:

      Stu:

      Your thinking is not correct. When we are editing, we are playing back directly from storage. If our storage speeds vary significantly, it will cause dropped frames or stuttery playback. That’s why the Data read/sec is so important. Also, when we render or export, we are both reading and writing files – reading the source media and writing the new output files.

      The reason I noticed this at all was because I was copying files between drives, which means I was both reading and writing.

      Larry

  3. Mark Mayer says:

    Maybe check the order you are loading extensions?

    —Mark stuck in 1999

  4. Bruce G says:

    Thanks, Larry. I have indeed just lately been experiencing dropped frames. Your article gave me a place to start to rectify.

Leave a Reply to Bruce G Cancel reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.