I, like many others, am looking forward to the release of Apple’s latest powerhouse: the iMac Pro. Promised to breathe fire and leave skid marks across every digital highway on the planet, what’s a geek not to like?
Except… well, as we move into the world of 4K, HDR, and 360° VR video, a fast computer is important but no longer the critical factor. Our storage is. And, as we increasingly move into a collaborative environment, our ability to share projects and media with team members has a direct effect on whether we can meet our budgets and deadlines.
Here’s the problem: Apple focuses on creating computers that are powerful and flexible, combined with its legendary ease of use. All of which I fully approve. But, to achieve that speed Apple is increasingly using internal SSD drives as their main storage.
SSD drives have many advantages. When compared to traditional spinning media, they are:
Today, most computers sport 250 GB – 1 TB internal SSD drives. The new iMac Pro can be expanded to 4 TB, though pricing hasn’t yet been announced. It is blindingly fast, but its storage isn’t big enough. Which means that we need to connect it to external storage – and that’s where the problem lies.
HERE’S THE CHALLENGE
For editing HD media, any computer released in the last six years will be fine, though older systems will have problems playing or compressing HEVC (H.265) media.
However, when compared to HD:
In other words, a storage system that works perfectly fine for HD will be overwhelmed in both capacity and bandwidth when moving into these new formats. Stepping up to 4K HDR files could generate media storage requirements that are sixteen TIMES what you are using now, with bandwidth requiring hundreds of megabytes per second.
How are we going to handle this?
Yes, we can compress this larger media by working with proxies but this is something you need to think about before you commit to that exciting new HDR project.
For example, when I create new training, I always work in Apple ProRes 4444 because it yields the highest image quality for screen captures. My recent Final Cut Pro X 10.3 training – which had 220 movies in it – generated almost 6 TB of data during production; FAR beyond what any currently affordable SSD can handle.
Capacity. The amount of data your storage device can hold. Measured in TB (Terabytes).
Bandwidth. The speed that your storage device can transfer data between itself and your computer. Generally measured in mb/s (megabits per second), this number is more useful when converted to MB/s (MegaBytes Per Second) by dividing mbps by 8. Thus, 1000 mbps equals 125 MB/sec.
Protocol. How a storage device connects to your computer. Common options are: Ethernet, Thunderbolt 3, Thunderbolt 2, Thunderbolt 1, FibreChannel.
Here are some numbers you can use to estimate bandwidth from your storage:
NOTE: Here’s a link to an Apple White Paper that provides more detail on ProRes data rates.
As every manufacturer will tell you, there is lots of variability in these numbers, but these are good to use for estimates and planning.
NOTE: To get maximum benefit from any protocol, say Thunderbolt 3, both the computer and storage device must support the same protocol. Recently, I was testing the speed of Thunderbolt 2 devices, only to discover that my computer only supported Thunderbolt 1, which meant that the devices were running slower than they were designed. Or, another example, to get the maximum speed from WiFi, both your computer and WiFi receiver need to support the latest protocols.
I WORK ALONE
If you are a solo editor, then your goal should be to purchase the fastest RAID using a connection protocol that is supported by your computer:
The more devices (drives) a RAID contains, the faster it will go and the more it will store. While there are other protocols – iSCSI, FireWire, eSATA – Thunderbolt will have faster performance, lower costs and better market support.
When purchasing storage my advice is to buy as much storage capacity as you can afford. There is no such thing as “too much storage.”
I WORK IN A GROUP
When you need to share media between members of a local workgroup, things get more complex. And, when you need to share files with members outside the local workgroup, you’ll need professional help in figuring out the best way to set this up.
When I’m creating new training, for example, three of us work on it – all accessing shared storage. Since Thunderbolt is not a “shareable” protocol, we need to use either Ethernet or FibreChannel. Since wiring my office with fiber is way too expensive, I’m forced to use Ethernet.
When sharing files over a network, HD is relatively simple because the bandwidth needs are so small (comparatively). But problems arise as I move into larger format files.
More personal examples:
Last night, I needed to transfer 3.2 TB of data from local storage to a network server via WiFi. The process took 21 hours. Backing up that data via 1 Gb Ethernet, which I need to do every night, took seven hours.
1 Gb Ethernet is not fast enough for 4K HDR media. However, rewiring my office to support 10 Gb Ethernet – which WOULD be fast enough – requires:
NOTE: The new iMac Pro is announced to support 10 Gb Ethernet natively. It is the only Mac to do so.
As you can imagine, all this gear ain’t cheap. And wiring for fiber, though potentially faster, is even more expensive, as are the switches.
What I’m currently doing is tailoring the video formats I create and edit to match the speed of my storage. Which is an interesting thought: My computers are more than fast enough, but my storage is not – especially when I’m editing in a workgroup.
Increasingly, projects that used to fit comfortably in a terabyte, are now expanding into dozens, even hundreds of terabytes.
The speed of our computers is more than fast enough – especially the newer ones with high-performance GPUs – to handle almost all these new media formats.
But storage bandwidth – and how we connect our gear – is not keeping up. If you plan to expand into all this new technology be sure to budget for a major expense in storage capacity, storage bandwidth and faster network infrastructure.
Otherwise, you’ll be all ready to edit – and waiting for your media to arrive.
27 Responses to The Future of Video Editing Rests on Storage Bandwidth← Older Comments
Great article. What are your thoughts on using MS’ Surface Studio desktop as a content creation device? Do you see stylus / dial / touch UI paradigms replacing KB / Mouse for professional content creation?
Everyone has an opinion – and they are all different – on what’s the “best” user interface.
While I expect each to find success, for editors that are focused on speed and throughput, i don’t think you’ll find anything faster than keyboard and mouse.
What is the most efficient way to import 4K RAW files (Cinemadng) that were recorded with an Atamos Shogun Inferno External Recorder and SONY FS5, and create proxies to facilitate editing a project in Adobe Premiere Pro?
I followed the workflow outlined on Adobe’s website and spoke with their Tech Support staff, who said I should import 15 clips at a time (I have hundreds of clips), but the process is still taking an inordinately long time.
I was initially trying to import the files from a WD 4TB 3.0 wireless drive, to which I transferred the files from the my SSD cards. I purchased a Samsung SSD 2TB 3.1 2nd Gen. drive with the hope to speed up the importing/creating proxy process. I need to get my footage edited in the next couple of weeks, so I was wondering if you had any suggestions on how to make the process as efficient as possible. Thanks for you help!
Buying the faster drive was smart – WiFi will be too slow for files this big. A Thunderbolt version of the Samsung will be an even better option in the future.
That being said, I don’t know the answer. I forwarded your question to the product team at Adobe and I’ll let you know what I find out if they get back to me.
You might also reach out to Sony Support to see if they have any ideas. AT a guess, it may be easiest to transcode these into a friendlier format, but that’s just a thought. I’ll let you know what else I learn.
Hi Larry, Great info as always. A couple questions.
Why does anything above Thunderbolt 1 matter when running any SATA drives, in RAID or not, when the data rate is maxed out by the SATA limitation of 600 MB/s ?
Related to that, there is a nice looking hardware RAID enclosure that specifically uses USB 3.2 gen 2 (also known as USB superspeed 10 Gbps), the Areca ARC-8050U3-6 6-Bay USB Type-C RAID Enclosure. Is there a reason to NOT buy this device because of the USB connection?
I am thinking of buying this as it seems like it would be considerably more reliable and faster as compared to an OWC (though I don’think OWC has a 6 bay, but if they did)
You are confusing the speed of individual drives with the aggregate speed of all the drives on a RAID. The RAID does not use SATA to communicate with the computer, it uses Thunderbolt, which means the RAID is faster than any single drive in it. So, in general, a single spinning hard drive transfers data around 150 – 200 MB/sec. A single SSD, that is not NVMe, transfers data between 400 – 550 MB/second. The RAID, then, becomes the sum of all drives in it, less the number of drives reserved for parity data.
As a note, NVMe SSD drives don’t use SATA as their communications protocol, which allows them to transfer data around 2500 MB/second.
From what I’ve read and been told, USB is optimized for transferring small files, think Word and Excel. Thunderbolt is optimized for transferring large files, think media. Both will work, Thunderbolt will be more efficient.