Understanding Storage and Media Bandwidth [u]

Posted on by Larry

[ This article was first published in November, 2008Updated: July 2009, May 2010, and significantly revised in June 2017. ]

Now that the shift to digital media is complete and the on-going race to every higher resolutions continues to pick up speed, it is increasingly important that we understand the role codecs and storage plays in the overall performance of our editing systems.

There are three key concepts we need to keep in mind:

  1. Storage requirements change as frame size, frame rate and codecs change.
  2. Hard disks are not infinite in speed, nor unlimited in storage.
  3. Dropped frame errors and stuttery performance are generally caused by a storage system that is too slow for the media you are editing.

STORAGE

Its seems obvious, but it is very easy to ignore. File sizes and bandwidth requirements increase as frame size, frame rate and bit depth increase. The table below illustrates this.

Also, as you’ll see, different codecs create files of significantly different sizes.

HARD DRIVE SPEEDS

Hard drive “bandwidth” is the speed at which a storage device transfers data between itself and the computer. We also call this a drive’s “data transfer rate.” This speed is dependent upon:

Here are several things to keep in mind:

  1. In the past, protocols like FireWire and USB 2 were slower at transferring data than the hard disk. So, how we connected the drive made a difference in its speed. Today, protocols like Thunderbolt and USB 3.1 Gen 2 are far faster than a single hard drive. So the speed of our storage is based on the number of drives they contain more than how the drives are connected to the computer.
  2. A single spinning media hard drive transfers data between 100 – 150 MB/second.
  3. A single SSD drive transfers data around 400 MB/second.
  4. RAIDs are devices that contain multiple drives; either spinning or SSD. The speed of a RAID is dependent upon how it is configured.
  5. USB 3.1 Gen 1 is FAR slower than USB 3.1 Gen 2.
  6. As frame sizes increase, file sizes get bigger and bandwidth needs to be faster.
  7. As frame rates increase, file sizes get bigger and bandwidth needs to be faster.
  8. As bit-depth increases, file sizes get bigger and bandwidth needs to be faster.
  9. Most editing codes – ProRes, DNxHD, GoPro Cineform, R3D – create large files which are optimized for editing performance.
  10. The more data you store on a hard drive the slower it goes. A drive is fastest when it is empty. When a drive is totally full, it neither plays back, nor records. I strongly recommend keeping about 20% free space on your drive.
  11. FireWire 800 drives are fine for standard definition video and single-stream HD editing. They will not be fast enough for editing multicam, 4K, RAW or extended bit-depth files.

In general, RAID 0 is the fasted configuration, while RAID 5 is the best balance between performance and guarding against data loss due to a drive failure. To determine the approximate bandwidth of a RAID, multiply the number of drives by the average speed of a drive. For example, a RAID 5 containing four spinning media disks will transfer data around 600 MB/second. Using four SSD’s will transfer data around 1,200 MB/second.

NOTE: By the way, if terms like RAID 0 or RAID 5 confuse you, read this short explanation of how RAIDS are defined.

Ideal Data Transfer Rates By Storage Connection Type
Connection Data Transfer Speed
ThunderBolt 3
About 3,000 MB / second
ThunderBolt 2
About 1,400 MB / second
USB 3.1 Gen 2
About 1,000 MB / second
10-Gig Ethernet
About 1,000 MB / second
USB 3.1 Gen 1
About 450 MB / second
1-Gig Ethernet
105 MB / second
FireWire 800
70 – 80 MB / second
Formats Too Slow to Use
USB 2.0
10 – 15 MB / second
FireWire 400
20 – 25 MB / second
iSCSI 75 – 95 MB / second

 

Keep in mind that these speeds define the size of the “pipe” that carries the data. The actual data speed is dependent upon how many drives are connected, how the drives are configured and how full the drives are.

With this as a background, let’s see how this applies to video editing.

VIDEO FORMAT DATA TRANSFER REQUIREMENTS

Here’s how to read the table below.

Format indicates the frame size, frame rate and codec.

Store One Hour indicates how much hard disk space it takes to store an hour of material in a particular format. This allows you to estimate what your total storage needs are based upon the amount of material shot. Remember, to add 20% for general storage overhead and free space.

Single Stream indicates how much data, on average per second, is required by a single stream of that format. (Measured in megabytes per second: MB/s.)

Editing indicates the approximate bandwidth needed for normal “single-stream” editing in that format.

Multicam indicates the approximate bandwidth needed for a 4-camera multicam edit in that format. (Using proxy files will decrease this number significantly.)

Here’s the key point: The table below helps you to match the required data rate of your video format with the speed of your storage, indicated in the table above.

Bandwidth by Video Format
Video Format Store 1 Hour Single Stream
(MB / second)
Editing
(MB / second)
Multicam
(MB / second)
SD
DV NTSC / PAL
13 GB
3.75
7.5
15
DVCPRO-50
27 GB
7.5
15
30
Uncompressed 8-bit (Beta SP)
72 GB
20.2
41
85
Uncompressed 10-bit (DigiBeta)
96 GB
26.7
54
110
ProRes 422 (NTSC or PAL)
19.5 GB
5.25
12
24
ProRes 422 HQ
28.1 GB
7.8
16
32
HD
HDV (25 mbps) 60i
13 GB
3.75
7.5
15
AVCHD 1080p/30
Up to 10.8 GB
1.5 – 3.0
6.0
12
AVCCAM 1080p/30
Up to 10.8 GB
1.5 – 3.0
6.0
12
AVC-Intra 1080p/30
45 GB
12.5
25
50
XDCAM HD (50 mbps)
28 GB
7.75
16
32
XDCAM EX 1080p/30
19 GB
5.2
12
24
DVCPROHD 1080p/30

45 GB

12.5
25
50
ProRes 422 (Proxy) 1080p/30 20 GB 5.6 12 24
ProRes 422 1080p/30 66 GB 18.1 36 72
ProRes 422 (HQ) 1080p/30 99 GB 27.5 55 110
ProRes 4444 (no alpha)
1080p/30
148 GB 41.25 85 170
R3D 1080p/30 *
137 GB
38
76
150
HDCAM 1080p/60
834 GB
237
474
948
4K AND BEYOND
UHD/30 ProRes Proxy 82 GB 22.75 46 92
UHD/30 ProRes 422 265 GB 73.5 150 300
4K/24 R3D * 586 GB 163 326 650
6K/24 R3D * 660 GB 183 360 720
Sony RAW 4K * 520 GB 300 600 1200

 

NOTES

* R3D and Sony files have a huge number of variations based upon compression rate, frame size, frame rate, and the amount of the sensor used. Consult the websites for specific Sony and RED cameras to determine the exact data rates for the media format you plan to shoot.

All AVC video formats are converted (transcoded) to ProRes422 during ingest into Final Cut Pro X. So, while the AVC source video is small, the converted ProRes are much larger.

ProRes is a variable bit-rate encoder, so file sizes vary depending upon format, image size, and frame rate. The HD specs for ProRes are  taken from Apple’s ProRes white paper. File sizes decrease for 720p files, or slower frame rates.

I’ve discovered that understanding the relationship between hard disk speeds and video formats greatly simplifies storage budgeting and solving dropped frame problems.

UPDATE – Nov. 17, 2008

Rich Roddman sent me the following note:

You have R3D files listed as not needing to create Pro Res files as they are imported into Final Cut. That is not quite true, while you can just drop the proxy file in the timeline to view or edit, if you use the Log & Transfer tool to import the R3D files they will transcode them into Pro Res (HQ). The 4K files are converted to 2K in in width, 3K import at 3K and 2K stays at 2K. Much like P2, it is not a fast process even with an 8 core tower.

Rob Naim adds:

There is a great widget from a UK company called Digital Heaven than can be helpful when working out video storage issues that you’ve discussed in your current newsletter.
Have a look at it here: http://www.digital-heaven.co.uk/videospace/

It doesn’t do data rates but does do storage needed for length of video and its free.

Larry replies: Thanks for both your thoughts.

 


Bookmark the permalink.

7 Responses to Understanding Storage and Media Bandwidth [u]

  1. James Prescott says:

    You say “10 – 15 MB / second” for USB 1.0 but that is impossible. USB 1.0 & 1.1 have a theoretical max of only 1.5 MB / second (12Mbit/s).

  2. Olaf Timm says:

    You say “75 – 95 MB / second” for iSCSI (Ethernet) but that is IMHE -at least for a single Gbit nic- impossible.
    Net transfer rate should be around 40MB/s per single nic, with bonding of 2 Gbit nics you may realize the described rate of around 80MB/s (at 9000 MTU).

  3. swaroop says:

    Dear sir, I want to transfer my 24 hours live video channel to a remote location by using 2mb private leased line. can it be possible ? my concern is with out any kind of mpeg2 or mpeg4 compression by just s-video out to ethernet and again that side ethernet to svideo. please suggest me

  4. We’re a bunch of volunteers and starting a new scheme in our community.
    Your site offered us with valuable information to work on. You’ve performed a formidable activity and
    our whole neighborhood will be thankful to you.

  5. Kane Peterson says:

    You values for AVC-Intra are incorrect. While this format is based on the AVC/H.264 compression the datarate is much higher. It runs at 50mb/s or 100mb/s (not down in the 12mb/s to 24mb/s rates like AVCHD or AVCCam)

  6. pablo says:

    Hello Larry, is there an updated version of this article that includes the latests codecs?
    also, is there any way to get an alert on this conversation?
    thank you very much

Leave a Reply

Your email address will not be published. Required fields are marked *

Larry Recommends:

FCPX Complete

NEW & Updated!

Edit smarter with Larry’s latest training, all available in our store.

Access over 1,900 on-demand video editing courses. Become a member of our Video Training Library today!

JOIN NOW

Subscribe to Larry's FREE weekly newsletter and save 10%
on your first purchase.