Significant Test Result Differences Between AJA System Test Lite and Blackmagic Disk Speed Test

Update – Nov. 28, 2022. I’ve heard back from OWC, Blackmagic Design and AJA. All three are looking into this problem.

Update – Dec. 5, 2022. Blackmagic PR wrote: “Thanks so much for sharing this information – I’ve passed this along to the Blackmagic Design team. As always, I’ll keep you posted as soon as I hear back from them.” Since then, I’ve heard nothing.

Update – Dec. 11, 2022>. Here’s an updated article looking at the differences between measurement tools. So far, no developer is able to explain these differences. Assume whatever test results you are looking at are wrong. (Errors also exist in Apple Activity Monitor.)

Update – Dec. 30, 2022. AJA is much more responsive. Today their tech support emailed me: “I did forward your test results to Product Development for review and they did acknowledge the Lite version has an issue. I don’t have any information on why there is a discrepancy between the full AJA system test and BMD, however I will let you know as soon as I have update from Product Development. By the way, thanks for all your work testing and documenting the test results, very much appreciated.”

This last week, I was testing the speed and capacity of the new OWC Thunderblade SSD RAID. (Read that review here.) This hardware speed demon is designed to maximize data transfer speeds using Thunderbolt 3 or 4. Except, I get significantly different results when testing storage speeds using AJA System Test Light versus Blackmagic Disk Speed Test.

Since we use both these tools daily to measure and verify the performance of our system, these differences trouble me — a lot.

EXECUTIVE SUMMARY

There is a 40% difference when measuring write speeds and a 22% difference when measuring read speeds between these two software tools.

Some variation during tests is expected, but this amount of difference is ridiculous. Clearly, one – or both- of these software tools is wildly wrong. But… which?

That, I have not yet determined.

MY COMPUTER

THE TESTS

I created a RAID 0 using an 8 TB OWC Thunderblade SSD, which uses four NVMe SSDs. This RAID format maximizes performance.

I ran four tests using each software to measure the read and write speeds of the Thunderblade.

NOTE: As part of my Thunderblade review, I ran twelve speed tests on each software and got very similar results. I used four tests here simply to save space.

(Click to see larger image.)

Same day, same computer, same storage, same data file size (4 GB), same connection (Thunderbolt 4)  — wildly different results.

THE DETAILS

• AJA System Test Lite consistently recorded write speeds 44% slower than Blackmagic Disk Speed Test.
• AJA System Test Lite consistently recorded read speeds 21.8% slower than Blackmagic Disk Speed Test.

NOTE: While the difference between read and write speeds is due to the Thunderblade, the differences in speed measurements are due to the software design.

Another troubling aspect of these results is that AJA System Test Lite reported a much wider fluctuation in speed measurements – especially in write speeds – than Blackmagic.

NOTE: The lower the standard deviation, which is measured in MB/second, the more consistent the speed. BMD reported far more consistent speeds than AJA. However, consistency is not the same as accuracy.

Here are the detailed results of my tests.

SUMMARY

Clearly, there is a significant difference between these two tools – and there shouldn’t be. Some variation is to be expected, but not this much. The problem is that we don’t know which one is the most accurate.

As I mentioned at the beginning,  I reached out to OWC, AJA and Blackmagic Design to get their interpretation of these results. All three said they would look into it and get back to me. For now, however, be careful relying on either of these software tools until we learn more.

EXTRA CREDIT

Apple’s Activity Monitor can be used to measure data transfer rates. However, unlike AJA or BMD’s tools, Activity Monitor can’t be isolated to report results on a single device. Activity Monitor shows aggregate disk or network traffic, which would almost always inflate the results.