We tested the PCI-Express SSDs using a system consisting of an Intel Core i5 and an ASRock Z97 Extreme6 motherboard. This board is one of the few that can still offer an M.2-slot usage of the full bandwidth, despite its Z97-chipset. The tests are run under Windows 8.1, while we of course did check whether or not using Windows 10 would result in a performance difference or not.
We use Iometer to determine the random read and write speeds with 4k data blocks. The performance with small data blocks is very important, because in Windows (and other operating systems) 4k is the most commonly used block size used for hard disks and SSDs. We run the tests with a queue depth of 1, 2, 4, 8, 16 and 32. This queue depth indicates how many simultaneous instructions are sent to the SSD. The performance at low queue depths is very important for consumer applications, and with QD32 SSDs can show off what they're capable off.
Iometer also measures the read and write speed with 1 MB data blocks. We use QD32, but here the number of parallel requests matters very little since the SSD controller is very good at distributing these over multiple SSD channels.
We also simulate the access patterns of a file server and a database server with Iometer. The database test entirely consists of random 8 kB operations, 67 percent of which is reading and 33 percent is writing. The file server tests is 80 percent reading and 20 percent writing, with the following transfer sizes: 10% 512 bytes, 5% 1 kB, 5% 2 kB, 60% 4 kB, 2% 8 kB, 4% 16 kB, 4% 32 kB and 10% 64 kB. Both the file server test and the database test are done with queue depth 1, 2, 4, 8, 16 and 32.
All Iometer tests are conducted with completely random data, so that SSD controllers using compression tricks don't have an edge. Each test runs for at least 30 seconds, which is much longer than comparable tests that are part of the various benchmark tools.
The popular AS SSD benchmark also tests performance with 4k data blocks (QD1 and QD64) and the sequential read and write performance. The advantage of running a test that's similar to Iometer is that AS SSD nicely converts the scores to a total score that provides a good impression of the general performance of an SSD. Secondly, AS SSD is easy to install on your own computer which allows you to compare scores.
Real-world benchmarks: PCMark
For consumer applications the real-world benchmarks PCMark7 and PCMar8 are much more important. PCMark7 simulates the hard disk access of actual software and indicates what the performance is in different scenarios. The total score is indicative of general use, and the subscores tell you what the speed is under different circumstances.
PCMark7 uses traces based on modern software from the Windows 7 era. PCMark7 runs the traces in real-time, so performance gains you wouldn't notice in real-life you won't see here.
The latest version of PCMark7 also has something called the Raw scores. These scores exclude the idle time in the traces of the various tests, so the scores don't correlate directly to real-life performance. They do clearly show the performance differences between SSDs.
We also run PCMark8, the latest version of the benchmark, and it also uses traces. PCMark8 uses traces of Adobe Photoshop, Adobe Illustrator, Adobe Indesign, Adobe After Effects, Microsoft Word, Microsoft Excel, Microsoft PowerPoint, World of Warcraft and Battlefield 3. The total score is the average score of each component. The traces run in real time, including the idle time. For this reason, the scores are very similar, just like in PCMark7. This is how it is in real life, with normal applications you won't notice much difference between different SSDs.
Continuous activity tests
We conduct two continuous activity tests in which we run a workload for 30 minutes on the SSD and measure the average performance for each minute. First we do this with the Iometer 4k random write test, and the the Iometer database simulation. Both continuous activity tests are done with QD32 and a file that takes up 75 percent of the available flash memory. For normal consumer use these results aren't very important, but for professional applications - in servers for example - these scores are essential, more so than any other benchmark results.
Consistency refers to how much the SSDs performance may fluctuate from one second to the next. In order to measure consistency, we once again run an Iometer 4k random write QD32 workload, but rather than taking a snapshot of the drive's performance each minute, we record an average for each second. Admittedly, performance consistency is mainly something that is relevant for the enterprise market. Those who put an SSD in a heavy duty database server will want to be sure that the performance of the used storage is predictable, as big ups and downs may result in erratic behaviour for certain workloads. For consumer workloads, performance consistency matters very little, although it will certainly influence the performance of SSDs when putting them in RAID 0. After all, when using a RAID 0 array, blocks of data will be written to two or more disks, which means that with regard to performance, the slowest drive (the "weakest link") will determine the overall level of performance. When one of the drives has poor performance consistency, the overall performance of a RAID 0 array will be relatively low, seeing as how the odds of one of the two drives having a performance "dip" are relatively high.
Unfortunately we are unable to measure the power consumption of M.2 and PCI-Express SSDs at the time of writing. This is because of the absence of the power connector that we can loop through an amp meter. However, we are still working on a solution to this problem.