Benchmarking Enterprise SSDs

Text-only Preview

Enterprise SSDs
The SSD head-to-head challenge, a worst case
SQLIO workload demonstration showcasing the
difference between SSDs.
SSDs benchmark scores may appear
Preconditioning by writing data to
The analysis reconciles benchmark
very fast but may mislead users
completely fil al NAND blocks so
data to actual user performance,
under an enterprise workload over
over time and under true enterprise
wear leveling and FLASH manage-
a period of time. The chal enge is to
workload environments.
capture actual consistent performance
ment routines are engaged.
over time.
Proper aspects of managing the host
data flow and internal NAND equates
to a realistic analysis of overal data.

Benchmarks are used extensively to report on performance but the traditional
benchmarks which are used to report differences for spinning HDD’s are not
effective at benchmarking SSD performance differences.
HDD’s are chal enged by design with rotational latency and seek times both of
which are easily measured in a short amount of time with existing benchmark
scripts. SSD’s tested with these scripts wil general y give a ‘best case’ perfor-
mance but don’t show the effects of running under enterprise conditions.
SSD’s have no spin or seek involved so the access results using HDD bench-
marks show very high IO rates and low latency results.
So how is an SSD different?
SSD’s are chal enged more with the aspects of managing the host data flow
and internal NAND than with accessing the data.
1. Accessing an SSD with mixed reads and writes rather than 100% read
or 100% write patterns
2. Running constant IO to the devices
3. Running large amounts of write data to fil the open NAND blocks so
wear leveling and FLASH management routines are engaged
4. Running constant random write-only test to determine devices capabili-
ties under worst case load
It is an easy, and now a much used line that says “Not al SSD’s are created
equal.” In use under heavy workloads, such as those found in an enterprise
environment, this adage is absolutely the case and easily visible under test
conditions that are readily reproducible.
This white paper wil discuss the concepts of the enterprise workspace, and
how an SSD’s performance can be impacted by these environments. Also
presented is data from 3 different competitive SATA SSD devices in a worst
case benchmark environment running over a prolonged period of time. This will
demonstrate how an SSD, which is used under an enterprise workload, can be
affected by the use environment. Then we wil recommend testing techniques
for SSD devices, al owing you to gauge their true behavior and performance
profile under enterprise level operating conditions.
Whitepaper on the difference between SSDs

The Problem with “Benchmarks”
Benchmarks are useful tools for testing devices as long as the benchmark and the resultant data are looked at with the
proper focus on how the device wil ultimately be deployed.
A benchmark is not necessarily useful if the test does not accurately represent the environment within which the tested
device wil be deployed. Unique to SSDs, and unlike HDDs, duration is also an important factor to consider in running
a benchmark. It is an unfortunate truth that many benchmarks are run in a short period of time and the results are not
demonstrative of the actual real-time experience for a working environment with SSD drives.
Current benchmark tests are optimized around testing HDDs and have overtime become very good at identifying HDD
results. HDD benchmarks are focused on identifying the chal enges with HDDs such as rotational latency and seek
times associated with spinning the media and moving the read/write heads across the surface of the disks. These
benchmarks are effective at demonstrating some of the relative strengths of SSDs as compared to HDDs, such as sus-
tained bandwidth and maximum read IOPS.
Challenging an SSD Drive with Benchmarks
It goes without saying that SSD’s are not the same as HDD’s and this is very true in the benchmarking. HDD bench-
marks are focused on finding the performance aspects of the drives and where they are weak such as rotational laten-
cy time and seek time. As SSD’s do not spin or seek, these normal access time specifications that can be measured in
a relatively short period of time do not apply to SSD’s. Shown in chart A.
SSD New (Empty)
SSD Working (Full)
Random Read IO

Very Fast

Very Fast
Random Write IO

Very Fast

Sequential Read

Very Fast

Very Fast
Sequential Write

Very Fast

Mixed Reads and Writes


Chart A. Performance aspects of storage drives.
To chal enge an SSD properly it is important to recognize where SSD’s perform wel and where they are chal enged to
perform wel .
Whitepaper on the difference between SSDs

The Demo
The demonstration associated with this white paper lines up three competing SATA SSD drives that are in production
and shipping to customers and graphical y il ustrates the results of running 100% write IO over time.
Each of the 3 SSD’s tested is shown in a tachometer style
gauge at the top of the test screen with each SSD hav-
ing a different colored indicator needle. This tachometer
graphic shows a representation of the actual IO’s per sec-
ond (IOPS) occurring on each device every second. At
the bottom of the screen, is a strip chart graph that plots
the IOPS from each of the 3 test drives over time. The line
colors correspond to the needle color in the tachometer
gauges at the top of the application (Green for SSD1, Red
for SSD2 and Blue for SSD3).
You can see the point in time where SSD from vendor B
slows down to manage wear leveling and block relocation
Figure 1. SQLIO IO Test
activity in order to make space available for the incoming
write data from the host.
Continuing the demo testing over time you can see as shown in the graphics below that the steady state performance
wil stabilize over time to a measureable pattern of access.
Shown below are the access measurements for the 3 SSD drives using block size of 4K for the database record access
(un-aligned IO at a queue depth of 8).
This demo is a tool for highlighting how differ-
ent SSD’s are not always equal in capability
when tested to the worst case access capa-
bilities and watching their behavior over time.
Figure 2. Random 4K access – 100% write IO
Whitepaper on the difference between SSDs

Demo Test Setup
There are many different enterprise environments and no single configuration can please everyone, however for this
demonstration we have selected a database environment for the purposes of shedding light on this topic and keeping
the benchmark within the range of actual user experience.
In this case we use a standard PC system loaded with a windows Vista operating system and 3 SATA SSD competi-
tive drives. Each drive is formatted with an NTFS file system, write caching is disabled in the device manager and then
loaded with a 30GB SQL database file.
The host runs a tool from Microsoft cal ed SQLIO.EXE that generates IO to any logical devices and generates access
requests to the SQL database files as fast as the file system can manage to enable measuring the storage path capability.
System and Software Specification

Gateway PC
Intel E4700 2.6GHz processor
4GB memory
Microsoft Vista 64bit
SSD Controller
SSD Drives
Drive X = Vendor A: MACH8 IOPS
Drive Y = Vendor B: Readily available Enterprise version SSD
Drive Z = Vendor B: Readily available Enterprise version SSD
Recognized on system by default
Write cache disabled on all 3 devices at the device manager
Test Software
Microsoft SQLIO.exe
Test File
30GB SQLIO database file
(created at program start by the SQLIO.exe application)
Monitoring Software
Custom application using M.S. PDH Perform API
Windows Perfmon.exe software will work equally in this test
Chart B. System and Software Specification
Configuring SQLIO.exe for the Test
The access pattern given to the SQLIO.EXE application is set for a continuous run of IO with two running command
windows of the program. One command window is set for random reads and the other for random writes.
Write Window:
SQLIO.exe -kW -s600 –frandom –o8 –b8 -LS -FSQLmon3.txt
The SQLmon3.txt file referenced above defines the device to access
x:\testfile.dat 2 0x0 30000

y:\testfile.dat 2 0x0 30000
z:\testfile.dat 2 0x0 30000
Final y for monitoring purposes a simple tool was added for graphical y plotting the IO in real time on the screen. Note:
the Perfmon.exe application that comes with windows could easily be used for this purpose as wel .
Whitepaper on the difference between SSDs

SSD Benchmark Concepts
Baseline the behavior at 100% Random Write over time
Measuring the performance of the drive as it is being written to with random data wil show how the behavior of
the SSD can change as it is being fil ed. This invokes the internal management schemes of the SSD to continue
to wear level and manage the FLASH.
We agree, this may not be ‘real world’ but this is a benchmark intended to get to the weak spots as quickly as
possible and expose the capabilities of the device.
Note: this test can be used as the preconditioning step below
Precondition the SSD under test
In this case pre-condition is the idea that an SSD should be fil ed past the raw capacity of the NAND/FLASH
memory with random write data in order to get the device into the mode of managing wear leveling, “scatter
gather” for retired blocks and error handling
This can be accomplished by fil ing the drive with write data past the raw capacity of the drive which should ac-
count for the typical over-provisioning in many SSD designs.
Mixed Read and Write operations
Another interesting weak spot in many SSD designs is mixed IO operations where the drive is mixing read and
write during the test. In many designs you wil see a bathtub curve effect in benchmarks showing 50/50 ratio of
writes and reads can be the bottom of a curve of performance from either pure read or write operations.
STEC recommends running a range of difference to see how this might be impacting your user environment.
STEC uses read ratio’s of 100%, 80%, 60%, 40%, 20% and al write.
Aligned IO vs. un-Aligned IO
IO alignment can have a significant impact on the performance of an SSD even when ful y
pre-conditioned with random write data.
FLASH architectures widely employed today use a 4K page write data. If using un-aligned write operations then
in many cases the device wil be impacted by the Read-Modify-Write schemes needed to span data across two
pages in the flash.
A good measure wil test both aligned and un-aligned capabilities. Which measure to pay attention to wil de-
pend on your operating system and file format capability.
A range of access patterns at different block sizes and queue depths
It is important to test the SSD at different block sizes and at different queue depths.
This type of measure is important in terms of the way the device wil be used in the customer environment and
what performance expectations can be seen depending on use model.
Whitepaper on the difference between SSDs

100% Random Write I/O Over Time
STEC recommends this type of test to both pre-condition a drive for general benchmark tests and also to baseline the
capability of an SSD in the worst case access methods where SSD’s struggle 100% Random write IO wil initial y run
pretty quickly assuming the drive is unfil ed at the start of the test. Over time and depending on the raw capacity of the
drive, once al the raw blocks in the device have been written to the SSD, it wil then start to engage its wear leveling
and FLASH management algorithms and the impact of those algorithms can be measured and evaluated.
As is visible from the data below, the actual IOPS under 100% write operations are achieving from ~2,700 IOPS to as
low as 170 IOPS.
Figure 3. Access performance as SSD’s are filled with write data
As you can see from this example of the effect of fil ing an SSD with random write data this chart demonstrates how a
working SSD can lose performance as it moves into wear leveling or other internal tasks for managing the operations of
the SSD.
Pre-Conditioning: empty drive vs. filled drive: As we have been describing, an empty SSD wil run much faster than
a fil ed SSD. Preconditioning the SSD with random write data sets the device up to be able to return accurate ‘steady
state’ performance results that the device is capable of delivering over the useful life of the device.
Pre-conditioning is done by over-writing the drives RAW capacity with random write data.
Note: Sequential write data wil be insufficient to force the device into the block al ocation and wear leveling algorithms
that wil tend to impact performance.

Whitepaper on the difference between SSDs

Mixed read and write ratios: Another facet of SSD performance is the drives ability to perform in a mixed read/write mode.
It is typical to call out the absolute performance of an SSD when performing 100% reads or 100% writes as these are
easy to measure. What is not typically described in the spec sheets is that when mixing both reads and writes at the
same time to an SSD, most devices actually slow down the most under this load due to write priority and having to ser-
vice the write operations which take time to complete.
As you can see from the test data shown, the
typical SSD drive demonstrates a bathtub
curve ef ect at the mixed IO ratios with a large up-
swing in performance at the 100% write operation.
Note:STEC designs have been architected to
address these issues.

Figure 4. Mixed Read and Write benchmark
Aligned vs. un-Aligned IO: Aligned IO can have a tremendous impact on SSD performance and endurance. Aligned
IO for an SSD gives efficiency to the device for managing the NAND writes and can also boost SSD endurance by re-
ducing the number of Read-Modify-Write operations that cause extra writes to occur in the background on the SSD.
Virtually all FLASH today writes a page of data at a time at a 4K page size. Aligning the IO to the page size, or a multiple
of 4K, offers the SSD the ability to maintain
efficient write management without the need
to constantly invoke schemes of read-modify-
write methods to store data across two flash
pages due to mis-alignment.
Unfortunately this may be out of the control of
the SSD integrator depending on the operat-
ing system and file system/partition format
so measuring both aligned and un-aligned
performance is a reasonable measure.
Figure 5. Mixed Read and Write benchmark
Whitepaper on the difference between SSDs

Varying IO block sizes: This is where we get to a benchmark result that starts to look similar to tests run on regular HDD drives
It is important to have all the pre-conditioning and environment decisions on alignment done before these tests are per-
formed in order to see the actual steady state performance an SSD device is capable of achieving.
The STEC data shown is demonstrating consis-
tent level performance at all block sizes across
the range of read/write ratios without the bath-
tub curve in mixed operations that affect other
SSD designs.
Figure 6. Benchmark test at varied block sizes and Queue Depth of 32
Varying Queue Depths: Queue depth is an important factor for systems and storage devices Efficiencies can be gained
from increasing queue depth to the SSD devices that allow for more efficient handling of write operations and may also
help reduce write acceleration that can affect the endurance life of the SSD.
This benchmark demonstrates the capability
of the device at different queue depths and
indicates to the user the performance which
can be expected from this SSD depending on
the use model for queue depth and read/write
ratio. This can also help the system integra-
tor adjust the system parameters to manage
queue depth settings to an optimum for the
device being integrated.
Figure 7. IOPS Performance with queue depth variance measured at different Read/Write ratio’s
8K data (unaligned)

Whitepaper on the difference between SSDs

The drive that may benchmark with the fastest raw read or write results from existing tools may be misleading as to
performance that wil be experienced in a working environment. SSD performance depends heavily on the workload
and time in use to show actual performance and can be significantly different from any artificial y high output from the
benchmarks and the drive specifications.
The measure for how an SSD wil work in a real world environment is to test the drive in the same environment the de-
vice wil be deployed and select the device that demonstrates capability in the target environment.
Benchmark Recommendations
1. Avoid running benchmarks on empty drives. Empty SSD’s don’t have to read data from flash as they are al-
ready fil ed with Zero’s.
2. Condition SSDs before testing with block level data. ie: Random write changing data to al blocks level data in
the SSD more than once.
3. Always run mixed read and write tests. Mixed IO is hard to manage and maintain speed.
4. Run tests over extended periods of time. Extended read and write testing al ows the drive to fil up and starts to
engage wear leveling algorithms and other internal algorithms that may impact performance.
5. Where possible run real world environments (using a file system and real data).
Whitepaper on the difference between SSDs