We ran fio on i.MX 8M Mini Evaluation Kit to test the nvme. The PCIe is Gen2 x1. We found the write bandwidth can reach to 400MB/s, while read bandwidth is only 260MB/s as the results shown below. We tested the same nvme on other platforms. Both write and read can reach to 400MB/s. We want to know what will be the constrain of nvme reading on i.MX 8M Mini?
./fio -ioengine=libaio -bs=64k -direct=1 -thread -rw=read -filename=/dev/nvme0n1 -name="BS 64KB read test" -iodepth=16 -runtime=60
BS 64KB read test: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 thread
Jobs: 1 (f=1): [R(1)][100.0%][r=264MiB/s][r=4227 IOPS][eta 00m:00s]
BS 64KB read test: (groupid=0, jobs=1): err= 0: pid=6066: Thu Apr 28 20:43:32 2022
read: IOPS=4217, BW=264MiB/s (276MB/s)(15.4GiB/60004msec)
slat (usec): min=18, max=2612, avg=28.66, stdev= 6.76
clat (usec): min=380, max=32820, avg=3759.64, stdev=332.16
lat (usec): min=441, max=32855, avg=3788.30, stdev=331.20
clat percentiles (usec):
| 1.00th=[ 3130], 5.00th=[ 3359], 10.00th=[ 3425], 20.00th=[ 3490],
| 30.00th=[ 3720], 40.00th=[ 3752], 50.00th=[ 3752], 60.00th=[ 3785],
| 70.00th=[ 3818], 80.00th=[ 4015], 90.00th=[ 4080], 95.00th=[ 4146],
| 99.00th=[ 4359], 99.50th=[ 4424], 99.90th=[ 4555], 99.95th=[ 4686],
| 99.99th=[15533]
bw ( KiB/s): min=253061, max=270976, per=100.00%, avg=270249.08, stdev=2158.40, samples=119
iops : min= 3954, max= 4234, avg=4222.50, stdev=33.72, samples=119
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=79.04%, 10=20.94%, 20=0.01%, 50=0.01%
cpu : usr=6.26%, sys=31.53%, ctx=122462, majf=0, minf=257
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=253070,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=264MiB/s (276MB/s), 264MiB/s-264MiB/s (276MB/s-276MB/s), io=15.4GiB (16.6GB), run=60004-60004msec
Disk stats (read/write):
nvme0n1: ios=252614/0, sectors=32334592/0, merge=0/0, ticks=919387/0, in_queue=919388, util=99.95%
./fio -ioengine=libaio -bs=64k -direct=1 -thread -rw=write -filename=/dev/nvme0n1 -name="BS 64KB read test" -iodepth=16 -runtime=60
BS 64KB read test: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=16
fio-3.36
Starting 1 thread
Jobs: 1 (f=1): [W(1)][100.0%][w=398MiB/s][w=6366 IOPS][eta 00m:00s]
BS 64KB read test: (groupid=0, jobs=1): err= 0: pid=628: Thu Apr 28 20:47:54 2022
write: IOPS=6357, BW=397MiB/s (417MB/s)(23.3GiB/60003msec); 0 zone resets
slat (usec): min=32, max=1904, avg=126.17, stdev=32.66
clat (usec): min=1632, max=5095, avg=2386.68, stdev=227.02
lat (usec): min=1739, max=5216, avg=2512.85, stdev=224.06
clat percentiles (usec):
| 1.00th=[ 1909], 5.00th=[ 2008], 10.00th=[ 2073], 20.00th=[ 2212],
| 30.00th=[ 2311], 40.00th=[ 2343], 50.00th=[ 2376], 60.00th=[ 2409],
| 70.00th=[ 2474], 80.00th=[ 2540], 90.00th=[ 2671], 95.00th=[ 2769],
| 99.00th=[ 2966], 99.50th=[ 3064], 99.90th=[ 3294], 99.95th=[ 3818],
| 99.99th=[ 4883]
bw ( KiB/s): min=406355, max=408399, per=100.00%, avg=407263.99, stdev=398.32, samples=119
iops : min= 6349, max= 6380, avg=6363.22, stdev= 6.27, samples=119
lat (msec) : 2=4.62%, 4=95.33%, 10=0.04%
cpu : usr=31.84%, sys=54.83%, ctx=81509, majf=0, minf=1
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,381462,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
WRITE: bw=397MiB/s (417MB/s), 397MiB/s-397MiB/s (417MB/s-417MB/s), io=23.3GiB (25.0GB), run=60003-60003msec
Disk stats (read/write):
nvme0n1: ios=46/381026, sectors=2120/48771328, merge=0/0, ticks=11/854635, in_queue=854646, util=100.00%
