SPDK (Storage Performance Development Kit) is an optimized storage reference architecture. It is initiated and developed by Intel.
SPDK provides a set of tools and libraries for writing high performance, scalable, user-mode storage applications. It achieves high performance by moving all of the necessary drivers into userspace and operating in a polled mode, like DPDK.
All these changes make software latency the major contributor to the whole latency stack in nowadays.
Two key changes for SPDK to reduce latency caused by software stack:
Following is the software stack of SPDK:
Provides direct, zero-copy data transfer to and from NVMe SSDs. It controls NVMe devices by directly mapping the PCI BAR into the local process and performing MMIO. I/O is submitted asynchronously via queue pairs.
User space application that presents block devices over the network using RDMA. It requires an RDMA-capable NIC with its corresponding OFED software package installed to run.
Implementation of the established specification for block traffic over Ethernet. Current version uses the kernel TCP/IP stack by default.
This generic block device abstraction is the glue that connects the storage protocols to the various device drivers and block devices. Also provides flexible APIs for additional customer functionality (RAID, compression, dedup, and so on) in the block layer.
A persistent, power-fail safe block allocator designed to be used as the local storage system backing a higher level storage service, typically in lieu of a traditional filesystem.
This is a virtual device that VMs or databases could interact with.
Adds basic filesystem functionality like filenames on top of the blobstore.
It extends SPDK to present virtio storage controllers to QEMU-based VMs and process I/O submitted to devices attached to those controllers
A framework for writing asynchronous, polled-mode, shared-nothing server applications.
The event framework is intended to be optional; most other SPDK components are designed to be integrated into an application without specifically depending on the SPDK event library. The framework defines several concepts - reactors, events, and pollers.
SPDK build/deployment is tested on LS2088.
# git clone git://dpdk.org/dpdk
# export RTE_TARGET=arm64-dpaa2-linuxapp-gcc
# export RTE_SDK=/code/dpdk
# make T=arm64-dpaa-linuxapp-gcc CONFIG_RTE_KNI_KMOD=n CONFIG_RTE_LIBRTE_PPFE_PMD=n
CONFIG_RTE_EAL_IGB_UIO=n install -j 4
# git clone https://github.com/spdk/spdk
# cd spdk
# sudo ./scripts/pkgdep.sh
# make -j8
check NVMe status
# sudo lspci -vn | sed -n '/NVM Express/,/^$/p'
You should see lines like
# modprobe uio
# modprobe uio_pci_generic
# echo -n "8086 2700 8086 3900" > /sys/bus/pci/drivers/uio_pci_generic/new_id
# echo -n "0000:01:00.0" > /sys/bus/pci/drivers/nvme/unbind
# echo -n "0000:01:00.0" > /sys/bus/pci/drivers/uio_pci_generic/bind
# modprobe vfio-pci
# cd <SPDK_ROOT_DIR>
# sudo ./examples/nvme/identify/identify
This app should give you the detail disk info of attached NVMe storage.
# sudo ./examples/nvme/perf/perf -q 128 -s 4096 -w write -t 60 -c 0xFF -o 2048 -r 'trtype:PCIe traddr:0000:01:00.0'
This will give SPDK performance data.
With prior described HW/SW settings, following data are achieved (performance in MBps):
# git clone https://github.com/axboe/fio --branch fio-3.3
# cd fio
# cd spdk
# ./configure --with-fio=<path-to-fio-src> --enable-debug
# make DPDK_CONFIG=arm64-armv8a-linuxapp-gcc
# cd fio
# LD_PRELOAD=../spdk/examples/nvme/fio_plugin/fio_plugin ./fio --name=nvme --numjobs=1
--filename="trtype=PCIe traddr=0000.01.00.0 ns=1" --bs=4K --iodepth=1
--ioengine=../spdk/examples/nvme/fio_plugin/fio_plugin --direct=1 --sync=0 --norandommap --group_reporting
--size=10% --runtime=3 -rwmixwrite=30 --thread=1 --rw=r
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.