SPDK¶

liblightnvm provides a kernel-bypassing backend implemented using Intel SPDK.

Installing SPDK¶

On a recent Ubuntu LTS, run the following to install SPDK from sources:

# Make sure system is up to date
sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade

# Clone SPDK into /opt/spdk
sudo git clone https://github.com/spdk/spdk /opt/spdk
sudo chown -R $USER:$USER /opt/spdk

# Go to the repository
cd /opt/spdk

# Checkout the v18.07 release
git checkout v18.07
git submodule update --init

# Install dependencies
sudo ./scripts/pkgdep.sh
sudo apt-get install uuid-dev

# Configure and build it
./configure
make -j $(nproc)

# Check that it works
./test/unit/unittest.sh

# Install DPDK
cd /opt/spdk/dpdk
sudo make install

# Install SPDK
cd /opt/spdk
sudo make install

Output from the last unittest.sh command should yield:

=====================
All unit tests passed
=====================

Unbinding devices and setting up memory¶

By running the command below 8GB of hugepages will be configured and the device detached from the Kernel NVMe driver:

sudo HUGEMEM=8192 /opt/spdk/scripts/setup.sh

This should output similar to:

0000:01:00.0 (1d1d 2807): nvme -> vfio-pci

If anything else that the above is output from setup.sh, for example:

0000:01:00.0 (1d1d 2807): nvme -> uio_generic

Or:

Current user memlock limit: 16 MB

This is the maximum amount of memory you will be
able to use with DPDK and VFIO if run as current user.
To change this, please adjust limits.conf memlock limit for current user.

## WARNING: memlock limit is less than 64MB
## DPDK with VFIO may not be able to initialize if run as current user.

Then consult the notes on Enabling ``VFIO`` without Limits.

Re-binding devices¶

Run the following:

sudo /opt/spdk/scripts/setup.sh reset

Should output similar to:

0000:01:00.0 (1d1d 2807): vfio-pci -> nvme

Device Identifiers¶

Since devices are no longer available in /dev, then the PCI ids are used using SPDK notation, such as traddr:0000:01:00.0, e.g. using the CLI:

sudo nvm_dev info traddr:0000:01:00.0

And using the API it would be similar to:

...
struct nvm_dev *dev = nvm_dev_open("traddr:0000:01:00.0");
...

Build liblightnvm with SPDK support¶

With SPDK in place, configure the liblightnvm build with:

make spdk_on configure build

Linking your source with liblightnvm and SPDK¶

Invoke like so:

gcc hello.c -o hello \
  -fopenmp \
  -llightnvm \
  -lspdk_nvme \
  -lspdk_util \
  -lspdk_log \
  -lspdk_env_dpdk \
  -lrte_bus_pci \
  -lrte_eal \
  -lrte_mempool \
  -lrte_mempool_ring \
  -lrte_pci \
  -lrte_ring \
  -lrt \
  -ldl \
  -lnuma \
  -luuid

The above compiles the example from the quick-start guide, note that the code has a hardcoded device identifier, you must change this to match the SPDK identifier.

Enabling `VFIO` without limits¶

If nvme is rebound to uio_generic, and not vfio, then VT-d is probably not supported or disabled. In either case try these two steps:

Verify that your CPU supports VT-d and that it is enabled in BIOS.
Enable your kernel by providing the kernel option intel_iommu=on. If you have a non-Intel CPU then consult documentation on enabling VT-d / IOMMU for your CPU.
Increase limits, open /etc/security/limits.conf and add:

*    soft memlock unlimited
*    hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited

Once you have gone through these steps, then this command:

dmesg | grep -e DMAR -e IOMMU

Should contain:

[    0.000000] DMAR: IOMMU enabled

And this this command:

find /sys/kernel/iommu_groups/ -type l

Should have output similar to:

/sys/kernel/iommu_groups/7/devices/0000:00:1c.5
/sys/kernel/iommu_groups/5/devices/0000:00:17.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1d.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.6
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.4

And SPDK setup:

sudo HUGEMEM=8192 /opt/spdk/scripts/setup.sh

Should rebind the device to vfio-pci, eg.:

0000:01:00.0 (1d1d 2807): nvme -> vfio-pci

Inspecting and manually changing memory avaiable to SPDK aka HUGEPAGES¶

The SPDK setup script provides HUGEMEM and NRHUGE environment variables to control the amount of memory available via HUGEPAGES. However, if you want to manually change or just inspect the HUGEPAGE config the have a look below.

Inspect the system configuration by running:

grep . /sys/devices/system/node/node0/hugepages/hugepages-2048kB/*

If you have not yet run the setup script, then it will most likely output:

/sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages:0
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:0
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/surplus_hugepages:0

And after running the setup script it should output:

/sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages:1024
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:1024
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/surplus_hugepages:0

This tells that 1024 hugepages, each of size 2048kB are available, that is, a total of two gigabytes can be used.

One way of increasing memory available to SPDK is by increasing the number of 2048Kb hugepages. E.g. increase from two to eight gigabytes by increasing nr_hugespages to 4096:

echo "4096" > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

After doing this, then inspecting the configuration should output:

/sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages:4096
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages:4096
/sys/devices/system/node/node0/hugepages/hugepages-2048kB/surplus_hugepages:0