Vectorized IO to NAND media

With the basics of obtaining device information and constructing addresses in place, one can dive into the task of creating commands for doing vectorized IO.

As the section on background information describes, then there are a handful of constraints to handle for IOs to NAND media to succeed.

Erase before write

The first constraint to handle is that a chunk must be erased before it can be written. We do so by constructing a chunk-address:

nvm_addr s20_to_gen /dev/nvme0n1 0 0 20 0

Yielding:

naddrs: 1
addrs:
  - {val: 0x0000001400000000, pugrp: 00, punit: 00, chunk: 0020, sectr: 0000}

With the address we can now construct the erase command:

nvm_cmd erase /dev/nvme0n1 0x0000001400000000

On success, yielding:

# nvm_cmd_erase: {pmode: SNGL}
naddrs: 1
addrs:
  - {val: 0x0000001400000000, pugrp: 00, punit: 00, chunk: 0020, sectr: 0000}

On error, yielding:

# nvm_cmd_erase: {pmode: SNGL}
naddrs: 1
addrs:
  - {val: 0x0000001400000000, pugrp: 00, punit: 00, chunk: 0020, sectr: 0000}
nvm_addr_erase: Input/output error
nvm_ret { result(0xff), status(3) }

If an erase fails, it is most likely due to bad media. This is where the information retrieved with nvm_cmd rprt is useful, that is, to avoid using known bad chunks.

Note

C API for performing erases using chunk-adressing is done with nvm_cmd_erase

Write

The two primary constraints for issuing writes are that they must be contiguous within a chunk and span ws_min sectors.

Note

C API for performing write using vectorized IO with addressing at sector-level is done using nvm_cmd_write, note that the payload must be aligned to sector size, the helper function nvm_buf_alloc is provided for convenience

Minimum Write

For the geometry in figure X, the ws_min dictates that 24 sectors must be written of each 4096 bytes, a command satisfying the minimum-write constraint thus contains 24 addresses in sequential with a payload of 96Kbytes.

The command can be constructed via the CLI as:

nvm_cmd write /dev/nvme0n1 \
0x0000001400000000 0x0000001400000001 0x0000001400000002 0x0000001400000003 \
0x0000001400000004 0x0000001400000005 0x0000001400000006 0x0000001400000007 \
0x0000001400000008 0x0000001400000009 0x000000140000000a 0x000000140000000b \
0x000000140000000c 0x000000140000000d 0x000000140000000e 0x000000140000000f \
0x0000001400000010 0x0000001400000011 0x0000001400000012 0x0000001400000013 \
0x0000001400000014 0x0000001400000015 0x0000001400000016 0x0000001400000017

The CLI creates an arbitrary payload, so we do not concern us with the payload at this point. The result of the command is:

# nvm_cmd_write: {pmode: SNGL}
naddrs: 24
addrs:
  - {val: 0x0000001400000000, pugrp: 00, punit: 00, chunk: 0020, sectr: 0000}
  - {val: 0x0000001400000001, pugrp: 00, punit: 00, chunk: 0020, sectr: 0001}
  - {val: 0x0000001400000002, pugrp: 00, punit: 00, chunk: 0020, sectr: 0002}
  - {val: 0x0000001400000003, pugrp: 00, punit: 00, chunk: 0020, sectr: 0003}
  - {val: 0x0000001400000004, pugrp: 00, punit: 00, chunk: 0020, sectr: 0004}
  - {val: 0x0000001400000005, pugrp: 00, punit: 00, chunk: 0020, sectr: 0005}
  - {val: 0x0000001400000006, pugrp: 00, punit: 00, chunk: 0020, sectr: 0006}
  - {val: 0x0000001400000007, pugrp: 00, punit: 00, chunk: 0020, sectr: 0007}
  - {val: 0x0000001400000008, pugrp: 00, punit: 00, chunk: 0020, sectr: 0008}
  - {val: 0x0000001400000009, pugrp: 00, punit: 00, chunk: 0020, sectr: 0009}
  - {val: 0x000000140000000a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0010}
  - {val: 0x000000140000000b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0011}
  - {val: 0x000000140000000c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0012}
  - {val: 0x000000140000000d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0013}
  - {val: 0x000000140000000e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0014}
  - {val: 0x000000140000000f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0015}
  - {val: 0x0000001400000010, pugrp: 00, punit: 00, chunk: 0020, sectr: 0016}
  - {val: 0x0000001400000011, pugrp: 00, punit: 00, chunk: 0020, sectr: 0017}
  - {val: 0x0000001400000012, pugrp: 00, punit: 00, chunk: 0020, sectr: 0018}
  - {val: 0x0000001400000013, pugrp: 00, punit: 00, chunk: 0020, sectr: 0019}
  - {val: 0x0000001400000014, pugrp: 00, punit: 00, chunk: 0020, sectr: 0020}
  - {val: 0x0000001400000015, pugrp: 00, punit: 00, chunk: 0020, sectr: 0021}
  - {val: 0x0000001400000016, pugrp: 00, punit: 00, chunk: 0020, sectr: 0022}
  - {val: 0x0000001400000017, pugrp: 00, punit: 00, chunk: 0020, sectr: 0023}

Optimal Write

For this specific device then ws_min is the same as ws_min. However, an improvement of round-trip-time can be obtained by increasing the amount of work done by a single command, that is, increase the number of addresses to a multiple of ws_opt and less than or equal to write-naddrs-max. In our case we can construct a command with 48 addresses.

Abiding to all of the above mentioned constraints a write command can be constructed as:

nvm_cmd write /dev/nvme0n1 \
0x0000001400000018 0x0000001400000019 0x000000140000001a 0x000000140000001b \
0x000000140000001c 0x000000140000001d 0x000000140000001e 0x000000140000001f \
0x0000001400000020 0x0000001400000021 0x0000001400000022 0x0000001400000023 \
0x0000001400000024 0x0000001400000025 0x0000001400000026 0x0000001400000027 \
0x0000001400000028 0x0000001400000029 0x000000140000002a 0x000000140000002b \
0x000000140000002c 0x000000140000002d 0x000000140000002e 0x000000140000002f \
0x0000001400000030 0x0000001400000031 0x0000001400000032 0x0000001400000033 \
0x0000001400000034 0x0000001400000035 0x0000001400000036 0x0000001400000037 \
0x0000001400000038 0x0000001400000039 0x000000140000003a 0x000000140000003b \
0x000000140000003c 0x000000140000003d 0x000000140000003e 0x000000140000003f \
0x0000001400000040 0x0000001400000041 0x0000001400000042 0x0000001400000043 \
0x0000001400000044 0x0000001400000045 0x0000001400000046 0x0000001400000047

Successfully yielding:

# nvm_cmd_write: {pmode: SNGL}
naddrs: 48
addrs:
  - {val: 0x0000001400000018, pugrp: 00, punit: 00, chunk: 0020, sectr: 0024}
  - {val: 0x0000001400000019, pugrp: 00, punit: 00, chunk: 0020, sectr: 0025}
  - {val: 0x000000140000001a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0026}
  - {val: 0x000000140000001b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0027}
  - {val: 0x000000140000001c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0028}
  - {val: 0x000000140000001d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0029}
  - {val: 0x000000140000001e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0030}
  - {val: 0x000000140000001f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0031}
  - {val: 0x0000001400000020, pugrp: 00, punit: 00, chunk: 0020, sectr: 0032}
  - {val: 0x0000001400000021, pugrp: 00, punit: 00, chunk: 0020, sectr: 0033}
  - {val: 0x0000001400000022, pugrp: 00, punit: 00, chunk: 0020, sectr: 0034}
  - {val: 0x0000001400000023, pugrp: 00, punit: 00, chunk: 0020, sectr: 0035}
  - {val: 0x0000001400000024, pugrp: 00, punit: 00, chunk: 0020, sectr: 0036}
  - {val: 0x0000001400000025, pugrp: 00, punit: 00, chunk: 0020, sectr: 0037}
  - {val: 0x0000001400000026, pugrp: 00, punit: 00, chunk: 0020, sectr: 0038}
  - {val: 0x0000001400000027, pugrp: 00, punit: 00, chunk: 0020, sectr: 0039}
  - {val: 0x0000001400000028, pugrp: 00, punit: 00, chunk: 0020, sectr: 0040}
  - {val: 0x0000001400000029, pugrp: 00, punit: 00, chunk: 0020, sectr: 0041}
  - {val: 0x000000140000002a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0042}
  - {val: 0x000000140000002b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0043}
  - {val: 0x000000140000002c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0044}
  - {val: 0x000000140000002d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0045}
  - {val: 0x000000140000002e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0046}
  - {val: 0x000000140000002f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0047}
  - {val: 0x0000001400000030, pugrp: 00, punit: 00, chunk: 0020, sectr: 0048}
  - {val: 0x0000001400000031, pugrp: 00, punit: 00, chunk: 0020, sectr: 0049}
  - {val: 0x0000001400000032, pugrp: 00, punit: 00, chunk: 0020, sectr: 0050}
  - {val: 0x0000001400000033, pugrp: 00, punit: 00, chunk: 0020, sectr: 0051}
  - {val: 0x0000001400000034, pugrp: 00, punit: 00, chunk: 0020, sectr: 0052}
  - {val: 0x0000001400000035, pugrp: 00, punit: 00, chunk: 0020, sectr: 0053}
  - {val: 0x0000001400000036, pugrp: 00, punit: 00, chunk: 0020, sectr: 0054}
  - {val: 0x0000001400000037, pugrp: 00, punit: 00, chunk: 0020, sectr: 0055}
  - {val: 0x0000001400000038, pugrp: 00, punit: 00, chunk: 0020, sectr: 0056}
  - {val: 0x0000001400000039, pugrp: 00, punit: 00, chunk: 0020, sectr: 0057}
  - {val: 0x000000140000003a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0058}
  - {val: 0x000000140000003b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0059}
  - {val: 0x000000140000003c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0060}
  - {val: 0x000000140000003d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0061}
  - {val: 0x000000140000003e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0062}
  - {val: 0x000000140000003f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0063}
  - {val: 0x0000001400000040, pugrp: 00, punit: 00, chunk: 0020, sectr: 0064}
  - {val: 0x0000001400000041, pugrp: 00, punit: 00, chunk: 0020, sectr: 0065}
  - {val: 0x0000001400000042, pugrp: 00, punit: 00, chunk: 0020, sectr: 0066}
  - {val: 0x0000001400000043, pugrp: 00, punit: 00, chunk: 0020, sectr: 0067}
  - {val: 0x0000001400000044, pugrp: 00, punit: 00, chunk: 0020, sectr: 0068}
  - {val: 0x0000001400000045, pugrp: 00, punit: 00, chunk: 0020, sectr: 0069}
  - {val: 0x0000001400000046, pugrp: 00, punit: 00, chunk: 0020, sectr: 0070}
  - {val: 0x0000001400000047, pugrp: 00, punit: 00, chunk: 0020, sectr: 0071}

Using vectorized IO we have with a single command successfully written a payload of 48 x 4096 bytes = 192 KB.

Read

Reads have fewer constraints than writes. The granularity of a read is a single sector (the smallest addressable unit) and can be performed non-contiguously.

The primary constraint for a read to adhere to is that the block which is read from must be closed. That is, all pages within the block must have been written.

It might be that the constraint can be relaxed, that is, instead of writing all sectors in the chunk, then only mw_cunits sectors ahead of the read must be written.

We have so far written a total of 72 sectors, the first 24 sectors in one command, the remaining 48 sectors in a second command. Thus additional 6072 sectors must be written before the chunk changes state to closed.

Specifying all those addresses via the CLI is tedious, we will therefore take a sneak peak at virtual blocks and execute:

nvm_vblk erase /dev/nvme0n1 0x0402008e00000000
nvm_vblk write /dev/nvme0n1 0x0402008e00000000
vblk:
  dev: {pmode: 'SNGL'}
  nblks: 1
  nmbytes: 24
  pos_write: 0
  pos_read: 0
  flags: 0x08c8
naddrs: 1
addrs:
  - {val: 0x0402008e00000000, pugrp: 04, punit: 02, chunk: 0142, sectr: 0000}
nvm_vblk_erase: {elapsed: 0.0050, mb: 24.00, mbsec: 4759.59}

vblk:
  dev: {pmode: 'SNGL'}
  nblks: 1
  nmbytes: 24
  pos_write: 0
  pos_read: 0
  flags: 0x08c8
naddrs: 1
addrs:
  - {val: 0x0402008e00000000, pugrp: 04, punit: 02, chunk: 0142, sectr: 0000}
nvm_buf_alloc: {elapsed: 0.000004}
nvm_buf_fill: {elapsed: 0.013511}
nvm_vblk_write: {elapsed: 0.8329, mb: 24.00, mbsec: 28.81}

What these two commands actually do will be described in the following section on virtual blocks. For now all we need to know is that the chunk is now fully written / closed and we can start reading from it without concern about mw_cunits.

Note

C API for performing write using vectorized IO with addressing at sector-level is done using nvm_cmd_read, the received payload must be stored in a sector-aligned buffer, the helper function nvm_buf_alloc is provided for convenience

Minimal Read

We can read a single sector:

nvm_cmd read /dev/nvme0n1 0x0402008e00000000
# nvm_cmd_read: {pmode: SNGL}
naddrs: 1
addrs:
  - {val: 0x0402008e00000000, pugrp: 04, punit: 02, chunk: 0142, sectr: 0000}

The data read from device can be written to a file system using the -o FILE option:

nvm_cmd read /dev/nvme0n1 0x0402008e00000000 -o /tmp/dump.bin
# nvm_cmd_read: {pmode: SNGL}
naddrs: 1
addrs:
  - {val: 0x0402008e00000000, pugrp: 04, punit: 02, chunk: 0142, sectr: 0000}

The payload is then available by inspection, e.g. with hexdump:

hexdump /tmp/dump.bin -C -n 128
00000000  41 42 43 44 45 46 47 48  49 4a 4b 4c 4d 4e 4f 50  |ABCDEFGHIJKLMNOP|
00000010  51 52 53 54 55 56 57 58  59 5a 41 42 43 44 45 46  |QRSTUVWXYZABCDEF|
00000020  47 48 49 4a 4b 4c 4d 4e  4f 50 51 52 53 54 55 56  |GHIJKLMNOPQRSTUV|
00000030  57 58 59 5a 41 42 43 44  45 46 47 48 49 4a 4b 4c  |WXYZABCDEFGHIJKL|
00000040  4d 4e 4f 50 51 52 53 54  55 56 57 58 59 5a 41 42  |MNOPQRSTUVWXYZAB|
00000050  43 44 45 46 47 48 49 4a  4b 4c 4d 4e 4f 50 51 52  |CDEFGHIJKLMNOPQR|
00000060  53 54 55 56 57 58 59 5a  41 42 43 44 45 46 47 48  |STUVWXYZABCDEFGH|
00000070  49 4a 4b 4c 4d 4e 4f 50  51 52 53 54 55 56 57 58  |IJKLMNOPQRSTUVWX|
00000080

Maximum Read

Same as a write, a read has an upper bound on the number of addresses in a single command. However, since there is no ws_min to align to, then the read can utilize all NVM_NADDR_MAX, that is 64 whereas the write was confined to the even multiple less than or equal to NVM_NADDRS_MAX, that was, 48.

nvm_cmd read /dev/nvme0n1 \
0x0000001400000000 0x0000001400000001 0x0000001400000002 0x0000001400000003 \
0x0000001400000004 0x0000001400000005 0x0000001400000006 0x0000001400000007 \
0x0000001400000008 0x0000001400000009 0x000000140000000a 0x000000140000000b \
0x000000140000000c 0x000000140000000d 0x000000140000000e 0x000000140000000f \
0x0000001400000010 0x0000001400000011 0x0000001400000012 0x0000001400000013 \
0x0000001400000014 0x0000001400000015 0x0000001400000016 0x0000001400000017 \
0x0000001400000018 0x0000001400000019 0x000000140000001a 0x000000140000001b \
0x000000140000001c 0x000000140000001d 0x000000140000001e 0x000000140000001f \
0x0000001400000020 0x0000001400000021 0x0000001400000022 0x0000001400000023 \
0x0000001400000024 0x0000001400000025 0x0000001400000026 0x0000001400000027 \
0x0000001400000028 0x0000001400000029 0x000000140000002a 0x000000140000002b \
0x000000140000002c 0x000000140000002d 0x000000140000002e 0x000000140000002f \
0x0000001400000030 0x0000001400000031 0x0000001400000032 0x0000001400000033 \
0x0000001400000034 0x0000001400000035 0x0000001400000036 0x0000001400000037 \
0x0000001400000038 0x0000001400000039 0x000000140000003a 0x000000140000003b \
0x000000140000003c 0x000000140000003d 0x000000140000003e 0x000000140000003f

Successfully yielding:

# nvm_cmd_read: {pmode: SNGL}
naddrs: 64
addrs:
  - {val: 0x0000001400000000, pugrp: 00, punit: 00, chunk: 0020, sectr: 0000}
  - {val: 0x0000001400000001, pugrp: 00, punit: 00, chunk: 0020, sectr: 0001}
  - {val: 0x0000001400000002, pugrp: 00, punit: 00, chunk: 0020, sectr: 0002}
  - {val: 0x0000001400000003, pugrp: 00, punit: 00, chunk: 0020, sectr: 0003}
  - {val: 0x0000001400000004, pugrp: 00, punit: 00, chunk: 0020, sectr: 0004}
  - {val: 0x0000001400000005, pugrp: 00, punit: 00, chunk: 0020, sectr: 0005}
  - {val: 0x0000001400000006, pugrp: 00, punit: 00, chunk: 0020, sectr: 0006}
  - {val: 0x0000001400000007, pugrp: 00, punit: 00, chunk: 0020, sectr: 0007}
  - {val: 0x0000001400000008, pugrp: 00, punit: 00, chunk: 0020, sectr: 0008}
  - {val: 0x0000001400000009, pugrp: 00, punit: 00, chunk: 0020, sectr: 0009}
  - {val: 0x000000140000000a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0010}
  - {val: 0x000000140000000b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0011}
  - {val: 0x000000140000000c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0012}
  - {val: 0x000000140000000d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0013}
  - {val: 0x000000140000000e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0014}
  - {val: 0x000000140000000f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0015}
  - {val: 0x0000001400000010, pugrp: 00, punit: 00, chunk: 0020, sectr: 0016}
  - {val: 0x0000001400000011, pugrp: 00, punit: 00, chunk: 0020, sectr: 0017}
  - {val: 0x0000001400000012, pugrp: 00, punit: 00, chunk: 0020, sectr: 0018}
  - {val: 0x0000001400000013, pugrp: 00, punit: 00, chunk: 0020, sectr: 0019}
  - {val: 0x0000001400000014, pugrp: 00, punit: 00, chunk: 0020, sectr: 0020}
  - {val: 0x0000001400000015, pugrp: 00, punit: 00, chunk: 0020, sectr: 0021}
  - {val: 0x0000001400000016, pugrp: 00, punit: 00, chunk: 0020, sectr: 0022}
  - {val: 0x0000001400000017, pugrp: 00, punit: 00, chunk: 0020, sectr: 0023}
  - {val: 0x0000001400000018, pugrp: 00, punit: 00, chunk: 0020, sectr: 0024}
  - {val: 0x0000001400000019, pugrp: 00, punit: 00, chunk: 0020, sectr: 0025}
  - {val: 0x000000140000001a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0026}
  - {val: 0x000000140000001b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0027}
  - {val: 0x000000140000001c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0028}
  - {val: 0x000000140000001d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0029}
  - {val: 0x000000140000001e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0030}
  - {val: 0x000000140000001f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0031}
  - {val: 0x0000001400000020, pugrp: 00, punit: 00, chunk: 0020, sectr: 0032}
  - {val: 0x0000001400000021, pugrp: 00, punit: 00, chunk: 0020, sectr: 0033}
  - {val: 0x0000001400000022, pugrp: 00, punit: 00, chunk: 0020, sectr: 0034}
  - {val: 0x0000001400000023, pugrp: 00, punit: 00, chunk: 0020, sectr: 0035}
  - {val: 0x0000001400000024, pugrp: 00, punit: 00, chunk: 0020, sectr: 0036}
  - {val: 0x0000001400000025, pugrp: 00, punit: 00, chunk: 0020, sectr: 0037}
  - {val: 0x0000001400000026, pugrp: 00, punit: 00, chunk: 0020, sectr: 0038}
  - {val: 0x0000001400000027, pugrp: 00, punit: 00, chunk: 0020, sectr: 0039}
  - {val: 0x0000001400000028, pugrp: 00, punit: 00, chunk: 0020, sectr: 0040}
  - {val: 0x0000001400000029, pugrp: 00, punit: 00, chunk: 0020, sectr: 0041}
  - {val: 0x000000140000002a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0042}
  - {val: 0x000000140000002b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0043}
  - {val: 0x000000140000002c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0044}
  - {val: 0x000000140000002d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0045}
  - {val: 0x000000140000002e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0046}
  - {val: 0x000000140000002f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0047}
  - {val: 0x0000001400000030, pugrp: 00, punit: 00, chunk: 0020, sectr: 0048}
  - {val: 0x0000001400000031, pugrp: 00, punit: 00, chunk: 0020, sectr: 0049}
  - {val: 0x0000001400000032, pugrp: 00, punit: 00, chunk: 0020, sectr: 0050}
  - {val: 0x0000001400000033, pugrp: 00, punit: 00, chunk: 0020, sectr: 0051}
  - {val: 0x0000001400000034, pugrp: 00, punit: 00, chunk: 0020, sectr: 0052}
  - {val: 0x0000001400000035, pugrp: 00, punit: 00, chunk: 0020, sectr: 0053}
  - {val: 0x0000001400000036, pugrp: 00, punit: 00, chunk: 0020, sectr: 0054}
  - {val: 0x0000001400000037, pugrp: 00, punit: 00, chunk: 0020, sectr: 0055}
  - {val: 0x0000001400000038, pugrp: 00, punit: 00, chunk: 0020, sectr: 0056}
  - {val: 0x0000001400000039, pugrp: 00, punit: 00, chunk: 0020, sectr: 0057}
  - {val: 0x000000140000003a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0058}
  - {val: 0x000000140000003b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0059}
  - {val: 0x000000140000003c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0060}
  - {val: 0x000000140000003d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0061}
  - {val: 0x000000140000003e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0062}
  - {val: 0x000000140000003f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0063}

Non-Contiguous Read

Reading 64 different sectors in arbitrary order:

nvm_cmd read /dev/nvme0n1 \
0x000000140000001a 0x000000140000001d 0x000000140000001f 0x0000001400000009 \
0x000000140000002e 0x0000001400000000 0x000000140000000b 0x000000140000000d \
0x000000140000002f 0x0000001400000001 0x000000140000003c 0x000000140000003b \
0x000000140000003e 0x0000001400000036 0x0000001400000030 0x000000140000001e \
0x000000140000003d 0x0000001400000026 0x0000001400000037 0x0000001400000024 \
0x0000001400000035 0x000000140000000e 0x0000001400000008 0x0000001400000004 \
0x000000140000001c 0x0000001400000023 0x0000001400000012 0x0000001400000006 \
0x0000001400000018 0x0000001400000013 0x0000001400000005 0x0000001400000031 \
0x0000001400000027 0x000000140000002c 0x0000001400000017 0x0000001400000010 \
0x0000001400000039 0x0000001400000028 0x0000001400000007 0x0000001400000015 \
0x000000140000000a 0x0000001400000032 0x000000140000000f 0x0000001400000002 \
0x000000140000003f 0x000000140000002b 0x0000001400000033 0x0000001400000020 \
0x0000001400000038 0x000000140000002d 0x000000140000001b 0x0000001400000003 \
0x0000001400000025 0x000000140000000c 0x000000140000003a 0x0000001400000022 \
0x0000001400000021 0x0000001400000019 0x0000001400000014 0x0000001400000029 \
0x0000001400000034 0x0000001400000011 0x0000001400000016 0x000000140000002a

Successfully yielding:

# nvm_cmd_read: {pmode: SNGL}
naddrs: 64
addrs:
  - {val: 0x000000140000001a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0026}
  - {val: 0x000000140000001d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0029}
  - {val: 0x000000140000001f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0031}
  - {val: 0x0000001400000009, pugrp: 00, punit: 00, chunk: 0020, sectr: 0009}
  - {val: 0x000000140000002e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0046}
  - {val: 0x0000001400000000, pugrp: 00, punit: 00, chunk: 0020, sectr: 0000}
  - {val: 0x000000140000000b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0011}
  - {val: 0x000000140000000d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0013}
  - {val: 0x000000140000002f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0047}
  - {val: 0x0000001400000001, pugrp: 00, punit: 00, chunk: 0020, sectr: 0001}
  - {val: 0x000000140000003c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0060}
  - {val: 0x000000140000003b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0059}
  - {val: 0x000000140000003e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0062}
  - {val: 0x0000001400000036, pugrp: 00, punit: 00, chunk: 0020, sectr: 0054}
  - {val: 0x0000001400000030, pugrp: 00, punit: 00, chunk: 0020, sectr: 0048}
  - {val: 0x000000140000001e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0030}
  - {val: 0x000000140000003d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0061}
  - {val: 0x0000001400000026, pugrp: 00, punit: 00, chunk: 0020, sectr: 0038}
  - {val: 0x0000001400000037, pugrp: 00, punit: 00, chunk: 0020, sectr: 0055}
  - {val: 0x0000001400000024, pugrp: 00, punit: 00, chunk: 0020, sectr: 0036}
  - {val: 0x0000001400000035, pugrp: 00, punit: 00, chunk: 0020, sectr: 0053}
  - {val: 0x000000140000000e, pugrp: 00, punit: 00, chunk: 0020, sectr: 0014}
  - {val: 0x0000001400000008, pugrp: 00, punit: 00, chunk: 0020, sectr: 0008}
  - {val: 0x0000001400000004, pugrp: 00, punit: 00, chunk: 0020, sectr: 0004}
  - {val: 0x000000140000001c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0028}
  - {val: 0x0000001400000023, pugrp: 00, punit: 00, chunk: 0020, sectr: 0035}
  - {val: 0x0000001400000012, pugrp: 00, punit: 00, chunk: 0020, sectr: 0018}
  - {val: 0x0000001400000006, pugrp: 00, punit: 00, chunk: 0020, sectr: 0006}
  - {val: 0x0000001400000018, pugrp: 00, punit: 00, chunk: 0020, sectr: 0024}
  - {val: 0x0000001400000013, pugrp: 00, punit: 00, chunk: 0020, sectr: 0019}
  - {val: 0x0000001400000005, pugrp: 00, punit: 00, chunk: 0020, sectr: 0005}
  - {val: 0x0000001400000031, pugrp: 00, punit: 00, chunk: 0020, sectr: 0049}
  - {val: 0x0000001400000027, pugrp: 00, punit: 00, chunk: 0020, sectr: 0039}
  - {val: 0x000000140000002c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0044}
  - {val: 0x0000001400000017, pugrp: 00, punit: 00, chunk: 0020, sectr: 0023}
  - {val: 0x0000001400000010, pugrp: 00, punit: 00, chunk: 0020, sectr: 0016}
  - {val: 0x0000001400000039, pugrp: 00, punit: 00, chunk: 0020, sectr: 0057}
  - {val: 0x0000001400000028, pugrp: 00, punit: 00, chunk: 0020, sectr: 0040}
  - {val: 0x0000001400000007, pugrp: 00, punit: 00, chunk: 0020, sectr: 0007}
  - {val: 0x0000001400000015, pugrp: 00, punit: 00, chunk: 0020, sectr: 0021}
  - {val: 0x000000140000000a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0010}
  - {val: 0x0000001400000032, pugrp: 00, punit: 00, chunk: 0020, sectr: 0050}
  - {val: 0x000000140000000f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0015}
  - {val: 0x0000001400000002, pugrp: 00, punit: 00, chunk: 0020, sectr: 0002}
  - {val: 0x000000140000003f, pugrp: 00, punit: 00, chunk: 0020, sectr: 0063}
  - {val: 0x000000140000002b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0043}
  - {val: 0x0000001400000033, pugrp: 00, punit: 00, chunk: 0020, sectr: 0051}
  - {val: 0x0000001400000020, pugrp: 00, punit: 00, chunk: 0020, sectr: 0032}
  - {val: 0x0000001400000038, pugrp: 00, punit: 00, chunk: 0020, sectr: 0056}
  - {val: 0x000000140000002d, pugrp: 00, punit: 00, chunk: 0020, sectr: 0045}
  - {val: 0x000000140000001b, pugrp: 00, punit: 00, chunk: 0020, sectr: 0027}
  - {val: 0x0000001400000003, pugrp: 00, punit: 00, chunk: 0020, sectr: 0003}
  - {val: 0x0000001400000025, pugrp: 00, punit: 00, chunk: 0020, sectr: 0037}
  - {val: 0x000000140000000c, pugrp: 00, punit: 00, chunk: 0020, sectr: 0012}
  - {val: 0x000000140000003a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0058}
  - {val: 0x0000001400000022, pugrp: 00, punit: 00, chunk: 0020, sectr: 0034}
  - {val: 0x0000001400000021, pugrp: 00, punit: 00, chunk: 0020, sectr: 0033}
  - {val: 0x0000001400000019, pugrp: 00, punit: 00, chunk: 0020, sectr: 0025}
  - {val: 0x0000001400000014, pugrp: 00, punit: 00, chunk: 0020, sectr: 0020}
  - {val: 0x0000001400000029, pugrp: 00, punit: 00, chunk: 0020, sectr: 0041}
  - {val: 0x0000001400000034, pugrp: 00, punit: 00, chunk: 0020, sectr: 0052}
  - {val: 0x0000001400000011, pugrp: 00, punit: 00, chunk: 0020, sectr: 0017}
  - {val: 0x0000001400000016, pugrp: 00, punit: 00, chunk: 0020, sectr: 0022}
  - {val: 0x000000140000002a, pugrp: 00, punit: 00, chunk: 0020, sectr: 0042}

It is worth mentioning that vectorized reads can be non-contiguous not only within a chunk but also scattered across chunks, in different PUNIT, and PUGRP.