Poor SLOG performance after joyent_20170831T155808Z

[aaltonenp]

I'm observing poor SLOG write performance after joyent_20170831T155808Z, with both KVM VM and NFS.

Here are tests done with builds from 20170831T155808Z, the next one after that and the latest as of writing this. Something seem to be slowing down this hardware with builds since 20170913T233706Z.

Configuration:

[root@smartostest ~]# sysinfo
{
  "Live Image": "20170831T155808Z",
  "System Type": "SunOS",
  "Boot Time": "1592684536",
  "SDC Version": "7.0",
  "Manufacturer": "Supermicro",
  "Product": "X8DTU",
  "Serial Number": "1234567890",
  "SKU Number": "To Be Filled By O.E.M.",
  "HW Version": "1234567890",
  "HW Family": "Server",
  "Setup": "false",
  "VM Capable": true,
  "CPU Type": "Intel(R) Xeon(R) CPU E5606 @ 2.13GHz",
  "CPU Virtualization": "vmx",
  "CPU Physical Cores": 2,
  "UUID": "534d4349-0002-6190-2500-619025006a6b",
  "Hostname": "smartostest",
  "CPU Total Cores": 4,
  "MiB of Memory": "12279",
  "Zpool": "zones",
  "Zpool Disks": "c2t4d0,c2t5d0",
  "Zpool Profile": "striped",
  "Zpool Creation": 1586432120,
  "Zpool Size in GiB": 449,
  "Disks": {
    "c2t4d0": {"Size in GB": 128},
    "c2t5d0": {"Size in GB": 500}
  },
  "Boot Parameters": {
    "console": "vga",
    "vga_mode": "115200,8,n,1,-",
    "root_shadow": "$5$2HOHRnK3$NvLlm.1KQBbB0WjoP7xcIwGnllhzp2HnT.mDO7DpxYA",
    "smartos": "true",
    "boot_args": "",
    "bootargs": ""
  },
  "Network Interfaces": {
    "igb0": {"MAC Address": "00:25:90:61:6b:6a", "ip4addr": "10.0.0.12", "Link Status": "up", "NIC Names": ["admin"]},
    "igb1": {"MAC Address": "00:25:90:61:6b:6b", "ip4addr": "", "Link Status": "down", "NIC Names": []}
  },
  "Virtual Network Interfaces": {
  },
  "Link Aggregations": {
  }
}

[root@smartostest ~]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          c2t5d0    ONLINE       0     0     0
        logs
          c2t4d0    ONLINE       0     0     0

Performance tests:

joyent_20170831T155808Z:

Windows KVM performance:

[Write]
Sequential 1MiB (Q=  8, T= 1):   149.913 MB/s [    143.0 IOPS] < 55386.42 us>
Sequential 1MiB (Q=  1, T= 1):    64.597 MB/s [     61.6 IOPS] < 16142.39 us>
    Random 4KiB (Q= 32, T=16):    26.432 MB/s [   6453.1 IOPS] < 77200.15 us>
    Random 4KiB (Q=  1, T= 1):     0.916 MB/s [    223.6 IOPS] <  4442.17 us>

NFSv3 performance (from Windows VM on another host over 1GbE):

[Write]
Sequential 1MiB (Q=  8, T= 1):   100.451 MB/s [     95.8 IOPS] < 82710.29 us>
Sequential 1MiB (Q=  1, T= 1):    45.088 MB/s [     43.0 IOPS] < 23189.28 us>
    Random 4KiB (Q= 32, T=16):     9.366 MB/s [   2286.6 IOPS] <194694.71 us>
    Random 4KiB (Q=  1, T= 1):     0.815 MB/s [    199.0 IOPS] <  5018.86 us>

joyent_20170913T233706Z:

Windows KVM performance:

[Write]
Sequential 1MiB (Q=  8, T= 1):     9.018 MB/s [      8.6 IOPS] <887448.27 us>
Sequential 1MiB (Q=  1, T= 1):     9.856 MB/s [      9.4 IOPS] <105942.94 us>
    Random 4KiB (Q= 32, T=16):     1.852 MB/s [    452.1 IOPS] <528647.35 us>
    Random 4KiB (Q=  1, T= 1):     0.866 MB/s [    211.4 IOPS] <  4701.18 us>

NFSv3 performance:

[Write]
Sequential 1MiB (Q=  8, T= 1):     8.598 MB/s [      8.2 IOPS] <883684.03 us>
Sequential 1MiB (Q=  1, T= 1):     9.018 MB/s [      8.6 IOPS] <114214.35 us>
    Random 4KiB (Q= 32, T=16):     3.160 MB/s [    771.5 IOPS] <341843.43 us>
    Random 4KiB (Q=  1, T= 1):     0.782 MB/s [    190.9 IOPS] <  5222.82 us>

joyent_20200617T233107Z:

Windows KVM performance:

[Write]
Sequential 1MiB (Q=  8, T= 1):    19.922 MB/s [     19.0 IOPS] <400486.87 us>
Sequential 1MiB (Q=  1, T= 1):    19.084 MB/s [     18.2 IOPS] < 54444.21 us>
    Random 4KiB (Q= 32, T=16):     1.717 MB/s [    419.2 IOPS] <511276.32 us>
    Random 4KiB (Q=  1, T= 1):     0.856 MB/s [    209.0 IOPS] <  4740.78 us>

NFSv3 performance:

[Write]
Sequential 1MiB (Q=  8, T= 1):    59.345 MB/s [     56.6 IOPS] <138754.11 us>
Sequential 1MiB (Q=  1, T= 1):    45.714 MB/s [     43.6 IOPS] < 22910.62 us>
    Random 4KiB (Q= 32, T=16):     1.877 MB/s [    458.3 IOPS] <443586.94 us>
    Random 4KiB (Q=  1, T= 1):     0.773 MB/s [    188.7 IOPS] <  5284.06 us>

NFSv3 performance, sync=disabled:

[Write]
Sequential 1MiB (Q=  8, T= 1):   107.567 MB/s [    102.6 IOPS] < 77117.50 us>
Sequential 1MiB (Q=  1, T= 1):    87.457 MB/s [     83.4 IOPS] < 11969.17 us>
    Random 4KiB (Q= 32, T=16):     8.537 MB/s [   2084.2 IOPS] <207805.83 us>
    Random 4KiB (Q=  1, T= 1):     4.353 MB/s [   1062.7 IOPS] <   938.37 us> 

[papertigers]

@jasonbking / @jjelinek / @KodyKantor any of you have any pointers on this one?