extremely poor i40e rx performance under certain condition

[youzhongyang]

We first noticed this issue when msys2 cp took a few minutes to copy a 2.5GB file from Windows host to a SmartOS server (running Samba), but the Windows 'copy' can finish it in a few seconds. Packet capturing showed that msys2 cp does SMB2 WRITE requests of size 128k, but 'copy' does SMB2 WRITE requests of size 1M; in the 'cp' case, there are frequent packet re-transmissions which causes the poor performance.

We are able to reproduce this issue on a variety of servers running SmartOS, even with 20190912 image downloaded from smartos website. However, on servers equipped with ixgbe cards, either cp or 'copy' works well ('copy' performs slightly better than cp, given its larger request size).

We've investigated for a few weeks but were unable to figure out the root cause.

Here is another way to reproduce the issue using dd:

dd against a server with i40e cards:

root@yyang-deb9-64:/devel/Btnas_stage8/scratch# dd if=/dev/zero of=zero bs=1M count=2000 oflag=sync
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 151.127 s, 13.9 MB/s

root@yyang-deb9-64:/devel/Btnas_stage8/scratch# stat zero
  File: zero
  Size: 2097152000      Blocks: 1          IO Block: 1048576 regular file
Device: 33h/51d Inode: 3276928     Links: 1
Access: (0644/-rw-r--r--)  Uid: (4294967294/ UNKNOWN)   Gid: (4294967294/ UNKNOWN)
Access: 2019-10-15 15:50:57.770090694 -0400
Modify: 2019-10-15 15:53:28.896779126 -0400
Change: 2019-10-15 15:53:28.896779126 -0400
 Birth: -

dd against a server with ixgbe cards:

root@yyang-deb9-64:/devel/Btnas1_c/scratch# dd if=/dev/zero of=zero bs=1M count=2000 oflag=sync
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 6.52858 s, 321 MB/s

root@yyang-deb9-64:/devel/Btnas1_c/scratch# stat zero
  File: zero
  Size: 2097152000      Blocks: 1          IO Block: 1048576 regular file
Device: 29h/41d Inode: 613696      Links: 1
Access: (0644/-rw-r--r--)  Uid: (4294967294/ UNKNOWN)   Gid: (4294967294/ UNKNOWN)
Access: 2019-10-15 15:52:34.984938355 -0400
Modify: 2019-10-15 15:52:41.513543159 -0400
Change: 2019-10-15 15:52:41.513543159 -0400
 Birth: -