Move Linux CI tests from Travis to Cirrus #1070

asomers · 2019-05-26T23:08:09Z

No description provided.

asomers · 2019-05-27T04:37:00Z

@glebpom Do you have any idea with the AF_ALG tests are failing when run on Cirrus instead of Travis? It looks like bind is returning ENOENT.

glebpom · 2019-05-27T09:04:42Z

@asomers it probably means that there is no kernel support (or kernel modules are not loaded) of algorithms which we use in tests on the host machine. Is it possible to invoke cat /proc/crypto of the build VM?

asomers · 2019-05-27T13:11:56Z

@glebpom here's the contents of /proc/crypto. Do you see what you need?

cat /proc/crypto
name         : xts(aes)
driver       : xts-aes-aesni
module       : aesni_intel
priority     : 401
refcnt       : 1
selftest     : passed
internal     : no
type         : skcipher
async        : yes
blocksize    : 16
min keysize  : 32
max keysize  : 64
ivsize       : 16
chunksize    : 16
walksize     : 16
name         : ctr(aes)
driver       : ctr-aes-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : no
type         : skcipher
async        : yes
blocksize    : 1
min keysize  : 16
max keysize  : 32
ivsize       : 16
chunksize    : 16
walksize     : 16
name         : cbc(aes)
driver       : cbc-aes-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : no
type         : skcipher
async        : yes
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 16
chunksize    : 16
walksize     : 16
name         : ecb(aes)
driver       : ecb-aes-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : no
type         : skcipher
async        : yes
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 0
chunksize    : 16
walksize     : 16
name         : gcm(aes)
driver       : generic-gcm-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : no
type         : aead
async        : yes
blocksize    : 1
ivsize       : 12
maxauthsize  : 16
geniv        : <none>
name         : __generic-gcm-aes-aesni
driver       : __driver-generic-gcm-aes-aesni
module       : aesni_intel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : yes
type         : aead
async        : no
blocksize    : 1
ivsize       : 12
maxauthsize  : 16
geniv        : <none>
name         : rfc4106(gcm(aes))
driver       : rfc4106-gcm-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : no
type         : aead
async        : yes
blocksize    : 1
ivsize       : 8
maxauthsize  : 16
geniv        : <none>
name         : __gcm-aes-aesni
driver       : __driver-gcm-aes-aesni
module       : aesni_intel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : yes
type         : aead
async        : no
blocksize    : 1
ivsize       : 8
maxauthsize  : 16
geniv        : <none>
name         : __xts(aes)
driver       : __xts-aes-aesni
module       : aesni_intel
priority     : 401
refcnt       : 1
selftest     : passed
internal     : yes
type         : skcipher
async        : no
blocksize    : 16
min keysize  : 32
max keysize  : 64
ivsize       : 16
chunksize    : 16
walksize     : 16
name         : __ctr(aes)
driver       : __ctr-aes-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : yes
type         : skcipher
async        : no
blocksize    : 1
min keysize  : 16
max keysize  : 32
ivsize       : 16
chunksize    : 16
walksize     : 16
name         : __cbc(aes)
driver       : __cbc-aes-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : yes
type         : skcipher
async        : no
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 16
chunksize    : 16
walksize     : 16
name         : __ecb(aes)
driver       : __ecb-aes-aesni
module       : aesni_intel
priority     : 400
refcnt       : 1
selftest     : passed
internal     : yes
type         : skcipher
async        : no
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 0
chunksize    : 16
walksize     : 16
name         : __aes
driver       : __aes-aesni
module       : aesni_intel
priority     : 300
refcnt       : 1
selftest     : passed
internal     : yes
type         : cipher
blocksize    : 16
min keysize  : 16
max keysize  : 32
name         : aes
driver       : aes-aesni
module       : aesni_intel
priority     : 300
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 16
min keysize  : 16
max keysize  : 32
name         : aes
driver       : aes-asm
module       : aes_x86_64
priority     : 200
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 16
min keysize  : 16
max keysize  : 32
name         : crct10dif
driver       : crct10dif-generic
module       : kernel
priority     : 100
refcnt       : 2
selftest     : passed
internal     : no
type         : shash
blocksize    : 1
digestsize   : 2
name         : crc32c
driver       : crc32c-generic
module       : kernel
priority     : 100
refcnt       : 6
selftest     : passed
internal     : no
type         : shash
blocksize    : 1
digestsize   : 4
name         : ecb(arc4)
driver       : ecb(arc4)-generic
module       : kernel
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : blkcipher
blocksize    : 1
min keysize  : 1
max keysize  : 256
ivsize       : 0
geniv        : <default>
name         : arc4
driver       : arc4-generic
module       : kernel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 1
min keysize  : 1
max keysize  : 256
name         : aes
driver       : aes-generic
module       : kernel
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 16
min keysize  : 16
max keysize  : 32
name         : des3_ede
driver       : des3_ede-generic
module       : kernel
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 8
min keysize  : 24
max keysize  : 24
name         : des
driver       : des-generic
module       : kernel
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 8
min keysize  : 8
max keysize  : 8
name         : sha224
driver       : sha224-generic
module       : kernel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 28
name         : sha256
driver       : sha256-generic
module       : kernel
priority     : 0
refcnt       : 2
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 32
name         : sha1
driver       : sha1-generic
module       : kernel
priority     : 0
refcnt       : 4
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 20
name         : md5
driver       : md5-generic
module       : kernel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 64
digestsize   : 16
name         : digest_null
driver       : digest_null-generic
module       : kernel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : shash
blocksize    : 1
digestsize   : 0
name         : compress_null
driver       : compress_null-generic
module       : kernel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : compression
name         : ecb(cipher_null)
driver       : ecb-cipher_null
module       : kernel
priority     : 100
refcnt       : 1
selftest     : passed
internal     : no
type         : blkcipher
blocksize    : 1
min keysize  : 0
max keysize  : 0
ivsize       : 0
geniv        : <default>
name         : cipher_null
driver       : cipher_null-generic
module       : kernel
priority     : 0
refcnt       : 1
selftest     : passed
internal     : no
type         : cipher
blocksize    : 1
min keysize  : 0
max keysize  : 0

glebpom · 2019-05-27T13:42:28Z

From what I see, the algorithm is supported. I will take a deeper look later today

glebpom · 2019-05-28T20:13:30Z

@asomers I was able to reproduce the problem by creating the GKE cluster, which cirrus uses internally. Linux kernel returns ENOENT while trying to bind AF_ALG socket. Manual loading of all modules on the host machine didn't help. I have also checked the libkcapi (a reference AF_ALG user-space implementation) tests, and it also failed. Here is the strace output of libkcapi tests:

bind(3, {sa_family=AF_ALG, sa_data="skcipher\0\0\0\0\0\0\0\0\0\0\0\0\0\0cbc(aes)\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 88) = -1 ENOENT (No such file or directory)
write(2, "libkcapi - Error: AF_ALG: bind f"..., 50libkcapi - Error: AF_ALG: bind failed (errno: -2)

It seems like the problem is somewhere in GCE virtualization and Linux kernel.

asomers · 2019-05-28T20:27:48Z

Fabulous 😏 . The kernel documentation doesn't mention anything about ENOENT for AF_ALG sockets. I fear that this is one of those "Use the source" moments. Do you have any more ideas, or shall I just disable that test in CI?

BTW, on my personal Debian VM, test_af_alg_aead fails with EINVAL and test_af_alg_cipher passes.

glebpom · 2019-05-29T07:12:57Z

Here it is: https://github.com/torvalds/linux/blob/63bdf4284c38a48af21745ceb148a087b190cd21/crypto/af_alg.c#L44
It returns ENOENT if algorithm name and type combination cannot be found. In fact, it exists (according to /proc/crypto).

BTW, on my personal Debian VM, test_af_alg_aead fails with EINVAL and test_af_alg_cipher passes.

AEAD is supported staring from kernel version 3.19. If you have jessie, could it be 3.16?

Do you have any more ideas, or shall I just disable that test in CI?

Switching underlying VMs to another OS/kernel versions may work. Not sure if it's possible for for open-source projects. Contacting google cloud support may also help.

asomers · 2019-05-29T13:37:34Z

Here it is: https://github.com/torvalds/linux/blob/63bdf4284c38a48af21745ceb148a087b190cd21/crypto/af_alg.c#L44
It returns ENOENT if algorithm name and type combination cannot be found. In fact, it exists (according to /proc/crypto).

So the algorithm exists according to /proc/crypto, but for some reason it's not available to bind? My only guess is that this has something to do with the container's capabilities.

BTW, on my personal Debian VM, test_af_alg_aead fails with EINVAL and test_af_alg_cipher passes.

AEAD is supported staring from kernel version 3.19. If you have jessie, could it be 3.16?

Nope, I have Debian 9.6, kernel 4.9.0-8.

Do you have any more ideas, or shall I just disable that test in CI?

Switching underlying VMs to another OS/kernel versions may work. Not sure if it's possible for for open-source projects. Contacting google cloud support may also help.

RDIL

I believe this is the issue @asomers.

RDIL · 2019-06-07T14:20:21Z

.cirrus.yml

+task:
+  name: Linux
+  container:
+    - image: rust:1.24.1


Suggested change

- image: rust:1.24.1

image: rust:1.24.1

RDIL · 2019-06-07T14:20:35Z

.cirrus.yml

+task:
+  name: Linux/GNU x86_64 stable
+  container:
+    - image: rust:latest


Suggested change

- image: rust:latest

image: rust:latest

fkorotkov · 2019-06-07T14:21:11Z

.cirrus.yml

+task:
+  name: Linux
+  container:
+    - image: rust:1.24.1


I believe it shouldn't be a YAML list:

Suggested change

- image: rust:1.24.1

image: rust:1.24.1

hehehe I beat you to it

fkorotkov · 2019-06-07T14:21:19Z

.cirrus.yml

+task:
+  name: Linux/GNU x86_64 stable
+  container:
+    - image: rust:latest


Suggested change

- image: rust:latest

image: rust:latest

asomers · 2019-06-07T14:25:44Z

Thanks for the rapid feedback @fkorotkov @RDIL !

Hopefully this will get us more up-to-date VMs.

It looks like the SIGALARM is interrupting poll(2)

* kmod tests must run exclusively, because they load and unload a module with a constant name. * A few tests were doing some variant of chdir, but weren't taking the CWD_MTX. * The kmod tests read files by path relative to CWD, so they need the CWD_MTX. But they don't need it exclusively, so convert the CWD_MTX into an RwLock. * Tests that do change the cwd need to change it back when they're done.

--test-threads > 1

I don't know why they fail. It's probably either a kernel bug or an undocumented restriction. According to /proc/crypto these algorithms _are_ supported on Cirrus's VMs.

asomers · 2021-02-07T02:35:58Z

superseded by #1350

asomers marked this pull request as ready for review May 26, 2019 23:08

asomers force-pushed the cirrus_ci2 branch from 4ccaeaa to acaf703 Compare June 7, 2019 13:41

RDIL suggested changes Jun 7, 2019

View reviewed changes

RDIL mentioned this pull request Jun 7, 2019

FAILED_PRECONDITION: Unknown field null on 'container' YAML node! cirruslabs/cirrus-ci-docs#325

Closed

fkorotkov suggested changes Jun 7, 2019

View reviewed changes

RDIL approved these changes Jun 7, 2019

View reviewed changes

asomers added 15 commits July 25, 2020 14:08

Switch Linux/GNU x86_64 tests from Travis to Cirrus

2face49

Hopefully this will get us more up-to-date VMs.

Hopefully fix test_alarm and test_poll in CI

ef8d239

It looks like the SIGALARM is interrupting poll(2)

Test the theory that test_poll fails due to a thread conflict

ef4df89

Check how many CPUs the Cirrus Linux images have

efe113c

Make the cross scripts compatible with Cirrus

fcda7c4

cat /proc/crypto during Cirrus Linux tests

8767023

WIP skip should_panic tests, which sometimes cause problems in CI with

6bdb886

--test-threads > 1

Skip the af_alg tests in Cirrus CI

09a68c4

I don't know why they fail. It's probably either a kernel bug or an undocumented restriction. According to /proc/crypto these algorithms _are_ supported on Cirrus's VMs.

Use ci/script.sh on Cirrus

b876579

Cirrus doesn't accept matrix items with a single entry

2811caa

Fix typo in .cirrus.yml

41ae1c3

matrix's children should not be a YAML list.

f33f1f7

container's children shouldn't be a YAML list

0c61a08

yaml: env should be a mapping, not a list

dacabbc

asomers added 7 commits July 25, 2020 14:10

YAML: fix missing space

5631c8f

Remove bashism in ci/script.sh

3501f9d

Use cross's install.sh

50cf6c1

fixup

8167464

Try to debug PATH failure

01542c8

put ~/.cargo/bin in PATH

2f6fbe0

Oddly, Cirrus's nodes don't have /root/.cargo/env

ab0295e

asomers force-pushed the cirrus_ci2 branch from a41c415 to ab0295e Compare July 25, 2020 20:13

asomers closed this Feb 7, 2021

asomers deleted the cirrus_ci2 branch February 7, 2021 02:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move Linux CI tests from Travis to Cirrus #1070

Move Linux CI tests from Travis to Cirrus #1070

asomers commented May 26, 2019

asomers commented May 27, 2019

glebpom commented May 27, 2019 •

edited

asomers commented May 27, 2019

glebpom commented May 27, 2019

glebpom commented May 28, 2019

asomers commented May 28, 2019

glebpom commented May 29, 2019

asomers commented May 29, 2019

RDIL left a comment

RDIL Jun 7, 2019

RDIL Jun 7, 2019

fkorotkov Jun 7, 2019

RDIL Jun 7, 2019

fkorotkov Jun 7, 2019

asomers commented Jun 7, 2019

asomers commented Feb 7, 2021

Move Linux CI tests from Travis to Cirrus #1070

Move Linux CI tests from Travis to Cirrus #1070

Conversation

asomers commented May 26, 2019

asomers commented May 27, 2019

glebpom commented May 27, 2019 • edited

asomers commented May 27, 2019

glebpom commented May 27, 2019

glebpom commented May 28, 2019

asomers commented May 28, 2019

glebpom commented May 29, 2019

asomers commented May 29, 2019

RDIL left a comment

Choose a reason for hiding this comment

RDIL Jun 7, 2019

Choose a reason for hiding this comment

RDIL Jun 7, 2019

Choose a reason for hiding this comment

fkorotkov Jun 7, 2019

Choose a reason for hiding this comment

RDIL Jun 7, 2019

Choose a reason for hiding this comment

fkorotkov Jun 7, 2019

Choose a reason for hiding this comment

asomers commented Jun 7, 2019

asomers commented Feb 7, 2021

glebpom commented May 27, 2019 •

edited