Add regression test infrastructure #2109

Thomasdezeeuw · 2021-03-13T17:24:00Z

Please the commit messages for details.

I still need to add lists for the following targets, but I got the major ones I think.

TODO:

aarch64-unknown-hermit
x86_64-unknown-hermit
x86_64-pc-solaris
x86_64-sun-solaris
sparcv9-sun-solaris
x86_64-fortanix-unknown-sgx
x86_64-unknown-illumos
asmjs-unknown-emscripten
wasm32-unknown-emscripten
wasm32-unknown-unknown
wasm32-wasi
Check symbols added after commit ed45c26.

TODO: add a bit to the contributing guide about adding to these lists.

Closes #2104.

rust-highfive · 2021-03-13T17:24:03Z

r? @JohnTitor

(rust-highfive has picked a reviewer for you, use r? to override)

bors · 2021-03-16T00:38:50Z

☔ The latest upstream changes (presumably #2113) made this pull request unmergeable. Please resolve the merge conflicts.

Thomasdezeeuw · 2021-03-16T10:34:08Z

@JohnTitor before I rebase could you review this first?

JohnTitor · 2021-03-16T19:18:47Z

One question is how we deal with the FreeBSD version difference but otherwise looks good to me!

Thomasdezeeuw · 2021-03-18T09:10:23Z

I think it's best to merge this as is and I'll try to find some time to add lists for the other targets.

For FreeBSD can make lists like freebsd-12.txt, freebsd-13.txt, etc where freebsd-13.txt includes freebsd-12.txt, etc.

JohnTitor · 2021-03-22T21:40:12Z

Seems the TODO in the OP isn't addressed, I'd like to see it in this PR.

Thomasdezeeuw · 2021-03-22T21:46:00Z

I don't really have the time to add all the lists for the targets any time soon, so it will take a while.

JohnTitor · 2021-03-22T21:52:44Z

Ah I mean the second TODO, add a bit to the contributing guide about adding to these lists. (sorry, I didn't notice we have two TODOs). It should be helpful for contributors. Adding all the targets isn't necessary for now.

Thomasdezeeuw · 2021-03-26T11:27:03Z

Ah I mean the second TODO, add a bit to the contributing guide about adding to these lists. (sorry, I didn't notice we have two TODOs). It should be helpful for contributors. Adding all the targets isn't necessary for now.

Done.

JohnTitor · 2021-03-27T03:39:26Z

Thank you for working on it!
@bors r+

bors · 2021-03-27T03:39:27Z

📌 Commit e864628 has been approved by JohnTitor

bors · 2021-03-27T03:39:34Z

⌛ Testing commit e864628 with merge 5fdb940...

…hnTitor Add regression test infrastructure Please the commit messages for details. I still need to add lists for the following targets, but I got the major ones I think. TODO: * aarch64-unknown-hermit * x86_64-unknown-hermit * x86_64-pc-solaris * x86_64-sun-solaris * sparcv9-sun-solaris * x86_64-fortanix-unknown-sgx * x86_64-unknown-illumos * asmjs-unknown-emscripten * wasm32-unknown-emscripten * wasm32-unknown-unknown * wasm32-wasi * Check symbols added after commit ed45c26. TODO: add a bit to the contributing guide about adding to these lists. Closes #2104.

bors · 2021-03-27T03:50:33Z

💔 Test failed - checks-actions

JohnTitor · 2021-03-27T04:35:12Z

Looks like the check isn't working well on i686-unknown-linux-musl.

This first step add the infrastructure to test if libc follows semantic versioning. In the build step it creates a test file which imports all functions, constants, etc. that are expected to be public. This file is generated from the files in the (not yet included) semver directory. These files include the function and constants expected to be public per target family, vendor, OS, etc. See the do_semver function in the build file of libc-test for the details.

Thomasdezeeuw · 2021-04-02T11:18:53Z

The failure for arm-linux-androideabi seems unrelated: thread 'main' panicked at 'failed to find successful test run', /tmp/runtest.rs:46:9 (https://github.com/rust-lang/libc/runs/2253378077?check_suite_focus=true).

JohnTitor · 2021-04-02T11:21:01Z

Hmm, but this happens twice (https://github.com/rust-lang/libc/runs/2245677639?check_suite_focus=true) on semver check so I guess we need some work there.

Thomasdezeeuw · 2021-04-02T11:22:18Z

Looking at

libc/ci/runtest-android.rs

Lines 42 to 47 in bd7e052

    
           stdout.lines().find(|l| 
        
               (l.starts_with("PASSED ") && l.contains(" tests")) || 
        
               l.starts_with("test result: ok") 
        
           ).unwrap_or_else(|| { 
        
               panic!("failed to find successful test run"); 
        
           });

, does it need to output something on Android?

Testing on Android (in ci/runtest-android.rs) seems to depend on the test outputting it was successful.

Thomasdezeeuw · 2021-04-02T11:25:50Z

I can't run it locally so hopefully 418c481 fixes it.

JohnTitor · 2021-04-02T11:27:01Z

@bors try

Add regression test infrastructure Please the commit messages for details. I still need to add lists for the following targets, but I got the major ones I think. TODO: * aarch64-unknown-hermit * x86_64-unknown-hermit * x86_64-pc-solaris * x86_64-sun-solaris * sparcv9-sun-solaris * x86_64-fortanix-unknown-sgx * x86_64-unknown-illumos * asmjs-unknown-emscripten * wasm32-unknown-emscripten * wasm32-unknown-unknown * wasm32-wasi * Check symbols added after commit ed45c26. TODO: add a bit to the contributing guide about adding to these lists. Closes #2104.

bors · 2021-04-02T11:27:08Z

⌛ Trying commit 418c481 with merge ed37f58...

bors · 2021-04-02T12:31:24Z

☀️ Try build successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Build commit: ed37f58 (ed37f58d5589649597d52c53fb78da10f1d13ab7)

Thomasdezeeuw · 2021-04-02T12:35:49Z

Thank you @bors, @JohnTitor CI is green 👍

JohnTitor · 2021-04-02T12:53:17Z

Thanks for working on it! @bors r+

bors · 2021-04-02T12:53:18Z

📌 Commit 418c481 has been approved by JohnTitor

bors · 2021-04-02T12:53:25Z

⌛ Testing commit 418c481 with merge 0a93b70...

bors · 2021-04-02T13:57:43Z

☀️ Test successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: JohnTitor
Pushing 0a93b70 to master...

semarie · 2021-04-03T07:03:15Z

I have the following error on OpenBSD:

+ cargo test --no-default-features --manifest-path libc-test/Cargo.toml --target x86_64-unknown-openbsd               
   Compiling libc-test v0.2.92 (/data/semarie/repos/rust/libc-rs/libc-test)
error[E0432]: unresolved imports `libc::EMULTIHOP`, `libc::ENOLINK`, `libc::WEXITED`, `libc::WNOWAIT`, `libc::WSTOPPED
`                                                           
   --> /data/semarie/repos/rust/build-libc/build/x86_64-unknown-openbsd/debug/build/libc-test-f95bb6991c27a551/out/sem
ver.rs:87:5                                                                                                           
    |            
87  |     EMULTIHOP,                                        
    |     ^^^^^^^^^ no `EMULTIHOP` in the root              
...                                                                                                                   
98  |     ENOLINK,                                                                                                    
    |     ^^^^^^^ no `ENOLINK` in the root                                                                            
...                                                         
431 |     WEXITED,                                                                                                    
    |     ^^^^^^^ no `WEXITED` in the root                                                                            
...                                                         
438 |     WNOWAIT,                                          
    |     ^^^^^^^ no `WNOWAIT` in the root                                                                            
439 |     WSTOPPED,                                                                                                   
    |     ^^^^^^^^ no `WSTOPPED` in the root                                                                          
    |                                                      
help: a similar name exists in the module                                                                             
    |                                                      
98  |     EMLINK,                                          
    |     ^^^^^^                                           
help: a similar name exists in the module                                                                             
    |                                                      
431 |     WIFEXITED,                                       
    |     ^^^^^^^^^                                        
help: a similar name exists in the module                                                                             
    |                                                      
439 |     WIFSTOPPED,                                      
    |     ^^^^^^^^^^                                       

error: aborting due to previous error                                                                                 

For more information about this error, try `rustc --explain E0432`.
error: could not compile `libc-test`                                                                                  

To learn more, run the command again with --verbose.                                                                  
warning: build failed, waiting for other jobs to finish...                                                            
error: build failed

it seems it is because the mentioned symbols are in unix.txt whereas OpenBSD (which is unix) doesn't have them. I checked that we really don't have them (it isn't just rust-libc missing them).

how to correct it ? move the symbols from unix.txt to target specific file ?

several symbols added to semver regression tests in "unix" aren't defined on OpenBSD. move these symbols in OS specific files.

unbreak openbsd after #2109 several symbols added to semver regression tests in "unix" aren't defined on OpenBSD. move these symbols in OS specific files.

Thomasdezeeuw · 2021-04-05T09:05:36Z

Sorry for the late response @semarie, but I see you've already fixed the problem, thanks.

The semver regression checks in rust-lang#2109 included the "SO_REUSEPORT" constant, which we do not yet have on illumos systems. Move it out to platform-specific files.

rust-highfive assigned JohnTitor Mar 13, 2021

rust-highfive added the S-waiting-on-review label Mar 13, 2021

JohnTitor added S-waiting-on-author and removed S-waiting-on-review labels Mar 15, 2021

Thomasdezeeuw marked this pull request as ready for review March 18, 2021 09:08

JohnTitor approved these changes Mar 27, 2021

View reviewed changes

bors added S-waiting-on-bors and removed S-waiting-on-author labels Mar 27, 2021

bors added S-waiting-on-review and removed S-waiting-on-bors labels Mar 27, 2021

Thomasdezeeuw added 5 commits March 27, 2021 10:27

Add Window semver list

c84bd4e

Add semver list for Apple's iOS and macOS

64950da

Add FreeBSD semver list

6384bbf

Add Redox semver list

2751674

Output PASSED 1 tests in semver test

418c481

Testing on Android (in ci/runtest-android.rs) seems to depend on the test outputting it was successful.

bors added S-waiting-on-bors and removed S-waiting-on-review labels Apr 2, 2021

bors merged commit 0a93b70 into rust-lang:master Apr 2, 2021

semarie added a commit to semarie/rust-libc that referenced this pull request Apr 4, 2021

unbreak openbsd after rust-lang#2109

95f34ad

several symbols added to semver regression tests in "unix" aren't defined on OpenBSD. move these symbols in OS specific files.

semarie added a commit to semarie/rust-libc that referenced this pull request Apr 4, 2021

unbreak openbsd after rust-lang#2109

08ec0d6

several symbols added to semver regression tests in "unix" aren't defined on OpenBSD. move these symbols in OS specific files.

bors added a commit that referenced this pull request Apr 4, 2021

Auto merge of #2136 - semarie:openbsd-semver, r=JohnTitor

3022d10

unbreak openbsd after #2109 several symbols added to semver regression tests in "unix" aren't defined on OpenBSD. move these symbols in OS specific files.

bors added a commit that referenced this pull request Apr 4, 2021

Auto merge of #2136 - semarie:openbsd-semver, r=JohnTitor

95e259f

unbreak openbsd after #2109 several symbols added to semver regression tests in "unix" aren't defined on OpenBSD. move these symbols in OS specific files.

Thomasdezeeuw deleted the issue_2104-regression_tests branch April 5, 2021 09:05

tarcieri mentioned this pull request May 12, 2021

musl: HWCAP_* consts missing on aarch64 targets #2171

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add regression test infrastructure #2109

Add regression test infrastructure #2109

Thomasdezeeuw commented Mar 13, 2021 •

edited

rust-highfive commented Mar 13, 2021

bors commented Mar 16, 2021

Thomasdezeeuw commented Mar 16, 2021

JohnTitor commented Mar 16, 2021

Thomasdezeeuw commented Mar 18, 2021

JohnTitor commented Mar 22, 2021

Thomasdezeeuw commented Mar 22, 2021

JohnTitor commented Mar 22, 2021

Thomasdezeeuw commented Mar 26, 2021

JohnTitor commented Mar 27, 2021

bors commented Mar 27, 2021

bors commented Mar 27, 2021

bors commented Mar 27, 2021

JohnTitor commented Mar 27, 2021

Thomasdezeeuw commented Apr 2, 2021

JohnTitor commented Apr 2, 2021

Thomasdezeeuw commented Apr 2, 2021

Thomasdezeeuw commented Apr 2, 2021

JohnTitor commented Apr 2, 2021

bors commented Apr 2, 2021

bors commented Apr 2, 2021

Thomasdezeeuw commented Apr 2, 2021

JohnTitor commented Apr 2, 2021

bors commented Apr 2, 2021

bors commented Apr 2, 2021

bors commented Apr 2, 2021

semarie commented Apr 3, 2021

Thomasdezeeuw commented Apr 5, 2021

Add regression test infrastructure #2109

Add regression test infrastructure #2109

Conversation

Thomasdezeeuw commented Mar 13, 2021 • edited

rust-highfive commented Mar 13, 2021

bors commented Mar 16, 2021

Thomasdezeeuw commented Mar 16, 2021

JohnTitor commented Mar 16, 2021

Thomasdezeeuw commented Mar 18, 2021

JohnTitor commented Mar 22, 2021

Thomasdezeeuw commented Mar 22, 2021

JohnTitor commented Mar 22, 2021

Thomasdezeeuw commented Mar 26, 2021

JohnTitor commented Mar 27, 2021

bors commented Mar 27, 2021

bors commented Mar 27, 2021

bors commented Mar 27, 2021

JohnTitor commented Mar 27, 2021

Thomasdezeeuw commented Apr 2, 2021

JohnTitor commented Apr 2, 2021

Thomasdezeeuw commented Apr 2, 2021

Thomasdezeeuw commented Apr 2, 2021

JohnTitor commented Apr 2, 2021

bors commented Apr 2, 2021

bors commented Apr 2, 2021

Thomasdezeeuw commented Apr 2, 2021

JohnTitor commented Apr 2, 2021

bors commented Apr 2, 2021

bors commented Apr 2, 2021

bors commented Apr 2, 2021

semarie commented Apr 3, 2021

Thomasdezeeuw commented Apr 5, 2021

Thomasdezeeuw commented Mar 13, 2021 •

edited