Testing

[utterances-bot]

This is a general purpose comment thread for the Testing post.

[leggettc18]

I'm not sure if I did something wrong, but by following along with this guide exactly I was receiving an error about fmt being undeclared after setting up the CompareMessage implementation. I fixed it by adding use core::fmt; and use core::fmt::Write; at the top of the file, and increasing the PANIC_LINE by 2 to compensate. Did I miss a step along the way or is there a missing step in the guide?

[phil-opp]

@leggettc18 You did nothing wrong, I forgot to mention these imports. Fixed in 1f97103. Thanks for reporting!

[Beidah]

I'm unable to run cargo xtest. Instead I keep keeping the message

error[E0463]: can't find crate for std
|
= note: the x86_64-blog_os-1954185736335484285 target may not be installed

cargo xbuild works fine, though.

[AntoineSebert]

Try cargo xtest --target x86_64-blog_os.json.

I have the same problem but couldn't figure out why the file isn't automatically found.

[phil-opp]

@AntoineSebert Do you have a build.target key defined in your .cargo/config file?

@Beidah Sounds like forgot a #![no_std] attribute (perhaps the conditional attribute in your lib.rs?). Or you accidentally use a dependency that uses std.

[AntoineSebert]

@phil-opp Yes it's there. The problem seems to have disappread today, and anyway it's not a big deal.

[AntoineSebert]

The recent improvements in the test framework now produce text outputs that are not colored in green, thus making the test result less visible within the terminal.
Is there a way to specify a color for the text when calling print/println macros ?

[phil-opp]

Yes, there is! You can use ANSI escape codes for coloring. For example, you can create a type to print something in green:

use core::fmt;

struct Green(&'static str);

impl fmt::Display for Green {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { 
        write!(f, "\x1B[32m")?; // prefix code
        write!(f, "{}", self.0)?;
        write!(f, "\x1B[0m")?; // postfix code
        Ok(())
    }
}

Now you can do serial_println!("{}", Green("[ok]")); at the end of your test.

I opened #603 for adding this to the post.

[meteor-lsw]

In the process of adding tests, I encountered some strange errors. In this test, I simply created a writer, but failed, and judging from the error message, it seems that it did not execute any tests at all, because I did not see the information printed by test_runner or panic_handler on the console.

lws@lsw:/document/project/flandre-os/FlandreOS$ cargo xtest test_default_writer
   Compiling flandre_os v0.0.1 (/document/project/flandre-os/FlandreOS)
    Finished dev [unoptimized + debuginfo] target(s) in 0.33s
     Running target/x86_64-flandre_os/debug/deps/flandre_os-d120248f92b70111
Building bootloader
   Compiling bootloader v0.6.0 (/work/tool/rust/cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/bootloader-0.6.0)
    Finished release [optimized + debuginfo] target(s) in 0.72s
Running: `qemu-system-x86_64 -drive format=raw,file=/document/project/flandre-os/FlandreOS/target/x86_64-flandre_os/debug/deps/bootimage-flandre_os-d120248f92b70111.bin -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none test_default_writer`
qemu-system-x86_64: -display none: drive with bus=0, unit=0 (index=0) exists
error: test failed, to rerun pass '--lib'

[meteor-lsw]

I did all kinds of printing tests and the printing function seemed to work normally, but when I tried to add some tests to ensure that these functions would not be damaged in future modifications, I got the above errors.

[bjorn3]

I think passing a test name to cargo xtest doesnt work. Try running just cargo xtest.

[phil-opp]

I think passing a test name to cargo xtest doesnt work. Try running just cargo xtest.

Yes, as you see from the "Running: …" output, the test_default_writer is directly passed as an argument to QEMU, which does not understand it and exits with an error. Passing the name of a single test is a feature of the default test runner and not available in our bare-metal environment because our kernel does not get any arguments passed when it's started.

What you can do is to run the tests of the library component through cargo xtest --lib and the tests of the binary through cargo xtest --bin blog_os. You can also run an individual integration test through cargo xtest --test test_name.

[xj42]

Now we can run the test using cargo xtest --test panic_handler. We see that it passes, which means that the reported panic info is correct. If we use a wrong line number in PANIC_LINE or panic with an additional character through panic!("{}x", MESSAGE), we see that the test indeed fails.

I tried adding the "x" as you indicated in the above paragraph. The test did not fail. WIth some more fooling about i see that the fn write_str(&mut self, s: &str) -> fmt::Result {} function is called 4 times. The third time it is called, it sets equals to true. Because there is no else setting false, the test does not work as expected.

I have submitted a pull request #610

[phil-opp]

@xj42 Thanks a lot for reporting! I created #611 to fix this.

[ta5een]

Hi, thank you for taking your time and effort into making this brilliant blog! I've learnt so much about operating systems and the Rust language in general! Can't wait to see how this blog unfolds and what new things you'll bring.

I just wanted to ask if it is the expected behaviour for serial_print! and serial_println! to only work in test mode? With cargo xtest, the serial print macros work as expected; they print to my host's terminal. However, with cargo xrun, nothing is shown in my host's terminal, nor my QEMU instance. Oddly enough, the system hasn't panicked, so it somehow worked (but the bytes don't show up).

It seems to me that perhaps this was the desired behaviour? If so, is there a way for me to send data to the serial port while the QEMU instance is running as normal?

[zhanlidegui]

[package.metadata.bootimage] test-args = ["-device", "isa-debug-exit,iobase=0xf4,iosize=0x04","-serial", "stdio","-display","none"] test-success-exit-code = 33 test-timeout = 120

This setting is only valid for testing

[Archelyst]

I’m using 1.61.0-nightly indeed. How do I upgrade? I tried `rustup update stable`, but that does not install a newer version. How come I need a not even released version of Rust for a tutorial that has been around for quite a while? Wouldn’t it work to somehow pin uart_16550 to an older version? (No idea how to do that, though…)

…

Am 18.05.2022 um 20:38 schrieb bjorn3 ***@***.***>: Which rustc version are you using? It should work with 1.62 nightly as this method will be stabilized in 1.61 which is currently in beta. I think you are using 1.61 nightly from just before it was marked as stabilized in 1.61, so updating to the latest nightly should fix this issue. — Reply to this email directly, view it on GitHub <#1009 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGD2ZFQS3HXYAWE7YJK6ZLVKU2LPANCNFSM5ARPOQNQ>. You are receiving this because you commented.

[bjorn3]

You can use rustup update nightly. This tutorial needs nightly as it uses several unstable features. Rustc forbids usage of unstable features on the stable and beta channel to prevent ossification of those features. Otherwise there is a great risk that we won't be able to fix problems with unstable features without breaking a large portion of the rust ecosystem.

[iivvoo]

Having similar issues as others, cargo build works fine, cargo test spews out tons of errors, starting with

   Compiling bit_field v0.10.1
   Compiling spin v0.5.2
   Compiling volatile v0.4.5
   Compiling bitflags v1.3.2
   Compiling volatile v0.2.7
   Compiling bootloader v0.9.23
error[E0152]: duplicate lang item in crate `core` (which `rustc_std_workspace_core` depends on): `sized`.

and later

error[E0277]: the size for values of type `()` cannot be known at compilation time
   --> /Users/ivo/.cargo/registry/src/github.com-1ecc6299db9ec823/volatile-0.4.5/src/lib.rs:809:50
    |
809 |     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
    |                                                  ^^^^^^^^^^^ doesn't have a size known at compile-time
    |
    = help: the trait `Sized` is not implemented for `()`
note: required by a bound in `Result`
   --> /Users/ivo/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/result.rs:504:17
    |
504 | pub enum Result<T, E> {
    |                 ^ required by this bound in `Result`

removing the panic = from Cargo.tom has no noticeable effect. The weird mix of versions being installed on cargo test also confuses me, some weird dependency issue?

[bjorn3]

At which step are you? Is the code public somewhere? And did you add

[build]
target = "x86_64-blog_os.json"

to .cargo/config.toml?

[iivvoo]

I have the target properly defined, I'm at the testing page, very first step, test_runner added to main.rs

build and run work just fine. I'll upload my code later but it should be nearly identical to the code in the blog posts so far.

[bjorn3]

Later on there is a note:

Note: There is currently a bug in cargo that leads to “duplicate lang item” errors on cargo test in some cases. It occurs when you have set panic = "abort" for a profile in your Cargo.toml. Try removing it, then cargo test should work. See the cargo issue for more information on this.

Maybe that is the problem?

[iivvoo]

I had it removed already, that's not the solution. Isn't the collection of versions being compiled suspicious?

    Compiling bitflags v1.3.2
   Compiling volatile v0.4.5
   Compiling spin v0.5.2
   Compiling bit_field v0.10.1
   Compiling bootloader v0.9.23
   Compiling volatile v0.2.7

volatile version 0.2.7 (correct) and 0.4.5 - where did that come from?

[iivvoo]

Okay I'm really confused - started comparing the blog-04 branch to my code (the branch is ahead of my code since I'm stuck at the start). blog-04 ran fine. Started changing my code and eventually it did work / compile properly at the point there I commented the panic="abort" out.

I'm 100% sure I tried this enabled/disabled a couple of times, not sure why it works this time. But at least I can continue :)

[Orkiin]

Maybe not for this stage but I think that it could be solved with threads, so when you introduce the threads in the kernel you could revisit the should_panic equivalent. You should still find a way to stop the work in these threads after sometime or it will loop forever until the timeout of the tests. If this can't be done I will appreciate any explanation. Forgive me for my ignorance I'm advance.

[phil-opp]

Good idea! With normal "user-space" Rust this would work, as you can deal with thread panics in the join function. However, this requires panic="unwind", but we set `panic="abort". Maybe we could still get something similar to work with a bit of trickery...

[bjorn3]

https://github.com/nbdd0121/unwinding is a rust implementation of a stack unwinder. It also provides a personality function and panic handler if requested. After that you should be able to implement your own catch_unwind using the r#try intrinsic. Never tried anything of this myself though.

[hai1222]

Qemu can not exit on mac, Is the port 0xf4 not work in macos?

[hai1222]

cpu:Apple M1

[phil-opp]

What's the full QEMU command that you're using?

[hai1222]

I run "cargo test" in ubuntu, and then downoad the bin file to mac, then run "qemu-system-x86_64 -drive format=raw,file=/Users/xxx/Downloads/bootimage-blog_os-2f7ee54200b7f9c8.bin"

[bjorn3]

You need to pass -device isa-debug-exit,iobase=0xf4,iosize=0x04 to qemu as indicated at https://os.phil-opp.com/testing/#exiting-qemu

[hai1222]

Thanks, I have update some code follow the article, I will try this later.

[hai1222]

The test case in main.rs and basic_boot.rs don't work. Also I download the code from github, then run cargo test, Only print test case from vag_buffer in console.

qemu-system-x86_64 -drive format=raw,file=/Users/hai1222/Downloads/bootimage-blog_os-1a05277e1c815ec0.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none
Running 3 tests
blog_os::vga_buffer::test_println_many...	[ok]
blog_os::vga_buffer::test_println_output...	[ok]
blog_os::vga_buffer::test_println_simple...	[ok]

What's the wrong with my code or enviroment? Thanks.

[haikuginger]

One option for allowing for multiple should_panic tests would also unlock the regular test suite to move past failed tests. You'd need to override (or replace?) the #[test_case] macro to allow for additional conditional compilation, and you'd need to be willing to invoke QEMU one time per test instead of for the whole suite though.

In short: start with a build flag that allows all the tests to be compiled, and emit their names through the serial port, but do not run them. Then, for each test case name received over the serial port, recompile with a config flag that will cause only that specific test case to be compiled and executed.

To lower the cost of multiple executions of QEMU, you could also skip the enumeration stage, and have that macro exclude specified tests on compilation. You'd run all tests until you get your first panic (tracking completions), then recompile with all the tests you already executed (successfully or not) skipped, and re-invoke.

[nyabinary]

Its extremely hard to find a tutor for Rust that does OS Development lol

[bingxinliu]

Hi! First of all appreciate for your absolutely awesome tutorial!

I met a problem at very beginning of this section. After replacing the default test framework by inserting below announcement and test_main function

#![feature(custom_test_frameworks)]
#![test_runner(crate::test_runner)]
#![reexport_test_harness_main = "test_main"]

Then I met the problem of “duplicate lang item” errors. So I remove all panic = "abort" settings. But after that the eh_personality error talked at section A Freestanding Rust Binary appears.
Any idea of how to go around this problem?

Thank you so much!

[bingxinliu]

Sorry, never mind. I forgot a line in target specification json file. After adding the panic strategy line, all things work smoothly!

"panic-strategy": "abort",

[ochudi]

On macOS, running cargo test gave a lot of errors; but it's in line with it not being able to find the crate for test:

error: could not compile `spin` (lib) due to 118 previous errors```

[CalvoM]

@ochudi any update on this?

[bjorn3]

Do you have

#![feature(custom_test_frameworks)]
#![test_runner(crate::test_runner)]
#![reexport_test_harness_main = "test_main"]

in main.rs?

[NightProg]

Hello, when i run the cargo test command i got this error :

qemu-system-x86_64: --format=json: invalid option
error: test failed, to rerun pass `--bin hello_os`

my config.toml file :

[target.'cfg(target_os = "linux")']
rustflags = ["-C", "link-arg=-nostartfiles"]

[target.'cfg(target_os = "windows")']
rustflags = ["-C", "link-args=/ENTRY:_start /SUBSYSTEM:console"]

[target.'cfg(target_os = "macos")']
rustflags = ["-C", "link-args=-e __start -static -nostartfiles"]

[unstable]
build-std-features = ["compiler-builtins-mem"]
build-std = ["core", "compiler_builtins"]

[build]
target = "x86_64-hello_os.json"

[target.'cfg(target_os = "none")']
runner = "bootimage runner"

my json file :

{
  "llvm-target": "x86_64-unknown-none",
  "data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128",
  "arch": "x86_64",
  "target-endian": "little",
  "target-pointer-width": "64",
  "target-c-int-width": "32",
  "os": "none",
  "executables": true,
  "linker-flavor": "ld.lld",
  "linker": "rust-lld",
  "panic-strategy": "abort",
  "disable-redzone": true,
  "features": "-mmx,-sse,+soft-float"
}

and yes i have :

#![feature(custom_test_frameworks)]
#![test_runner(crate::test_runner)]
#![reexport_test_harness_main = "test_main"]

Thanks you.

[NightProg]

It's fixed apparently the bug was clion that add the option --format=json to the command so i renamed test by t and its working

[CodeKitty-max]

I'm getting a segfault on the first step of adding the test_runner function and running the cargo test. Is there anything that could explain this behavior??

[olest]

I am seeing the same error signal: 11, SIGSEGV: invalid memory reference

[CodeKitty-max]

At what point do you get this?

[olest]

Same as you, after running cargo test

[phil-opp]

Could you try cloning this repo, checking out the post-04 branch, and then running cargo test?

[CodeKitty-max]

It now works. Is there something in particular that you suspect causing this

[mr-hespicable]

hi!
this project builds and tests fine on my home computer, but when i run it on github (using the workflow), i get this error:

Error: Failed to execute QEMU test command `"qemu-system-x86_64" "-drive" "format=raw,file=/home/runner/work/sketchOS/sketchOS/target/x86_64-sketchOS/debug/deps/bootimage-sketch_os-118bcc2728fa5037.bin" "-no-reboot" "-device" "isa-debug-exit,iobase=0xf4,iosize=0x04," "-serial" "stdio" "-display" "none"`: An I/O error occured: No such file or directory (os error 2)
error: test failed, to rerun pass `--lib`

My rust.yml looks like this:

name: Rust

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

env:
    CARGO_TERM_COLOR: always

jobs:
    build:
        runs-on: ubuntu-latest
        steps:
          
            - uses: actions/checkout@v4
            - name: Get Nightly Build
              run: rustup override set nightly
            - name: Add rust-src
              run: rustup component add rust-src --toolchain nightly-x86_64-unknown-linux-gnu
            - name: Add llvm-tools 
              run: rustup component add llvm-tools-preview
            - name: Install Bootimage
              run: cargo install bootimage
            - name: Build
              run: cargo build --target x86_64-sketchOS.json --verbose
            - name: Run tests
              run: cargo test --target x86_64-sketchOS.json --verbose

anyone know why this is happening?

[tsatke]

Correct me if I'm wrong, but it seems to me that qemu-system-x86_64 is not installed?

Try to add these steps before cargo test:

      # install QEMU
      - name: Install QEMU
        run: sudo apt update && sudo apt install -y qemu-system-x86
      - name: "Print QEMU Version"
        run: qemu-system-x86_64 --version

[mr-hespicable]

oh.
obviously, haha. thanks a ton! it works fine now... first successful build :D

[tsatke]

Congrats!

[cmanlh]

Hi, when I try to pass arguments "-display none" to QUME, and then run the cargo test, it will below error message.

QEMU 8.1.90 monitor - type 'help' for more information
qemu-system-x86_64: -serial stdio: qemu_add_wait_object: failed
qemu-system-x86_64: -serial stdio: could not connect serial device to character backend 'stdio'

[tsatke]

Can you share the full command line?

[cmanlh]

Hi,

PS D:\git\xos> cargo test
warning: unused variable: `tests`
  --> tests\basic_boot.rs:16:16
   |
16 | fn test_runner(tests: &[&dyn Fn()]) {
   |                ^^^^^ help: if this is intentional, prefix it with an underscore: `_tests`
   |
   = note: `#[warn(unused_variables)]` on by default

warning: function `test_runner` is never used
  --> tests\basic_boot.rs:16:4
   |
16 | fn test_runner(tests: &[&dyn Fn()]) {
   |    ^^^^^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: `xos` (test "basic_boot") generated 2 warnings (run `cargo fix --test "basic_boot"` to apply 1 suggestion)
    Finished test [unoptimized + debuginfo] target(s) in 0.14s
     Running unittests src\lib.rs (target\x86_64_xos\debug\deps\xos-17d47f4dbcc45181)
Building bootloader
    Finished release [optimized + debuginfo] target(s) in 0.14s
Running: `qemu-system-x86_64 -drive format=raw,file=D:\git\xos\target\x86_64_xos\debug\deps\bootimage-xos-17d47f4dbcc45181.bin -no-reboot -device isa-debug-exit,iobase=0xf4,iosize=0x04 -serial stdio -display none`
QEMU 8.1.90 monitor - type 'help' for more information
qemu-system-x86_64: -serial stdio: qemu_add_wait_object: failed
qemu-system-x86_64: -serial stdio: could not connect serial device to character backend 'stdio'
error: test failed, to rerun pass `--lib`

Caused by:
  process didn't exit successfully: `bootimage runner D:\git\xos\target\x86_64_xos\debug\deps\xos-17d47f4dbcc45181` (exit code: 1)    
note: test exited abnormally; to see the full output pass --nocapture to the harness.

[phil-opp]

Look like a QEMU-specific error, not sure what's the issue. Have you tried some other QEMU versions?

[wintertempest]

The section where we try to print the names of the test functions is not functioning correctly for me. Whenever I use core::any::type_name::(), it only prints the actual type, which is &dyn core::ops::function::Fn() instead of the function's name.

[bjorn3]

Can you show your Testable implementation? And your test_runner?

[wintertempest]

Hi, thanks for replying, this is my implementation

[wintertempest]

pub trait Testable {
fn run(&self) -> ();
}

impl Testable for T
where
T: Fn(),
{
fn run(&self) {
serial_print!("{}...\t", core::any::type_name::());
self();
serial_println!("[ok]");
}
}

#[cfg(test)]
fn test_runner(tests: &[&dyn Fn()]) {
serial_println!("Demarage de {} tests", tests.len());
for test in tests {
test.run();
}
serial_println!("Fin des tests");
exit_qemu(QemuExitCode::Success);
}

[bjorn3]

You need to change the function signature of test_runner to accept &[&dyn Testable] instead of &[&dyn Fn()].

[wintertempest]

Thank you very much, it was indeed the issue.

[Cooper-Pilot]

#[cfg(test)] never returns true when compiling with cargo t.
This is my cargo.toml:

[package]
name = "xarbaros"
version = "0.1.0"
edition = "2018"

#[profile.dev]
#panic = "abort"
#[profile.release]
#panic = "abort"

[dependencies.lazy_static]
version = "1.0"
features = ["spin_no_std"]

[dependencies]
spin = "0.5.2"
volatile = "0.2.6"
bootloader = "0.9.23"

main.rs:

#![no_std]
#![no_main]

#![feature(custom_test_frameworks)]
#![test_runner(crate::test_runner)]

#![reexport_test_harness_main = "test_main"]

use core::{panic::PanicInfo, fmt::Write};
mod vga_buffer;

#[no_mangle]
pub extern "C" fn _start() -> ! {
    println!("Hello World{}", "!");
    
    #[cfg(test)]
    test_main();
    
    loop {}
}

#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    loop {}
}

#[cfg(test)]
pub fn test_runner(tests: &[&dyn Fn()]) {
    println!("Running {} tests", tests.len());
    for test in tests {
        test();
    }
}

[toolateralus]

same problem, the tests never run/ get compiled

[AtomicGamer9523]

test_main doesn't exist

• it doesn't look like you have any tests to run

[Cooper-Pilot]

When I change #[cfg(test)] test_main(); to #[cfg(test)] println!("test");, I expect it to print “test” to the buffer when ran with cargo t, but the results are exactly the same as when ran with cargo r, that is it doesn’t print “test”. It seems to just be completely ignoring tests for me.

[toolateralus]

same here, and i also had some meaningless test cases and none ran.

[phil-opp]

Could you try with the post-04 branch of this repo?