Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Starving on crash (subproc_checkTimeLimit():522 pid=XXX has already been signaled due to timeout. Killing it with SIGKILL) #64

Open
CBenoit opened this issue Jan 11, 2022 · 0 comments
Open

Starving on crash (subproc_checkTimeLimit():522 pid=XXX has already been signaled due to timeout. Killing it with SIGKILL) #64

CBenoit opened this issue Jan 11, 2022 · 0 comments

Comments

@CBenoit
Copy link

CBenoit commented Jan 11, 2022 •
edited

Hello!

I'm currently trying to use honggfuzz for fuzzing a network interface using persistent fuzzing.

Here is my harness as of now (link refer to a specific commit): https://github.com/Devolutions/devolutions-gateway/blob/bf66f15933d9571574f2db8e65fd1c1019025551/fuzz/server/fuzz_targets/listeners_raw.rs

use honggfuzz::fuzz;
use server_fuzz::init;
use server_fuzz::oracles::raw::fuzz_listener;

fn main() {
    let rt = tokio::runtime::Builder::new_current_thread()
        .enable_all()
        .build()
        .unwrap();

    let listeners = rt.block_on(init());

    // At this point, sockets are binded and we can send data safely

    loop {
        fuzz!(|data: &[u8]| {
            for l in &listeners {
                fuzz_listener(data, l.addr().port());
                let _ = rt.block_on(l.handle_one());
            }
        })
    }
}

Note: the issue is the same regardless of the kind of tokio runtime used (new_current_thread() and new_multi_thread() both triggers the same behavior).

I'm running the fuzzing procedure with the following command:

$ RUSTFLAGS="-Z new-llvm-pass-manager=no -Z sanitizer=address" HFUZZ_RUN_ARGS="-t 10 -n 4 --tmout_sigvtalrm" cargo +nightly hfuzz run listeners_raw

-Z new-llvm-pass-manager=no is because of #61

I introduced a panic when receiving specific pattern of bytes on the listener side just to test out. This cause the program to crash quick enough:

Sz:1566 Tm:252us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/17
Sz:268 Tm:425us (i/b/h/e/p/c) New:0/0/0/0/0/2, Cur:0/0/0/0/0/3
Sz:264 Tm:2,827us (i/b/h/e/p/c) New:0/0/0/0/0/2, Cur:0/0/0/0/0/29
 rash (dup): 'hfuzz_workspace/listeners_raw/SIGABRT.PC.7ffff794b24c.STACK.f05f9f061.CODE.-6.ADDR.0.INSTR.mov____%eax,%ebp.fuzz' already exists, skipping
Sz:266 Tm:374us (i/b/h/e/p/c) New:0/0/0/3/0/4, Cur:0/0/0/3/0/2
Sz:452 Tm:918us (i/b/h/e/p/c) New:0/0/0/0/0/2, Cur:0/0/0/0/0/12
Crash (dup): 'hfuzz_workspace/listeners_raw/SIGABRT.PC.7ffff794b24c.STACK.f05f9f061.CODE.-6.ADDR.0.INSTR.mov____%eax,%ebp.fuzz' already exists, skipping
Crash (dup): 'hfuzz_workspace/listeners_raw/SIGABRT.PC.7ffff794b24c.STACK.f05f9f061.CODE.-6.ADDR.0.INSTR.mov____%eax,%ebp.fuzz' already exists, skipping
Sz:271 Tm:516us (i/b/h/e/p/c) New:0/0/0/1/0/59, Cur:0/0/0/3/0/13
 z:5026 Tm:346us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/1
Sz:78 Tm:543us (i/b/h/e/p/c) New:0/0/0/2/0/19, Cur:0/0/0/2/0/23
Sz:149 Tm:401us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/1
Sz:269 Tm:1,005us (i/b/h/e/p/c) New:0/0/0/0/0/4, Cur:0/0/0/0/0/11
Sz:372 Tm:665us (i/b/h/e/p/c) New:0/0/0/0/0/3, Cur:0/0/0/0/0/38
Sz:156 Tm:547us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/16
 z:8192 Tm:568us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/9
Sz:129 Tm:559us (i/b/h/e/p/c) New:0/0/0/0/0/2, Cur:0/0/0/0/0/5
Sz:279 Tm:510us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/16
Sz:264 Tm:548us (i/b/h/e/p/c) New:0/0/0/0/0/2, Cur:0/0/0/0/0/51
Sz:387 Tm:591us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/5
Sz:164 Tm:470us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/16
Sz:582 Tm:485us (i/b/h/e/p/c) New:0/0/0/0/0/2, Cur:0/0/0/0/0/5
 z:378 Tm:508us (i/b/h/e/p/c) New:0/0/0/1/0/0, Cur:0/0/0/1/0/8
Sz:82 Tm:315us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/12
Sz:273 Tm:654us (i/b/h/e/p/c) New:0/0/0/0/0/2, Cur:0/0/0/0/0/16
Sz:164 Tm:643us (i/b/h/e/p/c) New:0/0/0/0/0/1, Cur:0/0/0/0/0/16
Sz:136 Tm:573us (i/b/h/e/p/c) New:0/0/0/0/0/3, Cur:0/0/0/0/0/7
Crash (dup): 'hfuzz_workspace/listeners_raw/SIGABRT.PC.7ffff794b24c.STACK.f05f9f061.CODE.-6.ADDR.0.INSTR.mov____%eax,%ebp.fuzz' already exists, skipping
 2022-01-11T11:51:04-0500][W][41564] subproc_checkTimeLimit():529 pid=41571 took too much time (limit 10 s). Killing it with SIGVTALRM
[2022-01-11T11:51:04-0500][W][41565] subproc_checkTimeLimit():529 pid=41570 took too much time (limit 10 s). Killing it with SIGVTALRM
[2022-01-11T11:51:04-0500][W][41563] subproc_checkTimeLimit():529 pid=41568 took too much time (limit 10 s). Killing it with SIGVTALRM
[2022-01-11T11:51:05-0500][W][41564] subproc_checkTimeLimit():522 pid=41571 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41565] subproc_checkTimeLimit():522 pid=41570 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41564] subproc_checkTimeLimit():522 pid=41571 has already been signaled due to timeout. Killing it with SIGKILL
 2022-01-11T11:51:05-0500][W][41565] subproc_checkTimeLimit():522 pid=41570 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41564] subproc_checkTimeLimit():522 pid=41571 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41565] subproc_checkTimeLimit():522 pid=41570 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41563] subproc_checkTimeLimit():522 pid=41568 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41565] subproc_checkTimeLimit():522 pid=41570 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41564] subproc_checkTimeLimit():522 pid=41571 has already been signaled due to timeout. Killing it with SIGKILL
 2022-01-11T11:51:05-0500][W][41563] subproc_checkTimeLimit():522 pid=41568 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41564] subproc_checkTimeLimit():522 pid=41571 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41565] subproc_checkTimeLimit():522 pid=41570 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41563] subproc_checkTimeLimit():522 pid=41568 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41564] subproc_checkTimeLimit():522 pid=41571 has already been signaled due to timeout. Killing it with SIGKILL
[2022-01-11T11:51:05-0500][W][41565] subproc_checkTimeLimit():522 pid=41570 has already been signaled due to timeout. Killing it with SIGKILL
…-- continue --…

However, it appears the crashed threads are not able to continue fuzzing and I get the warning above ad vitam aeternam, and no progress can be made anymore. The behavior is the same regardless of --tmout_sigvtalrm.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@CBenoit