Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test stability #1426

Merged
merged 19 commits into from Jan 30, 2017
Merged

Improve test stability #1426

merged 19 commits into from Jan 30, 2017

Conversation

tagomoris
Copy link
Member

@tagomoris tagomoris commented Jan 19, 2017

To make tests stable on CI services.

Based on #1425, and fixes #1404.

@tagomoris tagomoris self-assigned this Jan 19, 2017
@tagomoris tagomoris force-pushed the improve-test-stability-201701 branch 2 times, most recently from f483fe0 to f127a16 Compare January 20, 2017 07:32
@tagomoris tagomoris force-pushed the improve-test-stability-201701 branch 17 times, most recently from d822a28 to 0e8a93a Compare January 26, 2017 09:01
@tagomoris
Copy link
Member Author

Currently I'm willing not to fix these 2 failures sometimes occurs.

@tagomoris tagomoris force-pushed the improve-test-stability-201701 branch 3 times, most recently from 334c234 to e16ca37 Compare January 27, 2017 10:17
@tagomoris
Copy link
Member Author

I'm expecting that this work is almost done.
I'll merge this change after 10x continuous CI Green (or Red only with known patterns).

@tagomoris tagomoris force-pushed the improve-test-stability-201701 branch from ff18e88 to 1de34ba Compare January 27, 2017 16:11
@tagomoris tagomoris force-pushed the improve-test-stability-201701 branch 5 times, most recently from 068c409 to 3015781 Compare January 29, 2017 11:21
@tagomoris
Copy link
Member Author

tagomoris commented Jan 29, 2017

3 (in 10) failures on CI (all on Travis CI), with 5 job. Known 2 (test_child_process about kill), unknown 3.
It can be tested by command line: bundle exec rake test TESTOPTS=-v TEST=test/{plugin/test_in_dummy.rb,plugin/test_output_as_buffered_retries.rb,plugin/test_output_as_buffered.rb,plugin_helper/test_child_process.rb}

DummyTest: 
test: value of auto increment key is not suspended after stop-and-start:F
Failure: test: value of auto increment key is not suspended after stop-and-start(DummyTest::doesn't suspend internal counters in default)
/home/travis/build/fluent/fluentd/test/plugin/test_in_dummy.rb:129:in `block (2 levels) in <class:DummyTest>'
     126:       end
     127: 
     128:       first_id2 = d2.events.first[2]['id']
  => 129:       assert_equal 0, first_id2
     130: 
     131:       assert !File.exist?(File.join(TEST_PLUGIN_STORAGE_PATH, 'json', 'test-01.json'))
     132:     end
<0> expected but was
<1>
BufferedOutputRetryTest: 
test: retry_timeout and retry_max_times will be ignored if retry_forever is true for exponential backoff:
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
BufferedOutputTest: 
 test: #write is called every time for each emits, and buffer chunk is purged:F
Failure: test: #write is called every time for each emits, and buffer chunk is purged(BufferedOutputTest::buffered output feature without any buffer key, flush_mode: immediate)
/Users/travis/build/fluent/fluentd/test/plugin/test_output_as_buffered.rb:756:in `block (3 levels) in <class:BufferedOutputTest>'
     753:           sleep 0.1 until @i.buffer.queue.size == 0 && @i.buffer.dequeued.size == 0
     754:         end
     755: 
  => 756:         assert_equal rand_records, ary.size
     757:         ary.reject!{|e| true }
     758:       end
     759:     end
/Users/travis/build/fluent/fluentd/test/plugin/test_output_as_buffered.rb:742:in `times'
/Users/travis/build/fluent/fluentd/test/plugin/test_output_as_buffered.rb:742:in `block (2 levels) in <class:BufferedOutputTest>'
<5> expected but was
<0>
ChildProcessTest: 
 test: can return exit status with signal code for child process killed by signal using on_exit_callback:F
Failure: test: can return exit status with signal code for child process killed by signal using on_exit_callback(ChildProcessTest::on_exit_callback is specified)
/home/travis/build/fluent/fluentd/test/plugin_helper/test_child_process.rb:683:in `block (2 levels) in <class:ChildProcessTest>'
     680:       # This test sometimes fails on TravisCI
     681:       #    with [nil, 11] # SIGSEGV
     682:       # or with [1, nil]  # ???
  => 683:       assert_equal [nil, 3, true, ""], [exit_status.exitstatus, exit_status.termsig, File.exist?(@temp_path), str] # SIGQUIT
     684:       # SIGSEGV looks a kind of BUG of ruby...
     685:     end
     686: 
<[nil, 3, true, ""]> expected but was
<[1, nil, true, ""]>
diff:
? [1, nil, 3, true, ""]

Failure: test: can return exit status with signal code for child process killed by signal using on_exit_callback(ChildProcessTest::on_exit_callback is specified)
/home/travis/build/fluent/fluentd/test/plugin_helper/test_child_process.rb:683:in `block (2 levels) in <class:ChildProcessTest>'
     680:       # This test sometimes fails on TravisCI
     681:       #    with [nil, 11] # SIGSEGV
     682:       # or with [1, nil]  # ???
  => 683:       assert_equal [nil, 3, true, ""], [exit_status.exitstatus, exit_status.termsig, File.exist?(@temp_path), str] # SIGQUIT
     684:       # SIGSEGV looks a kind of BUG of ruby...
     685:     end
     686: 
<[nil, 3, true, ""]> expected but was
<[nil, 11, true, ""]>
diff:
? [nil, 3 , true, ""]
?       11           

@tagomoris
Copy link
Member Author

There are no failures on TravisCI in 20 times CI (AppVeyor is still running).
I'll re-edit commits and merge into master.

@tagomoris tagomoris force-pushed the improve-test-stability-201701 branch from 90c3dbd to c4624cf Compare January 30, 2017 02:32
@tagomoris
Copy link
Member Author

Greeeeeeeen!

@tagomoris tagomoris merged commit d7182c5 into master Jan 30, 2017
@tagomoris tagomoris deleted the improve-test-stability-201701 branch January 30, 2017 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add timeouts to wait in shutting down in tests
1 participant