Project

General

Profile

Bug #1395

Online tests failing on test server

Added by Rogers, Chris almost 8 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Category:
Online reconstruction
Target version:
Start date:
09 January 2014
Due date:
% Done:

100%

Estimated time:
Workflow:
New Issue

Description

Several cascading issues:

  • rabbitmq-server was not running
  • celeryd processes lurking in the background and not dying properly
  • celery memory leak causing the tests to restart (resulting in endless loop)
  • reference reconstruction data sets appear to be out-of-spec

Files

imageReducePyCkovPlot_compared.root (153 KB) imageReducePyCkovPlot_compared.root Rogers, Chris, 09 January 2014 10:05
#1

Updated by Rogers, Chris almost 8 years ago

So I restarted rabbitmq-server and killed the lurking celeryd processes. This fixes the first issues, at least for now.

At r1021 I reduced the size of the test job to prevent the memory leak in celery from forcing a restart of the test. Now on my local machines tests are failing on KS test out-of-bounds.

#2

Updated by Rogers, Chris almost 8 years ago

Attached the failing KS tests, running on my local machine. Note all test failures in Ckov. Test output below:

Printing output to  /home/cr67/MAUS/maus_merge_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyTOFPlot_compared.root
ht01 KolmogorovTest 7.0 7.0 1.0 Pass
spx0 KolmogorovTest 89.0 89.0 1.0 Pass
spx1 KolmogorovTest 89.0 89.0 1.0 Pass
spx2 KolmogorovTest 89.0 89.0 1.0 Pass
spy0 KolmogorovTest 69.2 69.2 1.0 Pass
spy1 KolmogorovTest 69.2 69.2 1.0 Pass
spy2 KolmogorovTest 69.2 69.2 1.0 Pass
hnsp_0 KolmogorovTest 498.0 498.0 1.0 Pass
hnsp_1 KolmogorovTest 393.0 393.0 1.0 Pass
hnsp_2 KolmogorovTest 602.0 602.0 1.0 Pass
hpmthits000 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits001 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits010 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits011 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits100 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits101 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits110 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits111 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits200 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits201 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits210 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits211 KolmogorovTest 105.5 105.5 1.0 Pass
hslabhits00 KolmogorovTest 130.8 130.8 1.0 Pass
hslabhits01 KolmogorovTest 105.5 105.5 1.0 Pass
hslabhits10 KolmogorovTest 130.8 130.8 1.0 Pass
hslabhits11 KolmogorovTest 105.5 105.5 1.0 Pass
hslabhits20 KolmogorovTest 130.8 130.8 1.0 Pass
hslabhits21 KolmogorovTest 105.5 105.5 1.0 Pass
hspxy_0 KolmogorovTest 8.0 8.0 1.0 Pass
hspxy_1 KolmogorovTest 24.0 24.0 1.0 Pass
Printing output to  /home/cr67/MAUS/maus_merge_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyCkovPlot_compared.root
htof KolmogorovTest 12.0 12.0 1.0 Pass
tof_A KolmogorovTest 2.0 2.0 0.829120228286 Pass
tof_B KolmogorovTest 3.0 3.0 0.979272739059 Pass
hPMT1 KolmogorovTest 196.0 547.0 1.09408601943e-26 Failure
hPMT2 KolmogorovTest 196.0 537.0 3.21041603796e-23 Failure
hPMT3 KolmogorovTest 484.0 542.0 0.126613734547 Pass
hPMT4 KolmogorovTest 199.0 496.0 6.77215243484e-19 Failure
hPMT5 KolmogorovTest 506.0 500.0 1.0 Pass
hPMT6 KolmogorovTest 434.0 516.0 0.0234674232908 Failure
hPMT7 KolmogorovTest 543.0 533.0 1.0 Pass
hPMT8 KolmogorovTest 186.0 491.0 9.61630203037e-22 Failure
harr_time1 KolmogorovTest 4.0 4.0 1.0 Pass
harr_time2 KolmogorovTest 3.0 3.0 1.0 Pass
harr_time3 KolmogorovTest 3.0 3.0 1.0 Pass
harr_time4 KolmogorovTest 5.0 5.0 0.999999999915 Pass
harr_time5 KolmogorovTest 5.0 5.0 1.0 Pass
harr_time6 KolmogorovTest 8.0 7.0 1.0 Pass
harr_time7 KolmogorovTest 6.0 6.0 1.0 Pass
harr_time8 KolmogorovTest 5.0 5.0 0.999999986305 Pass
test file: /home/cr67/MAUS/maus_merge_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyCkovPlot.root passes: False
test file: /home/cr67/MAUS/maus_merge_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyTOFPlot.root passes: True
F
#3

Updated by Rogers, Chris almost 8 years ago

  • Assignee changed from Rogers, Chris to Rajaram, Durga

Just looking at the plots - it looks like a lot more PMT hits are coming out with "0" charge. I don't know why this might be - Durga, I think you need to check this one as you are closest to the Ckov development.

#4

Updated by Rogers, Chris almost 8 years ago

This now fails on the KS tests in Ckov, even on the test server (#51)

http://test.mice.rl.ac.uk/job/MAUS_online/51/changes

======================================================================
FAIL: Check that analyze_data_online makes good histos for full run
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/MAUS_online/tests/integration/test_analyze_data_online/test_analyze_data_online.py", line 163, in test_root_histos
    self.assertEquals(test_pass, True)
AssertionError: False != True
-------------------- >> begin captured stdout << ---------------------
Linking /home/jenkins/workspace/MAUS_online/third_party/install/share/test_data/test_data.cat to /home/jenkins/workspace/MAUS_online/tmp/test_analyze_data_online/test_data.cat
/home/jenkins/workspace/MAUS_online/tmp/test_analyze_data_online//test_data.cat_histos//raw False /home/jenkins/workspace/MAUS_online/tmp/test_analyze_data_online//test_data.cat_histos/ True
Running analyze online
Point your browser at http://localhost:9000/maus/
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 

Printing output to  /home/jenkins/workspace/MAUS_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyTOFPlot_compared.root
ht01 KolmogorovTest 7.0 7.0 1.0 Pass
spx0 KolmogorovTest 89.0 89.0 1.0 Pass
spx1 KolmogorovTest 89.0 89.0 1.0 Pass
spx2 KolmogorovTest 89.0 89.0 1.0 Pass
spy0 KolmogorovTest 69.2 69.2 1.0 Pass
spy1 KolmogorovTest 69.2 69.2 1.0 Pass
spy2 KolmogorovTest 69.2 69.2 1.0 Pass
hnsp_0 KolmogorovTest 489.0 498.0 1.0 Pass
hnsp_1 KolmogorovTest 384.0 393.0 1.0 Pass
hnsp_2 KolmogorovTest 593.0 602.0 1.0 Pass
hpmthits000 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits001 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits010 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits011 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits100 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits101 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits110 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits111 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits200 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits201 KolmogorovTest 130.8 130.8 1.0 Pass
hpmthits210 KolmogorovTest 105.5 105.5 1.0 Pass
hpmthits211 KolmogorovTest 105.5 105.5 1.0 Pass
hslabhits00 KolmogorovTest 130.8 130.8 1.0 Pass
hslabhits01 KolmogorovTest 105.5 105.5 1.0 Pass
hslabhits10 KolmogorovTest 130.8 130.8 1.0 Pass
hslabhits11 KolmogorovTest 105.5 105.5 1.0 Pass
hslabhits20 KolmogorovTest 130.8 130.8 1.0 Pass
hslabhits21 KolmogorovTest 105.5 105.5 1.0 Pass
hspxy_0 KolmogorovTest 8.0 8.0 1.0 Pass
hspxy_1 KolmogorovTest 24.0 24.0 1.0 Pass
Printing output to  /home/jenkins/workspace/MAUS_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyCkovPlot_compared.root
htof KolmogorovTest 12.0 12.0 1.0 Pass
tof_A KolmogorovTest 2.0 2.0 0.829120228286 Pass
tof_B KolmogorovTest 3.0 3.0 0.979272739059 Pass
hPMT1 KolmogorovTest 192.0 547.0 9.58365430666e-27 Failure
hPMT2 KolmogorovTest 195.0 537.0 1.19012776641e-22 Failure
hPMT3 KolmogorovTest 476.0 542.0 0.118861055944 Pass
hPMT4 KolmogorovTest 193.0 496.0 1.16591198559e-18 Failure
hPMT5 KolmogorovTest 498.0 500.0 0.999999999996 Pass
hPMT6 KolmogorovTest 425.0 516.0 0.021073812313 Failure
hPMT7 KolmogorovTest 535.0 533.0 1.0 Pass
hPMT8 KolmogorovTest 182.0 491.0 9.07227799321e-22 Failure
harr_time1 KolmogorovTest 4.0 4.0 1.0 Pass
harr_time2 KolmogorovTest 3.0 3.0 1.0 Pass
harr_time3 KolmogorovTest 3.0 3.0 1.0 Pass
harr_time4 KolmogorovTest 5.0 5.0 0.999999999915 Pass
harr_time5 KolmogorovTest 5.0 5.0 1.0 Pass
harr_time6 KolmogorovTest 8.0 7.0 1.0 Pass
harr_time7 KolmogorovTest 5.0 6.0 1.0 Pass
harr_time8 KolmogorovTest 5.0 5.0 0.999999986305 Pass
test file: /home/jenkins/workspace/MAUS_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyCkovPlot.root passes: False
test file: /home/jenkins/workspace/MAUS_online/tmp/test_analyze_data_online/test_data.cat_histos/end_of_run/4235/imageReducePyTOFPlot.root passes: True

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 63 tests in 2933.539s

FAILED (SKIP=16, failures=1)
 Kolmo Prob  h1 = tof01_ref, sum bin content =177  effective entries =177
 Kolmo Prob  h2 = tof01, sum bin content =218  effective entries =218
 Kolmo Prob     = 0.990583, Max Dist = 0.0443943
Build step 'Execute shell' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Sending e-mails to: chris.rogers@stfc.ac.uk durga@fnal.gov
Finished: FAILURE
#5

Updated by Rajaram, Durga almost 8 years ago

It turns out that some changes to the unpacker [ change to the pedestal data type from int to double ] changed the charge calculation, and hence the pmt plots.
I have updated the ckov reference root file and it passes on my machine.
Will have to see how it goes on the test server

#6

Updated by Rogers, Chris almost 8 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Fixed in r1023 (though there were a few more revisions round then that may have fixed some "incorrect commit" type issues

#7

Updated by Rajaram, Durga over 7 years ago

  • Target version changed from Future MAUS release to MAUS-v0.7.7

Also available in: Atom PDF