Bug #1330
Online reconstruction not catching change of run number in histogram production.
100%
Description
On rare occasions, after stopping and starting the DAQ, the online reconstruction will fail to pick up the change in run number.The histograms are not updated and the reducer log file does not report the start of the next run. Restarting the Online Recon solves the issue.
Unfortunately the log files are overwritten after restarting the Online Recon and as such, none have been saved. However the issue has been noted during the run change over 4993-4994 - the contents of "scalers.json" reports:
{ "run_number": 4993, "maus_event_type": "RunFooter"}
During the running period of 2nd to 5th August this behavior was recorded 3 times.
Please append log files and run numbers if found to happen again!
Files
Related issues
Updated by Hunt, Christopher over 10 years ago
- File celeryd.log celeryd.log added
- File maus-input-transform.log maus-input-transform.log added
- File maus-web.log maus-web.log added
- File mongodb.log mongodb.log added
- File reconstruct_daq_ckov_reducer.log reconstruct_daq_ckov_reducer.log added
Occured during change over from run 4996 to 4997, approx 1635 5th August. Log files are attached.
Updated by Hunt, Christopher over 10 years ago
- File reconstruct_daq_scalars_reducer.log reconstruct_daq_scalars_reducer.log added
- File reconstruct_daq_tof_reducer.log reconstruct_daq_tof_reducer.log added
- File reconstruct_monitor_reducer.log reconstruct_monitor_reducer.log added
Missed a couple of files.
Updated by Rogers, Chris over 10 years ago
Time stamps indicate that the reconstruction has not updated log files since 16:35. It is now 16:50 and data taking continues. maus-input-transform.log has received one spill from 4997 and then hung, indicating a hang up on the death(...) process...
Updated by Rogers, Chris over 10 years ago
- File micewww_target.tar.gz micewww_target.tar.gz added
- Target version set to Future MAUS release
Updated by Rogers, Chris about 10 years ago
Looks like one of the end of run celery.tasks is getting stuck in PENDING. I will add a time out and see if that fixes things.
Updated by Rogers, Chris about 10 years ago
I tried adding a timeout, for some reason it didn't take. So at the end of a run, I sleep for 10 seconds (hard-coded) then by hand kill all remaining reconstruction events. This is tested at integration level but I would really like to add unit tests to input_transform.py because, to be honest, getting this working is horrid without unit tests.
Updated by Rogers, Chris about 10 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
Fixed in r990
Updated by Rajaram, Durga about 10 years ago
- Target version changed from Future MAUS release to MAUS-v0.7.1