Bug #1157

Online reconstruction fails

Added by Rogers, Chris over 11 years ago. Updated over 11 years ago.

Richards, Alexander
Online reconstruction
Target version:
Start date:
14 October 2012
Due date:
% Done:


Estimated time:
Under Review


Cross posted to elog...

  • Reported error was that MAUS was making histograms but failing to produce any hits.
  • Assumed some problem with data structure so applied merge with trunk to online code. Still no luck.
  • Made a test_analyze_data_online at tests/integration/test_analyze_online/; hacked that inputter at bin/online/ so that it uses the offline inputter and bin/ so that it specifies a filed number and location. Downloaded data 04168.tar and attempted to reconstruct. No luck, and no daq_data tree at all was created.
  • Tried a different data file; 04010.tar. Still no luck. Prodded InputCppDAQData some - looks like everything is working okay but the line
    didn't seem to fill the "daq_data" branch.
  • Maybe there is something funny about single station tests. Tried 03513 then 03512 data files from March run. 03512 produced daq_data branch but still no histograms - tof reducer complains "No recon_events branch" or similar
  • Added some debugging output into the MapCppTOFDigits and ran input/transform/OutputCppRoot in single threaded mode. MapCppTOFDigits claims that it is indeed producing data...
  • That's as far as I got - I put the code back into multithreaded mode and InputCppOnlineDAQ.
  • So somehow it looks like it is producing reconstruction data but this is getting eaten somewhere...
    Sorry, that's as far as I got. It looks like we will have to do some more debugging next week...

Files (2.89 KB) Rajaram, Durga, 16 October 2012 05:46

Updated by Taylor, Ian over 11 years ago

During today's shift, I tracked down a part of the error. The software was attempting to read from DAQ_hostname = 'miceraid1a', which should now be 'miceraid5'. This allowed me to load data during runs, but still produced errors. I am looking at this, but hindered but lack of familiarity with the code.



Updated by Taylor, Ian over 11 years ago

Getting one error from MAUS.MapPyCkov().

Otherwise, I was briefly able to reconstruct some TOF information, but now no beam...



Updated by Rogers, Chris over 11 years ago

  • Category set to Online reconstruction
  • Assignee set to Richards, Alexander
  • Target version set to Future MAUS release

Alex, I think you should coordinate this - online is now your beast. If there is a problem in one of the reconstruction routines, then please bump it up to the relevant dev (ask if you aren't sure).

Online not working is a serious problem and I think we need to understand what went wrong and produce a post mortem type document. Understand how the system failed both at a code level and also at a management level (i.e. what are the systems that we need to put in place to make sure this doesn't happen again).


Updated by Rajaram, Durga over 11 years ago

Tried the following:
trunk rev# 829, using [ from onrec01:MAUS/maus/bin/online/ ]
changed Offline=True in the script
ran on 03513.000
  1. output=OutputCppRoot()
    no data in output root tree.
    I got the following errors while running
    Traceback (most recent call last):
      File "/home/durga/mtest/merge/src/common_py/", line 159, in HandleCppException
    ErrorHandler.CppError: In branch recon_events
    In branch part_event_number
    Missing required branch part_event_number converting json->cpp at PointerItem::SetCppChild
  2. with output=OutputPyJSON()

the errors go away; output json file seems fine [ has space points]

  1. with output=OutputPyImage(), reducer=ReducePyTOFPlot()

no errors & tof histograms are filled

  1. added MapPyReconSetup() before the other mappers & ouput=OutputCppRoot()

no errors & output root tree is filled
modified script attached (btw, bin/analyze_data_offline has MapPyReconSetup() by default)


Updated by Richards, Alexander over 11 years ago

This sounds like you have found the problem? As I am not very familiar with the data structure I can only give my two cents from the perspective of understanding generally how the cppRoot output is working. The error that you posted looks to me like during the conversion from Json to Cpp one of the processors decided that the Json document should contain the element `part_event_number' and either it couldn't be found or wasn't being set up in the cpp data structure.

I assume therefore that adding MapPyReconSetup() has created the correct entry in the Json hence the problem has disappeared.



Updated by Rogers, Chris over 11 years ago

Alex can you commit a fix to the online recon branch? I would still like some thoughts about how we should properly manage the update process...


Updated by Richards, Alexander over 11 years ago

You want me to commit the modified script as is or revert Offline=False first?



Updated by Rogers, Chris over 11 years ago

Please make offline=false (in fact you can probably remove the offline/online thing because we can use the file -> socket functionality that Yordan mentioned yesterday).


Updated by Richards, Alexander over 11 years ago

In that case would it be better to revert the whole file but just add MapPyReconSetup() and OutputCppRoot() since most of the additions seem based on this off/online switch?



Updated by Rogers, Chris over 11 years ago

Sounds fine


Updated by Richards, Alexander over 11 years ago

Fix in. I'm giving some thought about the best practise for using the control room branch.



Updated by Richards, Alexander over 11 years ago

  • Workflow changed from New Issue to Under Review

Updated by Rogers, Chris over 11 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Updated by Rogers, Chris over 11 years ago

  • Target version changed from Future MAUS release to MAUS-v0.4.1

Also available in: Atom PDF