On onrec03 - I checked out what I believe is the best code,

cd MAUS
bzr checkout bzr+ssh://bazaar.launchpad.net/~christopher-hunt08/maus/maus_scifi_recon_devel .offline_recon_tracker

I am now going through the install procedure. Note

• we are taking data tonight, it would be great to have this set up by this evening.
• I would like to move some code from lp:~chris-rogers/maus/1312b across as well;
• this has the InputPySocket and OutputPySocket stuff required to get mongodb out of the online reconstruction. I have been having trouble testing the online reconstruction which has blocked a bit (and I didn't manage to find the time to workaround my problems)...

I got a test fail. I am ignoring it:

======================================================================
FAIL: Check that PIDVarC pid
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mice/MAUS/.offline_recon_tracker/build/test_MapCppGlobalPID.py", line 214, in test_KL_PID
    self.assertTrue('recon_events' in spill_out)
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
{u'spill_number': 0, u'errors': {u'MapCppGlobalPID': u"<class 'ErrorHandler.CppError'>: In branch recon_events\nIn branch kl_event\nIn branch kl_cell_hits\nIn branch kl\nIn branch err_x\nMissing required branch err_x converting json->cpp at ValueItem::_SetCppChild"}, u'daq_event_type': u'', u'maus_event_type': u'Spill', u'run_number': 0}

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: Check that process fills global events from detector data
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mice/MAUS/.offline_recon_tracker/build/test_MapCppGlobalReconImport.py", line 98, in test_fill_Global_Event
    self.assertTrue('recon_events' in spill_out)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 587 tests in 346.813s

FAILED (SKIP=1, failures=2)
Traceback (most recent call last):
  File "/home/mice/MAUS/.offline_recon_tracker/src/common_py/docstore/root_document_store/_socket_manager.py", line 274, in _message_handler
    message = ROOT.TMessage()
AttributeError: FAIL Failed unit tests. Fatal error - aborting

Yes, they are known, Chris is working on the fix. Once he has got that sorted I will merge his code into the trunk and make a release.

Okay, I think everything is installed and working. I added a small script to do the offline recon, but using InputCppDAQOnlineData; this looks to be working okay, but I didn't have any run data to test with.

Nb: the online recon is not working yet, there is a small hiccup in some of the code... maybe... but the offline recon does appear to be working... but not very well tested... I will keep working on it.

What is not working with online recon?
I can look as well if it'll help

I am using /home/mice/MAUS/.offline_recon_tracker on onrec03 and pushing to bzr+ssh://bazaar.launchpad.net/~mice-lcr/maus/offrec/

I made a small offline job in bin/utilities/analyze_data_fast_turnaround/

Tip from Durga, add

                    '-SciFiDigitizationNPECut=2.0',
                    '-SciFiPRHelicalOn=False',

Seems to be required to make recon work in a reasonable time!

These are the cards that were given by the tracker group -- and they are in the CDB BatchIteration# = 2

simulation_geometry_filename = "Stage4.dat" 
reconstruction_geometry_filename = "Stage4.dat" 
SciFiDigitizationNPECut = 2.0
SciFiPRHelicalOn = False

I have placed these cards in

onrec03: MAUS/.offline_recon_tracker/bin/utilities/analyze_data_fast_turnaround/reco.cards

With field on, I imagine the SciFiPRHelicaolOn flag would need to be removed. I suggested to Adam and Chris H. at today's MAUS meeting that they set the flag on/off by querying the CDB and figuring out if the magnets were on

Humm, maybe I screwed something up...

I believe you're right. It does come from the tracker recon.
For one the output file size is ~50% larger (25 MB/100 evts trunk, 40 MB/100 events in the offline_recon_tracker branch)

I haven't looked into the guts of the output to see if for instance he finds more spacepoints now, etc.

If the reconstruction in Chis Hunt's branch is doing the 'right thing' and hence taking more time, it might be something we have to live with,
but would be good to know if it's by design.

Maybe it's just something stupid like two people trying to run at the same time on the same machine (even against the same file?).

I tried building a onrec03:MAUS/.maus_online_reco_mods, explicitly attempting to exclude Recon modifications but I didn't see any improvement. I do have online stuff in the code I am running, so I still don't rule out an issue there, although log file hints at Recon/SciFi being the issue. At least, the InputCppDAQOnline is set up to spit out a DAQ event every 0.5 seconds. For the record, I have reconstructed 210 DAQ events in 1009 seconds.

Just for information, I have left the new online recon running on onrec03 (.maus_online_reco_mods, which takes .maus_trunk as third party), with the production online recon running on onrec02. I think I will call it a night, and check log files in the morning. I might make it in for the end of the run tomorrow...

If we believe that Chris Hunt's branch

lp:~christopher-hunt08/maus/maus_scifi_recon_devel

bin/utilities/analyze_data_fast_turnaround/analyze_data_fast_turnaround.py

Hm...just ran 200 events from the scifi_recon_devel branch and tried loading it up from my trunk install. Able to browse & plot but the ROOT classdefs are different which might make physics shifter/analysis iffy (?)

Ok, I found the reason for the speed and output filesize differences between online_reco_mods & maus_trunk

[mice@miceonrec03 .maus_online_reco_mods]$ diff src/common_py/ConfigurationDefaults.py ../.maus_trunk/src/common_py/ConfigurationDefaults.py|grep SciFi
< SciFiClustExcept = 10000 # exception is thrown # Changed to accomodate bug in tracker reconstruction, Chris Heidt/Rogers, 21-06
> SciFiClustExcept = 100 # exception is thrown
[mice@miceonrec03 .maus_online_reco_mods]$

100 is also the parameter value in Chris Hunt's scifi_recon_devel branch, so I believe that's what it should be.

Have now modified

onreco3:.maus_online_reco_mods/src/common_py/ConfigurationDefaults.py

The speeds & output are now consistent between the online, scifi_devel and trunk branches

Tried running

python bin/utilities/analyze_data_fast_turnaround/analyze_data_fast_turnaround.py -configuration_file bin/utilities/analyze_data_fast_turnaround/config.py

Digging...

This means that you have a memory leak. There is a datacard which you can vary to fix it (I will head in now and look up the name when I get there) or you can recompile and change the #ifdef in src/common_cpp/DataStructure/Data.cc

It is strange.

After the run ended, I copied over all the 7273.00[0-6] files to onrec03:devel/data
and ran bin/analyze_data_offline.
It ran through all 6 chunks without any data reference count complaint.
Why does it complain when running InputCppDAQOnlineData? Hmm.

There is also a problem with the tracker readout.
There is an error from the VLSB on every event which caused an unpacking exception which means the entire spill gets discarded.

This is the reason the shifters reported empty onrec plots.

To "recover" the non-tracker i.e. TOF etc data, one has to reconstruct with Enable_VLSB_Unpacking=False

I have made an entry in the logbook
https://micewww.pp.rl.ac.uk/elog/MICE+Log/3347

I wanted to run the fast turnaround to at least see data from other detectors. To this end, I created a config

bin/utilities/analyze_data_fast_turnaround/config_notracker.py

[mice@miceonrec03 .maus_online_reco_mods]$ nohup python bin/utilities/analyze_data_fast_turnaround/analyze_data_fast_turnaround.py -configuration_file bin/utilities/analyze_data_fast_turnaround/config_notracker.py >& analyze_data_fast_turnaround_notracker.log &
[1] 4576

No RefCount errors so far [ ~1230 events & going ].

The output goes in maus_output_7274.root & the log file is as in the command listed above.

Could it be that it was happening before with the tracker errors because when an unpacking error happens, the error-trapping is not deleting a data ref when it should?

In single_thread.py I check for bad-data-input and if bad input, I disregard a run# change [ because InputCppDAQ does not spill or run# when an error happens ] but couldn't see a reason to delete data ref. I maybe missed something..

On the bright side, the fast turnaround reconstruction kept up with real data [ though with the tracker unpacking off ]

I have been running offline reconstruction on onrec03. Have been running against the trunk

MAUS/.maus_trunk

Because there is some field, I decided to turn the SciFiPRHelicalOn flag

SciFiPRHelicalOn=True

MAUS/.maus_trunk/bin/utilities/analyze_data_fast_turnaround/config_helical.py

Requires babysitting right now to re-start reconstruction after a run ends. Need to get PVs from EPICs to indicate end_run/start_run and such. Or poll & wait for the monitor

Rajaram, Durga wrote:

Requires babysitting right now to re-start reconstruction after a run ends. Need to get PVs from EPICs to indicate end_run/start_run and such. Or poll & wait for the monitor

I take that back. The monitor does catch end of run and stops and starts writing to a new xxxx_run#.root file when a new run starts.

So, finally after ~17,000 events, it died with a 'too many data refs' error.

Rajaram, Durga wrote:

So, finally after ~17,000 events, it died with a 'too many data refs' error.

To clarify, run 7289 had ended, and 7290 had automatically started reconstructing, and after a combined 17,000 events, it croaked.

I had started reconstructing 7289 maybe 30 minutes after the run had started. The question is did it catch up and reconstruct everything, or did it start/stop midway. I'll reconstruct that run again by hand to verify.

I re-ran 7289 by hand to pick up the spills I had missed [ because I started running the reconstruction 'late' ]

The reconstructed output from Runs 7285 (RefRun), 7286, 7287, 7288, 7289, 7290 are available on

onrec03:MAUS/.maus_trunk/output/

They were reconstructed with the trunk [ rev. 897 ], with the PRHelical=true, config listed in http://micewww.pp.rl.ac.uk/issues/1717#note-28

Though there's nothing special about running the reconstruction here, would it be possible to burden the physics shifter [ or the tracker group ] to look at these output files to make sure that the MLCR reconstruction works OK?

Need to think how [ automatic compaction/cron rsync? ] & where to [ datamover? IC web? micewww? ] we want to move reconstructed data...

Need to think how [ automatic compaction/cron rsync? ] & where to [

datamover?

Okay, but not easily accessible from big bad world

IC web?

Note this is a GRID site, so we probably buy all of the GRID issues if we try to do this.

micewww?

Not enough space there right now. We discussed running a web proxy over http from heplnm069 in the ppd meeting last week. There is a security advantage in running a web proxy rather than directly mounting heplnm069. There may be a disadvantage in terms of maintenance, I am not quite sure. This is what I would shoot for... maybe we don't want to permanently store recon data here though, I don't think we have enough space for that. Probably we can manage, say, last two centuries (or cron job eats oldest run when the storage runs low).

httpd is already running on heplnm069 and serving yum repos I think.