This is supported by the existing online running implementation. It is not well documented how to do this in offline mode, and some code development could make it more straightforward. Note that there will be some fiddling with the way the online running works as part of #1312

Hi Durga, you mentioned at the CM there was a way of hacking this by spoofing MAUS into thinking it was running online, could you explain how I go about this? Thanks.

Hi Adam -- thanks for the reminder. I've created an issue #1426 for updating the documentation for online deployment.
Presumably you're interested in running simulation and/or offline reconstruction?

Durga, would you like me to suggest something here? e.g. a bastardised online script?

Great!

Also giving a false negative message saying input-transform failed though there are no errors in the log.

Yes, the script just looks for existence of a process, doesn't check the return code. In principle in the control room the reconstruction runs forever and it is an error if any process (e.g. reconstruction) stops running. If you run e.g. test_analyze_data_online.py it also makes the same error message when the data runs out, and this is okay.

Okay, I will have a look at your script...

Inspecting the celery log shows that some tasks failed because of a hard time limit of 10s

Could try CELERYD_TASK_TIME_LIMIT parameter, which I guess should be added to src/common_py/mauscelery/celeryconfig.py see e.g. http://docs.celeryproject.org/en/latest/configuration.html#celeryd-task-time-limit

The reduce_output.log shows that the outputter was sent a SIGINT.
Not sure who sent the signal. Maybe when one of the workers finishes [ and the other is still working], the outputter is sent an interrupt?

That is exactly correct. If the reconstruction dies, the outputter is supposed to finish the current data and then end. Parallel script should poll workers every 10 seconds and make a line in the terminal output like "ouputter is still working, see logs".

The relevant handling is in src/common_py/framework/merge_output.py:

• line 343 _execute_inner_loop receives a SIGINT, sets keyboard_interrupt = True
• inner loop keeps going until a StopIteration (raised on no more events), which is raised up to line 352; if keyboard_interrupt == True the _execute_inner_loop function exits returning False
• execute keeps running unless _execute_inner_loop returns False, in which case the execution ends and we enter the finally statement (end_of_job, etc)

Distributed_offline seems to lose events.

My experience of setting up infrastructure for ROOT docstore shows that about 1 % of jobs get lost in passing over the socket, even when running on the same PC (localhost). I assumed Celery would have better efficiency, but maybe not? I have not got much trust in Celery, it seems quite flaky.

Ps: there is also a type_of_dataflow called multi_process that may be a better route to go down. It was in the test tests/integration/test_distributed_processing/_test_multithreaded_vs_single_threaded.py but didn't work reliably so I commented it out. This implementation uses celery for the task queue but holds reconstructed spills in memory (rather than placing them in mongodb). The problem I had was that spills were reconstructed twice or not-at-all. At least there the infrastructure is only celery, so it might be easier to handle.

• InputPySpillGenerator - MapPyBeamGenerator - ReducePyDoNothing - OutputCppRoot;
• utility script to split the root file;
• InputCppRoot - MapCppSimulation - MapPyRecon - etc - ReducePyDoNothing - OutputCppRoot
  where "utility script to split the root file" needs to be written

Deterministic MC running would be important for me, so that when I vary parameters and look at the results, I know I am comparing apples with apples. Could this be solved by considering a multi-threaded approach, instead of / in addition to full distributed processing?

Also, I believe Chris Hunt has already got a hack of the ROOT file stitching method working. Not the most elegant solution, but it works.

Ps: there is also a type_of_dataflow called multi_process that may be a better route to go down.

I started with multi_process initially but found that the output had only the header and footer and was missing actual spill data. Will look more closely. Losing events is worrisome though especially if we're looking at efficiencies and reproducibility

A more lightweight solution would be a stitch/unstitch on disk, rather than in memory - so we run 10 jobs with different random seeds and then stitch them together at the end, without having to worry about task queues, messaging, blah blah, all of the pain that comes from multiprocessing inline.

Or run a trio of jobs like

• InputPySpillGenerator - MapPyBeamGenerator - ReducePyDoNothing - OutputCppRoot;
• utility script to split the root file;
• InputCppRoot - MapCppSimulation - MapPyRecon - etc - ReducePyDoNothing - OutputCppRoot
  where "utility script to split the root file" needs to be written

The memory footprint will be large, and there's no sharing of common resources [ geometry for e.g. ], but from an implementation point of view running n jobs 'grid-like' and splitting/stitching might be the most straightforward. If we want true multithreading at the event-level it'll take some work.

I see that GEANT4 has some MPI-based parallelization examples in examples/extended/parallel/
http://geant4.web.cern.ch/geant4/UserDocumentation/Doxygen/examples_doc/html/Examples_MPI.html
Interesting..

Fine to spend some time trying to bludgeon the existing system. I would be very nervous to implement multithreading or anything more fancy than what we have right now. Let's say, this is a low priority task (we can probably win more with existing multiprocessing and some optimisation work, and we can't staff anything fancier - physics comes first right now).