Feature #725
Unpacker should make one line per spill
0%
Description
In MAUS we assume one line per spill as our data structure. InputCppDaqData however creates three lines per spill - start_of_burst, physics_event, end_of_burst. We need to fix this. Propose some structure like
{
"daq_data":{"start_of_burst":{<DATA>}, "physics_event":{<DATA>}, "end_of_burst":{<DATA>}}
}
Sounds sensible?
Files
Related issues
Updated by Rogers, Chris almost 12 years ago
After phone call with Yordan - the problem with this is that there is not a one to one mapping of event type to spill.
So e.g. there is one start_of_run event per run, etc.
So as a stop gap propose to add filter to the readNextEvent() function in InputCppDAQData. (Edit: filter will include only physics_event). But in long run, consider adding some filter type to the execution structure - for example, the start_of_run will contain the trigger information to which we want access.
Updated by Rogers, Chris almost 12 years ago
- File InputCppDAQData.hh InputCppDAQData.hh added
- File InputCppDAQData.cc InputCppDAQData.cc added
- File InputCppDAQData.i InputCppDAQData.i added
- File test_InputCppDAQData.py test_InputCppDAQData.py added
So I made a patch that addresses this issue - for Yordan to look at. Source code is in my branch at:
http://bazaar.launchpad.net/~chris-rogers/maus/devel/files/head:/src/input/InputCppDAQData/
Changes in:
src/input/InputCppDAQData/InputCppDAQData.cc src/input/InputCppDAQData/InputCppDAQData.hh src/input/InputCppDAQData/InputCppDAQData.i src/input/InputCppDAQData/test_InputCppDAQData.py
diffs attached
Updated by Rogers, Chris almost 12 years ago
- File InputCppDAQData.hh_diff InputCppDAQData.hh_diff added
- File InputCppDAQData.cc_diff InputCppDAQData.cc_diff added
- File InputCppDAQData.i_diff InputCppDAQData.i_diff added
- File test_InputCppDAQData.py_diff test_InputCppDAQData.py_diff added
Diffs attached this time...
Updated by Rogers, Chris almost 12 years ago
Add sample output file (structure only, all electronics unpacking disabled):
{"daq_data":[{"daq_event_type":"end_of_burst","event_data":null,"spill_num":-1}]} {"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":0},{"daq_event_type":"physics_event","event_data":null,"spill_num":0},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":0}]} {"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":1},{"daq_event_type":"physics_event","event_data":null,"spill_num":1},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":1}]} {"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":2},{"daq_event_type":"physics_event","event_data":null,"spill_num":2},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":2}]} {"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":3},{"daq_event_type":"physics_event","event_data":null,"spill_num":3},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":3}]} {"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":4},{"daq_event_type":"physics_event","event_data":null,"spill_num":4},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":4}]} {"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":5},{"daq_event_type":"physics_event","event_data":null,"spill_num":5},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":5}]} {"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":6},{"daq_event_type":"physics_event","event_data":null,"spill_num":6},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6}]}
Updated by Karadzhov, Yordan almost 12 years ago
The problem is that this is not going to work when we have "calibration event".
First because the calibration events come outside of the burst (start_of_burst <-> end_of_burst)
and second because calibration events don't have spill number.
I would suggest instead of having this complicated solution to:
Keep getCurentEvent unchanged.
Change only the name of getNextEvent to getNextDAQEvent.
Make another function called getNextPhysEvent which will skip all other types of event like start_of_burst, end_of_burst.
We may have getNextCalibEvent too.
Make possible the user to decide what to do (use getNextDAQEvent, getNextPhysEvent or getNextCalibEvent).
Updated by Karadzhov, Yordan almost 12 years ago
Can you remind here what is the motivation for the requirement to have only one line per spill.
Updated by Rogers, Chris almost 12 years ago
The whole simulation, reconstruction assumes that we have one json document per spill - the execution loop is something like
- while (input)
- input = get_input()
- map(input)
- waste cpu distributing documents that have no physics event (which then need to be rejected)
- need to make a data splitter to properly replicate the data structure in monte carlo (i.e. we need to add an execution type that splits monte carlo digitisted data into a physics_event, start_of_burst, end_of_burst
- probably in the end we will make a filter type to remove non-useful daq events from the real data structure (i.e. we filter start_of_burst, end_of_burst)
I guess also I just wasn't expecting it to go like that, and I naturally resist change somewhat!
Updated by Rogers, Chris almost 12 years ago
Another try at explaining:
It really makes the high level input/map/reduce framework much more complicated if we cannot guarantee one output per input. So either we are faced with doing some difficult algorithm to remove data from the execution stream or we just have stuff in the data stream that is sort of "interesting" but of not much use (calibration is of course important, but start_of_burst/end_of_burst I think is not really useful).
Updated by Karadzhov, Yordan almost 12 years ago
I do not agree that, we have to change the structure of the real data in order to make it the same as the simulation.
Also I do not agree that, only physics and calibration events will be useful.
That's why, I suggested to make user to decide what to do. This can be controlled by the datacards.
In any case high level applications will do different things when processing phys. events and when processing calib. events.
There is no way to avoid the check of the event type in the beginning before doing anything else.
Updated by Rogers, Chris almost 12 years ago
Fair enough. I agree monte carlo should follow data, not the other way round.
One more thing that occurs - MAUS assumes that reconstruction only ever depends on a single map() call. So you are not allowed, for example, to change some mapper depending on what happens during another map() call. Do we ever need e.g. the reconstruction of the physics_event to know about any of the other daq events?