Project

General

Profile

Feature #725

Unpacker should make one line per spill

Added by Rogers, Chris almost 10 years ago. Updated almost 9 years ago.

Status:
Rejected
Priority:
Normal
Category:
Input
Target version:
Start date:
04 October 2011
Due date:
% Done:

0%

Estimated time:
Workflow:

Description

In MAUS we assume one line per spill as our data structure. InputCppDaqData however creates three lines per spill - start_of_burst, physics_event, end_of_burst. We need to fix this. Propose some structure like

{
"daq_data":{"start_of_burst":{<DATA>}, "physics_event":{<DATA>}, "end_of_burst":{<DATA>}}
}

Sounds sensible?


Files

InputCppDAQData.hh (5.67 KB) InputCppDAQData.hh Rogers, Chris, 12 October 2011 10:51
InputCppDAQData.cc (10.2 KB) InputCppDAQData.cc Rogers, Chris, 12 October 2011 10:51
InputCppDAQData.i (341 Bytes) InputCppDAQData.i Rogers, Chris, 12 October 2011 10:51
test_InputCppDAQData.py (3.41 KB) test_InputCppDAQData.py Rogers, Chris, 12 October 2011 10:51
InputCppDAQData.hh_diff (1.7 KB) InputCppDAQData.hh_diff Rogers, Chris, 12 October 2011 10:52
InputCppDAQData.cc_diff (2.64 KB) InputCppDAQData.cc_diff Rogers, Chris, 12 October 2011 10:52
InputCppDAQData.i_diff (508 Bytes) InputCppDAQData.i_diff Rogers, Chris, 12 October 2011 10:52
test_InputCppDAQData.py_diff (1.93 KB) test_InputCppDAQData.py_diff Rogers, Chris, 12 October 2011 10:52

Related issues

Related to MAUS - Support #592: Code Review InputRealDataClosedKaradzhov, Yordan20 July 2011

Actions
#1

Updated by Rogers, Chris almost 10 years ago

After phone call with Yordan - the problem with this is that there is not a one to one mapping of event type to spill.

So e.g. there is one start_of_run event per run, etc.

So as a stop gap propose to add filter to the readNextEvent() function in InputCppDAQData. (Edit: filter will include only physics_event). But in long run, consider adding some filter type to the execution structure - for example, the start_of_run will contain the trigger information to which we want access.

#2

Updated by Rogers, Chris almost 10 years ago

So I made a patch that addresses this issue - for Yordan to look at. Source code is in my branch at:

http://bazaar.launchpad.net/~chris-rogers/maus/devel/files/head:/src/input/InputCppDAQData/

Changes in:

src/input/InputCppDAQData/InputCppDAQData.cc 
src/input/InputCppDAQData/InputCppDAQData.hh
src/input/InputCppDAQData/InputCppDAQData.i
src/input/InputCppDAQData/test_InputCppDAQData.py

diffs attached

#4

Updated by Rogers, Chris almost 10 years ago

Add sample output file (structure only, all electronics unpacking disabled):

{"daq_data":[{"daq_event_type":"end_of_burst","event_data":null,"spill_num":-1}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":0},{"daq_event_type":"physics_event","event_data":null,"spill_num":0},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":0}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":1},{"daq_event_type":"physics_event","event_data":null,"spill_num":1},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":1}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":2},{"daq_event_type":"physics_event","event_data":null,"spill_num":2},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":2}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":3},{"daq_event_type":"physics_event","event_data":null,"spill_num":3},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":3}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":4},{"daq_event_type":"physics_event","event_data":null,"spill_num":4},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":4}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":5},{"daq_event_type":"physics_event","event_data":null,"spill_num":5},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":5}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":6},{"daq_event_type":"physics_event","event_data":null,"spill_num":6},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6}]}
#5

Updated by Karadzhov, Yordan almost 10 years ago

The problem is that this is not going to work when we have "calibration event".
First because the calibration events come outside of the burst (start_of_burst <-> end_of_burst)
and second because calibration events don't have spill number.

I would suggest instead of having this complicated solution to:
Keep getCurentEvent unchanged.
Change only the name of getNextEvent to getNextDAQEvent.
Make another function called getNextPhysEvent which will skip all other types of event like start_of_burst, end_of_burst.
We may have getNextCalibEvent too.
Make possible the user to decide what to do (use getNextDAQEvent, getNextPhysEvent or getNextCalibEvent).

#6

Updated by Karadzhov, Yordan almost 10 years ago

Can you remind here what is the motivation for the requirement to have only one line per spill.

#7

Updated by Rogers, Chris almost 10 years ago

The whole simulation, reconstruction assumes that we have one json document per spill - the execution loop is something like

  1. while (input)
    1. input = get_input()
    2. map(input)
We assume that inputs are all equivalent. I guess if we have multiple input documents per spill we
  • waste cpu distributing documents that have no physics event (which then need to be rejected)
  • need to make a data splitter to properly replicate the data structure in monte carlo (i.e. we need to add an execution type that splits monte carlo digitisted data into a physics_event, start_of_burst, end_of_burst
  • probably in the end we will make a filter type to remove non-useful daq events from the real data structure (i.e. we filter start_of_burst, end_of_burst)

I guess also I just wasn't expecting it to go like that, and I naturally resist change somewhat!

#8

Updated by Rogers, Chris almost 10 years ago

Another try at explaining:

It really makes the high level input/map/reduce framework much more complicated if we cannot guarantee one output per input. So either we are faced with doing some difficult algorithm to remove data from the execution stream or we just have stuff in the data stream that is sort of "interesting" but of not much use (calibration is of course important, but start_of_burst/end_of_burst I think is not really useful).

#9

Updated by Rogers, Chris almost 10 years ago

We can discuss at phone call later?

#10

Updated by Karadzhov, Yordan almost 10 years ago

I do not agree that, we have to change the structure of the real data in order to make it the same as the simulation.
Also I do not agree that, only physics and calibration events will be useful.
That's why, I suggested to make user to decide what to do. This can be controlled by the datacards.
In any case high level applications will do different things when processing phys. events and when processing calib. events.
There is no way to avoid the check of the event type in the beginning before doing anything else.

#11

Updated by Rogers, Chris almost 10 years ago

Fair enough. I agree monte carlo should follow data, not the other way round.

One more thing that occurs - MAUS assumes that reconstruction only ever depends on a single map() call. So you are not allowed, for example, to change some mapper depending on what happens during another map() call. Do we ever need e.g. the reconstruction of the physics_event to know about any of the other daq events?

#12

Updated by Rogers, Chris almost 9 years ago

  • Status changed from Open to Rejected

Also available in: Atom PDF