After phone call with Yordan - the problem with this is that there is not a one to one mapping of event type to spill.

So e.g. there is one start_of_run event per run, etc.

So as a stop gap propose to add filter to the readNextEvent() function in InputCppDAQData. (Edit: filter will include only physics_event). But in long run, consider adding some filter type to the execution structure - for example, the start_of_run will contain the trigger information to which we want access.

Add sample output file (structure only, all electronics unpacking disabled):

{"daq_data":[{"daq_event_type":"end_of_burst","event_data":null,"spill_num":-1}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":0},{"daq_event_type":"physics_event","event_data":null,"spill_num":0},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":0}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":1},{"daq_event_type":"physics_event","event_data":null,"spill_num":1},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":1}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":2},{"daq_event_type":"physics_event","event_data":null,"spill_num":2},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":2}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":3},{"daq_event_type":"physics_event","event_data":null,"spill_num":3},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":3}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":4},{"daq_event_type":"physics_event","event_data":null,"spill_num":4},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":4}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":5},{"daq_event_type":"physics_event","event_data":null,"spill_num":5},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":5}]}
{"daq_data":[{"daq_event_type":"start_of_burst","event_data":null,"spill_num":6},{"daq_event_type":"physics_event","event_data":null,"spill_num":6},{"daq_event_type":"end_of_burst","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6},{"daq_event_type":"end_of_run","event_data":null,"spill_num":6}]}

The problem is that this is not going to work when we have "calibration event".
First because the calibration events come outside of the burst (start_of_burst <-> end_of_burst)
and second because calibration events don't have spill number.

I would suggest instead of having this complicated solution to:
Keep getCurentEvent unchanged.
Change only the name of getNextEvent to getNextDAQEvent.
Make another function called getNextPhysEvent which will skip all other types of event like start_of_burst, end_of_burst.
We may have getNextCalibEvent too.
Make possible the user to decide what to do (use getNextDAQEvent, getNextPhysEvent or getNextCalibEvent).

Can you remind here what is the motivation for the requirement to have only one line per spill.

The whole simulation, reconstruction assumes that we have one json document per spill - the execution loop is something like

1. while (input)
   1. input = get_input()
   2. map(input)

• waste cpu distributing documents that have no physics event (which then need to be rejected)
• need to make a data splitter to properly replicate the data structure in monte carlo (i.e. we need to add an execution type that splits monte carlo digitisted data into a physics_event, start_of_burst, end_of_burst
• probably in the end we will make a filter type to remove non-useful daq events from the real data structure (i.e. we filter start_of_burst, end_of_burst)

I guess also I just wasn't expecting it to go like that, and I naturally resist change somewhat!

Another try at explaining:

It really makes the high level input/map/reduce framework much more complicated if we cannot guarantee one output per input. So either we are faced with doing some difficult algorithm to remove data from the execution stream or we just have stuff in the data stream that is sort of "interesting" but of not much use (calibration is of course important, but start_of_burst/end_of_burst I think is not really useful).

We can discuss at phone call later?

I do not agree that, we have to change the structure of the real data in order to make it the same as the simulation.
Also I do not agree that, only physics and calibration events will be useful.
That's why, I suggested to make user to decide what to do. This can be controlled by the datacards.
In any case high level applications will do different things when processing phys. events and when processing calib. events.
There is no way to avoid the check of the event type in the beginning before doing anything else.

Fair enough. I agree monte carlo should follow data, not the other way round.

One more thing that occurs - MAUS assumes that reconstruction only ever depends on a single map() call. So you are not allowed, for example, to change some mapper depending on what happens during another map() call. Do we ever need e.g. the reconstruction of the physics_event to know about any of the other daq events?