MAUSOnlineReconstructionOverview » History » Revision 3
« Previous |
Revision 3/14
(diff)
| Next »
Jackson, Mike, 15 March 2012 13:47
Online Reconstruction Overview¶
Run numbers¶
Run numbers are assumed to be as follows:
- -N : Monte Carlo simulation of run N
- 0 : pure Monte Carlo simulation
- +N : run N
Transforming spills from an input stream (Input-Transform)¶
This is the algorithm used to transform spills from an input stream:
CLEAR document store run_number = None WHILE an input spill is available GET next spill IF spill does not have a run number # Assume pure MC spill_run_number = 0 IF (spill_run_number != run_number) # We've changed run. IF spill is NOT a start_of_run spill WARN user of missing start_of_run spill WAIT for current Celery tasks to complete WRITE result spills to document store run_number = spill_run_number CONFIGURE Celery by DEATHing current transforms and BIRTHing new transforms TRANSFORM spill using Celery WRITE result spill to document store DEATH Celery worker transforms
If there is no initial
start_of_run
spill (or no spill_num
in the spills) in the input stream (as can occur when using simple_histogram_example.py
or simulate_mice.py
) then spill_run_number will be 0, run_number will be None and a Celery configuration will be done before the first spill needs to be transformed.
Spills are inserted into the document store in the order of their return from Celery workers. This may not be in synch with the order in which they were originally read from the input stream.
Merging spills and passing results to an output stream (Merge-Output)¶
This is the algorithm used to merge spills and pass the results to an output stream:
run_number = None end_of_run = None is_birthed = FALSE last_time = 01/01/1970 WHILE TRUE READ spills added since last time from document store IF spill IS "end_of_run" end_of_run = spill IF spill_run_number != run_number IF is_birthed IF end_of_run == None end_of_run = {"daq_event_type":"end_of_run", "run_num":run_number} Send end_of_run to merger DEATH merger and outputter BIRTH merger and outputter run_number = spill_run_number end_of_run = None is_birthed = TRUE MERGE and OUTPUT spill Send END_OF_RUN block to merger DEATH merger and outputter
The Input-Transform policy of waiting for the processing of spills from a run to complete before starting processing spills from a new run means that all spills from run N-1 are guaranteed to have a time stamp < spills from run N.
is_birthed is used to ensure that there is no BIRTH-DEATH-BIRTH redundancy on receipt of the first spill.
Document store¶
Spills are stored in documents in a collection in the document store.
Documents are of form {"_id":ID, "date":DATE, "doc":SPILL}
where:
- ID: index of this document in the chain of those successfuly transformed. It has no significance beyond being unique in an execution of Input-Transform loop below. It is not equal to the spill_num (Python
string
) - DATE: date and time to the milli-second noting when the document was added (Python
timestamp
) - DOC: spill document. (Python
string
holding a valid JSON document)
Collection names¶
For Input-Transform,
- If configuration parameter
doc_collection_name
isNone
,""
, orauto
thenHOSTNAME_PID
, whereHOSTNAME
is the machine name andPID
the process ID, is used. - Otherwise the value of
doc_collection_name
is used. doc_collection_name
has default valuespills
.
For Merge-Output,
- If configuration parameter
doc_collection_name
isNone
,""
, or undefined then an error is raised. - Otherwise the value of
doc_collection_name
is used.
Miscellaneous¶
- Currently Celery timeouts are not used, transforming a spill takes as long as it takes.
- Celery task retries on failure option is not used. If the transformation of a spill fails first time it can't be expected to succeed on a retry.
- If memory leaks arise, e.g. from C++ code, look at Celery rate limitss, which allow the time or number of tasks before sub-process is killed and respawned, to be defined. Soft rate limits would allow
death
to be run on the transforms first.
Updated by Jackson, Mike over 11 years ago · 3 revisions