MICE Physics Devil¶
The "MICE physics devil" is a tool to check that the MICE operations and computing workflow worked. It performs a number of automated checks on the output data intended to support the physics shifter.
- Tarballs available to download from http://micewww.pp.rl.ac.uk/analysis/mice_physics_devil
The code is intended to install into the MAUS environment. To install, do:
wget mice_physics_devil-0.2.2.tar.gz tar -xzf mice_physics_devil-0.2.2.tar.gz cd mice_physics_devil source /path/to/maus/env.sh python setup.py install
To run, do
source /path/to/maus/env.sh cd /path/to/working/dir mkdir tmp # sorry, this is not automated mkdir tmp/output # sorry, this is not automated physics_devil -h # Gives a list of command line options physics_devil --start_run=6155 --end_run=6225 -d ReconCheck -d CdbCheck -d RawDataCheck # Run the scripts physics_devil --start_run=6155 --end_run=6225 -d RunReducer # Run the reducers - this can be very slow. Note physics_devil should append reducer output to the output for the other checks.
This will make output like
Running 6155 which is 1 out of 71 Running 6156 which is 2 out of 71 Running 6157 which is 3 out of 71 ...<etc>... Running 6221 which is 67 out of 71 Running 6222 which is 68 out of 71 Running 6223 which is 69 out of 71 Running 6224 which is 70 out of 71 Running 6225 which is 71 out of 71Detailed log file is available at
tmp/output/physics_devil.log. This will produce output like this for century 6100 and this for century 6200
- Each row corresponds to a MICE run
- Each column corresponds to a data output
- Download the RunConditionsSpreadsheet from http://mice.iit.edu/mico/run_condition_summary/RunConditionSummary.xls
- Open it in libre office or equivalent
- File > Save As
- Click "All Formats" and select "Text CSV .csv"
- File name should be as per
> Confirm File Format Dialogue click Use Text CSV Format > Ensure that Field Delimiter is , > Click Okay
- run: run number
- raw_data: is the raw data file available from mice data website
- cdb_entry: is there a cdb entry for the run <does not check that it is sane>
- run_conditions_spreadsheet_entry: is there a run conditions spreadsheet entry for the run <does not check that it is sane>
- n_errors: number of errors in the reconstructed data set
- n_daq_events: number of daq events in the reconstructed data set
- n_physics_events: number of physics events in the reconstructed data set (e.g. excluding calibrations, spill header/footer)
- n_recon_events: number of particle triggers in the reconstructed data set
- n_ckov_digits: number of cerenkov digits (PMT pulses)
- n_kl_cell_hits: number of kl cell hits (reconstructed cells hit)
- n_kl_digits: number of kl digits (PMT pulses)
- n_tof_digits: number of tof digits (PMT pulses)
- n_tof_slab_hits: number of tof slab hits (coincidence of 2 PMTs on a scintillator bar)
- n_tof_space_points: number of tof space points (calibrated, coincidence of 2 slab hits on a station)
- ReducePyScalersTable: Did ReducePyScalersTable run okay?
- ReducePyTOFPlot: Did ReducePyTOFPlot run okay?
- ReducePyCkovPlot: Did ReducePyCkovPlot run okay?
- ReducePyKLPlot: Did ReducePyKLPlot run okay?
- It would be nice to check for sanity in the cdb/run conditions spreadsheet.
- ReducePy<blah> are the (MAUS) scripts that make histograms. At the moment they seem to be running quite slow, so I have not run them on the data set over the weekend. There is a job to make these work a bit quicker. This is on me, but I won't get to it until the end of April.
- In the end I would like to make this a cron job that runs somewhere in PPD and automatically pushes output, including Reducer plots, to http://micewww.pp.rl.ac.uk/analysis/physics_shifts
Under the Hood¶
The code is all written in python
- Registered on launchpad as mice_physics_devil
- Checkout development snapshot my doing
bzr checkout lp:mice-physics-devil/trunk
- Code for
module1.module2.Classsits in file like
The code breaks down as:
- tests directory is empty!
- physics_devil/bin/physics_devil is a light script which calls physics_devil.PhysicsDevil class
- physics_devil.PhysicsDevil class contains the main run loop
- physics_devil.common module contains all of the common infrastructure
- physics_devil.daemonettes contains a set of classes for actually making the various checks
- physics_devil.common.Arguments contains calls for command line parsing and some other settings that are not accessible from the command line, e.g. file paths and the like. If you want to change this stuff it should work, but you have to mangle the code by hand.
- physics_devil.common.exception_handler makes a small function or two to redirect exceptions to the log files, not to the screen. Look at the log file in the first instance if the code is misbehaving
- physics_devil.common.DownloadData is a class that handles downloading reconstructed data sets from the Imperial webserver http://www.hep.ph.ic.ac.uk/micedata.
- It uses
wgetin "mirror" mode (choose to download based on client vs server time stamps)
- It uses
- physics_devil.common.RunDb is a very lightweight, very poorly implemented, serialisation for the outputs from all of the daemonettes. The idea is to make the output persistent so that we can watch for changes to the recon data set and then update the physics_devil output. My goal is to have this running as a daemon by the end of May or so.
Contains a set of checks to make on the data as it comes through.
- physics_devil.daemonettes.DaemonetteBase is the base class for all daemonettes. Contains a few functions which need to be overloaded for the code to work.
- physics_devil.daemonettes.CdbCheck checks that a cdb entry exists and that a run conditions spreadsheet entry exists. It doesn't yet do any sanity checks beyond that.
- physics_devil.daemonettes.RawDataCheck checks that a RawData file is available. It doesn't check that it can be unpacked.
- physics_devil.daemonettes.ReconCheck calculates reconstruction efficiencies.
- physics_devil.daemonettes.RunReducer runs the reducers. At the moment I am struggling to get this to work due to the processor requirements.
Making a Release¶To make a release:
- Update the README
- Check all the tests pass
- Check all the documentation is okay
- Update the version number in physics_devil/__init__.py
- Check that MANIFEST.in is okay (no new directories or entries in share)
python setup.py sdist
md5sum dist/mice_physics_devil-<version>.tar.gz > dist/mice_physics_devil-<version>.tar.gz.md5
scp dist//mice_physics_devil-<version>.tar.gz* micewww.pp.rl.ac.uk:/www/analysis/mice_physics_devil
- Update index.html with the new version