Project

General

Profile

Actions
History

MICE Physics Devil

The "MICE physics devil" is a tool to check that the MICE operations and computing workflow worked. It performs a number of automated checks on the output data intended to support the physics shifter.

The code is intended to install into the MAUS environment. To install, do:

wget mice_physics_devil-0.2.2.tar.gz
tar -xzf mice_physics_devil-0.2.2.tar.gz
cd mice_physics_devil
source /path/to/maus/env.sh
python setup.py install

To run, do

source /path/to/maus/env.sh
cd /path/to/working/dir
mkdir tmp                    # sorry, this is not automated
mkdir tmp/output             # sorry, this is not automated
physics_devil -h             # Gives a list of command line options
physics_devil --start_run=6155 --end_run=6225 -d ReconCheck -d CdbCheck -d RawDataCheck # Run the scripts
physics_devil --start_run=6155 --end_run=6225 -d RunReducer # Run the reducers - this can be very slow. Note physics_devil should append reducer output to the output for the other checks.

This will make output like

Running 6155 which is 1 out of 71
Running 6156 which is 2 out of 71
Running 6157 which is 3 out of 71
...<etc>...
Running 6221 which is 67 out of 71
Running 6222 which is 68 out of 71
Running 6223 which is 69 out of 71
Running 6224 which is 70 out of 71
Running 6225 which is 71 out of 71

Detailed log file is available at tmp/output/physics_devil.log. This will produce output like this for century 6100 and this for century 6200
  • Each row corresponds to a MICE run
  • Each column corresponds to a data output
To use the RunConditionsSpreadsheet checks, do
  • Download the RunConditionsSpreadsheet from http://mice.iit.edu/mico/run_condition_summary/RunConditionSummary.xls
  • Open it in libre office or equivalent
  • File > Save As
    • Click "All Formats" and select "Text CSV .csv"
    • File name should be as per physics_devil.common.Arguments.run_conditions_spreadsheet
      > Confirm File Format Dialogue click Use Text CSV Format > Ensure that Field Delimiter is , > Click Okay
Columns are:
  • run: run number
  • raw_data: is the raw data file available from mice data website
  • cdb_entry: is there a cdb entry for the run <does not check that it is sane>
  • run_conditions_spreadsheet_entry: is there a run conditions spreadsheet entry for the run <does not check that it is sane>
  • n_errors: number of errors in the reconstructed data set
  • n_daq_events: number of daq events in the reconstructed data set
  • n_physics_events: number of physics events in the reconstructed data set (e.g. excluding calibrations, spill header/footer)
  • n_recon_events: number of particle triggers in the reconstructed data set
  • n_ckov_digits: number of cerenkov digits (PMT pulses)
  • n_kl_cell_hits: number of kl cell hits (reconstructed cells hit)
  • n_kl_digits: number of kl digits (PMT pulses)
  • n_tof_digits: number of tof digits (PMT pulses)
  • n_tof_slab_hits: number of tof slab hits (coincidence of 2 PMTs on a scintillator bar)
  • n_tof_space_points: number of tof space points (calibrated, coincidence of 2 slab hits on a station)
  • ReducePyScalersTable: Did ReducePyScalersTable run okay?
  • ReducePyTOFPlot: Did ReducePyTOFPlot run okay?
  • ReducePyCkovPlot: Did ReducePyCkovPlot run okay?
  • ReducePyKLPlot: Did ReducePyKLPlot run okay?
Comments:
  • It would be nice to check for sanity in the cdb/run conditions spreadsheet.
  • ReducePy<blah> are the (MAUS) scripts that make histograms. At the moment they seem to be running quite slow, so I have not run them on the data set over the weekend. There is a job to make these work a bit quicker. This is on me, but I won't get to it until the end of April.
  • In the end I would like to make this a cron job that runs somewhere in PPD and automatically pushes output, including Reducer plots, to http://micewww.pp.rl.ac.uk/analysis/physics_shifts

Under the Hood

The code is all written in python

  • Registered on launchpad as mice_physics_devil
  • Checkout development snapshot my doing bzr checkout lp:mice-physics-devil/trunk
  • Code for module1.module2.Class sits in file like module1/module2/_class.py

The code breaks down as:

  • tests directory is empty!
  • physics_devil/bin/physics_devil is a light script which calls physics_devil.PhysicsDevil class
  • physics_devil.PhysicsDevil class contains the main run loop
  • physics_devil.common module contains all of the common infrastructure
  • physics_devil.daemonettes contains a set of classes for actually making the various checks

physics_devil.common

  • physics_devil.common.Arguments contains calls for command line parsing and some other settings that are not accessible from the command line, e.g. file paths and the like. If you want to change this stuff it should work, but you have to mangle the code by hand.
  • physics_devil.common.exception_handler makes a small function or two to redirect exceptions to the log files, not to the screen. Look at the log file in the first instance if the code is misbehaving
  • physics_devil.common.DownloadData is a class that handles downloading reconstructed data sets from the Imperial webserver http://www.hep.ph.ic.ac.uk/micedata.
    • It uses wget in "mirror" mode (choose to download based on client vs server time stamps)
  • physics_devil.common.RunDb is a very lightweight, very poorly implemented, serialisation for the outputs from all of the daemonettes. The idea is to make the output persistent so that we can watch for changes to the recon data set and then update the physics_devil output. My goal is to have this running as a daemon by the end of May or so.

physics_devil.daemonettes

Contains a set of checks to make on the data as it comes through.

  • physics_devil.daemonettes.DaemonetteBase is the base class for all daemonettes. Contains a few functions which need to be overloaded for the code to work.
  • physics_devil.daemonettes.CdbCheck checks that a cdb entry exists and that a run conditions spreadsheet entry exists. It doesn't yet do any sanity checks beyond that.
  • physics_devil.daemonettes.RawDataCheck checks that a RawData file is available. It doesn't check that it can be unpacked.
  • physics_devil.daemonettes.ReconCheck calculates reconstruction efficiencies.
  • physics_devil.daemonettes.RunReducer runs the reducers. At the moment I am struggling to get this to work due to the processor requirements.

Making a Release

To make a release:
  • Update the README
  • Check all the tests pass
  • Check all the documentation is okay
  • Update the version number in physics_devil/__init__.py
  • Check that MANIFEST.in is okay (no new directories or entries in share)
  • python setup.py sdist
  • md5sum dist/mice_physics_devil-<version>.tar.gz > dist/mice_physics_devil-<version>.tar.gz.md5
  • scp dist//mice_physics_devil-<version>.tar.gz* micewww.pp.rl.ac.uk:/www/analysis/mice_physics_devil
  • Update index.html with the new version

Updated by Rogers, Chris almost 8 years ago ยท 19 revisions