Bug #918

Configuration and distributed transforms

Added by Jackson, Mike over 12 years ago. Updated over 11 years ago.

Tunnell, Christopher
Online reconstruction
Target version:
Start date:
16 March 2012
Due date:
% Done:


Estimated time:
(Total: 0.00 h)
New Issue


When the MAUS configuration is loaded there are 4 absolute file paths in the configuration values. These are absolute due to their values being set in using os.environ.get("MAUS_ROOT_DIR").

The values and files that cite them are, for example:

"beam_file": "/home/michaelj/maus-bzr/maus/src/map/MapPyBeamMaker/test_g4bl.dat" 

"daq_data_path": "/home/michaelj/maus-bzr/maus/src/input/InputCppDAQData" 

"geometry_upload_directory": "/home/michaelj/maus-bzr/maus/files/geometry/upload" 

"geometry_download_directory": "/home/michaelj/maus-bzr/maus/files/geometry/download" 

Given pushes the MAUS configuration to Celery workers this would create a problem if the Celery workers were running in a MAUS directory that was in a different part of the file system.

The short-term solution is to recommend that MAUS deployments running on different hosts run in MAUS_ROOT_DIR directories with the same path.

A preferable solution would be that the components that use the above values calculate the absolute paths at the time they need them.


Bug #942: Geometry file handling should call environment variables locallyClosedLittlefield, Matthew16 March 2012

Bug #943: Beam maker file handling should call environment variables locallyClosedRogers, Chris16 March 2012

Bug #944: InputCppDAQData should handle environment variables locallyOpenKaradzhov, Yordan16 March 2012


Updated by Tunnell, Christopher about 12 years ago

I can't think of a good way to get MAUS to essentially compute what that value is on the fly. We already parse and do various operations when loading the ConfigurationDefaults as it is. We could (in principle) do a search and replace.

Am I correct in thinking that this is an anticipated problem? I added Chris Rogers in case he disagrees with what I'm about to say. For the most part, we're going to install MAUS when using it in a distributed fashion on some NFS partition... or shared home in whatever fashion that results in. For example, on the RAL PPD batch queue, your home is shared between interactive and batch machines. We'd probably do something similar since even though the overhead of NFS isn't desirable and there are numerous other undesirable things about NFS/AFS/etc., we aren't loading up MAUS that frequently so the burden of having many nodes load from the same HD isn't that high.

So my recommendation (which is very late, sorry) is to not worry since it fits with our use case. But if Chris Rogers or somebody is worried and the issue has already arisen somewhere, then we should define MAUS_ROOT_DIR_ENV to be forbidden in strings unless we want it search-replaced with os.getenviron('MAUS_ROOT_DIR')


Updated by Rogers, Chris about 12 years ago

I think Mike means just call


when opening the file (or whatever).


Updated by Jackson, Mike about 12 years ago


Also, this only applies to paths that would be used by mergers or code executed by them, though.


Updated by Rogers, Chris about 12 years ago

Okay. Probably easier to tell people just to use expandvars all the time (saves running into the problem of code moving into a merger later and then breaking in a funny situation that they didn't think of).


Updated by Tunnell, Christopher about 12 years ago

Cool!!! os.path.expandvar


Updated by Rogers, Chris over 11 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100


Also available in: Atom PDF