Bug #918
Configuration and distributed transforms
100%
Description
When the MAUS configuration is loaded there are 4 absolute file paths in the configuration values. These are absolute due to their values being set in ConfigurationDefaults.py using os.environ.get("MAUS_ROOT_DIR").
The values and files that cite them are, for example:
"beam_file": "/home/michaelj/maus-bzr/maus/src/map/MapPyBeamMaker/test_g4bl.dat" src/map/MapPyBeamMaker/MapPyBeamMaker.py "daq_data_path": "/home/michaelj/maus-bzr/maus/src/input/InputCppDAQData" src/input/InputCppDAQData/InputCppDAQData.cc "geometry_upload_directory": "/home/michaelj/maus-bzr/maus/files/geometry/upload" src/common_py/geometry/ConfigReader.py bin/utilities/upload_geometry.py "geometry_download_directory": "/home/michaelj/maus-bzr/maus/files/geometry/download" src/common_py/geometry/ConfigReader.py bin/utilities/download_geometry.py
Given Go.py pushes the MAUS configuration to Celery workers this would create a problem if the Celery workers were running in a MAUS directory that was in a different part of the file system.
The short-term solution is to recommend that MAUS deployments running on different hosts run in MAUS_ROOT_DIR directories with the same path.
A preferable solution would be that the components that use the above values calculate the absolute paths at the time they need them.
Subtasks
Updated by Tunnell, Christopher over 11 years ago
I can't think of a good way to get MAUS to essentially compute what that value is on the fly. We already parse and do various operations when loading the ConfigurationDefaults as it is. We could (in principle) do a search and replace.
Am I correct in thinking that this is an anticipated problem? I added Chris Rogers in case he disagrees with what I'm about to say. For the most part, we're going to install MAUS when using it in a distributed fashion on some NFS partition... or shared home in whatever fashion that results in. For example, on the RAL PPD batch queue, your home is shared between interactive and batch machines. We'd probably do something similar since even though the overhead of NFS isn't desirable and there are numerous other undesirable things about NFS/AFS/etc., we aren't loading up MAUS that frequently so the burden of having many nodes load from the same HD isn't that high.
So my recommendation (which is very late, sorry) is to not worry since it fits with our use case. But if Chris Rogers or somebody is worried and the issue has already arisen somewhere, then we should define MAUS_ROOT_DIR_ENV to be forbidden in strings unless we want it search-replaced with os.getenviron('MAUS_ROOT_DIR')
Updated by Rogers, Chris over 11 years ago
I think Mike means just call
os.path.expandvars(path)
when opening the file (or whatever).
Updated by Jackson, Mike over 11 years ago
Yes.
Also, this only applies to paths that would be used by mergers or code executed by them, though.
Updated by Rogers, Chris over 11 years ago
Okay. Probably easier to tell people just to use expandvars all the time (saves running into the problem of code moving into a merger later and then breaking in a funny situation that they didn't think of).
Updated by Rogers, Chris almost 11 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
Done