Support #697
Code Review - Geometry Handling system
100%
Description
The first iteration of the geometry handling system is now complete. The files which relate to the system are located in,
$MAUS_ROOT_DIR/src/common_py/geometry
$MAUS_ROOT_DIR/tests/py_unit/test_geometry
This is a first iteration and future work/development is going to be carried out which will entail the following:
- More development on the tests to make them much more robust as there are currently considered rudimentary.
- Soft coding the links and the setting of servers for the connection to the CDB.
- More functionality with respects to how we can download from the CDB i.e. adding a download geometry for pulses? (If desired)
- Extending the schema to incorporate detector information (Currently work is being carried out on the tracker)
- Also extending the XSLT to carry the information from runs/detectors into MAUS modules. A MAUS format needs to be created.
- Integration tests and tests for the executables.
- Changing the DownloadGeometryForRunNum.py as it currently downloads the current geometry for every run.
- The reason for this is there isn't any run information which has been entered recently which will match geometries entered recently. The code is ready to change though.
- Documentation/User guide
- Corrections from the code review.
- Fixing bugs.
And finally anything else that can be though of?
Files
Related issues
Updated by Rogers, Chris about 12 years ago
- Due date set to 28 September 2011
- Category set to common_py
Great - presumably files are in maus_littlefield?
Updated by Rogers, Chris about 12 years ago
So I tried running tests. Found few issues:
First the geometry tests weren't being executed by the test script. The scons script only pull code from tests/py_unit/ not tests/py_unit/*/. Need to edit line 634 of $MAUS_ROOT_DIR from
files = glob.glob('tests/py_unit/test_*.py')+glob.glob('tests/style/*.py')
to
files = glob.glob('tests/py_unit/*/test_*.py')+\ glob.glob('tests/py_unit/test_*.py')+glob.glob('tests/style/*.py')
Second, your run_tests.bash file was corrupted, presumably during a merge. All of this stuff
#<<<<<<< TREE # To run lcov:- # (a) Uncomment lines below # (b) Uncomment some CCFLAGS in SConstruct #if which lcov >& /dev/null; then # echo Clearing lcov # lcov -q -b ${MAUS_ROOT_DIR} --directory src --zerocounters -q #fi nosetests -v --with-coverage build #if which lcov >& /dev/null; then # echo Building lcov output # lcov -q -b ${MAUS_ROOT_DIR} --directory src --capture --output-file maus.info # genhtml -o doc/coverage/ maus.info #fi =======
and this line
#>>>>>>> MERGE-SOURCE
needs to be deleted. bzr inserts diff output into a file when it finds a conflict. Now I get errors like:
====================================================================== ERROR: Failure: ImportError (dynamic module does not define init function (initlibxml2)) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/cr67/G4MICE/MAUS/maus_littlefield/third_party/install/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py", line 390, in loadTestsFromName addr.filename, addr.module) File "/home/cr67/G4MICE/MAUS/maus_littlefield/third_party/install/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/home/cr67/G4MICE/MAUS/maus_littlefield/third_party/install/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/cr67/G4MICE/MAUS/maus_littlefield/build/test_gdml_to_maus_module.py", line 8, in <module> from geometry.GDMLtoMAUSModule import GDMLtomaus File "/home/cr67/G4MICE/MAUS/maus_littlefield/src/common_py/geometry/__init__.py", line 4, in <module> from geometry.CADImport import CADImport File "/home/cr67/G4MICE/MAUS/maus_littlefield/src/common_py/geometry/CADImport.py", line 6, in <module> import libxml2 ImportError: dynamic module does not define init function (initlibxml2)
I think we fixed this but I can't remember how...
Updated by Rogers, Chris about 12 years ago
- Assignee changed from Rogers, Chris to Littlefield, Matthew
Updated by Littlefield, Matthew about 12 years ago
I have fixed scons and taken out the corrupted code from run_tests.bash, when I run this I get this error,
test_digit (test_MapCppTOFDigitization.MapCppTOFDigitizationTestCase) ... python2.7: src/map/MapCppTOFDigitization/MapCppTOFDigitization.cc:152: std::string MapCppTOFDigitization::process(std::string): Assertion `mod != __null' failed.
run_tests.bash: line 29: 2716 Aborted nosetests --with-coverage -v build
Not sure what this is or if I'm doing this?
Updated by Rogers, Chris about 12 years ago
Well done for getting this in - looks really nice!
Comments:
Style-----
- If you put pylint disable directives at the top of the file, it disables for the entire file. Sometimes you need to do this, but often this is not correct. Better to disable per line by putting a comment at the line end. In particular, you disable the 80 character line limit. Please don't do this!
- Can you put a copyright notice at the top of your files? Something like:
# This file is part of MAUS: http://micewww.pp.rl.ac.uk:8080/projects/maus # # MAUS is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # MAUS is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with MAUS. If not, see <http://www.gnu.org/licenses/>.
I sorted out the latest cdb version as a third party library (together with real suds) and pushed it into bzr+ssh://bazaar.launchpad.net/~chris-rogers/maus/libxml2_test/
Also have a look at the attached test log. The errors where we have
CdbTemporaryError: Unable to contact CDB server at http://cdb.mice.rl.ac.uk/cdb/geometrySuperMouse?wsdl
are because I didn't use the suds testing mock-up. But stuff like
OSError: [Errno 2] No such file or directory: '/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_2.gdml'
Looks like genuine error...
Updated by Littlefield, Matthew about 12 years ago
I have had a look at the errors and I think that the problem is when I call environment variables. To call the MAUS_ROOT_DIR i use
os.environ['MAUS_ROOT_DIR']
All the errors where the file can't be found has this in it but it works on my machine? I think this isn't findind MAUS_ROOT_DIR on jenkins which way do you find this variable?
I'll change it and check it works to see if this is the problem.
FYI, I'm getting different errors on my machine to what you have seen....
Updated by Rogers, Chris about 12 years ago
For example src/common_py/geometry/testCases/testPacker/FileList.txt
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_2.gdml /home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_0.gdml /home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_3.gdml /home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/fastradModel_materials.xml /home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/fastradModel.gdml /home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_1.gdml /home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_4.gdml /home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_5.gdml
Updated by Rogers, Chris about 12 years ago
Note I completely removed the old cdb stuff and changed the build scripts to add suds and cdb in 44suds.bash 45cdb.bash
Updated by Rogers, Chris about 12 years ago
Also
echo $MAUS_ROOT_DIR /home/cr67/G4MICE/MAUS/maus_littlefield_merge
which is correct. Done using
source env.shas usual
Updated by Littlefield, Matthew about 12 years ago
OH, I forgot that I have hard coded the Filelist.txt woops I'll change it :-).
I noticed you removed the cdb stuff and I have since built the suds etc scripts on my machine and I no longer get the SuperMouseGeometry connection errors.
Updated by Littlefield, Matthew about 12 years ago
I have fixed some errors in the tests and added the legal blurb I still need to organise the pylint stuff.
I am getting errors from run_tests.bash which are different to the ones you have seen and was wondering if its a quick fix? If not maybe we could take a look at them at Fermilab? Ive attached the errors
Updated by Rogers, Chris about 12 years ago
All of the stuff in test_ReducePyMatplotlibHistogram.py, well, nothing to do with your code, so maybe something got overwritten. The other stuff is with the cdb tests, which I removed. Probably if you do scons -c; scons these tests should go away. The tests now live in Antony Wilson's code.
Updated by Rogers, Chris about 12 years ago
- Assignee changed from Littlefield, Matthew to Rogers, Chris
So... tests were failing on my machine, I fixed/hacked some stuff I didn't like:
- I replaced the cdb test server with some mock cdb classes because the actual cdb calls were failing on network issues.
- I found that because the GDML formatter overwrites the input files, someone Matt had accidentally overwritten his own test data. In fact, because GDML formatter overwrites input files, we can only run the test once without doing bzr revert. I changed to have an input folder and output folder for the data
The only test fail now on my machine is:
====================================================================== ERROR: method test_format ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/cr67/G4MICE/MAUS/maus_rogers/build/test_geometry/test_gdml_formatter.py", line 194, in test_format self.gdml.format() File "/home/cr67/G4MICE/MAUS/maus_rogers/src/common_py/geometry/GDMLFormatter.py", line 247, in format self.merge_maus_info(self.configuration_file) File "/home/cr67/G4MICE/MAUS/maus_rogers/src/common_py/geometry/GDMLFormatter.py", line 112, in merge_maus_info root_node = config.childNodes[2] IndexError: list index out of range -------------------- >> begin captured stdout << --------------------- Run information merged!
but I need to do another pass for more clean up. Sorry Matt, this will take another couple of days.
Updated by Rogers, Chris about 12 years ago
Should say - the reason I didn't fix that test is because I really didn't have a clue what the function was doing...
Updated by Littlefield, Matthew about 12 years ago
- File GDML_fastradModel.zip GDML_fastradModel.zip added
Heres the correct files :-)
Updated by Rogers, Chris about 12 years ago
I now have this passing in rogers build e.g. http://test.mice.rl.ac.uk/job/MAUS_rogers/196/
I did quite a bit of messing about - soft coding the URL for test server, moving the executables to bin/utilities, moving the file cache to files/geometry, separating the input directory and output directory for e.g. formatter, few other stuff. I want to finally make an integration test that checks that we can upload sample geometries to the test server, pull them back down again and run simulation against them before I do the merge.
Updated by Littlefield, Matthew about 12 years ago
That sounds good. Can I merge it with my local copy so I have the current version? Because I'm going to be doing some stuff with the tracker schema this week and would like my version to be current :-)
what is the branch name?
lp:maus_rogers?
or should I want till the maus release? May be better ...
Updated by Rogers, Chris about 12 years ago
Branch name is:
bzr+ssh://bazaar.launchpad.net/~chris-rogers/maus/devel/
I think it would be worth at least having a look at the code to see what I've done. You may have some objections, in which case let me know.
Nb: Try to keep your branch clean, there was a whole load of stuff which should not have been committed (temporary files, etc). Makes it quite a pain to merge!
Updated by Littlefield, Matthew about 12 years ago
OK Ill be cleaning up my branch :-) so its less painful.
When I try to merge is gives me a warning that I'm doing a criss-cross merge and I read bzr help and it looks scary I think Ill wait til the release because I don't want to mess it up :-)
Updated by Rogers, Chris about 12 years ago
Okay, well feel free to download my branch at least to have a look...
Updated by Rogers, Chris about 12 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
Now merged and will be in the next release