Project

General

Profile

Support #697

Code Review - Geometry Handling system

Added by Littlefield, Matthew about 11 years ago. Updated almost 11 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
common_py
Target version:
Start date:
12 September 2011
Due date:
28 September 2011
% Done:

100%

Estimated time:
Workflow:

Description

The first iteration of the geometry handling system is now complete. The files which relate to the system are located in,

$MAUS_ROOT_DIR/src/common_py/geometry
$MAUS_ROOT_DIR/tests/py_unit/test_geometry

This is a first iteration and future work/development is going to be carried out which will entail the following:

  • More development on the tests to make them much more robust as there are currently considered rudimentary.
  • Soft coding the links and the setting of servers for the connection to the CDB.
  • More functionality with respects to how we can download from the CDB i.e. adding a download geometry for pulses? (If desired)
  • Extending the schema to incorporate detector information (Currently work is being carried out on the tracker)
    • Also extending the XSLT to carry the information from runs/detectors into MAUS modules. A MAUS format needs to be created.
  • Integration tests and tests for the executables.
  • Changing the DownloadGeometryForRunNum.py as it currently downloads the current geometry for every run.
    • The reason for this is there isn't any run information which has been entered recently which will match geometries entered recently. The code is ready to change though.
  • Documentation/User guide
  • Corrections from the code review.
  • Fixing bugs.

And finally anything else that can be though of?


Files

test.log (66.4 KB) test.log Better add that test log Rogers, Chris, 20 October 2011 09:40
testLog (68.5 KB) testLog Littlefield, Matthew, 20 October 2011 14:43
GDML_fastradModel.zip (13.7 KB) GDML_fastradModel.zip Littlefield, Matthew, 29 October 2011 15:17

Related issues

Related to MAUS - Support #602: UploadGeometry.py - Code ReviewRejectedRogers, Chris29 July 2011

Actions
#1

Updated by Rogers, Chris about 11 years ago

  • Due date set to 28 September 2011
  • Category set to common_py

Great - presumably files are in maus_littlefield?

#2

Updated by Littlefield, Matthew about 11 years ago

Yeah they are :-)

#3

Updated by Rogers, Chris about 11 years ago

So I tried running tests. Found few issues:

First the geometry tests weren't being executed by the test script. The scons script only pull code from tests/py_unit/ not tests/py_unit/*/. Need to edit line 634 of $MAUS_ROOT_DIR from

files = glob.glob('tests/py_unit/test_*.py')+glob.glob('tests/style/*.py')

to

files = glob.glob('tests/py_unit/*/test_*.py')+\
        glob.glob('tests/py_unit/test_*.py')+glob.glob('tests/style/*.py')

Second, your run_tests.bash file was corrupted, presumably during a merge. All of this stuff

#<<<<<<< TREE
# To run lcov:-
# (a) Uncomment lines below
# (b) Uncomment some CCFLAGS in SConstruct

#if which lcov >& /dev/null; then 
#    echo Clearing lcov
#    lcov -q -b ${MAUS_ROOT_DIR} --directory src --zerocounters -q
#fi
nosetests -v --with-coverage build
#if which lcov >& /dev/null; then 
#    echo Building lcov output
#    lcov  -q -b ${MAUS_ROOT_DIR} --directory src --capture --output-file maus.info
#    genhtml -o doc/coverage/ maus.info
#fi

=======

and this line

#>>>>>>> MERGE-SOURCE

needs to be deleted. bzr inserts diff output into a file when it finds a conflict. Now I get errors like:

======================================================================
ERROR: Failure: ImportError (dynamic module does not define init function (initlibxml2))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/cr67/G4MICE/MAUS/maus_littlefield/third_party/install/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/cr67/G4MICE/MAUS/maus_littlefield/third_party/install/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/cr67/G4MICE/MAUS/maus_littlefield/third_party/install/lib/python2.7/site-packages/nose-1.1.2-py2.7.egg/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/cr67/G4MICE/MAUS/maus_littlefield/build/test_gdml_to_maus_module.py", line 8, in <module>
    from geometry.GDMLtoMAUSModule import GDMLtomaus
  File "/home/cr67/G4MICE/MAUS/maus_littlefield/src/common_py/geometry/__init__.py", line 4, in <module>
    from geometry.CADImport import CADImport
  File "/home/cr67/G4MICE/MAUS/maus_littlefield/src/common_py/geometry/CADImport.py", line 6, in <module>
    import libxml2
ImportError: dynamic module does not define init function (initlibxml2)

I think we fixed this but I can't remember how...

#4

Updated by Rogers, Chris about 11 years ago

  • Assignee changed from Rogers, Chris to Littlefield, Matthew
#5

Updated by Littlefield, Matthew about 11 years ago

I have fixed scons and taken out the corrupted code from run_tests.bash, when I run this I get this error,

test_digit (test_MapCppTOFDigitization.MapCppTOFDigitizationTestCase) ... python2.7: src/map/MapCppTOFDigitization/MapCppTOFDigitization.cc:152: std::string MapCppTOFDigitization::process(std::string): Assertion `mod != __null' failed.
run_tests.bash: line 29: 2716 Aborted nosetests --with-coverage -v build

Not sure what this is or if I'm doing this?

#6

Updated by Rogers, Chris almost 11 years ago

Well done for getting this in - looks really nice!

Comments:

Style
-----
  • If you put pylint disable directives at the top of the file, it disables for the entire file. Sometimes you need to do this, but often this is not correct. Better to disable per line by putting a comment at the line end. In particular, you disable the 80 character line limit. Please don't do this!
  • Can you put a copyright notice at the top of your files? Something like:
    #  This file is part of MAUS: http://micewww.pp.rl.ac.uk:8080/projects/maus
    # 
    #  MAUS is free software: you can redistribute it and/or modify
    #  it under the terms of the GNU General Public License as published by
    #  the Free Software Foundation, either version 3 of the License, or
    #  (at your option) any later version.
    # 
    #  MAUS is distributed in the hope that it will be useful,
    #  but WITHOUT ANY WARRANTY; without even the implied warranty of
    #  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    #  GNU General Public License for more details.
    # 
    #  You should have received a copy of the GNU General Public License
    #  along with MAUS.  If not, see <http://www.gnu.org/licenses/>.
    

I sorted out the latest cdb version as a third party library (together with real suds) and pushed it into bzr+ssh://bazaar.launchpad.net/~chris-rogers/maus/libxml2_test/

Also have a look at the attached test log. The errors where we have

CdbTemporaryError: Unable to contact CDB server at http://cdb.mice.rl.ac.uk/cdb/geometrySuperMouse?wsdl

are because I didn't use the suds testing mock-up. But stuff like
OSError: [Errno 2] No such file or directory: '/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_2.gdml'

Looks like genuine error...

#7

Updated by Rogers, Chris almost 11 years ago

test log attached...

#8

Updated by Littlefield, Matthew almost 11 years ago

I have had a look at the errors and I think that the problem is when I call environment variables. To call the MAUS_ROOT_DIR i use

os.environ['MAUS_ROOT_DIR']

All the errors where the file can't be found has this in it but it works on my machine? I think this isn't findind MAUS_ROOT_DIR on jenkins which way do you find this variable?

I'll change it and check it works to see if this is the problem.

FYI, I'm getting different errors on my machine to what you have seen....

#9

Updated by Rogers, Chris almost 11 years ago

For example src/common_py/geometry/testCases/testPacker/FileList.txt

/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_2.gdml
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_0.gdml
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_3.gdml
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/fastradModel_materials.xml
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/fastradModel.gdml
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_1.gdml
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_4.gdml
/home/matt/maus-littlefield/src/common_py/geometry/testCases/testGeometry/Step_5.gdml
#10

Updated by Rogers, Chris almost 11 years ago

Note I completely removed the old cdb stuff and changed the build scripts to add suds and cdb in 44suds.bash 45cdb.bash

#11

Updated by Rogers, Chris almost 11 years ago

Also

echo $MAUS_ROOT_DIR
/home/cr67/G4MICE/MAUS/maus_littlefield_merge

which is correct. Done using

 source env.sh
as usual

#12

Updated by Littlefield, Matthew almost 11 years ago

OH, I forgot that I have hard coded the Filelist.txt woops I'll change it :-).

I noticed you removed the cdb stuff and I have since built the suds etc scripts on my machine and I no longer get the SuperMouseGeometry connection errors.

#13

Updated by Littlefield, Matthew almost 11 years ago

I have fixed some errors in the tests and added the legal blurb I still need to organise the pylint stuff.

I am getting errors from run_tests.bash which are different to the ones you have seen and was wondering if its a quick fix? If not maybe we could take a look at them at Fermilab? Ive attached the errors

#14

Updated by Rogers, Chris almost 11 years ago

All of the stuff in test_ReducePyMatplotlibHistogram.py, well, nothing to do with your code, so maybe something got overwritten. The other stuff is with the cdb tests, which I removed. Probably if you do scons -c; scons these tests should go away. The tests now live in Antony Wilson's code.

#15

Updated by Rogers, Chris almost 11 years ago

  • Assignee changed from Littlefield, Matthew to Rogers, Chris

So... tests were failing on my machine, I fixed/hacked some stuff I didn't like:

  • I replaced the cdb test server with some mock cdb classes because the actual cdb calls were failing on network issues.
  • I found that because the GDML formatter overwrites the input files, someone Matt had accidentally overwritten his own test data. In fact, because GDML formatter overwrites input files, we can only run the test once without doing bzr revert. I changed to have an input folder and output folder for the data

The only test fail now on my machine is:

======================================================================
ERROR: method test_format
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/cr67/G4MICE/MAUS/maus_rogers/build/test_geometry/test_gdml_formatter.py", line 194, in test_format
    self.gdml.format()
  File "/home/cr67/G4MICE/MAUS/maus_rogers/src/common_py/geometry/GDMLFormatter.py", line 247, in format
    self.merge_maus_info(self.configuration_file)
  File "/home/cr67/G4MICE/MAUS/maus_rogers/src/common_py/geometry/GDMLFormatter.py", line 112, in merge_maus_info
    root_node = config.childNodes[2]
IndexError: list index out of range
-------------------- >> begin captured stdout << ---------------------
Run information merged!

but I need to do another pass for more clean up. Sorry Matt, this will take another couple of days.

#16

Updated by Rogers, Chris almost 11 years ago

Should say - the reason I didn't fix that test is because I really didn't have a clue what the function was doing...

#17

Updated by Littlefield, Matthew almost 11 years ago

Heres the correct files :-)

#18

Updated by Rogers, Chris almost 11 years ago

I now have this passing in rogers build e.g. http://test.mice.rl.ac.uk/job/MAUS_rogers/196/

I did quite a bit of messing about - soft coding the URL for test server, moving the executables to bin/utilities, moving the file cache to files/geometry, separating the input directory and output directory for e.g. formatter, few other stuff. I want to finally make an integration test that checks that we can upload sample geometries to the test server, pull them back down again and run simulation against them before I do the merge.

#19

Updated by Littlefield, Matthew almost 11 years ago

That sounds good. Can I merge it with my local copy so I have the current version? Because I'm going to be doing some stuff with the tracker schema this week and would like my version to be current :-)

what is the branch name?

lp:maus_rogers?

or should I want till the maus release? May be better ...

#20

Updated by Rogers, Chris almost 11 years ago

Branch name is:

bzr+ssh://bazaar.launchpad.net/~chris-rogers/maus/devel/

I think it would be worth at least having a look at the code to see what I've done. You may have some objections, in which case let me know.

Nb: Try to keep your branch clean, there was a whole load of stuff which should not have been committed (temporary files, etc). Makes it quite a pain to merge!

#21

Updated by Littlefield, Matthew almost 11 years ago

OK Ill be cleaning up my branch :-) so its less painful.

When I try to merge is gives me a warning that I'm doing a criss-cross merge and I read bzr help and it looks scary I think Ill wait til the release because I don't want to mess it up :-)

#22

Updated by Rogers, Chris almost 11 years ago

Okay, well feel free to download my branch at least to have a look...

#23

Updated by Rogers, Chris almost 11 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Now merged and will be in the next release

Also available in: Atom PDF