Project

General

Profile

Support #456

Code review - cadimport.py

Added by Rogers, Chris almost 12 years ago. Updated over 11 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
Littlefield, Matthew
Category:
common_py
Target version:
Start date:
19 May 2011
Due date:
30 May 2011
% Done:

0%

Estimated time:
Workflow:

Description

Email from Littlefield:

Ive just committed some code to my devel branch and i was wondering if you could take a look take a look at my units test because im not sure im doing them correclty and not entirely sure how to test the initialiser?

Files are

cadimport.py
test_cadimport.py

#1

Updated by Rogers, Chris almost 12 years ago

  • Assignee changed from Rogers, Chris to Littlefield, Matthew

General

Nice one. I like the fact that you laid out the comments more-or-less according to the guidelines.

CADImport.py main() function

The stuff at the bottom

geometry1 = CADImport("fastradModel.xml", "GDML2G4MICE.xsl", "", "OUTPUTFILE.txt")
geometry2 = CADImport("fastradModel.xml", "GDML2G4MICE.xsl")
geometry3 = CADImport("fastradModel.xml", "Merge.xsl", "FieldInfoTest.xml")

geometry1.XSLTParse()
geometry3.AppendMerge()

You should put this in a main() function like

def main():
  geometry1 = CADImport("fastradModel.xml", "GDML2G4MICE.xsl", "", "OUTPUTFILE.txt")
  geometry2 = CADImport("fastradModel.xml", "GDML2G4MICE.xsl")
  geometry3 = CADImport("fastradModel.xml", "Merge.xsl", "FieldInfoTest.xml")

  geometry1.XSLTParse()
  geometry3.AppendMerge()

Then add code like:

if name == '__main__':
main()

That way, if I do

python CADimport.py

The code in main runs. If I do

import CADimport

the code in main() does not run (which is what we want when we import a library). But if for some reason I decide I want to run the code in main() I can just call the main() function.

Comments

What I think is missing is a basic overview of what the class does. Coming to it with half a memory of our conversation I have a vague feeling that this does a merge between two files, but you really need to explain this to me. Most people haven't heard of xslt, you should at least explain what this does in a sentence. This can either go at the top of the class or at the top of the file (module).

        This method initialises the class and takes 7 parameters. The later 4 parameters are optional
        but if one of the latter needs to be initialised a blank "" needs to be placed to set preceeding 
        params to None. All arguments should be file names or paths.

        @param xmlin1,   first xml argument
        @param xsl,      xslt argument
        @param xmlin2,   second xml argument, Default None
        @param output,   output argument, Default None
        @param mergein,  template XSLT for append, Default Merge.xsl.in
        @param mergeout, destination for edited XSLT for append, Default Merge.xsl

So, typically the developer can see the arguments that are optional so you don't need to specify this. What you do need to specify is that xmlin1 is a file path to an xml filename that will be merged with xmlin2 or whatever...

Tests

Testing is a bit of a pain in code that is essentially making an interface to an external library. Tests I would like to see are like:

AssertMerge

Write a short file to $MAUS_ROOT_DIR/tmp, only has to be a couple of lines long. Check that all occurences of EDIT are replaced for cases like
  1. multiple EDIT commands on same line
  2. EDIT commands on different lines
  3. Error message if EDIT not found

XSLTParse

Write a short file to $MAUS_ROOT_DIR/tmp, only has to be a couple of lines long and check that it gets parsed correctly, or appropriate error if the file doesn't exist etc.

I don't see how the merge code works for when we have two files (maybe this code doesn't exist yet?)

#2

Updated by Tunnell, Christopher almost 12 years ago

  • You import the class you're definining? In CADIImport.
  • I don't know the answer, but can you use a standard XML parser with python? Python has it's own XML parser; is it harder to use? It just removes the overhead for a new person getting this running since we should probably outline what they need to install to get this running and I can't find that package on the PyPi package repository (ie. easy_install).
  • Worth deleting lines like "#outputExists = os.path.exists(output)" since version controls keep track of old code if you need to recover it and it just clutters it up
  • Does it pass pylint? There are examples of turning off pylint stuff in src/core/cdb if certain warnings don't apply
  • How do you want people to run this (even though it's an expert task)? It would be nice to have some tool that took a few command line arguments or datacards that lived in /bin/ in the end.
  • Can't actually run it since I don't have the required files which is fine for the moment

All looks nice! Glad to see progress

http://bazaar.launchpad.net/~matthew-littlefield/maus/devel/view/head:/tmp/test_CADImportFunctions.py

http://bazaar.launchpad.net/~matthew-littlefield/maus/devel/view/head:/src/core/CADImport.py

View files at links above.

#3

Updated by Tunnell, Christopher almost 12 years ago

To Rogers: what's required to close this issue? The code review is finished. Does he need to respond or something?

#4

Updated by Rogers, Chris almost 12 years ago

To Rogers: what's required to close this issue? The code review is finished. Does he need to respond or something?

Right - I'd like Matt to bounce this back when he's fixed the changes.

#5

Updated by Littlefield, Matthew almost 12 years ago

To Rogers:

  • All of the your commets from #1 down to the tests have been acted upon.
  • The tests have muddled me up a little. Ive written two test files one is in /tests/unit/test_CADImport.py
    which i think doesnt test anything really at all at the moment. The other is in /tmp which actually does test
    the class. Do I need to merge the to so the actual tests from /tmp have the try: except: style incoporated to
    turn these tests into proper unit tests?

To Tunnel:

  • Importing my class was me trying to solve a problem which has been resovled but I have tidied things up a bit
    more now and will be uploading more stuff later.
  • I didn't know python has its own parser but I've found this code on the net and it does everything I need it
    too. The libraries required are libxml2 and libxslt which are easily accessible. I managed to download and
    install them using the synaptic manager on ubuntu.
  • I will change the output exists stuff today.
  • I Haven't tried it on pylint yet, again I'll run it on pylint today.
  • I intend this to be a base class and I would like to write another class say "upload" which will have
    which will take arguments of the name of the gdml file to upload (or files) and it will then merge the files
    together and upload them all to the database in one go. Also I will write another class ("download") for
    users to use which has methods to download a Text File with the geometry in MICE Module format, this will
    use XSLTParse() in this class, so the users can have a local copy if they wish. Generally I think all users
    will have to do to download a geometry is to search the CDB find the ID number they need and write that into
    their datacards and the software downloads and merges all the necessary gdml information. I think we have 75%
    of this code written already, certainly by Ivan, I just need to bring it all together.
  • I'll upload another folder with the test gdml's in so you can see the results (should have thought of that
    before).

:-)

#6

Updated by Tunnell, Christopher almost 12 years ago

Small comment but not critical to implement: the more dependencies you have, the more we have to worry about support for that library and people figuring out how to install stuff. Some stuff we can ship with MAUS but it's not easy to gaurantee it works the same everywhere: for instance most people don't have Ubuntu and its package manager. They have scientific linux that can't do anything.

#7

Updated by Littlefield, Matthew almost 12 years ago

Thats a fair point I didn't consider that. I'm going to look at building a bash script, similar to the ones already in MAUS, so the dependencies needed will be built with MAUS becuase I have felt the torments of scientific linux and libraries.

#8

Updated by Littlefield, Matthew almost 12 years ago

  • Assignee changed from Littlefield, Matthew to Rogers, Chris
Ive added some more code to my devel branch (revision 501) would you please review
  • src/core/CADImport.py
    • In particular the constructor as I'm not entirely sure its the best way of instatianting the class object
  • tests/unit/test_CADImport.py
    • just too see if these tests are ok

and lastly ive added a couple of .bash scripts in the third party directory to build the libraries needed for the code above and just want to check they look ok?

Thanks

#9

Updated by Rogers, Chris almost 12 years ago

  • Assignee changed from Rogers, Chris to Littlefield, Matthew

Basic workflow

uploadGeometry.py is an executable script that uploads geometry into config db
  1. calls GDMLFormatter.formatter to manipulate paths so that they are correct
  2. calls GDMLtoCDB.gdmltocdb to upload to cdb
  3. calls packer to pack into a zip file for local cacheing
no download executable as yet
  1. GDMLtoCDB.downloader pulls data off the configdb
  2. CADImport.CADImport does the merge with an external geometry file containing e.g. field maps

Comments

uploadGeometry
  • STYLE argparse stuff is nice - but better to use src/common_py/Configuration.py and associated functions so the interface is common. This will get argparse interface some time.
  • STYLE uploadGeometry.py - please use consistent style for file name capitalisation
formatter:
  • BUG - hard coded file names should be changeable at runtime (Configuration file)
  • STYLE "obj" - better variable name please
GDMLtoCDB:
  • BUG - again server url should be soft coded
  • STYLE - notes field not commented
  • STYLE - change end of file string to be more formal. Probably should be an xml comment
  • STYLE - no colons (compatibility with smb), no spaces (sanity) in filenames
CADImport
  • BUG - mergout variable name clearly wrong. Should have been picked up in tests.
#10

Updated by Rogers, Chris over 11 years ago

  • Target version changed from Future MAUS release to MAUS-v0.1.0
#11

Updated by Littlefield, Matthew over 11 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Putting a new code review in which will cover this file.

#12

Updated by Rogers, Chris over 11 years ago

  • Status changed from Closed to Rejected
  • % Done changed from 100 to 0

Also available in: Atom PDF