Software sustainability evaluation (05/09/11)

This document presents a sustainability review of MAUS done by the Software Sustainability Institute as part of a Software Sustainability Institute / MAUS collaboration. The review was undertaken in the weeks of 28/08/11 and 05/09/11 and consists of a tutorial-based evaluation of MAUS from the perspectives of:

  • Users who download MAUS releases and use it to build applications.
  • Developers who work on MAUS itself, writing code to be submitted to the source code repository.

Please contact the reviewer at if you have any queries.

As the review included this wiki, certain pages cited in this review may have been updated or deleted since the review was completed.

Tutorial-based evaluation of the user experience

This is a tutorial-based evaluation intended to provide a pragmatic evaluation of usability of MAUS in the form of a reproducible record of experiences. This gives a developer a practical insight into how the software is approached and any potential technical barriers that prevent adoption.

Downloading and building MAUS

Summary of experience

I downloaded, built and installed the latest release of MAUS (

The machine I used ran Scientific Linux SL release 5.6 (Boron).

Starting from the MAUS wiki page, it was straightforward to see what to do:

I started to follow the instructions in Installing MAUS. The way the instructions were written implied that if gcc was available on my machine then all would be well:

"If you already have development tools and libraries installed like GCC, you can skip this section and continue to the next section... if you are unsure, continue.
To check if you have a compiler (which generally indicates that you have the rest) run at the command line the command gcc."

I ran gcc and got the same response as in the instructions

gcc: no input files

So, I proceeded to build MAUS

A error occurred
FATAL: gcc/g++ is required to compile code.

So, I did
yum install gcc-c++

and reran the build script. Another error occurred
Checking for libX11 ... no
configure: libX11 MUST be installed
FAIL: Failed to configure/make
WARNING: MAUS already setup
./tests/integration/install_then_build_then_test.bash: line 6: scons: command not found
./tests/integration/install_then_build_then_test.bash: line 7: scons: command not found

Seeing the reference to libX11 I did
yum install libX11

and was informed that
Package libX11-1.0.3-11.el5.i386 already installed and latest version.
Nothing to do

Now I had a dilemma. Running the build script would take forever. So, maybe it was an issue with scons? So I did
yum install scons

and then reran the build script. The libX11 message appeared but I ignored this as the build continued on. Another error occurred
Checking whether the C++ compiler worksyes
error: no result
Checking for C++ library json... no
!! Can't find jsoncpp which is needed.
You may install it by running:
     MAUS_ROOT_DIR=/home/michaelj/MAUS/maus-latest-release ./third_party/bash/52jsoncpp.bash

So, I followed this advioce and ran
MAUS_ROOT_DIR=/home/michaelj/MAUS/maus-latest-release ./third_party/bash/52jsoncpp.bash

This failed with
./third_party/bash/52jsoncpp.bash: line 57: test_lib_json: command not found

INFO: The package should be locally build now in your
INFO: third_party directory, which the rest of MAUS will
INFO: find.

I was now quite lost so decided to start again and install the recommended prerequisites as if the gcc check had failed. So I did
yum groupinstall "Development Tools" 
yum groupinstall "Development Libraries" 
yum install wget libX11-devel libXft-devel libXext-devel libXpm-devel libX11-devel

And then re-ran the problematic command
MAUS_ROOT_DIR=/home/michaelj/MAUS/maus-latest-release ./third_party/bash/52jsoncpp.bash
nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module
Ran 0 tests in 0.001s


At this point, I decided to start the entire build from scratch. It worked! This took about 3 hours in total.

I then tried rerunning the tests using

bash run_tests.bash

An error occurred

I did

Only when I did
export MAUS_ROOT_DIR=/home/michaelj/MAUS/maus-latest-release

did the tests run OK.


Build script:

  • It would be useful if the build script could in some way checkpoint its progress so if the build fails it could be restarted from the last successful checkpoint. The build script takes way too long to run from scratch each time.

Installing MAUS page:

  • The user should be told to run the yum instructions regardless. The gcc check is not good enough to check for all the required packages. This would avoid users running into the same issues as I did.
  • It would be useful to state how long the build takes so they can go off and do something else.
  • The size of the original directory in KB was of the order of 211MB and, once built, 2.4GB. It would be useful to state this in the prerequisites as I ran out of disk space during a build on another machine.
  • Given the time the build takes and the many tests run at the end, it would be useful if users were recommended to run typescript first so they have a log of what's happening. This would also make it easier, in case of problems, so submit the correct information to a ticket or e-mail list.
  • In "Troubleshooting", one hint is "I get errors in the test output: this is normal - if the tests finish with a line like OK, then the tests passed. If the tests finish with a line like FAILED (errors=1, failures=1) (or worse a segmentation fault), then the tests failed. Some of the tests are actually checking that errors are produced on bad input." An example of a failure message which is OK should be given so the user knows what to expect e.g.
    [ RUN ] CppErrorHandlerTest.HandleSquealTest
    Traceback (most recent call last):
      File "/home/michaelj/MAUS/maus-latest-release/src/common_py/", line 166,
        in HandleCppException
    ErrorHandler.CppError: a_test at exc::test
    [ OK ] CppErrorHandlerTest.HandleSquealTest (45 ms)
  • If a test does fail, it would be useful for the user to know what to do, even if it's just a request to get in touch with you.
  • "Introduction" lists 5 steps (Prerequisites, Getting code, Dependencies, Build, Test) but there are only 3 sections (Prerequisites, Getting the code, Dependencies and building MAUS). It would be better if these were consistent, the user may wonder where the Build and Test instructions are otherwise.
  • "Prerequisites"'s two "next section" links to are broken.
  • "Getting the code" has typos, "repository is meant to be unstable" -> "repository can be unstable", "post these instruction here" -> "...instructions..."
  • "Dependencies and building MAUS" says users should
    cd maus

    but for users using a release it should read
    cd maus-latest-release
  • The page lists the 5 commands executed by the build script. This is out-of-synch with the one in the release I used, which executes 6, and the current version in the repository, which executes 7. Such divergence could be avoided by just saying to users that they can more the script to see what it does and this page could just have a bulleted list with the main steps in English text.
  • The "This has worked on live display" list cites 6 platforms (each 64 or 32 bit) but the live display link at had only 3 platforms plus a "This project is currently disabled" message. It would be good if the reason for this was shown, along with the date it was disabled.


Using MAUS

Summary of experience

Once I'd downloaded MAUS there was no indication as to what to do next. There is a wiki page for MAUS User Documentation which states "There are several applications in the MAUS schema" but it does not say what these are or where to get them.

A search of the wiki using terms like "example", "application", "example application", "sample", "tutorial", "getting started" revealed open ticket #445 ("Some simulation examples would be useful").

In the wiki index there was an Examples link but this was just an empty page.

A search of JISC-USER revealed no examples of usage beyond a bounce from JISC-DEV about mentioning ...a demo script in 'bin'.... So, I looked at the bin directory. Looking at the source code confirmed these were sample applications, though I only knew these were samples as the code looked similar to the example in a MAUS IPAC paper I had read.

I successfuly ran:

cd bin/

And likewise for the other .py files in bin.

In terms of sample data, looking at the source code showed that uses a hard-coded JSON document and uses src/input/InputCppData/02873.003. The "Offline data access" section in MAUS User Documentation took me to a general page,, and to get data would involve setting up Grid certificates so I didn't proceed further.

In terms of writing applications, the examples in bin serve as a good starting point and I was able to customise these easily. According to the "Changing application modules" section in MAUS User Documentation, there is a list of components at Components. However there is not a direct correspondence between these and those cited in the sample code. The Data structure page provided an acceptable introduction to the JSON data structure passed around but examples would have been useful.

The release structure is a snapshot of the repository contents, which is very useful for users who want to make the transition to project members/developers.


Help users to get started:

  • Provide a "getting started" tutorial which takes users through writing their first application e.g. a simple map-reduce chain, so they can learn about the structure of applications and how to run these.
  • Provide a list of components that can be used to compose applications (e.g. the workers). Ideally, auto-generated API documentation would describe everything a user needs to use the components, and the component list could just link to those pages.
  • Supplying more sample data sets would be useful, either in the release, or online, without the need to use Grid certificates and register. This would lower a barrier to use.
  • In terms of understanding how MAUS works in enough detail to write an application, a good high-level overview was at the top of the Components page. This, with content from the IPAC paper, and the introductory section from Data structure would yield a good page-length user-level introduction to MAUS and its main concepts and how to construct applications that use it. However, the code example from IPAC would need to be consistent with the current code-base (it isn't at present).

MAUS User Documentation page:

  • State that the bin directory contains example applications along with a suggestion as to which one the user can run first (e.g. is a good candidate). Explain in general terms what it does and what it outputs.
  • In "Getting Help", it's preferable (to reduce the risk of inconsistent information) to have a single separate "Getting Help" page for all classes of user (release users, MAUS developers etc.
  • On such a "Getting Help" page, clarify the preferred means of asking for help - e-mail or ticket - so the user can select the option most likely to get their request addressed. Also, clarify whether any queries sent to the e-mail lists are entered as issues (this is useful information for project members too).
  • In "Configuration manipulation", the "List of valid materials" link is an undefined wiki page. Avoid undefined links and empty wiki pages if possible. If the content is work-in-progress, then state this (and, ideally an estimated completion date).
  • In "Configuration Manipulation" there are links to documents (, but it is unclear how to actually relate these to configuring MAUS.
  • In addition, examples of configuring MAUS using the specified settings would be useful.
  • These documents cite G4MICE which could be confusing for new users who expect to see MAUS and are unaware of its history.
  • It would be useful if these docs were dated and specified which releases they are applicable to.

Publications policy:

Components page:

  • There is a typo in the MC table, Simulation row, "default is default" should read "serial is default" (or similar).

Data structure page:

  • It would be useful if this page had some example documents.
  • There are some hyperlinks to "double" but other "double" are not hyperlinked. Same for the other types. Either do them all, or, preferably, to avoid too many hyperlinks, have a table at the start under the JSON link with "common types" used.
  • Retitling the "persistency_considerations" and explaining why this link is here and whether the user needs to read it would be useful.
  • The Spill table "triggers" name has type "trigger" but that section is empty.
  • The Triggers and Particles sections are empty. A user might think its a "todo" so an explanation, or example would be useful.
  • The link from "digitized hits" goes to "Digits" - change heading to "Digitized Hits - digits"
  • For consistency have a Type column for TOF, SciFi Tracker and EMR too.

Tutorial-based evaluation of the developer experience

Again, this is a tutorial-based evaluation intended to provide a pragmatic evaluation of usability of MAUS in the form of a reproducible record of experiences. This gives a developer a practical insight into how the software is approached and any potential technical barriers that prevent adoption.

From the Installing MAUS page it was clear what I needed to do as a developer to get hold of the source code.

I followed the link to Bzr usage via the link in "There is a MAUS-specific tutorial of using bzr here." and scrolled down to "Installing bzr".

I then ran

bzr --version
bash: bzr: command not found

So I

I then installed bzr by

sudo su -
su -c 'rpm -Uvh' 
warning: /var/tmp/rpm-xfer.ENlXbE: Header V3 DSA signature: NOKEY, key ID 217521f6
Preparing...                ########################################### [100%]
   1:epel-release           ########################################### [100%]

$ su -c 'yum install bzr'
Loaded plugins: kernel-module
epel                                                     | 3.7 kB     00:00
epel/primary_db                                          | 3.1 MB     00:06
Setting up Install Process

And did a check
Bazaar 2.1.1 -- a free distributed version-control tool

I then Googled for LaunchPad and followed the link to I then created a login

  • Clicked "Creating" an account
  • Entered full name, e-mail, password
  • Waited for confirmation e-mail
  • Entered 6 digit confirmation code
  • Clicked "Yes Sign Me Up"

Then I told bzr my LaunchPad ID

$ bzr whoami "Mike Jackson <>" 

The instructions then said I had to set up my SSH RSA key. So, I had to visit my LaunchPad profile. There was an "SSH keys+" button with "No SSH keys registered" so I clicked on "+" and was taken to the "Change your SSH keys" page.

I then clicked on "Importing your SSH key" and was taken to

On my server I ran

ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/michaelj/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): XXXXXX
Enter same passphrase again: XXXXXX
Your identification has been saved in /home/michaelj/.ssh/id_rsa.
Your public key has been saved in /home/michaelj/.ssh/
The key fingerprint is:

I pasted .ssh/ into the Change your SSH keys page form and clicked "Import Public Key". I then attempted to branch MAUS
bzr branch lp:maus
You have not informed bzr of your Launchpad ID, and you must do this to
write to Launchpad or access private data.  See "bzr help launchpad-login".
bzr: interrupted    

But I did the whoami command! So, after Googling, I tried
bzr launchpad-login
No Launchpad user ID configured

So I tried
bzr launchpad-login "Mike Jackson <>" 
bzr: ERROR: The user name mike jackson <> is not registered on Launchpad.

bzr launchpad-login "" 
bzr: ERROR: The user name is not registered on Launchpad.

So I logged into LaunchPad. At the top-right it says "Mike Jackson (michaelj-NNNN)" so assume that's my user ID. LaunchPad don't make this clear since their login page takes an e-mail address and password, not the user ID (unlike, for example, SourceForge).
Trying again
bzr launchpad-login "michaelj-NNNN" 
bzr branch lp:maus
Enter passphrase for key '/home/michaelj/.ssh/id_rsa':
Branched 615 revision(s).  

MAUS then built fine.


Installing MAUS page:

  • In "Bazaar repository (recommended for developers)" reword oddly-phrased sentence "Unstable code, this is, since..." to "The repository can be unstable since it's under development".
  • The, "branch" in "Please branch MAUS. The code is hosted on launchpad under the MAUS project..." is a hyper-link to Wikipedia. I'd prefer if this was not hyperlinked. It would be better if this was in the bzr tutorial.
  • Likewise for the "If you get a 'command not found'..." instruction which can also be folded into the bzr tutorial.
  • The "here" link in "If you run into connectivity issues, you can check the bazaar server status here." links to, which isn't Bazaar which may be confusing to new users.

Refactor the bzr instructions:

  • The pages Bzr usage and Installing Bazaar overlap. It would be more understandable, and remove redundancy, if these and the Installing MAUS page's "Bazaar repository (recommended for developers)" section was renamed to "Getting the latest code" and then a link to a new page which describes:
    • Bazaar and how it's used as the revision control system.
    • A link to Bazaar. Don't list the latest stable bzr release as this information could get out-of-synch with the bzr web site. Just state the minimum version number required.
    • LaunchPad, a link to the site, and a statement that this hosts MAUS's Bazaar repository.
    • How to create an SSH key, and why it's needed.
    • How to create a LaunchPad login, paste in the SSH key and get the user ID.
    • How to install Bazaar.
    • How to checkout the repository using Bazaar.
  • A troubleshooting section with common error messages
    • e.g. if a user has not registered a SSH key then the error is
      bzr launchpad-login "michaelj-NNNN" 
      bzr: ERROR: The user michaelj-NNNN has not registered any SSH keys with Launchpad.
      See <>
    • e.g. If the user has entered an incorrect SSH key or made a cut-and-paste error then the error is
      $ bzr branch lp:maus
      Permission denied (publickey).
      bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and 
      permissions, and report a bug if problems persist. 

Bzr usage page:

  • The page would be better pulled out into two pages, one on HOW-TOs for bzr-specific operations (e.g. branch, merge, diff etc) and one on the guidelines (i.e. how the repository is structured, the difference between trunks and branches, criteria for commiting to trunk etc).
  • The page cites "there are some style requirements in the style guide that are not (and cannot) be checked by". A link should be provided to the Coding style page.

Installing Bazaar page:

  • This states "Please find your system below then follow those instructions" but the only system listed is SL 4.8.

Provide a single page listing all the third-party packages:

  • A single page (perhaps "Prerequisites for developers") listing tools bundled in MAUS and those developers need to install themselves (with version numbers, links to web sites, and download pages) would help developers to get their development environment set up quickly. At present this information is both implicit or explicit and spread across various wiki pages.
  • A list I identified was as follows:
    Bundled in Linux
    valgrind - debugging, profiling, dynamic analysis
    gprof - GNU C++ profiler
    gcov - test coverage tool
    To be deployed by a user
    Bazaar - distributed version control
    Coverity - static code analyzer
     How to install?
    lcov - GUI for GCC's gcov test coverage tool
     How to install?
     Where to get and How to install?
    SWIG - Simplified Wrapper and Interface Generator
     Automatically creates Python wrappers for C++ code.
    Geant4 - geometry and tracking
     MC simulator of particles through matter.
     Assumes particles and events independent
    pylint - Python code static checker
     [Useful to have link to list of warning/message codes]
    cpplint - C++ code static checker
     Uses version bundled in Python.
    Scons - Software Construction Tool. 
     "Superior" Python alternative to Make.
    ROOT - large scale data analysis framework
    PyROOT - Python wrapper for ROOT
    Validictory - Python data validator
    Python coverage
    Minuit - multi-parameter FORTRAN function.
    PAW - Physics Analysis Workstation

Provide a single page summarising useful commands:

  • A page summarising useful commands for use when developing MAUS (e.g. how to run the Python and CPP style checkers or how to check test coverage) would be useful. Again, this information is currently spread across numberous wiki pages. This was a list of tools I determined:
  • Certain scripts are named on various pages e.g. with no mention as to where in the repository these are located. Ideally, all files cited should have absolute path names from the repository root.
  • This is a first draft of content for such a page:
    How to compile a single CPP file?
    -This rebuilds any changed files in src/ and puts them in build/
      scons build 
    How to recompile a Python file?
    -This rebuilds any changed files in src/ and puts them in build/
      scons build 
    -It also updates build/ to include any new files.
    -build/N.pyc files are created by Python when build/ files are imported.
    How to run unit tests
    -A single file.
     build/test_cpp_unit --gtest_filter=VirtualPlaneTest.*
     ./tests/cpp_unit/test_cpp_unit --gtest_filter=PolynomialTest*
     (same file)
     python build/
    -All files.
    How to run an integration test
    -A single file.
    -All files.
    How to run a CPP style check
    -A single file.
     python third_party/install/lib/python2.7/site-packages/ FILE
    -All files.
    How to run a Python style check
    -A single file.
     pylint FILE
    -All files.
    How to run all tests
    Build, test, run everything
    How to run Doxygen
    How to run Coverity on C++
    How to run lcov on C++
    How to run Python coverage
     coverage run
     coverage report
     coverage html resultDirectory

Refactor pages describing the design:

  • There is page detailing the MAUS Design. This though mixes the distribution structure, programming language, rationale for why the design is as it is and various thoughts. It's preferable to keep these separate so the reader can understand exactly what the state of the current design is and why. The Persistency considerations link from Data structure likewise mentions documentation (e.g. a JSON example), implementation information and design ideas.

Design page:


  • In Release schedule it could be more manageable to make the milestones actual milestones in the ticketing system, then this page could be auto-generated.
  • In Coding style there is a link to the coding standards which is a 58 page PDF document. This can provide intimidating to new developers or potential contributors. However, the fact that there is automated style-checking can mitigate this.
  • In contrast, Coding standards links to the Google style-guide. This could provide confusing. There should be a link to one set of standards only.
  • MAUS Developer Documentation "Contributing to MAUS" links to Bzr usage. See earlier comments on pulling out information on the guidelines on how repository usage is managed.
  • Coding style says to run tests/style/ and python ${MAUS_ROOT_DIR}/third_party/install/lib/ <filename1> <filename2>. What is the difference?
  • In the above, the path doesn't exist in the repository.
  • Code review could be usefully combined with guidelines on use of the repositry (commiting code to trunk) as a checklist that developers should go through on their own code, before submitting it to review or checking it.
  • It's not clear if code can be submitted by non-MAUS members i.e. can anyone with a LaunchPad login commit code to MAUS? This should be stated.
  • A map from Issue.Category (there are many) to the design/or directories/modules/components/third-party bundles in the code base would be useful to have on the wiki.
  • Additional build flags should be linked from the Installing MAUS page.
  • is important so should cited on the Data structure page, not buried within Persistency Considerations which can appear to be notes on work-in-progress or design ideas.
  • The shell instructions to build the docs cited in Release procedure should be added as a shell script in the repository.
  • In Jenkins a summary of what Jenkins does and runs, for new developers and interested users, would be useful.
  • A page on how the Doxygen run is initiated and how often its updated online would be useful. By chance I gound a reference to a CRON job at
  • The following pages didn't seem linked from any others.
  • Any pages relating to tickets should cite the associated ticket number.


From a user's perspective, bar initial build issues caused by ambiguous instructions (which are easily fixed) it was very straightforward to download and build. It is clear how to get help and support. The only glaring is how to proceed once MAUS has been built i.e. a reference to the sample applications in the bin directory and a tutorial on writing an application. A single page summarising the current design would be useful too.

From a developer's perspective, there is comprehensive information on all aspects of developing MAUS including how/where to get the code, the design, project standards and management, help and support. The main recommendations are:

  • The information could be more effectively structured on the wiki (one problem with wikis is they can evolve in haphazard ways) to differentiate between the current design of MAUS (how it is now and how it works), the design rationale (why it is as it is) and possible alternatives. At present these are intermixed which can make understanding the current state difficult.
  • A list of tools bundled in MAUS and those developers need to install themselves would be useful.
  • A page summarising useful commands for use when developing MAUS (e.g. how to run the Python and CPP style checkers or how to check test coverage) would be useful.