Sample event loop with some extra verbosity, printing memory src/common_cpp/DataStructure/Spill.hh memory allocations

emitting
Spill Default Constructor 0x621d8e0 <--- NEVER DESTRUCTED
Spill Copy Constructor 0x6a83ea0
Spill Copy Constructor 0x6a83f20
Spill Destructor 0x6a83ea0
Spill Copy Constructor 0x6a83ea0 <---- NEVER DESTRUCTED
Spill Destructor 0x6a83f20
TRANSFORM/MERGE/OUTPUT:  Processed 1 events so far, 1 events in buffer.
emitted
Spill Copy Constructor 0x6a83f20
Spill Destructor 0x6a83f20
transforming
Spill Copy Constructor 0x6a83f20
Spill Destructor 0x6a83f20
Spill Default Constructor 0x6ab4450
Spill Copy Constructor 0x6aae290
Spill Copy Constructor 0x6aae9f0
Spill Destructor 0x6aae290
Spill Destructor 0x6ab4450
Spill Copy Constructor 0x6ab4450
Spill Destructor 0x6ab4450
Spill Destructor 0x6aae9f0
Spill Default Constructor 0x6ab50b0
Spill Copy Constructor 0x6ab2ca0
Spill Copy Constructor 0x6a9b780
Spill Destructor 0x6ab2ca0
Spill Destructor 0x6ab50b0
Spill Copy Constructor 0x6ab50b0
Spill Copy Constructor 0x6ab2ca0
Spill Destructor 0x6ab50b0
Spill Destructor 0x6a9b780
Spill Copy Constructor 0x6a9b780
Spill Copy Constructor 0x6ab50b0
Spill Destructor 0x6a9b780
Spill Destructor 0x6ab2ca0
transformed
Spill Copy Constructor 0x6ab2ca0
Spill Destructor 0x6ab2ca0
Spill Destructor 0x6ab50b0
merging
merged
saving
OutputCppRoot::write_event a 4
Spill Default Constructor 0x6a9ed50
OutputCppRoot::write_event b 4
OutputCppRoot::write_event c 4
OutputCppRoot::write_event d 4
OutputCppRoot::write_event e 4
OutputCppRoot::write_event f 4
OutputCppRoot::write_event h 4
OutputCppRoot::write_event i 4 (destructing)
Spill Destructor 0x6a9ed50
OutputCppRoot::write_event j 4 (destructed)
OutputCppRoot::write_event k 4
saved

I found one... diff

=== modified file 'src/input/InputCppDAQData/InputCppDAQData.hh'
--- src/input/InputCppDAQData/InputCppDAQData.hh    2015-06-18 14:48:59 +0000
+++ src/input/InputCppDAQData/InputCppDAQData.hh    2015-07-05 05:21:13 +0000
@@ -113,12 +113,12 @@
   */
   MAUS::Data _emit_cpp() {
      if (this->readNextEvent()) {
-       MAUS::Data  *data_cpp  = new MAUS::Data; <--- THIS WAS NEVER DELETED
+       MAUS::Data  data_cpp;
        MAUS::Spill *spill_cpp = new MAUS::Spill;
-       data_cpp->SetSpill(spill_cpp);
-       this->getCurEvent(data_cpp);
+       data_cpp.SetSpill(spill_cpp);
+       this->getCurEvent(&data_cpp);
        _eventsCount++;
-       return *data_cpp;
+       return data_cpp;
      } else {
        throw(MAUS::Exception(Exception::recoverable,
                              "Failed to read next event",

The other one appears to be pre-existing

=== modified file 'src/map/MapPyGroup/MapPyGroup.py'
--- src/map/MapPyGroup/MapPyGroup.py    2014-07-24 10:23:22 +0000
+++ src/map/MapPyGroup/MapPyGroup.py    2015-07-05 19:40:46 +0000
@@ -155,6 +155,10 @@
         for worker in self._workers:
             if not (hasattr(worker, "can_convert") and worker.can_convert):
                 old_spill = converter.string_repr(nu_spill)
+                try:
+                    converter.del_data_repr(nu_spill)
+                except TypeError:
+                    pass
             else:
                 old_spill = nu_spill
             nu_spill = worker.process(old_spill)
@@ -163,7 +167,6 @@
             except TypeError:
                 pass
             old_spill = None
-
         return nu_spill

     def death(self):

lp:~chris-rogers/maus/1704a now in test.

Note that I added a data card data_maximum_reference_count which controls the maximum number of MAUS::Data objects allowed to exist. Should help with catching memory leaks I hope...

• I tried reconstructing 05679; test ran extremely quickly, suspect that MAUS was not doing anything (Step I data vs Step IV unpacker)
• I tried reconstructing 07155 (1 GB file, including tracker data) and it was still producing lots of data and working extremely slowly with lots of junk in the output. Maybe a leak deeper down in the datastructure or some issues in tracker recon?
• I tried reconstructing 06800 (Step IV, pre-tracker), I got through all 117 DAQ events okay with not too much leakage. I noted lots of things like:
  

  WARNING: Tracker RealDataDigitization: DAQ data size does not match Recon data size, aborting
  

• Now trying 06818...

That's great, Chris.
I see that 6818 doesn't have tracker data in it. I see a memory leakage when running against runs with tracker data [ I tried 7150 which is probably too large a run to try a load test on]. Since it looks like the framework is plugged as seen above, those leakages would have to be from the reconstruction side.