If I'm interpreting the usage output from the tests on Run 4000, it looks like the memory usage is now ~1.5 GB

[{'%mem': '49.8', 'sz': '494163', 'vsz': '1976652', 'pid': 28108, '%cpu': '99.8', 'etime': '03:49:12', 'rsz': '1471184'}]

For the EMR run 5679, it's higher -- possibly indicating a leak in the EMR code -

7863', 'vsz': '2431452', 'pid': 24692, '%cpu': '99.5', 'etime': '04:51:40', 'rsz': '1921228'}]
[{'%mem': '65.0', 'sz': '607863', 'vsz': '2431452', 'pid': 24692, '%cpu': '99.5', 'etime': '04:51:48', 'rsz': '1921228'}]
[{'%mem': '65.0', 'sz': '607863', 'vsz': '2431452', 'pid': 24692, '%cpu': '99.5', 'etime': '04:51:56', 'rsz': '1921228'}]
[{'%mem': '65.0', 'sz': '607863', 'vsz': '2431452', 'pid': 24692, '%cpu': '99.5', 'etime': '04:52:05', 'rsz': '1921228'}]

On my machine, I saw ~1.3 GB of usage (RSZ) for Run 4000.

valgrind log file for run 05810 attached. Note the job ran for 8 hours and then was killed. This was run on the 64 bit rogers_sl64 test server from the command

./install_build_test.bash ${NODE_THIRD_PARTY}
echo "Run unit test shuffle" 
source ${WORKSPACE}/env.sh
echo "Run load tests" 
python ${MAUS_ROOT_DIR}/tests/integration/test_load/test_load.py --run-number=05810 --valgrind=1 --max-time=28800 --max-mem-size=320000

I attach a valgrind log. Let me make some explanation.

I put some code in the top level Data object constructor to print out - that Data was constructed or deleted; the address of the constructed Data; the total number of Data objects in memory; the stack trace of the Constructor or Delete call. diff below, in case that helps.

There are a whole load of stuff like

==12062== Invalid read of size 8
==12062==    at 0x10D4ED0C: MAUS::DAQData::SetTriggerRequestArray(std::vector<MAUS::TriggerRequest*, std::allocator<MAUS::TriggerRequest*> >) (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D4F28B: MAUS::DAQData::~DAQData() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D4F788: MAUS::DAQData::~DAQData() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D57567: MAUS::Spill::~Spill() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D576E8: MAUS::Spill::~Spill() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D60ACA: MAUS::Data::~Data() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D60B88: MAUS::Data::~Data() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x1A122EBB: MAUS::PyConverter::py_del_data_repr(_object*, _object*) (in /home/cr67/MAUS/maus_merge/build/maus_cpp/converter.so)
==12062==    by 0x4F1E51C: PyEval_EvalFrameEx (ceval.c:4013)
==12062==    by 0x4F1EEE9: PyEval_EvalFrameEx (ceval.c:4099)
==12062==    by 0x4F1EEE9: PyEval_EvalFrameEx (ceval.c:4099)
==12062==    by 0x4F1FCE4: PyEval_EvalCodeEx (ceval.c:3253)
==12062==  Address 0x377918b0 is 0 bytes inside a block of size 32 free'd
==12062==    at 0x4C27FF2: operator delete(void*) (vg_replace_malloc.c:387)
==12062==    by 0x10D4ED11: MAUS::DAQData::SetTriggerRequestArray(std::vector<MAUS::TriggerRequest*, std::allocator<MAUS::TriggerRequest*> >) (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D4F28B: MAUS::DAQData::~DAQData() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D4F788: MAUS::DAQData::~DAQData() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D57567: MAUS::Spill::~Spill() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D576E8: MAUS::Spill::~Spill() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D60ACA: MAUS::Data::~Data() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x10D60B88: MAUS::Data::~Data() (in /home/cr67/MAUS/maus_merge/build/libMausCpp.so)
==12062==    by 0x1A129260: std::string* MAUS::PyObjectWrapper::unwrap<std::string>(_object*) (in /home/cr67/MAUS/maus_merge/build/maus_cpp/converter.so)
==12062==    by 0x1A129B19: _object* MAUS::PyConverter::py_wrap<std::string>(_object*, _object*) (in /home/cr67/MAUS/maus_merge/build/maus_cpp/converter.so)
==12062==    by 0x4F1E51C: PyEval_EvalFrameEx (ceval.c:4013)
==12062==    by 0x4F1EEE9: PyEval_EvalFrameEx (ceval.c:4099)
==12062== 

I think that the MAUS::Data involved is okay (it doesn't look like a double free there) so probably there is some lurking wickedness in the copy constructor for Spill or DAQData. I shall dig tomorrow.

== modified file 'src/common_cpp/DataStructure/Data.cc'
--- src/common_cpp/DataStructure/Data.cc    2012-09-27 10:59:51 +0000
+++ src/common_cpp/DataStructure/Data.cc    2014-07-24 16:11:03 +0000
@@ -21,15 +21,20 @@

 namespace MAUS {

+int Data::_reference_count = 0;
+
 Data::Data() : MAUSEvent<Spill>("Spill"), _spill(NULL) {
+    IncreaseRefCount();
 }

 Data::Data(const Data& data) : MAUSEvent<Spill>(), _spill(NULL) {
+    IncreaseRefCount();
     *this = data;
 }

 Data::~Data() {
-    if (_spill != NULL) {
+   DecreaseRefCount();
+   if (_spill != NULL) {
         delete _spill;
         _spill = NULL;
     }
@@ -64,5 +69,17 @@
   Data spill;
   return sizeof(spill);
 }
+
+void Data::IncreaseRefCount() {
+    _reference_count++;
+    std::cerr << "CONSTRUCT " << this << " " << _reference_count << std::endl;
+    std::cerr << MAUS::Exception().MakeStackTrace(1) << std::endl;
+}
+
+void Data::DecreaseRefCount() {
+    _reference_count--;
+    std::cerr << "DESTRUCT " << this << " " << _reference_count << std::endl;
+    std::cerr << MAUS::Exception().MakeStackTrace(1) << std::endl;
+}
 }

=== modified file 'src/common_cpp/DataStructure/Data.hh'
--- src/common_cpp/DataStructure/Data.hh    2014-04-22 15:27:03 +0000
+++ src/common_cpp/DataStructure/Data.hh    2014-07-24 14:03:56 +0000
@@ -89,8 +89,11 @@
     void SetEvent(Spill* spill) {SetSpill(spill);}

   private:
+    void IncreaseRefCount();
+    void DecreaseRefCount();
     Spill* _spill;
     std::string _event_type;
+    static int _reference_count;
     MAUS_VERSIONED_CLASS_DEF(Data)
 };
 }

Thanks for the fixes. But it fails on EMR runs [ log attached test_load_analyze_data_offline_05679.log ]
Looks like it's in the EMREvent destructor, the pointer's already gone when it tries to delete.
Is there a double free going on with the last even, with both single_thread.py and MapPyGroup.py trying to delete the event twice?

I had another look at this. E.g. run 04000 running against the trunk.

[{'%mem': '3.3', 'sz': '260621', 'vsz': '1042484', 'pid': 14158, '%cpu': '97.7', 'etime': '03:13', 'rsz': '548352'}]
... 4 hours later ...
[{'%mem': '4.2', 'sz': '294966', 'vsz': '1179864', 'pid': 14158, '%cpu': '99.8', 'etime': '04:44:17', 'rsz': '687640'}]

There is some 300 MB lost during this reconstruction, but that is not such a serious leak compared to the ~ 2000 MB program size. I also checked reference counting against spill by hard coding in a reference counter into spill constructor and destructor, results in memory.log - looks like the number of spill instances after running 100 spills is 1. So I think okay to close this issue. Any more memory leaks will have to be knocked off one by one.