Project

General

Profile

Bug #1704

Memory Leak

Added by Rogers, Chris about 6 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Data Structure
Target version:
Start date:
03 July 2015
Due date:
% Done:

100%

Estimated time:
Workflow:
New Issue

Description

Investigate possible memory leak (related to Input/Output moving off json). Valgrind log attached...


Files

test_load_analyze_data_offline_05810.log (10.1 MB) test_load_analyze_data_offline_05810.log Rogers, Chris, 03 July 2015 06:05
run_06818_resource_usage.png (5.68 KB) run_06818_resource_usage.png Rogers, Chris, 06 July 2015 08:22

Related issues

Related to MAUS - Bug #1706: Memory leaks preventing GRID runningClosedDobbs, Adam09 July 2015

Actions
#1

Updated by Rogers, Chris about 6 years ago

Sample event loop with some extra verbosity, printing memory src/common_cpp/DataStructure/Spill.hh memory allocations

emitting
Spill Default Constructor 0x621d8e0 <--- NEVER DESTRUCTED
Spill Copy Constructor 0x6a83ea0
Spill Copy Constructor 0x6a83f20
Spill Destructor 0x6a83ea0
Spill Copy Constructor 0x6a83ea0 <---- NEVER DESTRUCTED
Spill Destructor 0x6a83f20
TRANSFORM/MERGE/OUTPUT:  Processed 1 events so far, 1 events in buffer.
emitted
Spill Copy Constructor 0x6a83f20
Spill Destructor 0x6a83f20
transforming
Spill Copy Constructor 0x6a83f20
Spill Destructor 0x6a83f20
Spill Default Constructor 0x6ab4450
Spill Copy Constructor 0x6aae290
Spill Copy Constructor 0x6aae9f0
Spill Destructor 0x6aae290
Spill Destructor 0x6ab4450
Spill Copy Constructor 0x6ab4450
Spill Destructor 0x6ab4450
Spill Destructor 0x6aae9f0
Spill Default Constructor 0x6ab50b0
Spill Copy Constructor 0x6ab2ca0
Spill Copy Constructor 0x6a9b780
Spill Destructor 0x6ab2ca0
Spill Destructor 0x6ab50b0
Spill Copy Constructor 0x6ab50b0
Spill Copy Constructor 0x6ab2ca0
Spill Destructor 0x6ab50b0
Spill Destructor 0x6a9b780
Spill Copy Constructor 0x6a9b780
Spill Copy Constructor 0x6ab50b0
Spill Destructor 0x6a9b780
Spill Destructor 0x6ab2ca0
transformed
Spill Copy Constructor 0x6ab2ca0
Spill Destructor 0x6ab2ca0
Spill Destructor 0x6ab50b0
merging
merged
saving
OutputCppRoot::write_event a 4
Spill Default Constructor 0x6a9ed50
OutputCppRoot::write_event b 4
OutputCppRoot::write_event c 4
OutputCppRoot::write_event d 4
OutputCppRoot::write_event e 4
OutputCppRoot::write_event f 4
OutputCppRoot::write_event h 4
OutputCppRoot::write_event i 4 (destructing)
Spill Destructor 0x6a9ed50
OutputCppRoot::write_event j 4 (destructed)
OutputCppRoot::write_event k 4
saved
#2

Updated by Rogers, Chris about 6 years ago

I found one... diff

=== modified file 'src/input/InputCppDAQData/InputCppDAQData.hh'
--- src/input/InputCppDAQData/InputCppDAQData.hh    2015-06-18 14:48:59 +0000
+++ src/input/InputCppDAQData/InputCppDAQData.hh    2015-07-05 05:21:13 +0000
@@ -113,12 +113,12 @@
   */
   MAUS::Data _emit_cpp() {
      if (this->readNextEvent()) {
-       MAUS::Data  *data_cpp  = new MAUS::Data; <--- THIS WAS NEVER DELETED
+       MAUS::Data  data_cpp;
        MAUS::Spill *spill_cpp = new MAUS::Spill;
-       data_cpp->SetSpill(spill_cpp);
-       this->getCurEvent(data_cpp);
+       data_cpp.SetSpill(spill_cpp);
+       this->getCurEvent(&data_cpp);
        _eventsCount++;
-       return *data_cpp;
+       return data_cpp;
      } else {
        throw(MAUS::Exception(Exception::recoverable,
                              "Failed to read next event",
#3

Updated by Rogers, Chris about 6 years ago

The other one appears to be pre-existing

=== modified file 'src/map/MapPyGroup/MapPyGroup.py'
--- src/map/MapPyGroup/MapPyGroup.py    2014-07-24 10:23:22 +0000
+++ src/map/MapPyGroup/MapPyGroup.py    2015-07-05 19:40:46 +0000
@@ -155,6 +155,10 @@
         for worker in self._workers:
             if not (hasattr(worker, "can_convert") and worker.can_convert):
                 old_spill = converter.string_repr(nu_spill)
+                try:
+                    converter.del_data_repr(nu_spill)
+                except TypeError:
+                    pass
             else:
                 old_spill = nu_spill
             nu_spill = worker.process(old_spill)
@@ -163,7 +167,6 @@
             except TypeError:
                 pass
             old_spill = None
-
         return nu_spill

     def death(self):
#4

Updated by Rogers, Chris about 6 years ago

lp:~chris-rogers/maus/1704a now in test.

Note that I added a data card data_maximum_reference_count which controls the maximum number of MAUS::Data objects allowed to exist. Should help with catching memory leaks I hope...

#5

Updated by Rogers, Chris about 6 years ago

I pushed 1704a to the trunk. Should make things better, but:
  • I tried reconstructing 05679; test ran extremely quickly, suspect that MAUS was not doing anything (Step I data vs Step IV unpacker)
  • I tried reconstructing 07155 (1 GB file, including tracker data) and it was still producing lots of data and working extremely slowly with lots of junk in the output. Maybe a leak deeper down in the datastructure or some issues in tracker recon?
  • I tried reconstructing 06800 (Step IV, pre-tracker), I got through all 117 DAQ events okay with not too much leakage. I noted lots of things like:
    WARNING: Tracker RealDataDigitization: DAQ data size does not match Recon data size, aborting
    
  • Now trying 06818...
#6

Updated by Rogers, Chris about 6 years ago

Resource usage now looks flat on 06818...

#7

Updated by Rajaram, Durga about 6 years ago

That's great, Chris.
I see that 6818 doesn't have tracker data in it. I see a memory leakage when running against runs with tracker data [ I tried 7150 which is probably too large a run to try a load test on]. Since it looks like the framework is plugged as seen above, those leakages would have to be from the reconstruction side.

#8

Updated by Rogers, Chris about 6 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Looks okay. Tracker memory leak is a separate issue.

#9

Updated by Rajaram, Durga about 6 years ago

  • Target version changed from Future MAUS release to MAUS-v1.0.0

Also available in: Atom PDF