Project

General

Profile

Bug #1032

Occasional error while analyzing offline data file

Added by Pradhan, Neetish over 11 years ago. Updated over 11 years ago.

Status:
Closed
Priority:
High
Assignee:
Rogers, Chris
Category:
common_cpp
Target version:
MAUS-v0.3.1
Start date:
22 June 2012
Due date:
% Done:

100%

Estimated time:
Workflow:
New Issue

Description

There seems to be an occasional error that python throws when running through a data file. So far, I haven't been present when the error occured, so I can't say for sure, but it could be because the computer runs out of RAM. (I've noticed that ROOT has quite a memory leak in that it doesn't clear the RAM when you close a browser graph. So could it be linked with ROOT?) I am attaching the on-screen-display of one such run - the error is at the end.

(Using MAUS 0.2.4)

Files

run.txt (366 KB) run.txt Pradhan, Neetish, 22 June 2012 23:28
recon-valgrind.log (1.45 MB) recon-valgrind.log Rogers, Chris, 10 July 2012 12:10
#1

Updated by Pradhan, Neetish over 11 years ago

Edit: I meant to add that running MAUS seems to show a similar symptom as seen when running ROOT browser: the RAM used just keeps on increasing.

#2

Updated by Rogers, Chris over 11 years ago

I'll have a look. ROOT tends to be very unclear about memory ownership... I may have made a mistake somewhere.

#3

Updated by Rogers, Chris over 11 years ago

  • Category set to common_cpp
  • Assignee set to Rogers, Chris
  • Target version set to Future MAUS release
#4

Updated by Rogers, Chris over 11 years ago

  • Priority changed from Normal to High
#5

Updated by Rogers, Chris over 11 years ago

Okay, I finally managed to take a look at this... running now, looks like indeed memory usage is growing. I guess we need a load test...

#6

Updated by Rogers, Chris over 11 years ago

Looks like about 1 Gb per 400 spills (20 minutes running). Ouch.

#7

Updated by Rogers, Chris over 11 years ago

Valgrind log attached...

#8

Updated by Rogers, Chris over 11 years ago

Looks like the DAQData branch on the spill is not correctly deleted somehow.

#9

Updated by Rogers, Chris over 11 years ago

At least one problem - in DAQData there are a whole load of vectors of pointers. These pointers are never deleted.

#10

Updated by Rogers, Chris over 11 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

After r756, memory usage (from top) looks

Spill number VIRT RES SHARE TOTAL
44 732 222 41 995
1148 758 274 41 1032
1400 793 283 41 1117

So still some leakage but manageable. There is some error on the usage (I didn't bother to check at a particular point in the spill processing for example, memory is allocated and deallocated during this processing).

I will call this resolved, but I raised #1046 for implementing a more formal load test and then we can make some check on spill to spill memory usage. It will go into the next release.

#11

Updated by Rogers, Chris over 11 years ago

FYI - 1279m 768m 41m @ spill 2543 ... so still leaking but not so bad.

Edit to note - that is Event 2543, corresponds to Spill ~ 800

#12

Updated by Rogers, Chris over 11 years ago

I complete 3407, but in the end the total memory usage was about 2G which is too much really...

#13

Updated by Rogers, Chris over 11 years ago

  • Target version changed from Future MAUS release to MAUS-v0.3.1

Also available in: Atom PDF