Project

General

Profile

Support #1277

MAUS Tracker Recon: Speed and Memory Usage

Added by Santos, Edward over 8 years ago. Updated almost 7 years ago.

Status:
Open
Priority:
Normal
Category:
Tracker
Target version:
Start date:
20 May 2013
Due date:
% Done:

0%

Estimated time:
Workflow:
New Issue

Description

While setting up large data production, speed and memory usage became an issue to optimise. Here is the fractional time consumed by different parts of MAUS while running a simulation with 1 tracker only:

Time
Simulation (beam+G4): 31%
MCDigitization: 19%
Tracker Clustering: 30%
SpacePoint Recon: 9%
Pattern Recognition: 10%
Kalman Fit: 2%

This already included an optimisation in the MCDigitization which made it much faster. 2000 spills with 2 particle events each take about 35min to run on a i7 3.4GHz processor. The main point I have to make here is that looping over MiceModules is very time expensive but I don't see how to avoid it.

Memory consumption is abnormal. The 4000 particle events simulated consume about 5Gb of RAM while performing simulation and full tracker reconstruction. There seem to be memory leaks in the object processors. I'll attach the valgrind trace for a short 10 spill run.

Please leave your feedback/suggestions.


Files

valgrind_out.log (20.7 MB) valgrind_out.log Santos, Edward, 20 May 2013 15:55
datacard.py (3.33 KB) datacard.py Datacard file. Santos, Edward, 22 May 2013 11:59
HelicalTest_tracker0.dat (426 Bytes) HelicalTest_tracker0.dat To be saved in: /src/legacy/FILES/Models/Configurations Santos, Edward, 22 May 2013 11:59

Related issues

Related to MAUS - Bug #1278: Potential memory leak when running tracker reconstructionClosedSantos, Edward22 May 2013

Actions
#1

Updated by Santos, Edward over 8 years ago

Attaching the valgrind log.

#2

Updated by Rogers, Chris over 8 years ago

The main point I have to make here is that looping over MiceModules is very time expensive but I don't see how to avoid it.

Where in the code do you loop over MiceModules?

#3

Updated by Rogers, Chris over 8 years ago

Conversation with Ed

  • For MiceModules, looks like he is making lots of string compares and coordinate transformations in the clustering algorithm (MiceModules index by string, so to do a MiceModule look up requires inherent string compare). Recommend that clustering algorithm does a lookup from MiceModules at birth time in order to pull out the relevant data.
  • For memory leak - thanks for the warning, Rogers will have a look.
#4

Updated by Santos, Edward over 8 years ago

Adding datacard and geometry configuration file.

#5

Updated by Rogers, Chris over 8 years ago

I raised memory leak as #1278. I will try to run your test example against the tracker_devel branch.

#6

Updated by Rogers, Chris over 8 years ago

Is this closed?

#7

Updated by Rogers, Chris over 8 years ago

  • Category set to Tracker
  • Assignee set to Santos, Edward
  • Target version set to Future MAUS release
#8

Updated by Santos, Edward over 8 years ago

Mostly yes but I want to keep optimising for speed and maybe write down the "final" performance values here... I still find that the clustering is taking too much time. Let's leave it till the code review on Friday?

#9

Updated by Rogers, Chris over 8 years ago

Fine, no rush

#10

Updated by Rogers, Chris over 8 years ago

Comment from Ed (in person) - speed hog appears to be the JSON <-> CPP conversion

#11

Updated by Hunt, Christopher almost 7 years ago

  • Assignee changed from Santos, Edward to Hunt, Christopher

Ed's left so I'll take a look when I have time at the recon algorithms.

Also available in: Atom PDF