Project

General

Profile

Support #594

Mappers taking too long?

Added by Rogers, Chris about 12 years ago. Updated almost 12 years ago.

Status:
Closed
Priority:
Normal
Category:
common_py
Target version:
Start date:
21 July 2011
Due date:
% Done:

100%

Estimated time:
Workflow:

Description

I had a prod from Yordan to investigate the overhead in writing mappers. the specific question was - what is the overhead in making TOF reconstruction as 3 maps instead of 1. So I did a little investigation. I attach a script that generates a big input data set and then pushes it through a number of mappers. The test script is attached and I hope reasonably clear.

Output is also attached. Thing to see is that it took 75.94 s to do MapPyDoNothing once; and 219 s to do MapPyDoNothing 10 times. For comparison I also calculated the time to do (json -> string -> json) * number_of_maps directly. It took 15 s and 710 s respectively.

  • To do it properly I should repeat the study a thousand times - don't quite have the time.
  • The data set was comparable to a MICE run. Each spill was 1000 monte carlo primaries (~170 bytes each, so ~170 Kb in total). I did 1000 spills. (Say a MICE spill is presumably 1000 spills with 100 triggers, but each trigger is probably a lot bigger - I assumed a factor 10, might be a bit small still.

So in short, the overhead for splitting TOF code into 3 maps is probably going to be 5 minutes over the run - which shouldn't break us.

Is this good enough Yordan? Do you want further investigation?


Files

map_speed_test.py (2.87 KB) map_speed_test.py Rogers, Chris, 21 July 2011 15:52
map_speed_test.out (1.38 KB) map_speed_test.out Rogers, Chris, 21 July 2011 15:52
#1

Updated by Tunnell, Christopher about 12 years ago

Information: it should be a lot quicker within C++ (but should be checked) than in python. Also: we can probably play some tricks to pass JSON to Python mappers instead of the string to save on time if need be.

#2

Updated by Rogers, Chris about 12 years ago

Close this?

#3

Updated by Tunnell, Christopher about 12 years ago

Can Yordan comment? But otherwise, yeah, close

#4

Updated by Rogers, Chris about 12 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

I guess Yordan is happy with this...

#5

Updated by Rogers, Chris almost 12 years ago

Ping to pavel

Also available in: Atom PDF