Project

General

Profile

Feature #1617

Computing resources requirement

Added by Rogers, Chris over 7 years ago. Updated over 7 years ago.

Status:
Open
Priority:
Normal
Assignee:
Category:
bin(aries)
Target version:
-
Start date:
29 January 2015
Due date:
% Done:

0%

Estimated time:
Workflow:
New Issue

Description

Require estimate of the computing resources needed to run MAUS and G4BL for:

  • 1e7 particles at TOF1 (roughly 1e6 good muons)
  • 9 beamline settings

Resources should be

  • Disk space
  • computing power

Assume hand off at the G4BL - MAUS interface point, downstream of D2. Required by Monday 2nd February.


Files

measure_emittance_Kal1_PZ.pdf (34.9 KB) measure_emittance_Kal1_PZ.pdf Bayes, Ryan, 04 February 2015 17:35
measure_emittance_Kal1_PT.pdf (41.4 KB) measure_emittance_Kal1_PT.pdf Bayes, Ryan, 04 February 2015 17:35
#1

Updated by Dobbs, Adam over 7 years ago

Chris, one full working day's notice is not ok for a task like this. I'll start looking at it but it almost certainly won't be done by Monday.

#2

Updated by Rogers, Chris over 7 years ago

I understand - but the request for information came from computing group!

#3

Updated by Dobbs, Adam over 7 years ago

OK, well John Nugent is running a set of G4BL sims at the moment which should provide enough particles, but that won't be done until Monday, and will only be for one beam line setting (6mm 200MeV), and I will need to propagate to TOF1 after that, and try to figure out the CPU time etc.

Durga, why the Monday deadline? Also, are we interesting in the cost of running G4BL itself as well as core MAUS, or just MAUS once the G4BL beam has been generated?

#4

Updated by Rogers, Chris over 7 years ago

Looks like this has stalled as John Nugent's job failed and he is now on holiday.

#5

Updated by Rajaram, Durga over 7 years ago

I have asked Ryan if he has some numbers from the physics block challenge stuff he ran. Could be we can try to scale from that

#6

Updated by Bayes, Ryan over 7 years ago

I have looked at a simulation composed of 1231 G4beamline interface files. Each of the source interface files are 2.9 M in size. The resulting root files after being processed through MAUS are typically 14 MegaBytes in size. Currently the interface files describe the state of the beam 1 m prior to D2 (where the origin of the hall coordinate system is defined). We would like to move this interface position to 1 m downstream of the hall coordinate system origin.

I have used the virtual planes that I have embedded in the batch simulation to extract the number of muons in the simulation at various points in the beam line. Because there are no virtual planes defined in the simulation upstream of D2, I cannot give the absolute number of particles generated. Furthermore, my analysis has a cut applied (that I only just noticed) that requires that the primary particle is a muon; I think this means that muons sampled in the interface files will be counted, but muons resulting from decays in flight will not. Therefore, I don't have the number of particles at this time, but we are mostly interested in the number of muons, and this restrictive definition should be representative.

For the sum of all of the files there are;
  • 1193646 muons at z = 0 m,
  • 980907 muons at z = 1 m (in the hall coordinate system),
  • 55000 muons at TOF1, and
  • 35000 muons at TOF2.
#7

Updated by Rajaram, Durga over 7 years ago

Hi Ryan --greatly appreciate your getting these numbers at short notice.
Couple of quick questions to make sure I understand:
-- that's 55k muons after MAUSing 1231 G4BL inputs?
-- the 14MB reconstructed output per G4BL input - does the output contain digi+reco of trackers and EMR?

Also, do you happen to know what John's resource [ nodes, time] usage was in generating the G4BL?

#8

Updated by Bayes, Ryan over 7 years ago

Yes the 55k muons was the result of MAUSing all 1231 G4BL inputs
The 14MB reconstructed output files per G4BL input do contain the dig+reco of the trackers, but not the EMR.

I don't know the details pertaining to John's resource usage in the generation of the interface files.

#9

Updated by Rajaram, Durga over 7 years ago

Thanks, Ryan.

So, trying to put together some numbers:

  • 55k muons at TOF1 ~= 4k "good muons" [ Victoria's analysis showed ~8% yield from TOF1 triggers ]
  • So, disk usage for 4k "good muons":
    • Input: 1231*3MB = 3.7 GB
    • Output: 1231*14MB = 17.2 GB
      • this does not include EMR and Global, so let me estimate a 25% overhead from those = 22 GB
    • Input+Output = 4+22 = 26 GB
  • For 1e6 "good muons" = 26 * 1e6/4e3 = 6.5 TB
  • For 9 settings = 9*6.5 ~= 60 TB
  • Am I missing anything?
  • This does not include re-generation, re-processing, or generating input for specific data-taking conditions, or special runs.

-- Chris, is the physics requirement 1e6 "good muons" per setting? Or is is 1e7? [ 100x ]

#10

Updated by Rogers, Chris over 7 years ago

The requirement is 1e5 muons per setting, but then we need a factor 10 to do statistics analysis. At a guess we will do say 1000 big batch MC runs (say we have 10-20 papers, each of which wants 100 MC runs). Complete guess, I don't know how accurate it is...

#11

Updated by Rajaram, Durga over 7 years ago

Thanks, Chris. Extraneous to the matrix, do you imagine for each paper, 100 MC runs of 1e6 good muons?
I think it's safer here to overestimate than underestimate

#12

Updated by Rogers, Chris over 7 years ago

I don't think it is a crazy estimate - each paper has 10-20 plots, each of which describes 5 runs - that is typical I guess. Maybe some runs have lower statistics, but we probably have to redo some MC... if anything it is less than I would like, I was hoping to do more with MAUS...

#13

Updated by Bayes, Ryan over 7 years ago

I think that the 4k "good muons" is an extreme under-estimate.

I have also been doing the xboa analysis for the reconstructed tracks using the same batch simulation. The analysis gets of order 20k muons from the tracker reconstruction based on a subset of approximately 1000 files. I emphasize that this is the reconstructed analysis --- not a virtual plane analysis.

#14

Updated by Nebrensky, Henry over 7 years ago

Hi,

Yes the 55k muons was the result of MAUSing all 1231 G4BL inputs

do you have an idea of the CPU used for the MAUSing of the G4BL inputs?

#15

Updated by Rogers, Chris over 7 years ago

I think that the 4k "good muons" is an extreme under-estimate.

What is the transverse and momentum spread? We had a discussion of what goes into that estimate, notes are here. Good muons includes a transverse and momentum cut.

It needs another go...

#16

Updated by Bayes, Ryan over 7 years ago

Nebrensky, Henry wrote:

Hi,

Yes the 55k muons was the result of MAUSing all 1231 G4BL inputs

do you have an idea of the CPU used for the MAUSing of the G4BL inputs?

Not knowing the full specs on the machines involved anything I can offer might be a little specious, and I did not set up the scripts in such a way as to monitor the load (partially because I did not think of it at the time).

The "typical" job runs in less than 2 hours... although some jobs can take much more, due to the random number of particles that enter the channel.

#17

Updated by Bayes, Ryan over 7 years ago

Rogers, Chris wrote:

I think that the 4k "good muons" is an extreme under-estimate.

What is the transverse and momentum spread? We had a discussion of what goes into that estimate, notes are here. Good muons includes a transverse and momentum cut.

It needs another go...

Alright. I played with the cuts a little bit because the emittance was not behaving in the expected way. The default cuts selected pt between 0 and 150 MeV and pz less than 300 MeV. Reducing the fiducial region to only accept muons with pt between 30 MeV and 60 MeV reduces the number of muons to 4749. The only motivation for this cut was to reduce the tails (of the attached pt distribution) and the possibility of either having tracks that are too small or too big (although the p-value cut at 0.05 should also help with those issues). I also contemplated pz cuts (which is I think indicated in the note), but reducing the upper limit did not have the intended effect and I have not successfully added a lower bound on pz yet. The attached figure shows the pz distribution. After looking at the distribution I concluded that a further cut on the lower limit of pz would not be interesting.

I played other games of that type, but I realized the only motivating factor I had was trying to make my figure of merit (emittance) "look better" which is backwards.

#18

Updated by Kyberd, Paul over 7 years ago

I hate to be a bore but the deadline to get numbers to Peter Gronbech is the 7th.
I appreciate the work being done but are we on track - to have them by then?

#19

Updated by Rajaram, Durga over 7 years ago

Calculation and summary of MC Resource requirements

G4BL

  • For 55k muons at TOF1: 1231 files * 2.9 MB/file = 3.6 GB
  • For 1e7 muons at TOF1: 1e7/55e3 * 3.6 = 655 GB
  • Predicted drop in size of 1/5 by moving the interface point downstream => 131 GB
  • For 2000 such MC runs: 131 GB * 2e3 = 262e3 GB = 262 TB

MC+Reco

  • From Ryan Bayes' MAUS output (#1617#note-6): data size per muon at TOF1 ~300 kB [ 1231*14MB/55e3 ]
    • That does not include the EMR or global but includes a couple of hundred virtual planes
    • I ran some quick MAUS jobs throwing 1e3 muons at TOF1; I included the EMR, but no global and no virtual planes and I see 185 kB/muon ~60% lower than Ryan's, but maybe we do want to have several virtual planes in our simulation
  • For 1e7 muons at TOF1: 300 kB * 1e7 = 3 TB
  • For 2000 such MC runs: 3 TB * 2e3 = 6 PB

Total MC disk requirements

  • 262 TB (g4bl) + 6 PB (mc+reco) ~= 6.3 PB
    • there may be an overestimate by ~50% due to the numerous virtual planes in the output
    • on the other hand, we are not counting simulating samples of DAQ runs, and maybe special MC studies, so I think it's ok

CPU resources

  • In John's absence, the best reference I have so far is his CM38 talk, pg. 5
  • For 1e5 muons at TOF1, he quotes ~48 hours on ~100 CPUs
  • So, for 1e7 * 2000 MC runs: 4800 * 1e2 * 2e3 = 9.6e8 cpu hours
    • I don't know the flops/mips of the cputs that were used, so there may be a missing normalization, but that sounds like a lot and someone more versed in these things - Henry - should double-check
#20

Updated by Kyberd, Paul over 7 years ago

9.6e8 CPU hours corresponds to about 100,000 oore years. Which is about 40% of the LHCG grid resources. I would say it is at least a factor of 10 larger than is reasonable to ask for.

#21

Updated by Rajaram, Durga over 7 years ago

Paul - I agree, it sounds very high. What is our current request/allocation? The numbers John quoted, and from which I scaled, are from running on the Glasgow cluster but I don't know how much of a factor that is. What do you propose? If someone else has a better estimate please share.

#22

Updated by Rogers, Chris over 7 years ago

Maybe some non-optimal code somewhere? Probably my fault, it usually is...

#23

Updated by Kyberd, Paul over 7 years ago

The last request was put in by Dave Colling - I will ask him if he has a record of it

#24

Updated by Rajaram, Durga over 7 years ago

Rogers, Chris wrote:

Maybe some non-optimal code somewhere? Probably my fault, it usually is...

I think it's "pure" G4BL passed through the beamline geometry? Maybe it does point to a need to rethink and optimize how we do it.

#25

Updated by Nebrensky, Henry over 7 years ago

Yes the 55k muons was the result of MAUSing all 1231 G4BL inputs

...

The "typical" job runs in less than 2 hours... although some jobs can take much more, due to the random number of particles that enter the channel.

Being belatedly dim as usual: so 55k muons is 2 hrs? or 2460 hrs if you have 1 job/input?

Edit: I figure it's the 2hrs...

Also available in: Atom PDF