Project

General

Profile

Support #681

Tracker - simulating a large number of muons

Added by Belozertseva, Anastasia over 10 years ago. Updated over 10 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Simulation
Target version:
Start date:
23 August 2011
Due date:
% Done:

100%

Estimated time:
Workflow:

Description

Whenever I try to run simulate_mice.py with a large number of spills (no matter whether I set the number of spills to 3000 or 5000 or any other large number), in the terminal it says processing 1024 events instead of the number I set in the simulation. There seems to be some kind of cut off set somewhere that doesn't let me run more spills? There is no problem when I run less than 1024 spills though..

In addition, recently I have been getting the following warning: Illegal state BeamOn()- ignored, not sure where it came from..


Files

simulation.out (54.7 MB) simulation.out Tunnell, Christopher, 23 August 2011 22:32
#1

Updated by Rogers, Chris over 10 years ago

Are you running default simulate_mice.py (if not please attach simulate_mice.py? What version are you running?

#2

Updated by Rogers, Chris over 10 years ago

  • Category set to Simulation
  • Assignee set to Rogers, Chris
  • Target version set to Future MAUS release
#3

Updated by Tunnell, Christopher over 10 years ago

You are pointing out two things:

  1. You want to know if you are simulating only 1024 events when you ask to simulate 3000
  2. You want to know what the BeamOn() error is

Point one: 1024 events only?

To answer the first point: wait a little bit. I ran 2000 muons just to double check from the trunk. It should say that it processed 1024 events then after some time say that it processed 1024 more events then after some time that it processed the remainder.

It may seem weird and definitely looks weird but let me try to explain why it is like that. The way our dataflow works is that there are N events and 4 main operations (input, map, reduce, and output). A lot of physics programs run event 1 through all the operations so they load the first event, do stuff to it, then save it.

In Map-Reduce (http://en.wikipedia.org/wiki/MapReduce) world things are a little different. Instead of performing sequentially analyzing events (like most physics programs) we sequentially perform operations. The reason for this was related to performance and being able to parallelize later. So we want to send every event into the maps before we reduce.

But there's an issue. Think about how a computer works. If you're running 1 million events that are 1 MB each (we're smaller in memory but play along) then that's roughly 1 TB of memory. Most people only have about 1 GB of memory which is about a factor of 1000 off. The way I dealt with this problem was by buffering the input and maps by only doing 1024 at a time.

I can send you what the entire output should look like. Or if you can find the file load_test.py it shows you what would happen if you ran a million spills.

That clear?

Part two: the illegial state beamon

I have no idea. Can you send some output of what you're seeing? Are you using a release or the trunk?

#4

Updated by Tunnell, Christopher over 10 years ago

So Part 1 is just parts of our data flow. If you have suggestions on what type of output you want to see, then let me know. Those print statements are some of the first lines of code within MAUS and probably aren't that clear.

#5

Updated by Belozertseva, Anastasia over 10 years ago

I think I am running the trunk. I downloaded it back in June by doing bzr pull.

And not sure what you mean by the output - whatever I see in the terminal window when I run it? or the simulation.out file?

#6

Updated by Tunnell, Christopher over 10 years ago

I guess both. The simulation.out file (but a small version if you can repeat it by simulating only a few events).

Also you can post the terminal output by doing:

simulate_mice.py 1>maus_std 2>maus_err

where maus_std contains the standard terminal output and maus_err will contain errors

#7

Updated by Tunnell, Christopher over 10 years ago

When running:

bin/simulate_mice.py 1>maus_standard 2>maus_error

and in 'maus_standard' there is:

Welcome to MAUS:
        Process ID (PID): 3093
        Universally Unique ID (UUID): 0462e21e-8540-44eb-b057-669d8fb2ba71
        Program Arguments: ['bin/simulate_mice.py']
        Version: MAUS development version 0.0.7
INPUT: Reading some input
MAP: Setting up mappers
MAP: Processing 1024 events
MAP: Processing 976 events
MAP: Closing input and mappers
REDUCE: Setting up reducers
REDUCE: Reducing 1000 events in the 0 pass
REDUCE: Reducing 1000 events in the 1 pass
REDUCE: Merging 2 passes and reducing the 0 events left in the buffer

and the simulation output is temporarily here

#8

Updated by Tunnell, Christopher over 10 years ago

And maus_error is empty.

#9

Updated by Tunnell, Christopher over 10 years ago

Here's the output.

#10

Updated by Rogers, Chris over 10 years ago

You should update to the trunk (or maybe a recent version) and try again. Something like

bzr merge lp:maus -r MAUS-v0.0.6

If you get anything that mentions a conflict, you will need to resolve that. It means that a file that you changed has also been changed by someone else.

We have a very rapid development cycle in simulation stuff at the moment, and for the next few weeks - should settle after september, but meanwhile the code you are running has changed quite a lot since June.

#11

Updated by Belozertseva, Anastasia over 10 years ago

I just merged my version with the trunk, so I hope this will fix things!

#12

Updated by Belozertseva, Anastasia over 10 years ago

I just ran 50 spills, and maus_err is empty. Updating to the trunk fixed things! I should have done that in the first place, sorry..

Thanks for the help and for explaining in more detail the way the simulation runs.

#13

Updated by Rogers, Chris over 10 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Not always abvious - but worth keeping in touch with the trunk (say, merging every minor release) just so you can check that the code you are writing isn't broken by any new developments.

Also available in: Atom PDF