Support #681
Tracker - simulating a large number of muons
100%
Description
Whenever I try to run simulate_mice.py with a large number of spills (no matter whether I set the number of spills to 3000 or 5000 or any other large number), in the terminal it says processing 1024 events instead of the number I set in the simulation. There seems to be some kind of cut off set somewhere that doesn't let me run more spills? There is no problem when I run less than 1024 spills though..
In addition, recently I have been getting the following warning: Illegal state BeamOn()- ignored, not sure where it came from..
Files
Updated by Rogers, Chris over 12 years ago
Are you running default simulate_mice.py (if not please attach simulate_mice.py? What version are you running?
Updated by Rogers, Chris over 12 years ago
- Category set to Simulation
- Assignee set to Rogers, Chris
- Target version set to Future MAUS release
Updated by Tunnell, Christopher over 12 years ago
You are pointing out two things:
- You want to know if you are simulating only 1024 events when you ask to simulate 3000
- You want to know what the BeamOn() error is
Point one: 1024 events only?¶
To answer the first point: wait a little bit. I ran 2000 muons just to double check from the trunk. It should say that it processed 1024 events then after some time say that it processed 1024 more events then after some time that it processed the remainder.
It may seem weird and definitely looks weird but let me try to explain why it is like that. The way our dataflow works is that there are N events and 4 main operations (input, map, reduce, and output). A lot of physics programs run event 1 through all the operations so they load the first event, do stuff to it, then save it.
In Map-Reduce (http://en.wikipedia.org/wiki/MapReduce) world things are a little different. Instead of performing sequentially analyzing events (like most physics programs) we sequentially perform operations. The reason for this was related to performance and being able to parallelize later. So we want to send every event into the maps before we reduce.
But there's an issue. Think about how a computer works. If you're running 1 million events that are 1 MB each (we're smaller in memory but play along) then that's roughly 1 TB of memory. Most people only have about 1 GB of memory which is about a factor of 1000 off. The way I dealt with this problem was by buffering the input and maps by only doing 1024 at a time.
I can send you what the entire output should look like. Or if you can find the file load_test.py it shows you what would happen if you ran a million spills.
That clear?
Part two: the illegial state beamon¶
I have no idea. Can you send some output of what you're seeing? Are you using a release or the trunk?
Updated by Tunnell, Christopher over 12 years ago
So Part 1 is just parts of our data flow. If you have suggestions on what type of output you want to see, then let me know. Those print statements are some of the first lines of code within MAUS and probably aren't that clear.
Updated by Belozertseva, Anastasia over 12 years ago
I think I am running the trunk. I downloaded it back in June by doing bzr pull.
And not sure what you mean by the output - whatever I see in the terminal window when I run it? or the simulation.out file?
Updated by Tunnell, Christopher over 12 years ago
I guess both. The simulation.out file (but a small version if you can repeat it by simulating only a few events).
Also you can post the terminal output by doing:
simulate_mice.py 1>maus_std 2>maus_err
where maus_std contains the standard terminal output and maus_err will contain errors
Updated by Tunnell, Christopher over 12 years ago
When running:
bin/simulate_mice.py 1>maus_standard 2>maus_error
and in 'maus_standard' there is:
Welcome to MAUS: Process ID (PID): 3093 Universally Unique ID (UUID): 0462e21e-8540-44eb-b057-669d8fb2ba71 Program Arguments: ['bin/simulate_mice.py'] Version: MAUS development version 0.0.7 INPUT: Reading some input MAP: Setting up mappers MAP: Processing 1024 events MAP: Processing 976 events MAP: Closing input and mappers REDUCE: Setting up reducers REDUCE: Reducing 1000 events in the 0 pass REDUCE: Reducing 1000 events in the 1 pass REDUCE: Merging 2 passes and reducing the 0 events left in the buffer
and the simulation output is temporarily here
Updated by Tunnell, Christopher over 12 years ago
- File simulation.out simulation.out added
Here's the output.
Updated by Rogers, Chris over 12 years ago
You should update to the trunk (or maybe a recent version) and try again. Something like
bzr merge lp:maus -r MAUS-v0.0.6
If you get anything that mentions a conflict, you will need to resolve that. It means that a file that you changed has also been changed by someone else.
We have a very rapid development cycle in simulation stuff at the moment, and for the next few weeks - should settle after september, but meanwhile the code you are running has changed quite a lot since June.
Updated by Belozertseva, Anastasia over 12 years ago
I just merged my version with the trunk, so I hope this will fix things!
Updated by Belozertseva, Anastasia over 12 years ago
I just ran 50 spills, and maus_err is empty. Updating to the trunk fixed things! I should have done that in the first place, sorry..
Thanks for the help and for explaining in more detail the way the simulation runs.
Updated by Rogers, Chris over 12 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
Not always abvious - but worth keeping in touch with the trunk (say, merging every minor release) just so you can check that the code you are writing isn't broken by any new developments.