This is actually more complicated than I thought at first. A reduce is a binary function with one output. So ZxZ -> Z for example. But it seems like what we may want is actually a second 'map' phase that isn't parallelized (ie. internal state is allowed).

I'll have to give this a think...

I think essentially there's a design flaw in MAUS because we never actually want to do a reduce (ie. two inputs with one output like f(x,y ) = x+y. We actually want another 'map' but that isn't parallelized and retains internal state.

So what I will probably do (after thinking) is modify the reduce API to only take in one input. A lot of what we described this part of MAUS to do is the same -- functionality that sees all of the data -- but I think it would clarify things.

Can you think of anything better to call 'map' and 'reduce'?

At a very high-level, MAUS processing is similar to a map-reduce in that data is partitioned, mapped, then reduced (e.g. to a single output plot or file).

In the detail though, the match to the map-reduce concepts seems to break down at bit. Map is defined as Map(k1,v1)->list(k2,v2) whereas MAUS has Map(doc1)->doc2. Reduce is defined as Reduce(k2, list(v2))->list(v3) whereas MAUS wants, for example, Reduce(list(doc))->thing.

Mappers and reducers could be replaced with Transformers (no internal state) and Mergers (internal state)? These could apply:

• Transform(doc)->doc
• Merge(list(docs))->thing (e.g. plot, list(docs), file ...)

I was trying to think if we should abandon the Map-Reduce setup (like you say) but was waiting to talk to Chris Rogers before doing anything. I think you're right: we either need to follow more strictly the Map-Reduce setup which would mean that histogramming was a separate package (which is what I was originally thinking honestly) or move to what you suggest but avoid Map-Reduce naming.

I think I like your idea best. We could even just define every component to say if it can be parallelized. Normally single threaded programming is more like pipeline programming and this would facilitate that. However, we could probably start having a multipass with the intermediate state stored in a DB or something. Currently the temporary storage between map and reduce really worries me because it's a lot of I/O. We could have normal users do single-pass single-thread, and the control room and experts have a two pass setup where one pass is parallelized and the other isn't. This could even generalize to N passes.

I imagine that getting an answer to this question taking a week should not block your work? It just may require a little refactoring?

Yes, it wouldn't be a blocker.

Going back to the functools.reduce issue, according to the functools.reduce API, functools.reduce(function, iterable, initializer]) expects function to be of form function(m, n) where m is the currently aggregated information i.e. it processes n and updates m with the result.

It states that "If initializer is not given and iterable contains only one item, the first item is returned." This is the cause of the problems for us since function won't be called (not only that but the function is never called on the first argument of the iterable) meaning it won't be "reduced" in the way we may want. For example:

def test_int_reduce(x, y):
    print "In test_int_reduce: y: %d" % y
    x = x + y
    return x

functools.reduce(test_int_reduce, [1, 2, 3])
In test_int_reduce: y: 2
In test_int_reduce: y: 3
6
functools.reduce(test_int_reduce, [1, 2])
In test_int_reduce: y: 2
3
functools.reduce(test_int_reduce, [1])
1

functools.reduce(test_int_reduce, [1], 0)
In test_int_reduce: y: 1
In test_int_reduce: y: 2
1

def test_reduce(x, y):
    x = "%s%d" % (x, len(y))
    return x

functools.reduce(test_reduce, ["aa", "bbb", "cccc"])
'aa34'

functools.reduce(test_reduce, ["aa", "bbb", "cccc"], "")
'234'
functools.reduce(test_reduce, ["aa"], "")
'2'

Sounds like the API wasn't quite right to start with, so changing makes sense. Splitting into a separate package introduces more management overhead than we can probably sustain. So I think modifying the API is probably the correct way to go...

After a bit more thought, the way I would like to do this is to introduce a base class, one for C++ and one for Python. The base class defines the interface and handles

• Conversion from json to string
• Errors

The motivation for this is

• Makes the interface clear
• Allows to do error handling once

Might be slightly out of scope for Mike's work? But if we change the API, best to only do it once. Thoughts? Might be a Rogers job that one...

That's definitely outside of his job definition. I'll try to do it this week unless you want to do it (since you wrote the error handler for example).

Will take me a week or two to get to it... snowed this week.

I'll do it Thursday. At least everything but the errors part unless you can explain what you wanted. My other goal on Thursday relates to the RealData stuff (which I really want to rename DAQData for clarity!). This should take an hour to write and an hour to test.

Good code - lots of comments, easy to read. Where are the tests? I couldn't see anything in your branch. Looking through Go.py:

40: 'control_room_style' doesnt mean much to me, and the description doesnt help. Presumably you mean distributed across many PCs? Can you think of a better description/name

140: assert type_of_dataflow in possible_types_of_dataflow.keys()
...
148: else
149:     raise LookupError("bad type_of_dataflow: %s" % type_of_dataflow)

148: else
149:     raise LookupError("type_of_dataflow '"+str(type_of_dataflow)+"' should be one of "+str(possible_types_of_dataflow.keys()) % type_of_dataflow)

A small comment. It may be less confusing to have the print statements in the

while len(map_buffer) != 0

HINT: MAUS will process 1 event only at first...
TRANSFORM/MERGE/OUTPUT:  Processed 0 events so far, 1 events in buffer.
TRANSFORM/MERGE/OUTPUT:  Processed 1 events so far, 3 events in buffer.
TRANSFORM: Shutting down transformer

HINT: MAUS will process 1 event only at first...
TRANSFORM/MERGE/OUTPUT:  Processed 1 events so far,   3 events in buffer.
TRANSFORM/MERGE/OUTPUT:  Processed 4 events so far,   0 events in buffer.
TRANSFORM: Shutting down transformer

implemented