Project

General

Profile

Feature #936

Data representation conversion

Added by Rogers, Chris over 11 years ago. Updated about 11 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Richards, Alexander
Category:
Detector Integration
Target version:
MAUS-v0.4.0
Start date:
15 March 2012
Due date:
% Done:

100%

Estimated time:
Workflow:
Closed

Description

The aim is to implement an implicit conversion from one data representation to another at runtime. So at the moment we have, say, Module A, Module B and Module C each of which interfaces the data in a different. Say Module A and Module B represent the data using JsonCpp and Module C represents the data using Cpp. Then I would like to be able to determine at run time the data representation used by each Module and call a converter class to handle the interface between the modules. So at the moment the model is that everything interfaces by a string, like:

Module A -> string -> Module B -> string -> Module C -> ...

What I would like is

Module A -> JsonCpp -> Converter<JsonCpp, JsonCpp> -> JsonCpp -> Module B -> JsonCpp -> Converter<JsonCpp, NativeCpp> -> NativeCpp -> Module C -> ...

I would like to keep the API essentially the same however. So for example, we have in bin/simulate_mice.py

    # This input generates empty spills, to be filled by the beam maker later on
    my_input = MAUS.InputPySpillGenerator()

    # Create an empty array of mappers, then populate it
    # with the functionality you want to use.
    my_map = MAUS.MapPyGroup()
    my_map.append(MAUS.MapPyBeamMaker()) # beam construction
    my_map.append(MAUS.MapCppSimulation())  #  geant4 simulation
    my_map.append(MAUS.MapCppTrackerMCDigitization())  # SciFi electronics model
    # can specify datacards here or by using appropriate command line calls
    datacards = io.StringIO(u"")
    # Then construct a MAUS output component - filename comes from datacards
    my_output = MAUS.OutputPyJSON()

    # The Go() drives all the components you pass in, then check the file
    # (default simulation.out) for output

    MAUS.Go(my_input, my_map, MAUS.ReducePyDoNothing(), my_output, datacards)

Don't mind cosmetic changes, but basically I think this is a nice set up and I would like to keep it.

#1

Updated by Rogers, Chris over 11 years ago

So this necessitates a couple of things:

  1. New Converter abstract type and python bindings (some implementation already for this).
  2. Implementation of converter type for Converter<PyJson, CppJson>, Converter<NativeCpp, CppJson>, Converter<NativeCpp, PyJson>. I consider PyJson and CppJson to be different representations of the data (they look different in memory).
  3. Add the converter calls at runtime, determining the interface type of each module dynamically

The natural order is to start with 1 and end at 3, but probably want to think about how to do 3 first because it may impact on the abstraction layer somewhat - even if the actual implementation is done last.

The converter call needs to happen in a few places. For single threaded operation:
  • src/common_py/Go.py - PipelineSingleThreadDataflowExecutor.execute
  • src/common_py/Go.py - DataflowUtilities.buffer_input
  • src/map/MapPyGroup/MapPyGroup.py MapPyGroup.process
For multi processing operation:
  • src/common_py/Go.py - MultiProcessInputTransformDataflowExecutor.execute
  • src/common_py/Go.py - MultiProcessMergeOutputDataflowExecutor.execute
  • src/map/MapPyGroup/MapPyGroup.py MapPyGroup.process (same as above - this used to be handled differently, I think now it is handled the same multiprocess vs single process)

For testing, you will need to explicitly have the celery stuff set up to check the distributed processing operation. I will try to get you a login for the test machines and give you sudo access, so we can install the celery stuff there (or maybe ask Matt Robinson to do it, he may end up being in charge of that machine).

Note that this has probably changed in recent releases so you really want latest release I think. I added Mike as a watcher so he can comment.

#2

Updated by Richards, Alexander over 11 years ago

Hi Chris,
despite the pseudo code you gave looking like strongly typed c++ you actually want this implemented in the python right. So need a was of getting Converter<JsonCpp, NativeCpp> in python if I understand you correctly

--Alex

#3

Updated by Richards, Alexander over 11 years ago

sorry, just read it more closely, so your looking for c++ converters with python bindings got it. ;-)

--Alex

#4

Updated by Rogers, Chris over 11 years ago

I was imagining SWIGged C++ because it probably will run faster. But I don't know if SWIG can handle the templates properly.

#5

Updated by Rogers, Chris about 11 years ago

This is closed, right? If so can you close it? Cheers, Chris

#6

Updated by Richards, Alexander about 11 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100
  • Workflow changed from New Issue to Closed
#7

Updated by Rogers, Chris about 11 years ago

  • Target version changed from Future MAUS release to MAUS-v0.4.0

Also available in: Atom PDF