Project

General

Profile

Bug #933

InputCppDAQData and releases

Added by Jackson, Mike over 11 years ago. Updated over 10 years ago.

Status:
Closed
Priority:
Urgent
Category:
Online reconstruction
Target version:
Start date:
12 March 2012
Due date:
% Done:

100%

Estimated time:
Workflow:
New Issue

Description

./bin/user/online_dqp_celery_joy.py does not run in the MAUS release.

Using "online" mode - where the input is provided by,

    my_input = MAUS.InputCppDAQOnlineData()

fails with:
$  ./bin/user/online_dq_celery_joy.py -type_of_dataflow=multi_process_input_transform
Traceback (most recent call last):
  File "./bin/user/online_dq_celery_joy.py", line 73, in <module>
    run()
  File "./bin/user/online_dq_celery_joy.py", line 53, in run
    my_input = MAUS.InputCppDAQOnlineData()
AttributeError: 'module' object has no attribute 'InputCppDAQOnlineData'

Using "offline" mode - where the input is provided by,
    my_input = MAUS.InputCppDAQData()

fails with:
ERROR : InputCppDAQData is a base imput class and can not be used to access the DAQ data!
*** Use InputCppDAQOfflineData or InputCppDAQOnlineData instead. ***

But there is no InputCppDAQOnlineData class at present.


Files

InputCppDAQOnlineData.tarz (3.81 KB) InputCppDAQOnlineData.tarz Karadzhov, Yordan, 12 March 2012 15:15
#1

Updated by Rogers, Chris over 11 years ago

  • Assignee changed from Tunnell, Christopher to Karadzhov, Yordan
  • Target version set to Future MAUS release

Yordan can you look at this? But Mike, if you want this code to run, can you please make an application test? So add a directory like

tests/integration/test_user/test_online_dq_celery_joy

and put a script that calls the python code.

#2

Updated by Rogers, Chris over 11 years ago

Also, can you change the name to something sensible?

#3

Updated by Karadzhov, Yordan over 11 years ago

To run the InputCppDAQOnlineData you need a special version of the third party libraries that includes some DATE staff + MSQL.
This do not exist in the repository, only on the online reco machine.

#4

Updated by Rogers, Chris over 11 years ago

Yeah I realise this - but the InputCppDAQData isn't working for offline use. I was expecting to see an InputCppDAQOfflineData but it doesn't seem to have made it into the trunk...

#6

Updated by Jackson, Mike over 11 years ago

Sorry, my phrasing was odd.

InputCppDAQOfflineData exists - and I can run that fine.

The issue is InputCppDAQOnlineData which doesn't exist in the the codebase but is needed by clients (such as online_dq_celery_joy.py) that are in the release. And, there is no doc that explains where to get InputCppDAQOnlineData.

I was deploying my latest development branch on the control room machine (since the control room branch does not build).

#7

Updated by Rogers, Chris over 11 years ago

Yeah, sorry, my mistake, it's InputCppDAQOnlineData that was missing.

#8

Updated by Karadzhov, Yordan over 11 years ago

InputCppDAQOnlineData can't go into the trunk because it can't be build with the standard version of the third party.
It can't go into the control-room branch too because control-room branch uses the same third party as the trunk - that's why control-room branch does not build.

Try to copy and run the attacher on the online reco machine.
You will need also at add the following lines to the ConfigurationDefaults.py:

DAQ_hostname = 'miceraid1a'
DAQ_monitor_name = 'MICE Online Monitor'

#9

Updated by Karadzhov, Yordan over 11 years ago

BTW this class (InputCppDAQOnlineData) has never been tested.
If you have problems let me know.

#10

Updated by Rogers, Chris over 11 years ago

Mind if I add the code attached to the trunk and explicitly write "do not build" somewhere?

#11

Updated by Karadzhov, Yordan over 11 years ago

OK but first try it in the control room?

#12

Updated by Rogers, Chris over 11 years ago

okay...

#13

Updated by Rogers, Chris over 11 years ago

I think we need to make the test machine heplnm071 really look like the control room. I'm nervous about the amount of online code that is not tested, both Yordan and Mike's. Rogers job I guess, but I'm now landed with MOM job for next two weeks...

#14

Updated by Tunnell, Christopher over 11 years ago

Rogers, Chris wrote:

I think we need to make the test machine heplnm071 really look like the control room. I'm nervous about the amount of online code that is not tested, both Yordan and Mike's. Rogers job I guess, but I'm now landed with MOM job for next two weeks...

Not to be the pessimist, but it ain't going to happen that way. First, I think they're moving to SL so the Centos-ness of heplnm071 won't matter that much. Second, unless you can have a fake socket to test from (which I would bet would be a lot of work to get right) then you can't really make it exactly like the control room.

Having a 'test procedure' that MOMs can do before a run period that is fairly well documented, I'd say, would be the way to go. If we try to turn the DAQ on in Cosmic mode a week before a run, then we can learn a lot. The key here is that doucumenting this 'manual' (rather than automatic) procedure may deliver the same result of functioning code but at much less of a price in terms of man-power.

Just like it's hard to test Geant4 interfaces, it's hard to mock the DATE stuff.

My two cents...

#15

Updated by Rogers, Chris over 11 years ago

First, I think they're moving to SL so the Centos-ness of heplnm071 won't matter that much.

They are indeed moving to SL, we still want a machine designated as "online clone" for online testing. OS doesnt matter.

Second, unless you can have a fake socket to test from (which I would bet would be a lot of work to get right) then you can't really make it exactly like the
control room.

I guess it comes down to a judgement call - how much effort to make manual tests compared to automatic tests. Certainly the celery stuff we already have automatic tests which are not run because the environment is wrong - madness - so we need to set up the environment correctly. For DATE stuff, we have the situation where the developer can't even check his code works because we don't have an appropriate environment which is also bonkers.

Just like it's hard to test Geant4 interfaces, it's hard to mock the DATE stuff.

Well, we do indeed check GEANT4 interfaces - as in I make a G4Step and check that the stepping action behaves correctly etc.

#16

Updated by Jackson, Mike over 11 years ago

(Possibly naive) attempt to build InputCppDAQOnlineData in the control room,.

$ cd MAUS.new
$ wget http://micewww.pp.rl.ac.uk/attachments/848/InputCppDAQOnlineData.tarz
$ mv InputCppDAQOnlineData.tarz InputCppDAQOnlineData.tar.gz
$ gunzip InputCppDAQOnlineData.tar.gz 
$ tar -xf InputCppDAQOnlineData.tar 
$ cp -r control-room/src/input/InputCppDAQOnlineData/ maus-control-room/src/input/
$ cd maus-control-room/
$ source env.sh
$ scons build
...
scons: Building targets ...
g++ -o src/common_cpp/DataStructure/MausDataStructure.os -c -Wall -Dlong32='int' "-Dlong64='long long'" -O3 -m32 -pthread -rdynamic -fPIC -Ithird_party/install/include -Ithird_party/install/include/python2.7 -Ithird_party/install/include/root -Isrc/legacy -Isrc/common_cpp -Ithird_party/build/root_v5.30.03/include -Ithird_party/build/geant4.9.2.p04/include -I. src/common_cpp/DataStructure/MausDataStructure.cc
g++ -o src/input/InputCppDAQOnlineData/build/InputCppDAQOnlineData.os -c -Wall -Dlong32='int' "-Dlong64='long long'" -O3 -m32 -pthread -rdynamic -fPIC -Ithird_party/install/include -Ithird_party/install/include/python2.7 -Ithird_party/install/include/root -Isrc/legacy -Isrc/common_cpp -Ithird_party/build/root_v5.30.03/include -Ithird_party/build/geant4.9.2.p04/include -I. -Isrc/input/InputCppDAQOnlineData src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc
In file included from ./src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.hh:30,
                 from src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:18:
./src/input/InputCppDAQOnlineData/MDmonitoring.hh:5:31: error: unpacking/monitor.h: No such file or directory
In file included from src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:18:
./src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.hh:35:48: error: src/common_cpp/DAQ/UnpackEventLib.hh: No such file or directory
./src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.hh:49: error: expected class-name before '{' token
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: In constructor 'InputCppDAQOnlineData::InputCppDAQOnlineData(std::string, std::string)':
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: 'InputCppDAQData' has not been declared
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: expected `(' before 'InputCppDAQData'
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: expected `{' before 'InputCppDAQData'
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: At global scope:
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: expected constructor, destructor, or type conversion before '(' token
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: In member function 'bool InputCppDAQOnlineData::birth(std::string)':
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:27: error: 'InputCppDAQData' has not been declared
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: In member function 'bool InputCppDAQOnlineData::readNextEvent()':
src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:58: error: '_eventPtr' was not declared in this scope
scons: *** [src/input/InputCppDAQOnlineData/build/InputCppDAQOnlineData.os] Error 1
scons: building terminated because of errors.
#17

Updated by Karadzhov, Yordan over 11 years ago

As I said this code has never been tested.
Try to replace all includes in the header file with:

#include "src/input/InputCppDAQData/InputCppDAQData.hh"
#include "src/input/InputCppDAQOnlineData/MDmonitoring.hh"

#18

Updated by Karadzhov, Yordan over 11 years ago

I see that unpacking/monitor.h is missing too.
You may copy this file from the DATE installation on some of the daq machines.
/opt/date/monitoring/monitor.h

#19

Updated by Tunnell, Christopher over 11 years ago

It's called the wrong thing "InputCppDAQData", right? In what Mike is seeing?

The control room branch on launchpad I thought had these changes? and at one point we wrote down what we changed to get there. it should be in the wiki? I can hunt for it if we need.

Is it beyond the scope of Mike's work to ensure his code works with anything other than InputCppDAQData (the offline one)? My memory fades more and more, but I wrote some stuff down either in the control-room repo or in the wiki...

#20

Updated by Jackson, Mike over 11 years ago

Ticket #904?

#21

Updated by Rogers, Chris over 11 years ago

Is it beyond the scope of Mike's work to ensure his code works with anything other than InputCppDAQData (the offline one)?

I think it would be great if Mike could ensure his code works in the control room for online stuff.

#22

Updated by Jackson, Mike over 11 years ago

bin/user/online_dq_celery_joy cleaned and renamed to bin/user/reconstruct_daq.py

InputCppDAQOfflineData used in the above works fine in the control room.

#23

Updated by Jackson, Mike over 11 years ago

I'll be able to test the updates tomorrow, Friday or Monday.

#24

Updated by Jackson, Mike over 11 years ago

I saw that MAUS.new/maus-control-room had been updated and a compiled
version of InputCppDAQOnlineData is now in build/.

So, I tried to run

$MAUS_ROOT_DIR/bin/user/reconstruct_daq.py

but this failed with
failed to import InputCppDAQOnlineData

Likewise,
     
$ ipython
$ from InputCppDAQOnlineData import InputCppDAQOnlineData

In [4]: from InputCppDAQOnlineData import InputCppDAQOnlineData
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call
last)
/home/mice/MAUS.new/maus-control-room/<ipython-input-4-6643baeb60d7> in
<module>()
----> 1 from InputCppDAQOnlineData import InputCppDAQOnlineData

/home/mice/MAUS.new/maus-control-room/build/InputCppDAQOnlineData.py in
<module>()
     24                 fp.close()
     25             return _mod
---> 26     _InputCppDAQOnlineData = swig_import_helper()
     27     del swig_import_helper
     28 else:

/home/mice/MAUS.new/maus-control-room/build/InputCppDAQOnlineData.py in
swig_import_helper()
     20         if fp is not None:
     21             try:
---> 22                 _mod = imp.load_module('_InputCppDAQOnlineData',   
fp, pathname, description)
     23             finally:
     24                 fp.close()

ImportError:
/home/mice/MAUS.new/maus-control-room/build/_InputCppDAQOnlineData.so:
undefined symbol: _ZN15InputCppDAQData13readNextEventEv

#25

Updated by Karadzhov, Yordan over 11 years ago

This is a problem in the linking. The error message above means that InputCppDAQOnlineData is not linked to InputCppDAQOData.
I manage to fix this. It was due to an error in the sconscript file.
But now I have:
ImportError: /home/mice/MAUS.new/maus-control-room/build/_InputCppDAQOnlineData.so: undefined symbol: monitorSetNowait

Which probably means that it is not linked to libDATEMon.a and I do not know how to fix this.

#26

Updated by Rogers, Chris over 11 years ago

Is there anything stopping us rolling back to the previous online DAQ code? Feels like this one just isn't ready.

#27

Updated by Karadzhov, Yordan over 11 years ago

Ok, obviously libDATEMon.a was missing in /home/mice/MAUS.new/maus-control-room/third_party/install/lib

now you have to tell scons to link this library. This has to be easy???

#28

Updated by Rogers, Chris over 11 years ago

Normally you just do

env.Append(LIBS=DATEMon)

But this needs a shared library - can you make the DATE lib a shared library though? Something like

g++ -shared -o libDATEMon.so libDATEMon.a
#29

Updated by Karadzhov, Yordan over 11 years ago

Here is the summary of what I did:
1. copy /home/mice/monitoring/monitor.h
to /home/mice/MAUS.new/maus-control-room/third_party/install/include/unpacking/monitor.h

2. copy /home/mice/monitoring/libDATEMon.a and /home/mice/monitoring/libmysqlclient.so
to /home/mice/MAUS.new/maus-control-room/third_party/install/lib/

3. add line: localenv.Append(LIBS=['DATEMon', 'mysqlclient'])
to /home/mice/MAUS.new/maus-control-room/src/input/InputCppDAQOnlineData/sconscript

Seems like everything is Ok now.

#30

Updated by Rogers, Chris over 11 years ago

It links against a static library? Nice to know... well done.

#31

Updated by Karadzhov, Yordan over 11 years ago

BTW it does not work when linking against the shared library libDATEMon.so.

#32

Updated by Jackson, Mike over 11 years ago

Again, possibly naive test. reconstruct_daq now using InputCppDAQOnlineData...

$MAUS_ROOT_DIR/bin/user/reconstruct_daq.py 
Trying to add file 02873.003 from /home/mice/MAUS.new/maus-control-room/src/input/InputCppDAQData 
File 02873.003 is added 
python: src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:40: 
bool InputCppDAQOnlineData::birth(std::string): Assertion `configJSON.isMember("DAQ_hostname")' failed. 
Aborted 

Maybe expected if the DAQ isn't running?

#33

Updated by Rogers, Chris over 11 years ago

I am available for the next couple of days to make sure that stuff works online. If I can be any help in this please let me know.

#34

Updated by Rogers, Chris over 11 years ago

  • Priority changed from Normal to Urgent

We got to the point now where InputCppDAQOnlnieData runs but returns NULL for the event pointer (DAQ buffer). So progress. This obviously needs to work by Friday.

#35

Updated by Rogers, Chris over 10 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

I think this is fixed. It works for me.

#36

Updated by Rajaram, Durga over 10 years ago

  • Target version changed from Future MAUS release to MAUS-v0.6.0

Also available in: Atom PDF