Bug #933
InputCppDAQData and releases
100%
Description
./bin/user/online_dqp_celery_joy.py does not run in the MAUS release.
Using "online" mode - where the input is provided by,
my_input = MAUS.InputCppDAQOnlineData()
fails with:
$ ./bin/user/online_dq_celery_joy.py -type_of_dataflow=multi_process_input_transform Traceback (most recent call last): File "./bin/user/online_dq_celery_joy.py", line 73, in <module> run() File "./bin/user/online_dq_celery_joy.py", line 53, in run my_input = MAUS.InputCppDAQOnlineData() AttributeError: 'module' object has no attribute 'InputCppDAQOnlineData'
Using "offline" mode - where the input is provided by,
my_input = MAUS.InputCppDAQData()
fails with:
ERROR : InputCppDAQData is a base imput class and can not be used to access the DAQ data! *** Use InputCppDAQOfflineData or InputCppDAQOnlineData instead. ***
But there is no InputCppDAQOnlineData class at present.
Files
Updated by Rogers, Chris over 11 years ago
- Assignee changed from Tunnell, Christopher to Karadzhov, Yordan
- Target version set to Future MAUS release
Yordan can you look at this? But Mike, if you want this code to run, can you please make an application test? So add a directory like
tests/integration/test_user/test_online_dq_celery_joy
and put a script that calls the python code.
Updated by Karadzhov, Yordan over 11 years ago
To run the InputCppDAQOnlineData you need a special version of the third party libraries that includes some DATE staff + MSQL.
This do not exist in the repository, only on the online reco machine.
Updated by Rogers, Chris over 11 years ago
Yeah I realise this - but the InputCppDAQData isn't working for offline use. I was expecting to see an InputCppDAQOfflineData but it doesn't seem to have made it into the trunk...
Updated by Jackson, Mike over 11 years ago
Sorry, my phrasing was odd.
InputCppDAQOfflineData exists - and I can run that fine.
The issue is InputCppDAQOnlineData which doesn't exist in the the codebase but is needed by clients (such as online_dq_celery_joy.py) that are in the release. And, there is no doc that explains where to get InputCppDAQOnlineData.
I was deploying my latest development branch on the control room machine (since the control room branch does not build).
Updated by Rogers, Chris over 11 years ago
Yeah, sorry, my mistake, it's InputCppDAQOnlineData that was missing.
Updated by Karadzhov, Yordan over 11 years ago
InputCppDAQOnlineData can't go into the trunk because it can't be build with the standard version of the third party.
It can't go into the control-room branch too because control-room branch uses the same third party as the trunk - that's why control-room branch does not build.
Try to copy and run the attacher on the online reco machine.
You will need also at add the following lines to the ConfigurationDefaults.py:
DAQ_hostname = 'miceraid1a'
DAQ_monitor_name = 'MICE Online Monitor'
Updated by Karadzhov, Yordan over 11 years ago
BTW this class (InputCppDAQOnlineData) has never been tested.
If you have problems let me know.
Updated by Rogers, Chris over 11 years ago
Mind if I add the code attached to the trunk and explicitly write "do not build" somewhere?
Updated by Rogers, Chris over 11 years ago
I think we need to make the test machine heplnm071 really look like the control room. I'm nervous about the amount of online code that is not tested, both Yordan and Mike's. Rogers job I guess, but I'm now landed with MOM job for next two weeks...
Updated by Tunnell, Christopher over 11 years ago
Rogers, Chris wrote:
I think we need to make the test machine heplnm071 really look like the control room. I'm nervous about the amount of online code that is not tested, both Yordan and Mike's. Rogers job I guess, but I'm now landed with MOM job for next two weeks...
Not to be the pessimist, but it ain't going to happen that way. First, I think they're moving to SL so the Centos-ness of heplnm071 won't matter that much. Second, unless you can have a fake socket to test from (which I would bet would be a lot of work to get right) then you can't really make it exactly like the control room.
Having a 'test procedure' that MOMs can do before a run period that is fairly well documented, I'd say, would be the way to go. If we try to turn the DAQ on in Cosmic mode a week before a run, then we can learn a lot. The key here is that doucumenting this 'manual' (rather than automatic) procedure may deliver the same result of functioning code but at much less of a price in terms of man-power.
Just like it's hard to test Geant4 interfaces, it's hard to mock the DATE stuff.
My two cents...
Updated by Rogers, Chris over 11 years ago
First, I think they're moving to SL so the Centos-ness of heplnm071 won't matter that much.
They are indeed moving to SL, we still want a machine designated as "online clone" for online testing. OS doesnt matter.
Second, unless you can have a fake socket to test from (which I would bet would be a lot of work to get right) then you can't really make it exactly like the
control room.
I guess it comes down to a judgement call - how much effort to make manual tests compared to automatic tests. Certainly the celery stuff we already have automatic tests which are not run because the environment is wrong - madness - so we need to set up the environment correctly. For DATE stuff, we have the situation where the developer can't even check his code works because we don't have an appropriate environment which is also bonkers.
Just like it's hard to test Geant4 interfaces, it's hard to mock the DATE stuff.
Well, we do indeed check GEANT4 interfaces - as in I make a G4Step and check that the stepping action behaves correctly etc.
Updated by Jackson, Mike over 11 years ago
(Possibly naive) attempt to build InputCppDAQOnlineData in the control room,.
$ cd MAUS.new $ wget http://micewww.pp.rl.ac.uk/attachments/848/InputCppDAQOnlineData.tarz $ mv InputCppDAQOnlineData.tarz InputCppDAQOnlineData.tar.gz $ gunzip InputCppDAQOnlineData.tar.gz $ tar -xf InputCppDAQOnlineData.tar $ cp -r control-room/src/input/InputCppDAQOnlineData/ maus-control-room/src/input/ $ cd maus-control-room/ $ source env.sh $ scons build ... scons: Building targets ... g++ -o src/common_cpp/DataStructure/MausDataStructure.os -c -Wall -Dlong32='int' "-Dlong64='long long'" -O3 -m32 -pthread -rdynamic -fPIC -Ithird_party/install/include -Ithird_party/install/include/python2.7 -Ithird_party/install/include/root -Isrc/legacy -Isrc/common_cpp -Ithird_party/build/root_v5.30.03/include -Ithird_party/build/geant4.9.2.p04/include -I. src/common_cpp/DataStructure/MausDataStructure.cc g++ -o src/input/InputCppDAQOnlineData/build/InputCppDAQOnlineData.os -c -Wall -Dlong32='int' "-Dlong64='long long'" -O3 -m32 -pthread -rdynamic -fPIC -Ithird_party/install/include -Ithird_party/install/include/python2.7 -Ithird_party/install/include/root -Isrc/legacy -Isrc/common_cpp -Ithird_party/build/root_v5.30.03/include -Ithird_party/build/geant4.9.2.p04/include -I. -Isrc/input/InputCppDAQOnlineData src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc In file included from ./src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.hh:30, from src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:18: ./src/input/InputCppDAQOnlineData/MDmonitoring.hh:5:31: error: unpacking/monitor.h: No such file or directory In file included from src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:18: ./src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.hh:35:48: error: src/common_cpp/DAQ/UnpackEventLib.hh: No such file or directory ./src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.hh:49: error: expected class-name before '{' token src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: In constructor 'InputCppDAQOnlineData::InputCppDAQOnlineData(std::string, std::string)': src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: 'InputCppDAQData' has not been declared src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: expected `(' before 'InputCppDAQData' src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: expected `{' before 'InputCppDAQData' src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: At global scope: src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:21: error: expected constructor, destructor, or type conversion before '(' token src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: In member function 'bool InputCppDAQOnlineData::birth(std::string)': src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:27: error: 'InputCppDAQData' has not been declared src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc: In member function 'bool InputCppDAQOnlineData::readNextEvent()': src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:58: error: '_eventPtr' was not declared in this scope scons: *** [src/input/InputCppDAQOnlineData/build/InputCppDAQOnlineData.os] Error 1 scons: building terminated because of errors.
Updated by Karadzhov, Yordan over 11 years ago
As I said this code has never been tested.
Try to replace all includes in the header file with:
#include "src/input/InputCppDAQData/InputCppDAQData.hh"
#include "src/input/InputCppDAQOnlineData/MDmonitoring.hh"
Updated by Karadzhov, Yordan over 11 years ago
I see that unpacking/monitor.h is missing too.
You may copy this file from the DATE installation on some of the daq machines.
/opt/date/monitoring/monitor.h
Updated by Tunnell, Christopher over 11 years ago
It's called the wrong thing "InputCppDAQData", right? In what Mike is seeing?
The control room branch on launchpad I thought had these changes? and at one point we wrote down what we changed to get there. it should be in the wiki? I can hunt for it if we need.
Is it beyond the scope of Mike's work to ensure his code works with anything other than InputCppDAQData (the offline one)? My memory fades more and more, but I wrote some stuff down either in the control-room repo or in the wiki...
Updated by Rogers, Chris over 11 years ago
Is it beyond the scope of Mike's work to ensure his code works with anything other than InputCppDAQData (the offline one)?
I think it would be great if Mike could ensure his code works in the control room for online stuff.
Updated by Jackson, Mike over 11 years ago
bin/user/online_dq_celery_joy cleaned and renamed to bin/user/reconstruct_daq.py
InputCppDAQOfflineData used in the above works fine in the control room.
Updated by Jackson, Mike over 11 years ago
I'll be able to test the updates tomorrow, Friday or Monday.
Updated by Jackson, Mike over 11 years ago
I saw that MAUS.new/maus-control-room had been updated and a compiled
version of InputCppDAQOnlineData is now in build/.
So, I tried to run
$MAUS_ROOT_DIR/bin/user/reconstruct_daq.py
but this failed with
failed to import InputCppDAQOnlineData
Likewise,
$ ipython $ from InputCppDAQOnlineData import InputCppDAQOnlineData In [4]: from InputCppDAQOnlineData import InputCppDAQOnlineData --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/mice/MAUS.new/maus-control-room/<ipython-input-4-6643baeb60d7> in <module>() ----> 1 from InputCppDAQOnlineData import InputCppDAQOnlineData /home/mice/MAUS.new/maus-control-room/build/InputCppDAQOnlineData.py in <module>() 24 fp.close() 25 return _mod ---> 26 _InputCppDAQOnlineData = swig_import_helper() 27 del swig_import_helper 28 else: /home/mice/MAUS.new/maus-control-room/build/InputCppDAQOnlineData.py in swig_import_helper() 20 if fp is not None: 21 try: ---> 22 _mod = imp.load_module('_InputCppDAQOnlineData', fp, pathname, description) 23 finally: 24 fp.close() ImportError: /home/mice/MAUS.new/maus-control-room/build/_InputCppDAQOnlineData.so: undefined symbol: _ZN15InputCppDAQData13readNextEventEv
Updated by Karadzhov, Yordan over 11 years ago
This is a problem in the linking. The error message above means that InputCppDAQOnlineData is not linked to InputCppDAQOData.
I manage to fix this. It was due to an error in the sconscript file.
But now I have:
ImportError: /home/mice/MAUS.new/maus-control-room/build/_InputCppDAQOnlineData.so: undefined symbol: monitorSetNowait
Which probably means that it is not linked to libDATEMon.a and I do not know how to fix this.
Updated by Rogers, Chris over 11 years ago
Is there anything stopping us rolling back to the previous online DAQ code? Feels like this one just isn't ready.
Updated by Karadzhov, Yordan over 11 years ago
Ok, obviously libDATEMon.a was missing in /home/mice/MAUS.new/maus-control-room/third_party/install/lib
now you have to tell scons to link this library. This has to be easy???
Updated by Rogers, Chris over 11 years ago
Normally you just do
env.Append(LIBS=DATEMon)
But this needs a shared library - can you make the DATE lib a shared library though? Something like
g++ -shared -o libDATEMon.so libDATEMon.a
Updated by Karadzhov, Yordan over 11 years ago
Here is the summary of what I did:
1. copy /home/mice/monitoring/monitor.h
to /home/mice/MAUS.new/maus-control-room/third_party/install/include/unpacking/monitor.h
2. copy /home/mice/monitoring/libDATEMon.a and /home/mice/monitoring/libmysqlclient.so
to /home/mice/MAUS.new/maus-control-room/third_party/install/lib/
3. add line: localenv.Append(LIBS=['DATEMon', 'mysqlclient'])
to /home/mice/MAUS.new/maus-control-room/src/input/InputCppDAQOnlineData/sconscript
Seems like everything is Ok now.
Updated by Rogers, Chris over 11 years ago
It links against a static library? Nice to know... well done.
Updated by Karadzhov, Yordan over 11 years ago
BTW it does not work when linking against the shared library libDATEMon.so.
Updated by Jackson, Mike over 11 years ago
Again, possibly naive test. reconstruct_daq
now using InputCppDAQOnlineData
...
$MAUS_ROOT_DIR/bin/user/reconstruct_daq.py Trying to add file 02873.003 from /home/mice/MAUS.new/maus-control-room/src/input/InputCppDAQData File 02873.003 is added python: src/input/InputCppDAQOnlineData/InputCppDAQOnlineData.cc:40: bool InputCppDAQOnlineData::birth(std::string): Assertion `configJSON.isMember("DAQ_hostname")' failed. Aborted
Maybe expected if the DAQ isn't running?
Updated by Rogers, Chris over 11 years ago
I am available for the next couple of days to make sure that stuff works online. If I can be any help in this please let me know.
Updated by Rogers, Chris over 11 years ago
- Priority changed from Normal to Urgent
We got to the point now where InputCppDAQOnlnieData runs but returns NULL for the event pointer (DAQ buffer). So progress. This obviously needs to work by Friday.
Updated by Rogers, Chris over 10 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
I think this is fixed. It works for me.
Updated by Rajaram, Durga over 10 years ago
- Target version changed from Future MAUS release to MAUS-v0.6.0