Project

General

Profile

Bug #1467

Tracker cosmic data broken

Added by Dobbs, Adam over 7 years ago. Updated over 6 years ago.

Status:
Open
Priority:
Normal
Assignee:
Category:
Tracker
Target version:
Start date:
16 May 2014
Due date:
% Done:

0%

Estimated time:
Workflow:
New Issue

Description

The tracker cosmic data read in is currently broken in the tracker devel branch. The mapper responsible for digitising the data, MapCppTrackerDigits, throwing a 'Bad json document error. It is still passing its tests however, and I think the error does not originate there. Looking at the json data that arrives there I see:

{"daq_data":{"V830":{"channels":{"ch0":0,"ch1":0,"ch10":0,"ch11":0,"ch12":0,"ch13":0,"ch14":0,"ch15":0,"ch16":0,"ch17":0,"ch18":0,"ch19":0,"ch2":0,"ch20":0,"ch21":0,"ch22":0,"ch23":0,"ch24":0,"ch25":0,"ch26":0,"ch27":0,"ch28":0,"ch29":0,"ch3":0,"ch30":0,"ch31":0,"ch4":0,"ch5":0,"ch6":0,"ch7":0,"ch8":0,"ch9":0},"equip_type":0,"geo":0,"ldc_id":0,"phys_event_number":0,"time_stamp":0},"ckov":[],"emr":{"V1731":[],"dbb":[]},"kl":[],"tag":[],"tof0":[],"tof1":[],"tof2":[],"tracker0":[],"tracker1":[],"trigger":[],"trigger_request":[],"unknown":[]},"daq_event_type":"physics_event","errors":{},"maus_event_type":"Spill","run_number":1901,"spill_number":-1}

i.e. it does not seem to be getting given any tracker data, perhaps pointing the finger at InputCppDAQOfflineData. Though then it does seem strange that it still registers as bad document, rather than just as an empty one. Continuing to investigate...


Files

2014-10-08_maus-v0-7-0_cosmics.root (14.2 MB) 2014-10-08_maus-v0-7-0_cosmics.root Dobbs, Adam, 08 October 2014 14:13
VLPC_adc.png (14.3 KB) VLPC_adc.png Dobbs, Adam, 25 February 2015 18:11
Digits_npe.png (7.6 KB) Digits_npe.png Dobbs, Adam, 25 February 2015 18:11
maus_output.root (3.73 MB) maus_output.root Dobbs, Adam, 25 February 2015 18:11
output.txt (4.63 KB) output.txt Dobbs, Adam, 28 May 2015 12:39

Related issues

Related to MAUS - Bug #1563: Segmentation fault during VLSB unpackingOpenDobbs, Adam07 October 2014

Actions
#1

Updated by Dobbs, Adam about 7 years ago

OK, starting to look at this again.

At present the code runs but finds no data, not even tracker digits. Initial investigations show:

  • Tracker real digitization process really needs a rewrite following the upgrade to the MAUS API. Currently we convert from a MAUS Data object, to JSON, back to a MAUS Spill
  • The flaw however seems to enter in before then. The Data object seems to have empty arrays for the raw tracker data, which implies the bug is below the level of tracker digitization i.e. in the unpacking or some of the MAUS machinery that forms the Data object.

Work is ongoing...

#2

Updated by Rogers, Chris almost 7 years ago

Let me know if I can assist. Cheers...

#3

Updated by Dobbs, Adam almost 7 years ago

OK, I think the problem lies in both the InputCppDAQData module and in UnpackingEventLib (both in src/input/InputCppDAQData).

Firstly, it seems InputCppDAQData, whilst having a VLSB processor member to deal with regular tracker DAQ data, has no VLSB_C processor for implementing tracker cosmic DAQ data (though the processor itself is available).

Secondly, it does not look like the unpacking libraries implements VLSB_C fully either. There is a VLSB_CDataProcessor class there, but it looks decidedly incomplete when compared with the VLSBCppDataProcessor. In particular, only the process function is implemented, without any fill_daq_data() or reset() functions, implying it cannot save any data it extracts.

Chris, thanks for the offer, would you be able to take a look at the indicated files and tell me if you agree with my conclusions? After that we need to decide who is responsible for writing the fix (one of us from tracker I would guess).

#4

Updated by Rajaram, Durga almost 7 years ago

In case this is of help -- the unpacking library version changed at MAUS-v0.7.6.
To isolate the unpacker, it might be useful to see if it reconstructs with MAUS-v0.7.5?

#5

Updated by Dobbs, Adam almost 7 years ago

Update: I have written new methods into UnpackingEventLib to process VLSB_C data using the data structure classes without recourse to json. I did this by imitating what had already been done for VLSB objects and by looking at the old json based methods for the VLSB_C objects. I now have VLSB_C daq data getting to the tracker digitisation routines, but the reconstruction comes out as a mess. The next step is to alter the digitisation so that it bypasses the conversion to json too, to see if that takes care of it.

Additionally I have come across a few issues along the way:

1. In different sections of code, the manner of differentiating between trackers seems to change. In UnpackingEventLib, it seems to be done by assuming the following:

LDC_ID = 0 -> tracker 1
LDC_ID = 2 -> tracker 2

In the digitisation however (RealDataDigitisation), for organising the DAQ data it seems to done as:

board = floor(bank/4) (for VLPC)

OR

board = Geo - 1 (for VLPC_C)

then

if board < 4 -> tracker 1
else -> tracker 2

It then done differently again by using the calibration files for creating the SciFiDigit, again in RealDataDigitisation.

Perhaps I do not understand, but this makes me uncomfortable. Referring to David for his view.

2. In UnpackingEventLib.cc VLSBCppDataProcessor::Process, the following logic appears:

xPartEv = xVLSBFragment->GetEventNum(xWordCount)-1;
if (xPartEv > _tracker1_spill.size()) {
  _tracker1_spill.resize(xPartEv);
  _tracker0_spill.resize(xPartEv);
  _single_st_spill.resize(xPartEv);
}

however this lead to a seg fault when applied to VLSB_C data, as xPartEv can be zero. I believe the correct implementation should replace xPartEv with xPartEv + 1 everywhere in the if block (cannot test without tracker data...), which is how I implemented it for the VLSB_C's.

3. In RealDataDigitisation.cc the VLSB processing is disabled with a few code tweaks - I am curious why someone would do this. There should be no VLSB data except for the single station test and Step IV onwards anyway.

Durga, thanks for the suggestion, if I get stuck I may give it a try.

More to follow...

#6

Updated by Dobbs, Adam almost 7 years ago

Update: I have got reconstruction partially working. I think things are ok for tracker 1, but the calibration fails for tracker 2, and I think this is leading to the tracker 2 digits ending up in tracker 1.

I have made a branch to record my progress at https://code.launchpad.net/~phuccj/maus/cosmic_repair

David, could you take a look at the following and let me know what you think:

http://bazaar.launchpad.net/~phuccj/maus/cosmic_repair/view/head:/src/common_cpp/Recon/SciFi/RealDataDigitization.cc
VLSB_CCppDataProcessor::Process function (be careful of the name, there are a lot of similarly named functions.)

http://bazaar.launchpad.net/~phuccj/maus/cosmic_repair/view/head:/src/input/InputCppDAQData/UnpackEventLib.cc
RealDataDigitization::process_VLSB_c_2
RealDataDigitization::get_StatPlaneChannel

Work ongoing...

#7

Updated by Dobbs, Adam almost 7 years ago

Aside: Unable to download and install maus 0.7.5 - cdb third party library cannot be downloaded when using bazaar, and tar ball links are broken from maus wiki for both source and third party libraries.

#8

Updated by Rajaram, Durga almost 7 years ago

Yes, the problem is the cdb tarball source somehow disappeared/got mangled in Antony Wilson's branch
[ this is already an open issue #1471 ]
I then stuck the tarball in Micemine.
In addition, since ~0.8 the scons source disappeared from the pypi repository, and so we now get it from sourceforge.
The short answer - to get 0.7.5 to build -- is an ugly hack:
get

third_party/bash/40python_extras.bash
and
third_party/bash/44cdb.bash
from the trunk and stick them in the 0.7.5's
third_party/bash
directory and then build
Sorry! I wish I knew a better way to handle busted third party sources in old releases.

#9

Updated by Rogers, Chris almost 7 years ago

I was trying to save a MAUS copy of the old 3rd party stuff on heplnv156 in the third_party tarball - I guess some stuff got missed, python packages which don't come in the main 40python_extras.bash?

A test server job might be to disable $http_proxy, $https_proxy, then attempt to do the third party build; stuff that is available from heplnv156 should download without need for proxy; stuff from web should not download (fail to go through the proxy). This could be done, for example, at the release_candidate stage.

#10

Updated by Rajaram, Durga almost 7 years ago

Is this the /redmine_repos/ area on heplnv156 you're talking about, Chris?
If so, do you mind if I use that area to stuff some TP sources in there? Or let me know if you suggest a different machine/location.

#11

Updated by Rogers, Chris almost 7 years ago

I meant e.g. /www/maus/MAUS_release_version_0.9.1/third_party_libraries_incl_python.tar.gz on heplnv156. We have a backlog of these all the way back to 0.2.0 on heplnm069 e.g. /backups/rsync-weekly/weekly/www/maus/MAUS_release_version_0.7.5/third_party_libraries_incl_python.tar.gz but needs an scp to access - we don't have so much hard drive space on the web machine that we can keep many back copies. I see Adam has a login on heplnm069.

#12

Updated by Dobbs, Adam almost 7 years ago

Well, it seems I do. Out of curiosity, what is the purpose of that machine (it is missing from the list available on the computing wiki)? Pulled down the third party source, building at the moment...

#13

Updated by Rogers, Chris almost 7 years ago

Out of curiosity, what is the purpose of that machine (it is missing from the list available on the computing wiki)

Parry-riposte!

http://micewww.pp.rl.ac.uk/projects/computing-software/wiki/Computing_infrastructure#Responsibility-for-Hardware
http://micewww.pp.rl.ac.uk/projects/computing-software/wiki/Heplnm069_services

#14

Updated by Dobbs, Adam almost 7 years ago

Well struck ;)

#15

Updated by Dobbs, Adam almost 7 years ago

Update: Remembered that I have historic maus builds for the major releases stored on the RAL PPD cluster. Just ran maus 0.7.0, and it successfully completed running over the standard cosmics file (output attached). Will try to narrow when the fault entered to a specific revision next...

#16

Updated by Dobbs, Adam almost 7 years ago

Also working in maus 0.7.5...

#17

Updated by Adey, David almost 7 years ago

In the unpacking library here are two types of "data fragment" which correspond to the chunk of tracker data from each spill, one for the cosmic tests, in which the data from each VLSB board were packaged into the same "equipment" (i.e. under the same header), and one since then where the data from each VLSB were broken down into the memory banks, so under four separate headers.

In UnpackEventLib.cc there are two functions to process, both called VLSBDataProcessor::Process

The first uses the Non-cosmic MDfragmentVLSB and is intended to use the post-cosmic data.
The second (line 786) has a comment saying (actually they both do), only use with MDfragmentVLSB_C (for the cosmic data) as the argument, and seems to go and cast it as MDfragmentVLSB...

Both the cosmic and post-cosmic data types are in the equipment map from the unpacking, so the fragment should know what type it is and the processor should act accordingly.

My guess would be to remove the cast and let the function work on the MDfragmentVLSB_C type.

#18

Updated by Dobbs, Adam almost 7 years ago

Update: Broken in maus v0.7.6, so now know that the break occurs between 0.7.5 and 0.7.6. Will try David's solution above and see what happens, and also examine changes between those two versions...

#19

Updated by Rajaram, Durga almost 7 years ago

The third party unpacking library changed at 0.7.6
I've added Yordan to the watch list.
Yordan - do you have any idea how/why the unpacker change affects the tracker data?

#20

Updated by Dobbs, Adam almost 7 years ago

Update: have managed to get the json-converter-cpp path working in RealDataDigitisation, but the output is again wrong, with nearly everything being in the upstream tracker and the z positions being clearly out. Tried switching to one of the older calibration files, but no joy either.

Running out of steam on how to solve this... David, I think this needs the eye of someone who knows the DAQ system now, can I transfer ownership of this to you?

Yordan, I'll echo Durga's comments - do you know what might have caused the break between maus 0.7.5 and 0.7.6?

#21

Updated by Overton, Edward over 6 years ago

A quick mind dump.. Yordan found 1. to make the on-mon work in the mock data run.

1. Modify /third_party/build/unpacking-mice/build/unpacking-mice/src/MDfragmentVLSB.cpp so _madeOfParticles = false; => _madeOfParticles = true;

2. Readout now returns: {"daq_event_type":"","errors":{"bad_data_input":"InputCppDAQOfflineData says:ERROR in MDprocessManager::ProcessSubEvent : \nwhile processing particle event in board VLSB Geo 0\nThe status is : 2 Phys. Event \n1 skipped!"},"maus_event_type":"Spill","run_number":0,"spill_number":0}

This is down to a cast error in L487 in src/input/InputCppDaqData/UnpackEventLib.cc
IE: MDfragmentVLSB is not being passed to the VSLBCppDataProcessor.

#22

Updated by Dobbs, Adam over 6 years ago

Hi All,

I have made some progress with this. My latest attempts are on launchpad at:

https://code.launchpad.net/~phuccj/maus/real_data_fix

and may be branched with bazaar. After Yordan's unpacker fix in release 0.9.3 I now see some VLSB data for the single station test data, see VLPC_adc.png for example. I can also now get SciFiDigits to form from the data, however no higher level objects form from the digits (and the output file can lead to ROOT seg faults sometimes it seems). An example of the npe distribution I get from the digits is also attached. I think it needs someone with some DAQ expertises to have a look at the VLSB and digit data and check it is reasonable. Ed, could you take a look please? I also attach a root file with data from the first 3 DAQ events in run 4055.006 (where the pngs come from).

If anyone would like to try using the code, branch the above repository, build, and then run with:

./bin/analyze_data_offline.py --daq_data_path /path/to/data/04055 --daq_data_file 04055.006 --Enable_V1731_Unpacking=0 --Number_of_DAQ_Events=3

The data is available on the grid or the IC web store: http://gfe02.grid.hep.ph.ic.ac.uk:8301/MICE/Step1/04000/

Also, the code I have refactored also a DAQ expert's eye too. Ed, sorry but I think that is you again. Could you take a look at:

http://bazaar.launchpad.net/~phuccj/maus/real_data_fix/view/head:/src/common_cpp/Recon/SciFi/RealDataDigitization.cc

In particular the process_VLSB method, and check the steps look correct please? I am a bit dubious, as the single station data is currently reconstructing digits in several different stations...

#23

Updated by Overton, Edward over 6 years ago

I'm on a train right now, but will have a major look soon.

Some things I notice:
The vlpc ADC distribution looks OK. Mose adcs read pedastool, with a hit in a couple of channels. A log-y plot should show a long tail upwards, which is the real light.

Theres something odd in the didgits plot. We should have an npe (number of photoelectrons) around 10. I can see a bump there, so I guess that's OK. The problem is the npe peak at zero. There should be a cut at 2 npe, which looks like its not getting applied. This could be confusing the software...

#24

Updated by Dobbs, Adam over 6 years ago

Update. Following on from MDR3 we now have some tracker noise data taken using the new Step IV daq. At present we are seeing an issue where the tracker daq data is not appearing in the MAUS data structure at all after running analyze_data_offline.py. Running some of the dedicated unpacker executables though (found in third_party/build/unpacking-mice/bin/) shows reasonable looking tracker data. It therefore looks like there is a fault occurring somewhere between the unpacker and getting daq data into the MAUS datastructure.

I have been moving through the code in InputCppDAQData, InputCppDAQOfflineData, UnpackEventLib etc adding in cerr statements to see what is going on. A sample result is attached. Of particular interest are lines 35 - 40:

INFO: VLSBCppDataProcessor::Process Processing data
ERROR in MDprocessManager::ProcessSubEvent : 
while processing particle event in board VLSB Geo 0
The status is : 2
*** Unpacking exception in void  InputCppDAQData::getCurEvent(MAUS::Data *data) : 
DAQ Event skipped!

This is the only time VLSBCppDataProcessor::Process gets called during the whole run, so it seems significant. Yordan, does this mean anything to you?

#25

Updated by Karadzhov, Yordan over 6 years ago

I found this problem too. It is a bug in one of the versions of the unpacking which must be fixed now. Try running with the current version of

http://dpnc.unige.ch/~yordan/unpacking-mice_2.tarz

#26

Updated by Dobbs, Adam over 6 years ago

Brilliant, thank you Yordan, now seeing unpacked daq data in MAUS datastructure, and scifi digits too!

Next issue: Nothing above digits produced, seeing an error message

TRANSFORM/MERGE/OUTPUT:  Processed 6 events so far, 1 events in buffer.
Traceback (most recent call last):
  File "/home/adobbs/maus/shared-repo/dev/src/common_py/ErrorHandler.py", line 159, in HandleCppException
    raise CppError(error_message)
ErrorHandler.CppError: Failed to find SciFi plane in _geometry_map. at SciFiClusterRec::process_cluster

Chris Heidt, this seems to be in your domain, any thoughts?

Also available in: Atom PDF