Bug #1989


Added by Nugent, John over 4 years ago. Updated over 4 years ago.

Target version:
Start date:
09 October 2019
Due date:
% Done:


Estimated time:
New Issue



after requesting new MC for the scattering paper I pulled the data processed with a matching version of MAUS (v3.3.2). When I run the analysis with this data I encounter problems. For the analysis I select events with

  • 1 sp in TOF0
  • 1 sp in TOF1

Setting aside the rest of the selection for the time being, there are 0 events which fulfil the above criteria. This is not consistent with any of the previous processings. If I loosen this constraint to

  • >0 sp in TOF0
  • 1 sp in TOF1

the analysis runs as normal. (I have run over a subsample of all events, I cannot say that things are identical but most of the plots look fairly similar) The scattering distributions are filled and the rest of the plots for the analysis are made.

Prior to this point to the best of my knowledge I do not do anything which would bias the data. It is not conditional on the other selection criteria, each cut it applied independently to all events and a tag for each event set. I can also produce the TOF distibutions before the TOF cut is applied and they do not look unusual.

I have also run over the data with in $MAUS_ROOT_DIR/bin/examples and I see the same behaviour. This is a very simple script, is entirely indepedent of my code and works with the freshly processed data before I start tinkering with them. This makes me think that it is nothing that I do when selecting or reducing the files but really is something to do with the processing.

So essentially in order to run the analysis I have to allow events which have 2 sp in TOF0 to be included in the selection. This does not seem correct to me and I have never had this issue with any of the data before. Paolo you mentioned that this version of the data was run with the TOF-tracker fitter, has anything else changed? Are either of you aware of any other analysis which has encountered similar problems?

I am teaching in the lab all afternoon however when I return to this I will pull a previous version of the data (v3.1.2) and compare the raw files to those I am now currently using. If I discover anything I will let you but if you spot any differences in the processing please let me know.



07666_3.3.2.tar.gz (100 KB) 07666_3.3.2.tar.gz Franchini, Paolo, 22 October 2019 14:17
07666_3.1.2.tar.gz (100 KB) 07666_3.1.2.tar.gz Franchini, Paolo, 22 October 2019 14:17
3.2.2.png (55.9 KB) 3.2.2.png Nugent, John , 10 January 2020 15:37
3.1.2.png (103 KB) 3.1.2.png Nugent, John , 10 January 2020 15:37
312y.png (49.9 KB) 312y.png Nugent, John , 10 January 2020 16:58
332y.png (53.8 KB) 332y.png Nugent, John , 10 January 2020 16:58
MCSNote.pdf (2.04 MB) MCSNote.pdf Nugent, John , 13 January 2020 11:24
Runs_JN.txt (571 Bytes) Runs_JN.txt Runs to be reprocessed Franchini, Paolo, 13 January 2020 13:41

Updated by Franchini, Paolo over 4 years ago

Hi John,

I do not see the MapCppTrackerTOFCombinedFit doing anything nasty like adding TOF space points and nothing has really changed in the TOFs reconstruction.
I had run some code asking for just one space point and was not struggling to do that.


Updated by Franchini, Paolo over 4 years ago

I have downloaded the DATA for 07666 ( and can see that most of the events have

1 sp in TOF0
1 sp in TOF1

using GetTOF0SpacePointArraySize() and GetTOF1SpacePointArraySize() methods.
Rarely more than 1. Have you been checking all your datasets?


Updated by Nugent, John over 4 years ago

So I have done some more digging. If I run over 7666 I do see TOF sp. So I can reproduce what you are seeing Paolo. However the issue is if you make the TOF01 plot you see that only events below 26.2 ns are populated - well outside the time window for the scattering analysis, this is the electron peak. Now if I use the script exactly as it is, here there is no TOF selection unlike my analysis, the only modification that I make is on line 130 where I change between

if tof0_sp_size < 1 or tof1_sp_size != 1 :
return True


if tof0_sp_size != 1 or tof1_sp_size != 1 :
return True

(I am ignoring TOF2 for just now).

In the second case only the electron peak is populated in the TOF01 plot. In the first case the muon peak is also populated. All of the events which line under the muon peak have two TOF0 sp, I suspect that there is something going on here.

I have now run over all of the LiH 200 MeV/c data (the complete list of runs is shown in table 2a of the Note: and I see the same thing. This also agrees with what I get from my own analysis code which makes all of the scattering analysis plots.


Updated by Nugent, John over 4 years ago

I have pulled 07666 from 3.1.2 and am comparing to 3.3.2. At first looks in 3.1.2 there are 96k TOF0 events and in 3.3.2 there are 136k events. They have different distributions in slabX/Y hits and in _time. Their distributions in dt are very different. If it was just one or two events different then I would be inclined to think that the processing was more or less the same but 40k events is a reasonable number. Their overall array size is the same in both versions, also if I look at @size the number of events with 1 TOF0 sp appears to be different across the two versions.

If I look at _global_x there are a large number of events at -100 & -150 in 3.3.2 that do not exist in 3.1.2. Possibly this is something to do with a specific bar? A change of calibration for an edge bar or something?


Updated by Franchini, Paolo over 4 years ago

Thanks for the digging you have been doing. I was trying to re-reconstruct without the TOF/Tracker re-seeder but I am quite sceptical it's the culprit.

Indeed the TOF reconstruction code has been touched after release 3.1.2. I have added Viktor on the watchers list in case he can shed a light.


Updated by Rogers, Chris over 4 years ago

I'm being thick as usual, but I didn't quite follow.

  • Is the bug in MC or data?
  • Is the bug that the number of TOF0 space points has changed or is the bug that the TOF0 time has changed? (or something else)



Updated by Nugent, John over 4 years ago

Hi Chris,

the issue is in data and the issue is that all of the events that appear in the TOF plots from the script have 2 TOF0 spacepoints. So it would appear that there is something amiss with the number of spacepoints in TOF0. I have followed this further back up the chain and the issue does not appear if I plot the TOF plots in before the trackers cuts are implemented. Many events fail the tracker cuts for some reason and those which are left have 2 TOF0 spacepoints.

I have not had time to study what is going on in this script any further, though it does not appear too complicated. It counts the number of track points and discards tracks with less than a certain number. I will be at CERN all week for a T2K meeting so I will have another look next week. If I can figure out why all of the tracks are failing the cut I may be able to locate the source of the error (which may have nothing to do with the TOFs).



Updated by Rogers, Chris over 4 years ago

Thanks for the clarification. Could you remind me what are the tracker cuts?


Updated by Nugent, John over 4 years ago

It counts the number of track points and discards tracks with less than a certain number.


Updated by Franchini, Paolo over 4 years ago

Hi John,

I am a bit confused at this point because cannot reproduce your results. Have done some plotting with 07666 reconstructed with MAUS-v3.3.2. Please find attached some SP/Tracks plots. I have used some other code.
In the 2D plots you can see that the massive muon peak (~28 ns). TOF0 has most of the time 1 SP and TKU 1 track.
I am not doing any cut apart from selecting TOF0 and TOF1 events with more than 0 space points to compute the t.o.f.

The TOF01 distribution is slightly different from what I get from 3.1.2 but in terms of SP and tracks cannot see any relevant difference.


Updated by Franchini, Paolo over 4 years ago

Nugent, John wrote:

It counts the number of track points and discards tracks with less than a certain number.

If you get the size of scifitrackpoints() is either 15 or 0 I assume by definition. In few cases is 30 when the there are 2 reconstructed tracks.


Updated by Nugent, John over 4 years ago


so I have had some time over the last few days to do a bit more digging. I believe I have at least made some progress in partially identifying the problem. I can reproduce the plots that Paolo has posted so on the surface there are TOF sp and there are trkr tracks and I can populate a sensible TOF plot. The issue appears to be that while the selection cuts that I listed above are true there is also an intrisic cut that I apply to the data that I previously did not consider.

Once I have got a TOF1 sp I ask for an US track but in additional I also ask for

  • the z co-ordinate of the trackpoints for that track

as I want the trackpoint closest to the absorber to do the scattering measurement. Given that there is a track and a trackpoint I naturally assumed that this would not be an issue. It is. Tracks that I look at have their z co-ordinates as NAN and due to this implicit "selection cut" the rest of my analysis fails (if the analysis cannot get the co-ordinates of the US track it cannot propagate it DS and add the track to the final selection). I have posted two images for v3.2.2 and v3.1.2 data. They show the co-ordinates of all trackpoints in the data. You will notice that there is a significant different between both versions of the data (not the number of entries but the y-axis values, presumably the NAN values are not plotted).

So while the first thing that I noticed was the TOF plots being different and that is what the earlier posts in this thread were about actually I do not think this has anything to do with the problem, it was merely a simptom. Paolo I think that the reason that we redid the data processing was so that the track fitting method could be changed, could this contribute to the behaviour that I am seeing? i.e. some change in the tracker code from 3.1.2 to 3.3.2?

I have rerun my analysis with v3.1.2 of the data and all of plots are populated as expected, the code is identical to that that I run over v3.3.2 of the data so there really is something different in this version of the data. If I cannot find a way to resolve this issue over the next few working days then I am going to roll back to the last working version of the data and produce the Note with that. If either of you have any suggestions as to what has changed I would be very grateful to hear you both are probably more familiar with the tracker code than I am.



Updated by Franchini, Paolo over 4 years ago

Hi John,

thanks for looking at this.

Which method do you use to get the z coordinate? I agree that something has changed and will investigate on that, but in principle do you use the z coordinate just to understand if you are on the closest station to the absorber (Station 1)? Could you use something like spacepoints[k]->get_station() instead?


Updated by Nugent, John over 4 years ago

Hi Paolo,

to get the co-ordinate I use


Yes you are right I could use another method but the idea in the analysis was to be completely agnostic as to whether you are in the US/DS trkr so I cycle through each trackpoint by position until I get the one who's co-ordinates are closest to the absorber. Also if I look at the x and y co-ordinates for v3.3.2 vs v3.1.2 again I see a significant difference. This is a problem as this the information that will be used to propagate the track downstream and check if it is contained within the tracker it is also needed to make the plots for the Note showing the distributions in the tracker. Even if the x and y were the same I am just hesitant to use this data if something like the z co-ordinate is so strange.

Thank you for having a look Paolo anything you find would be much appreciated.



Updated by Franchini, Paolo over 4 years ago

Ok, so we agree that there are less reconstructed trackpoints and those you are interested in do not have position information (-nan).

Can you tell which are the cuts that constrain your sample to those tracks?


Updated by Nugent, John over 4 years ago

Franchini, Paolo wrote:

Ok, so we agree that there are less reconstructed trackpoints and those you are interested in do not have position information (-nan).

Agreed that is what I am seeing.

For the 4 plots that I have attached above I require a TOF1 sp as this is the starting point for my analysis but that is the only cut everything else comes later. I am working with the data before selection to try and make things as simple as possible.


Updated by Franchini, Paolo over 4 years ago

Please have a go with this reconstructed single run

which has the TOF/Tracker refitter been disabled on MAUS-v3.3.2. I do not see any NAN for the track's positions so this should be the culprit.
I am doing few more tests, but I will mostly likely rerun the reconstruction for your straight track runs keeping it disabled.


Updated by Nugent, John over 4 years ago

That works! Thanks for doing that Paolo.

I have had a scan through the plots obviously the statistics are limited but most plots look more or less as I would expect. So in terms of how to move forward Paolo could you please reprocess all of the scattering data with v3.3.2 without the TOF-tracker refitter? I think that this will be fastest way to progress unless you or Chris have a clear idea of why the refitter is breaking in this way. Whilst the data is being reprocessed I will have a look at the refitter code (could you please point me to where it is?) however given my lack of familiarity I doubt that I will come across anything quickly.

Unless there is a strong feeling that the refitter significantly improves the performance of the track reconstruction then I will proceed without it for the time being.

Thanks again for your efforts Paolo.



Updated by Franchini, Paolo over 4 years ago

Great, will sort out a quick reconstruction.

The refitter is in /src/map/MapCppTrackerTOFCombinedFit and shows a lot of these errors

Error in <TDecompLU::DecomposeLUCrout>: matrix is singular
Error in <TDecompLU::InvertLU>: matrix is singular, 0 diag elements < tolerance of 2.2251e-308

Haven't had much time to investigate, if you can have a look can be quite interesting.
The refitter was never really meant to work on straight tracks, so have overlooked that during the last reconstruction campaign.

Could you please attach here a list of the run numbers you need? cannot find your last note.


Updated by Nugent, John over 4 years ago

Great thanks I have attached the Note you can find the complete list of runs used in the analysis in table 2A.

I'm getting the feeling that it might be more hassle than it is worth but I will devote a couple of hours to looking through the code today. If I cannot resolve the issue on that time scale then I will leave it for the time being, I am quite keen to simple get to end of this analysis.



Updated by Nugent, John over 4 years ago

I am also reviewing the MC that I have for this analysis. If I remember correctly the MC productions 203-208 were done with out the refitter and then 230-235 were done with the fitter? 203-208 was stopped after we discovered this discrepancy at the Warwick workshop. If I have that the correct way round could you please restart the 203-208 jobs until the statistics match the previous production? They are mostly done but if this is the sample that is going to be used then it wouldn't hurt to top them up.



Updated by Franchini, Paolo over 4 years ago


I am now doing a dirty patch and hope to have some jobs on the Grid by the end of the day.
If you do not have any particular concern about the previous straight track reconstruction is probably the correct solution.

Will take care of the MC too, need to remember myself what was done back than but you should be correct (btw, if you have time try to process a small sample of the no refitter MC to see if there is any "NAN problem").


Updated by Nugent, John over 4 years ago

OK so I can't run any jobs just now due some technical trouble on the Glasgow linux cluster at the moment which has held me up since this morning. It may come back to life in the next few hours, if it does I will run a test job with the MC I still have all of the 203-208 production data on disk here. From memory the MC without the refitter looked good and I didn't see any NAN issues but I will double check later this afternoon if the cluster comes back to life.



Updated by Rogers, Chris over 4 years ago

Well done for figuring out the problem. Is it possible to run an analysis either on a few locally reconstructed files or on a test/small version of the production recon. I think that will help the process to converge.


Updated by Nugent, John over 4 years ago

Hi Chris,

Paolo sent me a test case yesterday which I ran the analysis over successfully. This was the reconstructed data. I have also independently tested the MC which never suffered from this problem so with both of those sorted I am just waiting for the full data reprocessing. I believe that Paolo submitted this list of jobs yesterday so it shouldn't be too much longer. Hopefully everything in the full set also looks sensible, I'll keep you posted. We may at last be closing in the end of this particular bug.



Updated by Rogers, Chris over 4 years ago

Great! Fingers crossed


Updated by Franchini, Paolo over 4 years ago

After some fiddling Dimitrije had to do modifing his script to use Singularity dockers for SL6, runs are appearing here

Considering a rough estimate they might be all done by Monday.


Updated by Nugent, John over 4 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF