Project

General

Profile

Bug #1483

Celery fails to import MapCpp modules

Added by Rogers, Chris over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Online reconstruction
Target version:
Start date:
05 June 2014
Due date:
% Done:

100%

Estimated time:
Workflow:
New Issue

Description

Presumably a consequence of #1376 changes


Related issues

Related to MAUS - Feature #1376: JSON Conversion OverheadClosedRogers, Chris20 November 2013

Actions
#1

Updated by Rogers, Chris over 7 years ago

Configuring Celery nodes and birthing transforms...
[{u'ctr': {u'status': u'error', u'error': [{u'message': u'No such transform: MapCppTOFDigits', u'error': u"<type 'exceptions.ValueError'>"}], u'run_headers': u''}}]
Clearing Globals
Traceback (most recent call last):
  File "/home/cr67/MAUS/maus_merge/bin/online/analyze_data_online_input_transform.py", line 67, in <module>
    run()
  File "/home/cr67/MAUS/maus_merge/bin/online/analyze_data_online_input_transform.py", line 64, in run
    MAUS.Go(my_input, my_map, reducer, output_worker, data_cards) 
  File "/home/cr67/MAUS/maus_merge/src/common_py/Go.py", line 131, in __init__
    self.get_job_footer())
  File "/home/cr67/MAUS/maus_merge/src/common_py/framework/input_transform.py", line 284, in execute
    self.process_event(spill)
  File "/home/cr67/MAUS/maus_merge/src/common_py/framework/input_transform.py", line 314, in process_event
    self.start_of_run(current_run_number)
  File "/home/cr67/MAUS/maus_merge/src/common_py/framework/input_transform.py", line 219, in start_of_run
    self.config_doc, self.config_id, self.run_number)
  File "/home/cr67/MAUS/maus_merge/src/common_py/framework/utilities.py", line 267, in birth_celery
    CeleryUtilities.validate_celery_response(response)
  File "/home/cr67/MAUS/maus_merge/src/common_py/framework/utilities.py", line 315, in validate_celery_response
    raise CeleryNodeException(failed_nodes)
framework.utilities.CeleryNodeException: Celery node(s) failed to configure: [(u'ctr', {u'status': u'error', u'error': [{u'message': u'No such transform: MapCppTOFDigits', u'error': u"<type 'exceptions.ValueError'>"}], u'run_headers': u''})]
#2

Updated by Rogers, Chris over 7 years ago

Reduces to:

cr67@ctr ~/MAUS/maus_merge $ python
Python 2.7.2 (default, Apr 23 2014, 17:30:14) 
[GCC 4.6.1] on linux3
Type "help", "copyright", "credits" or "license" for more information.
>>> __import__("MapPyDoNothing")
<module 'MapPyDoNothing' from '/home/cr67/MAUS/maus_merge/build/MapPyDoNothing.pyc'>
>>> __import__("MapCppSimulation")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named MapCppSimulation
>>> 
#3

Updated by Rogers, Chris over 7 years ago

  • Category changed from Data Structure to Online reconstruction
#4

Updated by Rogers, Chris over 7 years ago

So in the new setup, modules are put in e.g. _MapCppSimulation.so which has a module called _MapCppSimulation. This is imported into MAUS module. So the better way to do this code is to have getattr(MAUS, 'MapCppSimulation'). This means that we don't need to use the deprecated __import__ function.

I also modified the birth and death functions so they pass on exceptions more cleanly and forgive us if we return None.

I pushed to lp:~chris-rogers/maus/1483, let's run a test job and see what happens (though I note heplnm071 is playing up on jenkins).

#5

Updated by Rogers, Chris over 7 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Ack, stupid me, I got a circular dependency when I import MAUS. I think I tried to do this one time before and got the same problem... So back to the old way and __import__. I do something like (pseudocode)

try 
  import Module
except
  try
    import _Module
  except
    raise

Pushed to trunk and in production...

Also available in: Atom PDF