Project

General

Profile

Bug #1281

celery runs very slowly on my laptop

Added by Rogers, Chris over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Online reconstruction
Target version:
Start date:
29 May 2013
Due date:
% Done:

100%

Estimated time:
Workflow:
New Issue

Description

Running tests/integration/test_distributed_processing/test_multithreaded_vs_singlethreaded fails on my laptop because celery/RabbitMQ is running very slowly. There seems to be some hang up when the celery head node communicates with subprocesses that causes significant slowdown. In particular, I've found that the birth cycle gets hung up in communication between mauscelery.mausworker.sub_process_broadcast and mauscelery.mausprocess.process_birth but it looks like individual spill processing is also proceeding slowly.

I checked for this in control room branch in MLCR and things look okay - so it could be a feature of my laptop or the release branch.

#1

Updated by Rogers, Chris over 8 years ago

I checked this in MLCR release branch (0.5.1) and also see the same problem. So it is a difference between release build and control room build. Either

  1. the code is different
  2. the third party libraries versions have changed causing a problem
#2

Updated by Rogers, Chris over 8 years ago

Reading a diff of the code, it looks like there are no significant changes in the source code that could cause a massive slowdown.

Library versions:-

Control room branch

  • celery-2.5.5
  • pymongo 2.3
  • kombu 2.1.8

Release branch

  • celery 3.0.15
  • pymongo 2.4.2
  • kombu 2.5.6
  • amqp 1.0.8
  • billiard 2.7.3.21
#3

Updated by Rogers, Chris over 8 years ago

Reverting the libraries to the old version appears to have fixed things. How irritating.

#4

Updated by Rogers, Chris over 8 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Celery was reverted to 2.5.5 in r947

#5

Updated by Rogers, Chris over 8 years ago

  • Target version changed from Future MAUS release to MAUS-v0.5.4

Also available in: Atom PDF