Bug #1281
celery runs very slowly on my laptop
100%
Description
Running tests/integration/test_distributed_processing/test_multithreaded_vs_singlethreaded
fails on my laptop because celery/RabbitMQ is running very slowly. There seems to be some hang up when the celery head node communicates with subprocesses that causes significant slowdown. In particular, I've found that the birth cycle gets hung up in communication between mauscelery.mausworker.sub_process_broadcast
and mauscelery.mausprocess.process_birth
but it looks like individual spill processing is also proceeding slowly.
I checked for this in control room branch in MLCR and things look okay - so it could be a feature of my laptop or the release branch.
Updated by Rogers, Chris about 10 years ago
I checked this in MLCR release branch (0.5.1) and also see the same problem. So it is a difference between release build and control room build. Either
- the code is different
- the third party libraries versions have changed causing a problem
Updated by Rogers, Chris about 10 years ago
Reading a diff of the code, it looks like there are no significant changes in the source code that could cause a massive slowdown.
Library versions:-
Control room branch
- celery-2.5.5
- pymongo 2.3
- kombu 2.1.8
Release branch
- celery 3.0.15
- pymongo 2.4.2
- kombu 2.5.6
- amqp 1.0.8
- billiard 2.7.3.21
Updated by Rogers, Chris about 10 years ago
Reverting the libraries to the old version appears to have fixed things. How irritating.
Updated by Rogers, Chris about 10 years ago
- Status changed from Open to Closed
- % Done changed from 0 to 100
Celery was reverted to 2.5.5 in r947
Updated by Rogers, Chris about 10 years ago
- Target version changed from Future MAUS release to MAUS-v0.5.4