Skip to content

adds code to enable profiling of Celery tasks #165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
FredworkLemmas wants to merge 1 commit into
base: master
Choose a base branch
from
Open

adds code to enable profiling of Celery tasks #165

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
adds code to enable profiling of Celery tasks 
* as a workaround for multiprocessing refusal to fork daemon processes,
  incorporates changes to allow importing of Process and Pipe from
  billiard, the Celery project's fork of the multiprocessing module if
  MEMPROF_PREFERS_BILLIARD envvar is set to true
  • Loading branch information
@FredworkLemmas
FredworkLemmas committed Oct 24, 2017
commit b28621acae0972bb6138d1b3334d6dc829863469
37 changes: 32 additions & 5 deletions memory_profiler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,40 @@
import logging
import traceback
from signal import SIGKILL

import importlib
import distutils
from distutils.util import strtobool

# TODO: provide alternative when multiprocessing is not available
try:
from multiprocessing import Process, Pipe
except ImportError:
from multiprocessing.dummy import Process, Pipe

# import Process and Pipe from a multiprocessing library in the following order:
#
# billiard - if and only if MEMPROF_PREFERS_BILLIARD envvar is set and true
# multiprocessing
# multiprocessing.dummy
#
MULTILIBS = (
("billiard", (lambda: True if "MEMPROF_PREFERS_BILLIARD" in os.environ and
strtobool(os.environ["MEMPROF_PREFERS_BILLIARD"])
else False)),
("multiprocessing", None,),
("multiprocessing.dummy", None,)
)
for module_tuple in MULTILIBS:
multi_module_name, use_if_test = module_tuple
try:
multi_module = importlib.import_module(multi_module_name)
except ImportError:
if multi_module_name == "multiprocessing.dummy":
raise
continue
if use_if_test is None or use_if_test():
try:
Process = multi_module.Process
Pipe = multi_module.Pipe
break
except AttributeError:
continue

try:
from IPython.core.magic import Magics, line_cell_magic, magics_class
Expand Down