%matplotlib inline fails when converted to script

This might be known, but didn't see a bug for it. Consider a notebook test.pynb containing the magic %matplotlib inlineand converted to a script test.py:

jupyter nbconvert --to script test.ipynb

test.py now looks like

get_ipython().magic('matplotlib inline')

However, get_ipython() is undefined and the script will be unusable. nbconvert version 5.0.0

I'd be interested in working on resolving this issue, but would need some guidance since this is my first time around this project.

Magic commands don't work outside of IPython. When we convert a notebook to a .py script, we convert the special syntax to regular Python syntax so that it's a valid .py file, but we can't just remove all the magic commands entirely, because that would change the meaning of the code. If that is what you want, it would be possible (if a bit fiddly) to write a custom exporter to strip all magics, but I don't think we'd make that the default.

Otherwise, I think you just need to manually change those bits after converting it.

Yes, converting magic commands to regular python syntax makes sense to me. I think the bug here is that the python that results from that conversion is not usable. A notebook containing %matplotlib inline will not convert to a usable script; the script will raise an error that get_ipython() is not defined.

As you said, a possible fix would be to strip out the magic commands, but this has obvious shortcomings. Instead, perhaps nbconvert could add an import for get_ipython().

I'm not sure that would work… but if you try running it using ipython as your interpreter (`ipython your_file.py` instead of `python your_file.py`) does that work?

…

On Wed, Dec 28, 2016 at 10:56 Jacob Peacock ***@***.***> wrote: Yes, converting magic commands to regular python syntax makes sense to me. I think the bug here is that the python that results from that conversion is not usable. A notebook containing %matplotlib inline will not convert to a usable script; the script will raise an error that get_ipython() is not defined. As you said, a possible fix would be to strip out the magic commands, but this has obvious shortcomings. Instead, perhaps nbconvert could add an import for get_ipython(). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#503 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACXg6GPhZ5MlkaQ3DF6ekSDYYJscAGaIks5rMrDmgaJpZM4LWqCw> .

> ipython3 test.py
# [stacktrace]
# UnknownBackend: No event loop integration for 'inline'. 
# Supported event loops are: qt, qt4, qt5, gtk, gtk2, gtk3, tk, wx, pyglet, glut, osx

This basically hits ipython/ipython#9834. We could work around this by calling for a different backend, as get_ipython().magic('matplotlib <backend>'), but that choice of backend is then user dependent. (Perhaps it could be set to the local matplotlib default?)

I could also see the argument that notebooks are intended for GUI display, whereas one would expect a script to run successfully from a terminal, and thus it would be reasonable for nbconvert to remove commands specific to the GUI as part of the conversion process.

What we could do is just remove the backend, which will set the matplotlib backend to whatever is the default.

Ok, traced this down.

In nbconvert, this is happening in the ipython2python filter, which is calling IPython.core.inputsplitter.IPythonInputSplitter, which is parsing the ipython logic either through escaped_commands() or assign_from_magic() (not sure which, because those the escape chars are included as noted in a comment for backward compatibility, they're also included first in the list which would mean that they would be hit first if it iterates through the transformers).

But the key is, there's no way to use the current transformation machinery to accomplish this on its own, so we're either going to need to preprocess or postprocess special cases inside ipython2python.

#507 should do this in as efficient a manner as possible, as I was worried about what doing extra processing on every code cell would do to the performance of nbconvert overall on python script exports for large notebooks.

Filtering should only triggered if requested (only the python exporter currently does this, using the no_gui argument to ipython2python) and then defaults to use a precompiled regex (only overwritten if additional backends are requested to be removed).

Any solution so far
get_ipython().magic('matplotlib inline') is not converted to .py

I write 'exec(%matplotlib inline)' instead of get_ipython().magic('matplotlib inline')

I don't get any error, but there is no output I mean there is no output images.

Anyone can fix it please ???

Replacing %matplotlib inline in the notebook by 'exec(%matplotlib inline)' works for me, fixing the error and updating outputs as expected. My nbconvert command is:

jupyter nbconvert --template='script_template.tpl' --to python 'notebook.ipynb'

@hadrien-toma

Hi @hadrien-toma
I can run properly .ipynb file from jupyter notebook there's no problem
but I want to convert it to .py format, I did the following:-

I used this jupyter nbconvert --to python object_detection_tutorial.ipynb to convert from .ipynb file
to .py file
then I opened the new converted .py file
I removed this get_ipython().magic('matplotlib inline') and I added this 'exec(%matplotlib inline)'
I ran the script by python object_detection_tutorial.py
there is no error in a terminal, everything seems fine, but there is no output bounding images, there is no output at all. !!!!!!!!!!!!! any solution ?

@Abduoit

@Abduoit did you try running ipython object_detection_tutorial.py?

@tacaswell

Also @tacaswell and I discussed this at SciPy. There are unfortunately at least three ways to interpret (4 if you include a somewhat more esoteric use case that involves matplotlib as a means of determining control signals in a human-in-the-loop computational decision process).

What is the actual thing that you want to occur when you want to convert this to a script? Like what is the ideal way this would work out for you?

I also tried ipython I got same results, no change.

If you just want the images as png, you could try using the AGG backend with matplotlib.use("AGG") (I'm just guessing since that does png output… I don't know if it will actually work for your use case).

And I'm glad to be able to help. It's just sometimes the most help I can give is to point you to the people who know how to teach you about the details that you need to grok to solve your problem.

Could you come back here once you find a solution that works (and the reason you found that solution to work)?

Yeah sure, I ll give it try, then I ll be back here,
thx

@mpacer

If you want them to be displayed in the context of running a script, you'll then need to use an interactive backend with some GUI toolkit for displaying the images.
Or, at least that's how I understand it.

Yes, this is correct. Once you involve a GUI backend you now have an event loop (which is to first order a while True: loop) that needs to be run for the figure to be interactive. If you expect an interactive window, then you have to decide if you

want it to block the execution of the script for each figure you create, requiring it to be closed before the script continues.
block once at the end and you can see all of the figures created

If you do not want to block for GUI windows than you have to

save each figure to a file automatically, which requires communicating to the user what they are named
assume the script is saving figures and just run-through

(These may or may not be the same 4 options I discussed with @mpacer at scipy....)

@mpacer

@mpacer @Abduoit hi，could you pls tell me what's your results?

@li10141110

@li10141110 there's no result, the process finish and there is no output

@Abduoit

@Abduoit hai, I encounted the same problems recently，is there any new progress in your previous question？ I could not get the output image too .Thank you.

@Abduoit

@Abduoit @smalllittlewhite Hi, any progress on this? I am also struggling to get an output image. I changed 'matplotlib inline' to 'matplotlib auto' in the code. When running it via ipython, i don't get an error o anything but the images do not show. Eventually i would like to edit the code and then place the 'result' images into a new folder showing what its detected.

Here's a way to handle the magics, assuming your converter is turning magic commands into get_ipython().run_line_magic commands (which regular Python cannot execute) rather than scrubbing them entirely. Parse the exported Python file into a list of lines with e.g.:

with open('/path/to/exported/python/file') as f:
    code = f.read()
code = code.split('\n')

and then apply this to each line in code using:

def strip_line_magic(line,magics_allowed):
        matches = re.findall("run_line_magic\(([^]]+)", line)
        if matches and matches[0]: # This line contains the pattern
            match = matches[0]
            if match[-1] == ')':
                match = match[:-1] # Just because the re way is hard
            magic_kind,stripped = eval(match)
            if magic_kind not in magics_allowed:
                stripped = "" # If the part after the magic won't work, just get rid of it
        else:
            printd("No line magic pattern match in '%s'" % line)
            stripped = line
        return stripped

For example, if we make a list of magics that are likely to precede an otherwise-functioning line of code (e.g. %time, %timeit, etc.) and exclude from that list things that aren't expected to precede valid code (e.g. %matplotlib), then we can scrub the magics that won't work in Python and keep the ones where the magic is just incidental.

For example:

magics_allowed = ['time']
example1 = "get_ipython().run_line_magic('matplotlib', 'inline')" # 'inline' is not valid code without magic
example2 = "get_ipython().run_line_magic('time', 'x = np.random.rand(10000)'" # 'x = ...' is valid code without magic
strip_line_magic(example1,allowed) # Returns '"" since this line must be purged
strip_line_magic(example1,allowed) # Returns "x = np.random.rand(10000)", the code we want to run

slightly off topic - but hypothetically - using iterm - someone could rig up the terminal to actually show image at the command line. Here's the heavy lifting already done - just need to call out from python to command line https://github.com/johndpope/terminal-image-cli
Examples
$ image unicorn.jpg
$ cat unicorn.jpg | image

@Abduoit

@Abduoit did you figure this out?

I was having this issue in PyCharm. The fix for me was I didn't have the matplotlib package installed. Not sure if this applies to anyone else but figured I'd mention it in case it helps.

You might as well spawn InteractiveShell class to wrap ipython 'magic' calls.
I have done it by adding cell to notebook which will run in case of execution outside of ipython:

if '_' not in globals():
    print("Running outside of ipython")
    from IPython import InteractiveShell as get_ipython

not sure about matplotlib but it works like a charm for command line calls (actually executes them).

@Skitionek

I would have proposed added an "ipython" exporter that doesn't do anything with magic commands and creates a .ipy file, and then the python exporter could just rip out the magic commands. However, @Skitionek's workaround looks even better to me though I haven't tried it. This might work best if we can detect if magics are used, and then add the code block to the start if needed.

ipython worked for me. Thanks

Navigation Menu

Repository navigation

`%matplotlib inline` fails when converted to script #503

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

%matplotlib inline fails when converted to script #503

Description

Activity

jpeacock29 commented on Dec 28, 2016

takluyver commented on Dec 28, 2016

jpeacock29 commented on Dec 28, 2016

mpacer commented on Dec 28, 2016

jpeacock29 commented on Dec 28, 2016

mpacer commented on Jan 4, 2017

mpacer commented on Jan 5, 2017

mpacer commented on Jan 5, 2017

Abduoit commented on Aug 11, 2017

Abduoit commented on Aug 13, 2017

hadrien-toma commented on Aug 15, 2017

Abduoit commented on Aug 16, 2017

mpacer commented on Aug 16, 2017

mpacer commented on Aug 16, 2017

Abduoit commented on Aug 16, 2017

12 remaining items

mpacer commented on Aug 16, 2017

Abduoit commented on Aug 16, 2017

tacaswell commented on Aug 19, 2017

li10141110 commented on Sep 18, 2017

Abduoit commented on Sep 18, 2017

FfyFY commented on Jan 18, 2018

mzkasu commented on Feb 11, 2018

rgerkin commented on Mar 1, 2018

johndpope commented on May 9, 2018

shamik111691 commented on Mar 21, 2019

elsahill commented on Jul 15, 2019

Skitionek commented on Mar 17, 2020

t-makaro commented on Aug 14, 2020

Nafees-060 commented on Feb 13, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`%matplotlib inline` fails when converted to script #503