Global variables undefined in interactive use of embedded ipython shell

Original Launchpad bug 399627: https://bugs.launchpad.net/ipython/+bug/399627
Reported by: h-fangohr (Hans Fangohr).

The error can be reproduced as follows:

Start Python, and start embedded ipython session. Following ipython's manual, we do

fangohr@eta:~$ python
Python 2.4.3 (#1, Jun  8 2009, 14:09:06) 
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython.Shell import IPShellEmbed
>>> ipshell=IPShellEmbed()
>>> ipshell()

Within the just started ipython session, global variables are sometimes not visible. Two examples are:

Example 1:

In [1]: a=1

In [2]: def f(x):
   ...:     return a*x
   ...: 

In [3]: f(2)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/home/fangohr/<ipython console> 

/home/fangohr/<ipython console> in f(x)

NameError: global name 'a' is not defined

Example 2:

In [4]: b=1

In [5]: (lambda :b)()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

/home/fangohr/<ipython console> 

/home/fangohr/<ipython console> in <lambda>()

NameError: global name 'b' is not defined

There is no error if "b=1; (lambda :b)()" is put in a script, and this script is executed using ipython's "run" command.

There is no error if "b=1; (lambda :b)()" is executed interactively after having started ipython.

The only way the error seems comes up is if an embedded IPython shell is started from a Python prompt AND the commands are executed interactively at the prompt.

I find the same error when trying ipython-0.9.1 (with python2.4).

The bug was reported by Olivier Klein to the nmag team (http://nmag.soton.ac.uk).

Over at launchpad...

chairmanK wrote 20 hours ago:

Me Too. In addition to functions, this bug also appears in generator expressions.

The equivalent component in trunk seems to be IPython.frontend.terminal.InteractiveShellEmbed. But that's broken in other ways, and evidently hasn't been tested much. Does anyone know what its future is?

Could this be the same issue as #136?

This has been now fixed:

amirbar[ipython]> python
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from IPython import embed
>>> embed()
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: a = 1

In [2]: def f(x):
   ...:     return a*x
   ...: 

In [3]: f(3)
Out[3]: 3

In [4]: b = 1

In [5]: (lambda : b)()
Out[5]: 1

In [6]:

Can someone explain a little more what went into the fix of this? I still encounter the issue, but only when I'm one layer removed via a method call, and only if I run from a script, not interactively.

python 2.7.2 on OSX 10.6.8, ipython 0.11 and 0.12 both exhibit similar behavior (using 0.12 for this comment)

This is a problem with our project which features (prominently) an embedded IPython shell.

testembed.py

from IPython import embed

def hi():
    embed()

if __name__ == '__main__':
    #embed()
    hi()

Run this on the command line with python testembed.py and see this session:

Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import time

In [2]: def tim():
   ...:     print time.time()
   ...:     

In [3]: tim()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
[snip] in <module>()
----> 1 tim()

[snip] in tim()
      1 def tim():
----> 2     print time.time()
      3 

NameError: global name 'time' is not defined

In [4]:

However, comment out the call to hi() and replace it with the embed() call:

Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import time

In [2]: def tim():
    print time.time()
   ...:     

In [3]: tim()
1328639444.29

In [4]:

After poking around, I think it has to do with the stack_depth parameter used here and this block: https://github.com/ipython/ipython/blob/master/IPython/frontend/terminal/embed.py#L185

Thoughts?

This is somewhat complicated, but I believe that's a restriction in Python itself.

In the failing case you show, time is not actually put into the global namespace: because you called embed inside the hi function, new variables you create interactively are local to that function. Ideally, tim() should work as a closure, closing over the reference to the time module. However, closures only appear to work when the containing function is compiled in one go. As far as I can tell, there is no way to define a closure dynamically. This simple example fails:

def outer():
    import time
    exec("def inner(): return time.time()")
    return inner

outer()()

This is probably because nested scopes were added to Python relatively late (they were a future import in 2.1, and always on in 2.2).

Ok, I think I understand. Looks like we can't really do anything about this then, other than dumping what I would write interactively to a file and then reading that file back in. Probably too complex for what we want to be able to do interactively.

Thanks, btw.

Sorry to keep harping on here, but it just makes interactive sessions feel very clunky:

Regular python:

>>> d={'one':1, 'two':2}
>>> getkeys=lambda: d.keys()
>>> getkeys()
['two', 'one']

Regular IPython:

In [1]: d={'one':1, 'two':2}

In [2]: getkeys=lambda: d.keys()

In [3]: getkeys()
Out[3]: ['two', 'one']

Embedded IPython:

>>> from IPython import embed
>>> embed()
Python 2.7.2 (default, Aug 29 2011, 12:33:18) 
Type "copyright", "credits" or "license" for more information.

IPython 0.12.dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: d={'one':1, 'two':2}

In [2]: getkeys=lambda: d.keys()

In [3]: getkeys()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <module>()
----> 1 getkeys()

/Users/asadeveloper/Documents/Dev/code/pyon/bin/python in <lambda>()
----> 1 getkeys=lambda: d.keys()

NameError: global name 'd' is not defined

I guess there isn't much that can be done, but I don't get why we can create a dynamic closure in regular python/ipython but not the embedded version.

By the oddities of Python scoping, in standard Python/IPython, we're not actually creating a closure. d is a global variable, and the rules for accessing those work differently from closures. Each function keeps a reference to the global namespace where it was defined (getkeys.func_globals), so it can access any variables defined there.

In contrast, when you make a closure, Python attaches references to each variable over which it has closed, as determined by the compiler - but that only works when the inner and outer functions are compiled at the same time. It looks like this:

In [8]: def outer():
    a = 1
    def inner():
        return a
    return inner
   ...: 

In [9]: f = outer()

In [10]: f
Out[10]: <function __main__.inner>

In [11]: f.func_closure
Out[11]: (<cell at 0x9f5e344: int object at 0x9a830b0>,)

This is possibly done to save memory - if the closure carried a reference to the local scope where it was defined, none of the variables from that scope could be freed while the closure was live.

I'm using Python 2.7.2 and IPython 0.12.1 on OSX 10.7.3 and having this issue still. When I run ./manage.py shell of Django which invokes IPython.embed(), the issue occurs. However, manually invoking the embed() in Python shell or from simple script file, there is no problem.

There's not a lot we can directly do about that, but we should probably encourage third parties away from embed for non-trivial uses.

@takluyver

@takluyver Do you mean that better to use ipython directly in this case?

It's possible for Django to start IPython in a way that won't cause this problem, but that's not the way we currently make easy. The problem occurs when IPython starts with separate local and global namespaces. There's no reason that Django requires separate local and global namespaces, but that's what calling embed() inside a function implies.

For reference, here's the code in Django:
https://code.djangoproject.com/browser/django/trunk/django/core/management/commands/shell.py

debugshell: make globals() match locals() in interactive shell

I have a workaround posted at https://gist.github.com/limwz01/7e06cacf31d4ed83aa74134971030967 which is not as limited as the current approaches, but it has several caveats.

Allowing closures over local variables is done by using a transformer added
to shell.ast_transformers that creates an outer function scope wrapping the
cell code with parameters from local variables and which finally updates the
locals() and returns the last expression. For illustration, we have the
following generated when the code has statement0; statement1; last_expression and the local variables are x, y, z:

def _ipy_magic(_ipy_magic, x, y, z):
    statement0
    statement1
    return (last_expression, _ipy_magic.update(locals()))[0]
_ipy_magic(locals(), x, y, z)

exceptions raised in the shell further trigger a weird exception
executing.executing.NotOneValueFound during traceback exception formatting
and this disables pretty tracebacks
direct changes to local variables are only visible to code within the
shell session
outer scope variables must already have had a closure referencing it to be
visible (nonlocal does not work) and they become local variables (and
hence above restriction applies)
local variables named _ipy_magic will be overwritten
return statements without an enclosing function does not give an error and
results in unpredictable behaviour

What this means is that you can just paste in code as if you are really inside the function you had embedded in, and it should work most of the time. This is useful to test small snippets of code as it is or in the event a program throws an exception, check if a modified piece of code works without re-running the whole program.

Borrowing some of the above examples, these work:

In [1]: import time

In [2]: def tim():
   ...:     print(time.time())
   ...:

In [3]: tim()
1718358164.5522225

In [4]: a = 1

In [5]: (lambda: a)()
Out[5]: 1

@3b1b

I'm wondering if there is any change in Python or IPython that could provide a first-class work around for this issue. It even confuses and annoys @3b1b: https://youtu.be/rbu7Zu5X1zI?si=3gcC1r2k3WrfZu-I&t=1734

My workaround at https://github.com/limwz01/ipython_utils is clunky but works.. and I've been using it for a long while now.

@3b1b

Yeah, I'm in the process of watching @3b1b video, and I know this is an ipython limitation. I'd love a PR, I unfortunately dont have the time to dive into it.

My guess is that it's the same issue that affects ipdb where list comprehension don't work.

@limwz01, if you think your utils can be ported to be default in IPython, please send a PR.
Worse case we'll find some issues and that can be reverted, but at least we'll move forward on our comprehension of what it happening.

@Carreau

@Carreau I've packaged it as a separate pip installable package ipython_utils as I really need more people to test it in real world usage first. It does a lot of complicated stuff behind the scenes and right now it messes up the stack trace a little and I've had some minor problems with restoring the excepthook. You're welcome to try it out and give me some feedback

@gaogaotiantian

@gaogaotiantian recently pushed a similar solution to pdb in cpython itself: python/cpython#111094

His solution is a little too simplistic and did not handle all the edge cases, so it may not be suitable here. However, it shows that such a solution is actually acceptable.

His solution is a little too simplistic and did not handle all the edge cases, so it may not be suitable here.

Feel free to submit an issue about the edge cases that pdb can't handle. We might not be able to fix all of those because we need to make the pdb code as maintainable as possible (so playing with ast might not be an option), which means we need to balance the code complexity and the capability, but it would be good for us to at least know what are the cases that we can't handle.

@gaogaotiantian

@gaogaotiantian I use the following to illustrate the problems:

#!/usr/bin/env python3.13
g0 = 0
def main():
    import pdb
    pdb.Pdb().set_trace()
if __name__ == "__main__":
    main()

The following are the problems:

not re-using the same Cells for each local (this is really the biggest issue)

(Pdb) t = 1
(Pdb) f = lambda: t
(Pdb) t = 2
(Pdb) f()
1

not obtaining the last expression (a pdb test suggests otherwise, but it is actually wrong)

(Pdb) t=10; (lambda: t)()
<no output>

(it does return a value for global g; g=1; g. but I'm at a loss as to where that value comes from)

giving a weird result with shadowed variables (actually because it bails out):

(Pdb) g0 = 1
(Pdb) global g0; t=g0; t=(lambda: t)()
*** NameError: name 't' is not defined

not removing annotations

(Pdb) t:int = 1; (lambda: t)()
*** NameError: name 't' is not defined
* using indentation to set up block scope

(Pdb) print((lambda:len("""
... """))())
7

* not hoisting globals (this is arguable):

(Pdb) global g; g=1
(Pdb) t=2
(Pdb) g=(lambda: t)()
(Pdb) global g; g
1

I can't seem to edit my post to correct some typos and cross-reference the issue, but anyway the issue has been submitted here: python/cpython#126958

Use user_ns. It may be some dict, or locals() | globals(), for example.

from IPython import start_ipython
start_ipython(user_ns={})

or

from IPython.terminal.embed import InteractiveShellEmbed
InteractiveShellEmbed(user_ns={})()

or

import IPython
IPython.embed(user_ns={})

Result:

In [1]: a = 1

In [2]: def f(x):
   ...:     return a * x
   ...:

In [3]: f(2)
Out[3]: 2

This issue is about the problems of embedding a shell in a function scope, something you totally missed in your examples, @ur001 Edit: okay that wasn't exactly the initial poster's problem, but in a few posts down it becomes obvious that embedding in a function is the problem that is remaining after the "This has been now fixed" post.

Backport Pdb closure evaluation fix (list comprehension, generators) …

@limwz01

@limwz01 it appears that you problem was a different issue, related to pdb, fixed in Python 3.13 and backported to IPython in #14933.

Passing locals() scope explicitly is now required in Python 3.13 as per:

It looks like we could just ensure that embed defaults to pass a locals() in. Is there any example where

import IPython
IPython.embed(user_ns={})

or

import IPython
IPython.embed(user_ns=locals())

would not work?

Yes, definitely. What you are doing is running the shell with its globals set to the locals() at that point in question. That is bad because even global variables will now be "out of scope" in the new shell. Here's a piece of code that has all the types of variables:

import IPython
x0 = 100
def main():
    y0 = 200
    y1 = 300
    def f(z0, z1):
        nonlocal y1
        def g():
            nonlocal z1
            z1 += 1
        y1 += 1
        print(x0, y0, y1, z0, z1)
        IPython.embed(user_ns=locals())
        print(x0, y0, y1, z0, z1)
        IPython.embed(user_ns=locals())
    f(400, 500)
main()

Now, let's see what you can do in the first shell:

x0 cannot be accessed
no changes to the local variables are retained, i.e. changing any of the variables in the first shell wouldn't have any effect on the second print, or their values in the second shell.
g(); print(z1) changes only the real z1 and the shell's z1 is unchanged

Quoting from my readme at https://github.com/limwz01/ipython_utils?tab=readme-ov-file#limited-functionality-in-default-ipython-embedded-shell :

There are a few workarounds for this, when embedding into the scope of a function f:

Use a mocked-up globals() including what was in locals()

Use the real globals() and copy over locals() into the real globals()

Both of the above require us to patch all the nonlocal statements referring to locals in the scope of f (and parent functions) to become global. In the first workaround, we use IPython with a globals() namespace that have been updated with both the original frame's globals() and locals(), either by running IPython.start_ipython with user_ns as the mocked-up globals() or patching IPython.terminal.embed.InteractiveShellEmbed (instead of this, one could possibly keep calling globals().update(locals()) for every statement executed but that is really inefficient and only works if the created child functions do not modify the referenced variables). In this workaround, references to real globals in created functions will no longer access/modify the real global variable, but a copy, hence diverging from functions created by other means which use the real globals. In the second workaround, the real global namespace gets polluted and globals might have their values incorrectly overwritten by the locals, causing some existing functions to behave incorrectly.

Instead, one could use the (almost) complete solution that I developed (after pip install ipython_utils), and everything just works fine, you can even create a closure that works in the second shell:

import ipython_utils
x0 = 100
def main():
    y0 = 200
    y1 = 300
    def f(z0, z1):
        nonlocal y1
        def g():
            nonlocal z1
            z1 += 1
        y1 += 1
        print(x0, y0, y1, z0, z1)
        ipython_utils.embed([f, lambda: z1])
        print(x0, y0, y1, z0, z1)
        ipython_utils.embed([f, lambda: z1])
    f(400, 500)
main()

In the first shell:

def h():
    nonlocal z1
    z1 += 2

In the second shell:

print(z1)
g()
print(z1)
h()
print(z1)

And this prints:

500
501
503

If anyone can break my code, feel free to post an issue! EDIT: I've updated my code due to a backward-incompatible change in Python3.13.

Navigation Menu

Repository navigation

Global variables undefined in interactive use of embedded ipython shell #62

34 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Global variables undefined in interactive use of embedded ipython shell #62

Description

Activity

fperez commented on Dec 4, 2010

takluyver commented on Mar 23, 2011

takluyver commented on Mar 23, 2011

fperez commented on Nov 28, 2011

daf commented on Feb 7, 2012

takluyver commented on Feb 7, 2012

daf commented on Feb 7, 2012

daf commented on Feb 7, 2012

takluyver commented on Feb 7, 2012

liokm commented on Apr 24, 2012

takluyver commented on Apr 24, 2012

liokm commented on Apr 24, 2012

takluyver commented on Apr 24, 2012

34 remaining items

limwz01 commented on Jun 14, 2024

Erotemic commented on Oct 13, 2024

limwz01 commented on Oct 13, 2024

Carreau commented on Oct 14, 2024

limwz01 commented on Oct 27, 2024

limwz01 commented on Nov 6, 2024

gaogaotiantian commented on Nov 6, 2024

limwz01 commented on Nov 18, 2024

limwz01 commented on Nov 19, 2024

ur001 commented on Dec 20, 2024

limwz01 commented on Dec 21, 2024

krassowski commented on Jul 4, 2025

limwz01 commented on Jul 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions