Decide how to handle str/unicode

There's a long discussion on this topic in the mypy tracker: python/mypy#1141

I'm surfacing it here because I can never remember whether that discussion is here, or in the typeshed repo, or in the mypy tracker.

(Adding str, bytes, unicode, Text, basestring as additional search keywords.)

A new proposal: use the existing triad bytes-str-unicode.

Let me try to explain the new proposal with more care.

Gradual Byting

I am most interested in solving this issue for straddling code; my assumption is that most of the interest in type annotations for Python 2 has to do with that. (This is the case at Dropbox, and everyone who has enough Python 2 code to want to annotate it probably should be thinking about porting to Python 3 anyway. :-)

In the proposal, str has a position similar to the one that Any has in the type system as a whole -- i.e. assuming we have three variables b, s, t, with respective types bytes, str, (typing.)Text, then: b is compatible with s, s is compatible with b and t, t is compatible with s, but b and t are not compatible with each other. IOW the relationships between the three types are not expressible using subtyping relationships only. (It's actually a little more complicated, I'll spell out the actual rules I'm proposing below.)

Before we get to that, I'd like to discuss the use cases for the proposal. In straddling code we often have the problem that some Python 2 code is vague about whether it works on bytes or text or both. The corresponding Python 3 code may work only on bytes, or only on Text, or on both as long as they are used consistently (i.e. AnyStr), or possibly on Union[bytes, Text]. To find such cases, maybe we could just type-check the code twice, once in Python 2 mode and once in Python 3 mode. If it type-checks cleanly in both modes, it should run correctly in both Python versions too (insofar as type-checking cleanly can ever say anything about running without errors :-).

However, when we have a large code base, it is usually a struggle to make it type-check cleanly even in one mode, and typically we start with Python 2. So if we have code that runs correctly using Python 2 and type-checks cleanly in Python 2 mode, and we want to port it to Python 3, requiring it to type-check cleanly in Python 3 mode is setting the bar very high (as high as expecting it to run correctly using Python 3).

Therefore I am proposing a gradual approach. Similar to the way we start by type-checking an untyped program (which by definition should type-check cleanly, since all types are Any -- even though in practice there are some holes in that theory), I propose to start with a Python 2 program that uses str for all string types, and type-checks cleanly that way, and gradually change the program to replace each occurrence of str with either bytes or Text (or one or the rarer alternatives like AnyStr or Union[str, Text]). That way we can gradually tighten the type annotations, keeping the code type-check clean as we go.

Just like, when I define a function with def f(x: Any), I can call f(1), f('') and f([0]) and it's all the same to the type checker, and f's body I can use x+1, x() or x[0], the idea here is that a function defined with def g(s: str) can be called as g(''), g(b'') or g(u''), and in g's body I can use s+b'xxx', s+'yyy' or s+u'zzz'.

The actual details are a bit subtle. I'm proposing (in builtins; recall that this is for Python 2 only):

class bytes with (mostly) the methods currently present on str, with arguments of type bytes and returning bytes (as appropriate).
class str(bytes) with overloaded methods that return str if the other argument is a str, returning bytes for `bytes (more or less).
class unicode unchanged from its current definition, keeping typing.Text as a pure alias for it.

The subclassing relationship between bytes and str makes str acceptable where bytes is required. In mypy we can add a "promotion" from bytes to str to enable compatibility in the other direction. Mypy (in Python 2 mode) has an existing promotion from str to unicode that accepts str where unicode is required. I don't actually propose to make unicode acceptable where str is required (this is a deviation from the "str is like Any" idea). Because promotions are not transitive (unlike subclassing), bytes is not acceptable where unicode is required, nor the other way around.

There is still a lot more to explain. I want to show in detail what happens in various cases, and why I think that is right. I need to explain the concept of "text in spirit" to motivate why I am okay with the difference between these rules and the actual workings of Python 2. I want to go over some examples involving container types (since that's where the "AsciiBytes" proposal went astray). And I need to give some guidelines for stub authors and changes to existing stubs. (E.g. I think that Python 2 getattr() may have to be changed to accept unicode.)

[But that will have to wait until tomorrow.]

t [Text] is compatible with s [str]

This contradicts with this part of the proposal:

I don't actually propose to make unicode acceptable where str is required

Also, the example with def g(s: str) suggests that it can be called as g(u''), with
a unicode argument. This should be clarified and made consistent across the proposal, as
otherwise things get confusing.

Because promotions are not transitive (unlike subclassing)

Mypy actually considers the promotions int -> float and float -> complex
transitive, and int can be promoted to complex. We could change the language
to something like "these promotions are not transitive" or we could perhaps treat
the int -> complex promotion as a separate promotion.

Other notes:

I'd assume that str methods would return unicode if the other argument is unicode.
Currently this is left unspecified. It could be useful to have table of the result types
of s1 + s2 for all combinations of str, bytes and unicode (9 cases).
AnyStr would have to range over str, bytes and unicode. This means that we may want
to give different meanings to IO[str] and IO[bytes], for example.
List[Any] is compatible with List[int] and vice versa in mypy (though PEP 484/483
seems to be silent on this), but should List[str] be compatible with List[bytes], and
vice versa? I'd argue that List[str] and List[bytes] should be incompatible, similar to
how List[int] and List[float] are incompatible, but I don't have a strong opinion on
this.

This contradicts with this part of the proposal

That's why I wrote It's actually a little more complicated. I am having a hard time summarizing the proposal briefly and also writing it up in detail without contradictions between the two. In case of conflict the detailed version should win and the summary seen as a hint at most. Maybe we'll have to use more vagueness in the summary to avoid confusing experts who know the terminology.

the example with def g(s: str) suggests that it can be called as g(u'')

More imprecision in the summary. :-( It really can't, unless g() is implemented in C in a certain way, e.g. getattr(x, u"foo"). But for a Python function this is wrong. Actually, for a Python function, the other way around is also wrong. But nevertheless the promotion allows it. Just like the promotion from int to float is technically wrong in Python 2, as shown here:

def f(a):
    # type: (float) -> float
    return a/2
assert f(3) == 1.5  # Fails, it returns 1

I will try to spec out the true compatibility as a bunch of tables.

str methods would return unicode if the other argument is unicode

Yes. There are already some overloads like that. The bigger difference will be that these overloads won't exist for bytes+unicode.

AnyStr would have to range over str, bytes and unicode. This means that we may want to give different meanings to IO[str] and IO[bytes], for example.

Yes. In fact IO[unicode] would only be obtainable by calling io.open().

should List[str] be compatible with List[bytes], and vice versa?

I think not (so we agree here). This will lead to some of the same issues as I ran into when trying to implement the AsciiBytes idea, but the issues will much less common.

[In the next installment I will try to construct the tables of compatibilities. I will also talk about the concept of "text in spirit".]

Text in Spirit

(This is still pretty messy. But I promised I would explain the concept.)

I'll sometimes say that some variable in Python 2 is "text in spirit". For example, in getattr(x, name), name is "text in spirit". In this case I mean two things with this: first, that in Python 3 the name argument to getattr() has type str, not bytes. Second, that even in Python 2, the name is an identifier, and even though you can write getattr(x, '\xff\x01'), that would be useless.

Note that text encoded as bytes is not "text in spirit". The requirement is that the corresponding Python 3 API uses str, and the Python 2 API supports bytes or unicode, though not necessarily all bytes or all unicode -- e.g. getattr() only accepts a unicode name if it contains only ASCII characters, even though it doesn't make that requirement when the argument is str.

Basically the point of "text in spirit" is to make the argument that an API should not use bytes even though it may accept non-ASCII str instances. But I have to do more exploration before I decide how important this concept is.

Here's another table that could be useful -- if we define def f(s: s1) -> None: ..., is a call with an argument of type s2 valid, when s1 and s2 range over str, bytes and unicode.

Compatibility Tables

[UPDATE: made function calls primary, per Jukka's suggestion below]

Let's start by stating the compatibility between expressions of types bytes, str, text and functions with arguments of those types. Each row corresponds to a declared argument type; each column corresponds to the type of an expression passed in for that argument.

Argument type	xb: bytes	xs: str	xt: Text
arg_b: bytes	Yes (same)	Yes (str <: bytes)	No
arg_s: str	Yes (promotion)	Yes (same)	???
arg_t: Text	No	Yes (promotion)	Yes (same)

The above table also describes compatibility of expressions with variables (assuming the type checker, like mypy currently, doesn't just change the type of the variable). Note that I'm not decided yet whether to allow passing a Text value to a str argument, but I'm inclined to put "No" there, even though that breaks the illusion of "str as the Any of string types" (IOW gradual byting :-).

Next let's describe the return type for expressions of the form x + y where x and y can each by of type bytes, str, or Text.

x	yb: bytes	ys: str	yt: Text
xb: bytes	bytes	bytes	ERROR
xs: str	bytes	str	Text
xt: Text	ERROR	Text	Text

Note that this table is more regular and I'm pretty confident about it.

Irregularities

`encode()` and `decode()`

For bizarre reasons, in Python 2 both str and unicode support both encode() and decode(). This makes no sense, e.g. u'abc'.decode('utf8') is equivalent to u.abc'.encode('ascii').decode('utf8'), and 'abc'.encode('utf8') really means 'abc'.decode('ascii').encode('utf8').

I propose to rationalize this to the extent possible, as follows:

bytes only support .decode(), and it returns unicode
unicode only supports .encode(), and it returns bytes (not str!)
str supports .encode(), returning bytes, and .decode(), returning unicode

This would mean complete removal of unicode.decode() from the stubs, since it basically always means some terrible misunderstanding happened. For variables declared as bytes, it would likewise remove the encode() method, whose use would point to a similar (but opposite) misunderstanding. For str we remain generous (since using str means the code probably hasn't received enough attention from the straddling police).

`str()` and `repr()`

The return types of bytes.__str__() and bytes.__repr__() are still str, because that's how they are constrained by object. (FWIW these are examples of methods returning "text in spirit" strings.)

A reason we might prefer to use a call instead of an assignment as a basis for the table is that some type checkers like to infer a new type from assignment, and thus arbitrary assignments are considered correct -- they just redefine the type of a variable.

OK, edited the text.

If we don't do the Text -> str promotion (which seems reasonable), then the original "gradual byting" story may need tweaking, as the first step would be to annotate with str and Text only (not just str, because unicode literals wouldn't be compatible with it), and the gradual byting migration would migrate some str types to bytes (or maybe unicode). Also, the gradual migration may involve changing some '' literals to b'' literals

Example first phase annotation where we'd need Text:

def utf8_len(x: Text) -> int:
    return len(x.encode('utf8'))

utf8_len(u'\u1234')

Here we'd need to use Text unless we include the Text -> str promotion.

@gvanrossum

@gvanrossum @JukkaL I would like to join your discussion.

Here is a summary of the current ASCII types and gradual byting proposals as I understand them.

Rationale

(This piece is here to make sure we’re solving the same problem.)

The changes in text and binary data handling in Python 3 is one of the major reasons people cannot run their Python 2 code on Python 3. These changes were:

Disabled implicit str to unicode and vice versa conversions using the ASCII encoding
Some missing methods or methods with different semantics for the text and binary classes

The most viable approach to porting to Python 3 is via porting to Python 2+3.

Therefore we need a way to make this transition from a Python 2 program with implicit conversions to a Python 2 program with less implicit conversions to a Python 2+3 program that runs on both versions but still contains some implicit conversions to a Python 3 program.

ASCII Types

The idea is to introduce two new types, one new types compatibility rule and special type inference rules for binary and text literals.

The new types are:

class typing.ASCIIBytes(bytes): ...
class typing.ASCIIText(typing.Text): ...

The ASCIIBytes type is compatible with ASCIIText for Python 2.

If a text or binary literal contains only ASCII characters then type checkers should infer the corresponding ASCII types instead of regular text / binary types.

Pros

People can be very precise about their Python 2 types and implicit conversions. This precision is much needed for forbidding implicit conversions eventually while porting from Python 2 to Python 2+3 to Python 3.

Cons

List[ASCIIBytes] is not compatible with List[str] that causes lots of errors according to Guido’s experiments. As a workaround, we might make ASCIIBytes compatible with any TypeVar that is constrained by ASCIIText or Text despite it’s variance. The cost of this workaround is more false negatives for unsafe modifications of invariant collections of strings.
People will have to write ASCII types for their ASCII-preserving functions (e. g. for a user-defined equivalent of str.upper() .

Gradual Byting

A proposal by Guido described here.

The idea is to distinguish the str in type hints from bytes and Text (unicode in Python 2). Then there are new types compatibility rules:

bytes is compatible with str
str is compatible with bytes
Text is maybe compatible with str (??? undecided yet)
str is compatible with Text

Questions

How the issue with List[bytes] being not a subtype of List[str] is resolved here? (see the similar issue in the ASCII types proposal)
If Text is not compatible with str then str is no longer a safe staring point in the Python 3 migration process. Won’t it confuse people?
What should be the type of functions like getattr() that do implicit text to bytes conversion?

What do you think of the workaround for invariant collections of ASCIIBytes?

Could you please answer my questions about the gradual byting proposal?

I still have to investigate this, but the problem with typeshed is likely that we don’t have an easy standard way to determine whether a specific function needs Unicode or Text.

I think remdragon's fork was the right approach, and it's too bad it went stale so quickly. When it was fresh I tried it (combined with the one-liner mypy tweak) on our codebase and there was only the one minor regression (easily fixable, and I wrote way too many words about it, whoops!) I think that's the way forward.

As I understand it, typeshed is maintained by hand, so I don't know that there's any way around a big "Textification" commit that is also written by hand. In cases where it's unclear, fall back to AnyStr and mypy should keep accepting the things that it accepts today.

I propose we land remdragon's fork (after a fix and rebase) first—which should cause no change in behavior. Then make the changes in mypy to drop the str promotion (presumably behind a config flag), and include the new typeshed.

Did I miss anything? What do you think about that approach?

EDIT: Changed Text to AnyStr

Is this issue still relevant? I think there is a consensus on how to handle this and it has been implemented by all type checkers supporting Python 2.

I don't think anything is going to change, especially now Python 2 has reached its end of life (and acknowledging that many projects and companies are still working on porting their legacy Python 2 code to Python 3).

I'm not sure what is the consensus on how to handle it? I believe we changed some language in PEP 484 about not promoting str to unicode but I don't believe mypy made a change to match (and the PEP was intentionally not requiring it, just allowing it).

FWIW I migrated reasonably-sized codebase from Python 2.7 to 3.5 in late 2016 and I would claim what typing and mypy provided back then was already good enough, as I had no runtime errors related to str/bytes/unicode/text handling as far as I remember once mypy was satisfied.

The consensus from what I understand (which might be wrong) is:

PEP 484: Text is an alias for str (py3) or unicode (py2)
PEP 484: Text in argument types accept both str and unicode (py2)
unicode can't be assigned to str, whether the reverse holds true is up to type checkers.

That said I have no particular stake in this discussion, since I'm fortunate enough not to work with Python 2 anymore.

Mypy treats Text as an alias to str (Python 3) or unicode (Python 2). In Python 2, str is considered as a subtype of unicode (but not vice versa). Mypy is not going to change this, since Python 2 has reached end of life, and because any changes would likely be very disruptive.

I don't think that it's important to change anything or to reach consensus any more. I'm fine with closing this issue.

Sounds good. Closing.

Enchant: Spell checking library for Enchant WIP

Navigation Menu

Repository navigation

Decide how to handle str/unicode #208

Gradual Byting

Text in Spirit

Compatibility Tables

Irregularities

`encode()` and `decode()`

`str()` and `repr()`

Rationale

ASCII Types

Pros

Cons

Gradual Byting

Questions

97 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Decide how to handle str/unicode #208

Description

Activity

gvanrossum commented on Aug 10, 2016

gvanrossum commented on Aug 17, 2016

Gradual Byting

JukkaL commented on Aug 17, 2016

gvanrossum commented on Aug 17, 2016

gvanrossum commented on Aug 17, 2016

Text in Spirit

JukkaL commented on Aug 17, 2016

gvanrossum commented on Aug 17, 2016

Compatibility Tables

gvanrossum commented on Aug 17, 2016

Irregularities

encode() and decode()

__str__() and __repr__()

JukkaL commented on Aug 17, 2016

gvanrossum commented on Aug 18, 2016

JukkaL commented on Aug 18, 2016

vlasovskikh commented on Aug 21, 2016

Rationale

ASCII Types

Pros

Cons

Gradual Byting

Questions

97 remaining items

gvanrossum commented on Jan 3, 2019

whatisaphone commented on Jan 7, 2019

srittau commented on Apr 25, 2020

gvanrossum commented on Apr 25, 2020

jstasiak commented on Apr 25, 2020

srittau commented on Apr 25, 2020

JukkaL commented on Apr 27, 2020

gvanrossum commented on Apr 27, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`encode()` and `decode()`

`str()` and `repr()`