Skip to content

Fixed support for Indian locales #64226

Closed
Closed
Fixed support for Indian locales#64226
Assignees
serhiy-storchaka
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@serhiy-storchaka
serhiy-storchaka
opened on Dec 19, 2013
BPO 20027
Nosy @malemburg, @loewis, @serhiy-storchaka
Files
  • locale_devanagari_3.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2013-12-26.19:24:04.296>
created_at = <Date 2013-12-19.19:42:08.692>
labels = ['type-bug', 'library']
title = 'Fixed support for Indian locales'
updated_at = <Date 2013-12-26.19:24:04.295>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2013-12-26.19:24:04.295>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2013-12-26.19:24:04.296>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2013-12-19.19:42:08.692>
creator = 'serhiy.storchaka'
dependencies = []
files = ['33234']
hgrepos = []
issue_num = 20027
keywords = ['patch']
message_count = 10.0
messages = ['206636', '206680', '206684', '206685', '206687', '206689', '206695', '206953', '206954', '206955']
nosy_count = 4.0
nosy_names = ['lemburg', 'loewis', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20027'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

    Activity

    1. serhiy-storchaka commented on Dec 19, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 19, 2013
      MemberAuthor

      The locales alias table contains invalid entries for devanagari modifiers (see bpo-5815):

      'ks_in@devanagari':                     'ks_IN@devanagari.UTF-8',
'sd':                                   'sd_IN@devanagari.UTF-8',

      Here is a patch which fixes aliases for these locales.

    2. serhiy-storchaka
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Dec 19, 2013

    3. malemburg commented on Dec 20, 2013

      @malemburg
      malemburg
      on Dec 20, 2013
      Member

      On 20.12.2013 12:19, Serhiy Storchaka wrote:

      Added file: http://bugs.python.org/file33231/locale_devanagari_2.patch

      See my message on bpo-20034:

      There is some recent activity in glibc related to these. Here's a
      patch that adds the sd_IN@devanagari locale to glibc:
      http://sourceware.org/cgi-bin/cvsweb.cgi/libc/localedata/locales/sd_IN@devanagari.diff?cvsroot=glibc&r1=NONE&r2=1.1

      So they will start working once platforms adopt the new
      glibc versions.

      The @-modifier is applied to the locale, not the encoding, because
      the locale uses a different script, as opposed to limiting itself
      to part of an encoding. This looks reasonable, even though I'm
      not sure it conforms to standards.

      Since all this is still very much in flux, perhaps we ought
      to wait a bit more and let the dust settle ?!

    4. serhiy-storchaka commented on Dec 20, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 20, 2013
      MemberAuthor

      Ubuntu 12.04 supports Kashmiri and Sindhi locales (requires language-pack-sd-base and language-pack-sd-base packages).

      $ locale -a
...
ks_IN
ks_IN@devanagari
ks_IN.utf8
ks_IN.utf8@devanagari
...
sd_IN
sd_IN@devanagari
sd_IN.utf8
sd_IN.utf8@devanagari
...

      Current Python doesn't support all of these locales:

      $ LC_ALL=ks_IN ./python -c 'import locale; print(locale.getlocale())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/locale.py", line 556, in getlocale
    return _parse_localename(localename)
  File "/home/serhiy/py/cpython/Lib/locale.py", line 465, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: ks_IN
$ LC_ALL=ks_IN@devanagari ./python -c 'import locale; print(locale.getlocale())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/locale.py", line 556, in getlocale
    return _parse_localename(localename)
  File "/home/serhiy/py/cpython/Lib/locale.py", line 465, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: ks_IN@devanagari
$ LC_ALL=ks_IN.utf8 ./python -c 'import locale; print(locale.getlocale())'
('ks_IN', 'utf8')
$ LC_ALL=ks_IN.utf8@devanagari ./python -c 'import locale; print(locale.getlocale())'
('ks_IN', 'UTF-8')
$ LC_ALL=sd_IN ./python -c 'import locale; print(locale.getlocale())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/locale.py", line 556, in getlocale
    return _parse_localename(localename)
  File "/home/serhiy/py/cpython/Lib/locale.py", line 465, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: sd_IN
$ LC_ALL=sd_IN@devanagari ./python -c 'import locale; print(locale.getlocale())'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/locale.py", line 556, in getlocale
    return _parse_localename(localename)
  File "/home/serhiy/py/cpython/Lib/locale.py", line 465, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: sd_IN@devanagari
$ LC_ALL=sd_IN.utf8 ./python -c 'import locale; print(locale.getlocale())'
('sd_IN', 'utf8')
$ LC_ALL=sd_IN.utf8@devanagari ./python -c 'import locale; print(locale.getlocale())'
('sd_IN', 'utf8')

      After applying the patch Python supports all ks_IN and sd_IN locales.

    5. malemburg commented on Dec 20, 2013

      @malemburg
      malemburg
      on Dec 20, 2013
      Member

      On 20.12.2013 15:55, Serhiy Storchaka wrote:

      After applying the patch Python supports all ks_IN and sd_IN locales.

      Well, yes, but only because you are removing the @-modifiers. I don't
      think that's correct, since e.g. the string formatting used for
      numbers is different with the modifier.

      If you keep the modifiers, but move them to the end of the locale
      string you should get the correct behavior, e.g.

      • 'sd': 'sd_IN@devanagari.UTF-8',
        + 'sd': 'sd_IN.UTF-8@devanagari',

      (modulo perhaps the spelling of "UTF-8")

    6. serhiy-storchaka commented on Dec 20, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 20, 2013
      MemberAuthor

      Well, yes, but only because you are removing the @-modifiers. I don't
      think that's correct, since e.g. the string formatting used for
      numbers is different with the modifier.

      All the @-modifiers except euro are applied to the locale, not the encoding.
      And Python removes all the @-modifiers, e.g. latin and cyrillic which specify
      the script.

      If you keep the modifiers, but move them to the end of the locale
      string you should get the correct behavior, e.g.

      • 'sd': 'sd_IN@devanagari.UTF-8',
      • 'sd': 'sd_IN.UTF-8@devanagari',

      (modulo perhaps the spelling of "UTF-8")

      Recent the locale.alias file changes these entities:

      sd: sd_IN.UTF-8
      sd_IN.utf8: sd_IN.UTF-8
      sd@devanagari: sd_IN@devanagari.UTF-8
      sd_IN@devanagari: sd_IN@devanagari.UTF-8
      sd_IN@devanagari.utf8: sd_IN@devanagari.UTF-8

    7. malemburg commented on Dec 20, 2013

      @malemburg
      malemburg
      on Dec 20, 2013
      Member

      On 20.12.2013 16:24, Serhiy Storchaka wrote:

      Serhiy Storchaka added the comment:

      > Well, yes, but only because you are removing the @-modifiers. I don't
      > think that's correct, since e.g. the string formatting used for
      > numbers is different with the modifier.

      All the @-modifiers except euro are applied to the locale, not the encoding.
      And Python removes all the @-modifiers, e.g. latin and cyrillic which specify
      the script.

      That's not quite correct. The modifiers are used to determine the
      correct mapping, so you'll often find them on the left side, but
      not necessarily on the right side.

      There are several cases where the modifiers are kept around,
      since they have implications on the way number or dates are
      formatted.

      For the Indian "devanagari" locales we have to keep them,
      because the locale formatting of number and dates depends
      on them.

      > If you keep the modifiers, but move them to the end of the locale
      > string you should get the correct behavior, e.g.
      >
      > - 'sd': 'sd_IN@devanagari.UTF-8',
      > + 'sd': 'sd_IN.UTF-8@devanagari',
      >
      > (modulo perhaps the spelling of "UTF-8")

      Recent the locale.alias file changes these entities:

      sd: sd_IN.UTF-8
      sd_IN.utf8: sd_IN.UTF-8
      sd@devanagari: sd_IN@devanagari.UTF-8
      sd_IN@devanagari: sd_IN@devanagari.UTF-8
      sd_IN@devanagari.utf8: sd_IN@devanagari.UTF-8

      I'm not sure I can parse this comment :-)

      Looking at bpo-20034 I think we are saying that the new updated
      local.alias file contains these entries:

      sd: sd_IN.UTF-8
      sd_IN.utf8: sd_IN.UTF-8
      sd@devanagari: sd_IN@devanagari.UTF-8
      sd_IN@devanagari: sd_IN@devanagari.UTF-8
      sd_IN@devanagari.utf8: sd_IN@devanagari.UTF-8

      So my example is wrong with the new locale.alias file. Instead,
      sd will map directly to sd_IN.UTF-8.

      Still, I think the makelocalalias.py script should correct
      the non-standard locale names from sd_IN@devanagari.UTF-8
      to sd_IN.UTF-8@devanagari in order to match the output
      of "locale -a".

    8. serhiy-storchaka commented on Dec 20, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 20, 2013
      MemberAuthor

      Updated patch to tip. The makelocalalias.py script now corrects
      the non-standard locale names.

    9. serhiy-storchaka commented on Dec 26, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 26, 2013
      MemberAuthor

      Could you please make a decision about last patch, Marc-Andre?

    10. malemburg commented on Dec 26, 2013

      @malemburg
      malemburg
      on Dec 26, 2013
      Member

      On 26.12.2013 19:59, Serhiy Storchaka wrote:

      Could you please make a decision about last patch, Marc-Andre?

      Looks good. Thanks, Serhiy.

    11. python-dev commented on Dec 26, 2013

      @python-dev
      python-devmannequin
      on Dec 26, 2013
      Mannequin

      New changeset aad582f717da by Serhiy Storchaka in branch '2.7':
      Issue bpo-20027: Fixed locale aliases for devanagari locales.
      http://hg.python.org/cpython/rev/aad582f717da

      New changeset 7615c009e925 by Serhiy Storchaka in branch '3.3':
      Issue bpo-20027: Fixed locale aliases for devanagari locales.
      http://hg.python.org/cpython/rev/7615c009e925

      New changeset fff3f28733b4 by Serhiy Storchaka in branch 'default':
      Issue bpo-20027: Fixed locale aliases for devanagari locales.
      http://hg.python.org/cpython/rev/fff3f28733b4

    12. serhiy-storchaka
      closed this as completedon Dec 26, 2013
    13. serhiy-storchaka
      self-assigned this
      on Dec 26, 2013
    14. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions