Generate Idle help from Doc/library/idle.rst

@birkenfeld

BPO	16893
Nosy	@birkenfeld, @terryjreedy, @ned-deily, @serwy, @merwok, @asvetlov, @roseman, @rovitotv, @vadmium, @zware
Files	issue16893_makefiles.diff: Adjust the Doc makefiles to add idledoc target issue16893_makefiles.v2.diff: Version 2 help.txt: rendered help text issue16893.v3.diff: Version 3, with changes to Doc/library/idle.rst issue16893-v4.diff sphinxview.py htmlhelp.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2015-09-23.04:10:23.175>
created_at = <Date 2013-01-08.16:45:12.163>
labels = ['expert-IDLE', 'type-feature', 'docs']
title = 'Generate Idle help from Doc/library/idle.rst'
updated_at = <Date 2015-09-23.04:10:23.173>
user = 'https://github.com/zware'

bugs.python.org fields:

activity = <Date 2015-09-23.04:10:23.173>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2015-09-23.04:10:23.175>
closer = 'terry.reedy'
components = ['Documentation', 'IDLE']
creation = <Date 2013-01-08.16:45:12.163>
creator = 'zach.ware'
dependencies = []
files = ['28754', '29111', '29112', '29133', '36975', '40405', '40530']
hgrepos = []
issue_num = 16893
keywords = ['patch']
message_count = 44.0
messages = ['179362', '179615', '179708', '179710', '179846', '179954', '179959', '179960', '179961', '179962', '179963', '180096', '180104', '182115', '182298', '182300', '182301', '182546', '185706', '185726', '185729', '185732', '186035', '186040', '229706', '229712', '232105', '232233', '232798', '250132', '250153', '250194', '250238', '250434', '250439', '251180', '251182', '251191', '251284', '251285', '251389', '251398', '251399', '251403']
nosy_count = 13.0
nosy_names = ['georg.brandl', 'terry.reedy', 'ned.deily', 'roger.serwy', 'eric.araujo', 'asvetlov', 'markroseman', 'THRlWiTi', 'docs@python', 'Todd.Rovito', 'python-dev', 'martin.panter', 'zach.ware']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue16893'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

Lib/idlelib/help.txt and Doc/library/idle.rst contain almost exactly the same information, only formatted a bit differently. This issue is to suggest the merger of the two files, by way of creating help.txt from idle.rst as a part of the build or install process.

This could be as simple as merely copying the one to the other, but that of course leaves in place all reST directives that mean nothing to a simple text document. Thus, idle.rst should be processed to create help.txt. This could be done with sphinx/docutils, but to avoid having the CPython build process (not just the doc build process) depend on those tools, I think a very simple dedicated script could be created to do the processing without much trouble.

As far as I can tell, this new script would only have to do a few simple steps:

Remove comments (including index markers, etc.)
Remove reST roles (replace ":role:`" with "`", possibly remove "`" as well)
Remove backslash escapes
Remove (or undouble) double colons
Replace "#." with "1.", etc.

Thoughts?

I think getting this issue fixed makes sense, it would save time and remove duplication.

I like the proposition in general, but files should be synchronized first.

Note that Sphinx' "make text" output already should be suitable. We already update the pydoc topics with that on every release, so we could just as well do the same for the IDLE doc. No need for another separate script.

Andrew,
Zachary and I worked on another issue together to sync idle.rst with help.txt here:
http://bugs.python.org/issue5066

bpo-5066 is ready for commit if you have time by the way. Thanks!

Georg:

Note that Sphinx' "make text" output already should be suitable. We
already update the pydoc topics with that on every release, so we could
just as well do the same for the IDLE doc. No need for another
separate script.

I take it this would mean generating help.txt and then checking it in? Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available. That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.

Regenerating idle.txt and committing it is fine to me.

I take it this would mean generating help.txt and then checking it in? Otherwise, users who built their own Python would likely run into issues with IDLE not finding its help file, or would be required to have sphinx available.

Yes, it will be checked in.

That's not quite what I had in mind, but if we already do it for pydoc topics, it sounds fine to me.

I don't think we have to worry about it getting out of date quickly. Automatically generating the IDLE help at run time from a documentation source file is not posssible anyway, since the doc sources are not available in a standard location for installed Pythons.

I don't think we have to worry about it getting out of date quickly.

Fair point :)

Automatically generating the IDLE help at run time from a documentation
source file is not posssible anyway, since the doc sources are not
available in a standard location for installed Pythons.

This isn't quite what I meant either; I had intended it to be done at either Python build or install time. Your method is much simpler, though, I like it.

That said, here's the diff between the (now) current Lib/idlelib/help.txt and the Sphinx make text output of Doc/library/idle.rst. I've not found where we've automated the pydoc topics generation, else I'd try to provide a patch to do this as well.

@echo

The unified diff is not very helpful; I think somebody has to put the files side by side and merge.

The pydoc topics are built with a custom Sphinx builder implemented in tools/sphinxext/pyspecific.py -- but if we just want the vanilla text builder output it should be enough to add a Doc/Makefile target:

idledoc: BUILDER = text
idledoc: SOURCES = library/idle
idledoc: build
@echo "Build finished; now copying build/text/library/idle.txt to ../Lib/idlelib/help.txt."
@cp build/text/library/idle.txt ../Lib/idlelib/help.txt

and I add a step to PEP-101 to run this.

To make it actually work, replace "library/idle" by "library/idle.rst".

Ok. In case you're curious or might find it useful elsewhere, I've attached sphinxview.py, which renders the HTML into a text widget (with formatting, it's not converted to plain text).

This looks more maintainable than I expected, so I will definite try it out.

In msg185726 Ned said "Ironically, the Tk text widget is perfectly capable of rendering richer text styles." He was and is correct. Mark Roseman offered to take up the challenge. In spite of my skepticism, he went ahead and built sphinxview around HTMLParser. This reduced the core of the problem to "for each tag, content, and end tag, what state change or text insertion is needed. The result is simple enough for me to understand and even edit.

I sent Mark a detailed comparison of the Firefox rendering of idle.html and his tk rendering. He replied with a new version whose output is essentially identical to Firefox's (minus links), including the green example box. I intend to use a third version and will work on a full patch.

When running in a repository clone, Idle could access ../../Doc/build/html/library/idle.html. But for installations, idle.html needs to be in idlelib. I guess the only was to guarantee this everywhere (excepting overt downstream removals) is to commit it into the repository, unlike most other files built from .rst files).

To help make sure this happens, I would like to add a comment at the top of idle.rst. The devguide sentence on comments is unclear to me. Will the following work?
.. # After editing this, also build html, copy idle.html to idlelib,
test, and upload or commit the html diff also.

It would be nice if the copy and commit process were added to the release process, but I will go ahead even without that. A nicely rendered, slightly out-of-date .html will be a great improvement over the current situation.

I'd be happy to work on making an 'idledoc' target for the Doc/ makefiles that works with Mark's viewer. I can't guarantee any kind of timeframe, but the idledoc target wouldn't be necessary right off the bat; it could be done after the main bulk of the patch is committed.

As far as reST comments, the syntax is:

"""
.. This is a comment.
This is still the same comment.
And more of the same comment.

This is no longer part of the comment
"""

See http://docutils.sourceforge.net/docs/user/rst/quickref.html#comments

New changeset 4038508240a1 by Terry Jan Reedy in branch '2.7':
Issue bpo-16893: Replace help.txt with idle.html for Idle doc display.
https://hg.python.org/cpython/rev/4038508240a1

New changeset 52510bc368f8 by Terry Jan Reedy in branch '2.7':
Issue bpo-16893: include new files
https://hg.python.org/cpython/rev/52510bc368f8

New changeset 2d808b72996d by Terry Jan Reedy in branch '3.4':
Issue bpo-16893: Replace help.txt with idle.html for Idle doc display.
https://hg.python.org/cpython/rev/2d808b72996d

New changeset e66fbfa282c6 by Terry Jan Reedy in branch '2.7':
Issue bpo-16893: whitespace in idle.html.
https://hg.python.org/cpython/rev/e66fbfa282c6

New changeset 9b79a4901069 by Terry Jan Reedy in branch '3.4':
Issue bpo-16893: whitespace in idle.html.
https://hg.python.org/cpython/rev/9b79a4901069

New changeset 1107e3ee6103 by Terry Jan Reedy in branch '2.7':
Issue bpo-16893: whitespace in help.py.
https://hg.python.org/cpython/rev/1107e3ee6103

I committed a patch that works well enough for release as it. I attached the diff for changed files in case anyone wants to view in Rietveld.

Still to do for this issue:

Automate getting the 2.7 and earliest 3.x idle.html copied to idlelib and whitespace-normalized. (Commit/push should stay manual.) Several lines have 1 to many blanks at the end, while the file lacks a final \n. See e66fbfa282c6. (These anomalies come from Sphinx; idle.rst has been normalized to be checked in.)

I have a .bat file that builds any and copies as needed. I will to add a pass through Idle's trailing-whitespace deleter, which did just what was needed to commit. (Patchcheck.normalize_docs_whitespace does a here unneeded comparision (always unequal), creates an unwanted and nuisance .bak, and does not appear to work when the last list has no '\n'.)

When I have automation working for me, I would like to add make targets for others to use (Zach's patch 4).

Edit idle.rst

I will push NEWS and idlelib.NEWS entries separately, along with others for Idle.

I opened new issue bpo-25198 for further viewer improvements (and bug fixes, if any).

New changeset e749080fa0f9 by Terry Jan Reedy in branch '2.7':
Issue bpo-16893: finish deprecation.
https://hg.python.org/cpython/rev/e749080fa0f9

New changeset ff0270e9bdfb by Terry Jan Reedy in branch '3.4':
Issue bpo-16893: finish deprecation.
https://hg.python.org/cpython/rev/ff0270e9bdfb

New changeset 855484b55da3 by Terry Jan Reedy in branch '2.7':
Issue bpo-16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/855484b55da3

New changeset df987a0bc350 by Terry Jan Reedy in branch '3.4':
Issue bpo-16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/df987a0bc350

New changeset f08437278049 by Terry Jan Reedy in branch '3.5':
Issue bpo-16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/f08437278049

New changeset 09ebed6a8cb8 by Terry Jan Reedy in branch 'default':
Issue bpo-16893: Add idlelib.help.copy_strip() to copy-rstrip Doc/.../idle.html.
https://hg.python.org/cpython/rev/09ebed6a8cb8

NEWS entries are done.

I added a function, copy_strip, to help.py to rstrip (in bytes mode) idle.html as it copies to help.html. (I changed the name to better remember which is which.) In my .bat file, with Doc as current directory, I use
..\pcbuild\python_d.exe -c "from idlelib.help import copy_strip; copy_strip()"
I could change the if __name__ block so that
..\pcbuild\python_d.exe -m idlelib.help copy_strip
would work. Either way, I would like to add 'idlehelp' to makefiles. But not tonight.

The tests seem to have grown a new Deprecation Warning (triggered when run with python -bWall). Looks like this may be a side effect of revision 2d808b72996d.

[160/392] test_idle
/media/disk/home/proj/python/cpython/Lib/idlelib/EditorWindow.py:88: DeprecationWarning: EditorWindow.HelpDialog is no longer used by Idle.
It will be removed in 3.6 or later.
It has been replaced by private help.HelpWindow

  helpDialog = HelpDialog()  # singleton instance, no longer used

New changeset 26e819909891 by Terry Jan Reedy in branch '2.7':
Issue bpo-16893: Move idlelib.EditorWindow.HelpDialog deprecation warning
https://hg.python.org/cpython/rev/26e819909891

New changeset c607004a98bf by Terry Jan Reedy in branch '3.4':
Issue bpo-16893: Move idlelib.EditorWindow.HelpDialog deprecation warning
https://hg.python.org/cpython/rev/c607004a98bf

Thank you for reporting this. My general problem is that idlelib was only informally private until PEP-434 and I only started adding 'private' and 'deprecated' to the code a week ago. So I am trying to make changes in existing versions while not breaking even unlikely uses of the current API. The specific problem is the the class init method *is* used -- to create an unused instance -- on import.

  helpDialog = HelpDialog()  # singleton instance, no longer used

The patch fixes the problem by moving the deprecation to the show method, which Idle really does not now call.

Through the contributions of several people, Zack's key idea, that Idle documentation should have a single source, idle.rst, has come to fruition. So I think it time to close this. I already mentioned bpo-25198 for improving the tkinter viewer. I also opened bpo-25218 for revising Zack's patch and possibly using an 'idledoc' target in the release process.

Navigation Menu

Repository navigation

Generate Idle help from Doc/library/idle.rst #61097

23 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Generate Idle help from Doc/library/idle.rst #61097

Description

Activity

zware commented on Jan 8, 2013

rovitotv commented on Jan 11, 2013

asvetlov commented on Jan 11, 2013

birkenfeld commented on Jan 11, 2013

rovitotv commented on Jan 13, 2013

zware commented on Jan 14, 2013

asvetlov commented on Jan 14, 2013

birkenfeld commented on Jan 14, 2013

zware commented on Jan 14, 2013

birkenfeld commented on Jan 14, 2013

birkenfeld commented on Jan 14, 2013

23 remaining items

roseman commented on Sep 8, 2015

terryjreedy commented on Sep 8, 2015

terryjreedy commented on Sep 11, 2015

zware commented on Sep 11, 2015

python-dev commented on Sep 21, 2015

terryjreedy commented on Sep 21, 2015

python-dev commented on Sep 21, 2015

python-dev commented on Sep 22, 2015

terryjreedy commented on Sep 22, 2015

vadmium commented on Sep 23, 2015

python-dev commented on Sep 23, 2015

terryjreedy commented on Sep 23, 2015

terryjreedy commented on Sep 23, 2015

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions