IDLE: add a token browser

[serhiy-storchaka]

Seeing how source is split into tokens helps in learning how Python's tokenizer works and in debugging tokenizer-dependent behavior, but IDLE offers no way to inspect the token stream.

I propose a Token Browser added as an extension on the Tools menu (Shell and editor). It shows the output of tokenize for the current window in a ttk.Treeview — one row per token with its type, string, and range.

Interaction:

• Selecting a row highlights the corresponding source in the editor.
• Double-clicking a row moves the editor cursor to that token.
• The browser follows the editor: a selection selects every overlapping token row, and the cursor alone selects the single token containing it.
• Scope is the editor selection if there is one, else the Shell's current input, else the whole window.

A PR will follow.

Linked PRs

• gh-152941: Add a token browser to IDLE #152946

[terryjreedy]

I want to convert the existing path and module browsers to ttk Treeview, and even made a start on the latter. They could be included with the new browsers. Perhaps call the new menu 'Browse'.

Policy note: a justification for an Python-specific IDE is that it can do very Python-syntax specific things like these browsers.

[terryjreedy]

I do not want to add new feature as extensions. One of the major things I and a couple of others did nearly a decade ago was to integrate the then extension features into the idlelib code base. It was a big and sometimes tricky patch, but well worth while. I would have to look back at the issue to explain why. Since then, new features have been written into the code base. The extension mechanisn was left for 3rd parties; the zdummy extension only exists for testing.

[terryjreedy]

If the above is not properly documented, please suggest where it should be. Readme? Changes in .py files?