Alternative: render from_markdown via markdown-it-py

[MattFisher]

What this is

An alternative implementation of Post.from_markdown() that delegates parsing to markdown-it-py (a CommonMark parser) plus the standard footnote plugin, with a small renderer (mdrender.py) that maps the syntax tree onto Substack's node schema. Node construction is centralised in a new nodes.py module so the (undocumented) schema lives in one place.

Why I'm opening it

First off — this approach wasn't discussed beforehand, and I realise it's a bigger change than a typical PR, so I'm putting it up purely for your consideration; entirely your call on whether it's a direction you want to take.

I started out adding footnote support to the existing hand-rolled parser (#56). As that grew — footnotes, fenced/inline-code edge cases, multi-paragraph definitions — it started to feel like we were re-implementing a Markdown parser. So I prototyped this alternative to see how it compared, and it turned out significantly simpler:

• from_markdown() drops from a ~270-line hand-rolled parser to a few lines delegating to the renderer (net post.py ≈ −215 lines).
• Footnotes come essentially for free from the footnote plugin (including multi-paragraph definitions), rather than via bespoke pre-parse text extraction.
• A real CommonMark parser brings correctness for free (nested structures, edge cases) and removes the fragile overlapping-regex inline parsing.

Trade-offs (flagging honestly)

• Adds two runtime dependencies: markdown-it-py and mdit-py-plugins (both widely used and well maintained). This is the main thing to weigh.
• Two intentional, CommonMark-correct behaviour changes vs the old parser (tests updated to match, with comments):
  • Consecutive > lines are one paragraph; blank > lines split paragraphs (standard CommonMark). The old parser made one paragraph per line.
  • Footnote definitions that are never referenced are dropped (not appended to the end).
• parse_inline() / tokens_to_text_nodes() remain as public helpers (still used by the manual footnote() builder), but from_markdown() no longer relies on them.

Tests

• All existing from_markdown/parse_inline tests pass (the two semantic-difference tests above were updated).
• Added test_from_markdown_features.py with end-to-end coverage of every feature listed in the from_markdown() docstring (headings 1–6, bold/italic/bold+italic/inline-code/strikethrough, links, images, linked images, code blocks with/without language, blockquotes, bullet/ordered lists, horizontal rules, paragraphs).
• Footnote tests include references/definitions, named labels, numbering, multi-paragraph definitions, and code-span safety. 86 passing (excludes the pre-existing live-API tests that require credentials).

Relationship to #56

This is an alternative to #56 — if you'd prefer this approach, #56 can be closed in its favour; if not, no harm done and #56 stands on its own.

[ma2za]

Hi, thanks for this. I merged #56 first because it was the lower-risk incremental path for footnotes and kept the existing from_markdown output shape intact.

I like the direction here. Moving Markdown rendering onto markdown-it-py is the cleaner long-term approach, but I don’t want to merge this PR as-is yet because it changes a couple of things beyond the parser swap:

• poetry install / poetry check currently fails because the new Markdown dependencies were added to pyproject.toml but the lockfile is not in sync.
• Image rendering changes shape from the existing nested captionedImage -> image2.attrs structure to a flattened captionedImage node. That may be valid, but it is a compatibility change from the current library behavior, so I’d prefer this PR preserve the existing image schema unless we have verified the new one against Substack.

If you want to keep going with this PR, I think the next step is to rebase/merge main now that #56 is merged, regenerate the lockfile, and make the renderer preserve the current draft node shapes while still using markdown-it-py for parsing.

[MattFisher]

Awesome I’ll do that

…

On Sat, 27 Jun 2026 at 7:23 pm, Paolo Mazza ***@***.***> wrote: *ma2za* left a comment (ma2za/python-substack#61) <#61 (comment)> Hi, thanks for this. I merged #56 <#56> first because it was the lower-risk incremental path for footnotes and kept the existing from_markdown output shape intact. I like the direction here. Moving Markdown rendering onto markdown-it-py is the cleaner long-term approach, but I don’t want to merge this PR as-is yet because it changes a couple of things beyond the parser swap: - poetry install / poetry check currently fails because the new Markdown dependencies were added to pyproject.toml but the lockfile is not in sync. - Image rendering changes shape from the existing nested captionedImage -> image2.attrs structure to a flattened captionedImage node. That may be valid, but it is a compatibility change from the current library behavior, so I’d prefer this PR preserve the existing image schema unless we have verified the new one against Substack. If you want to keep going with this PR, I think the next step is to rebase/merge main now that #56 <#56> is merged, regenerate the lockfile, and make the renderer preserve the current draft node shapes while still using markdown-it-py for parsing. — Reply to this email directly, view it on GitHub <#61?email_source=notifications&email_token=AAARHQUG3RZP6NVMHMGQXDD5B6G7ZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOBRGYZTMNJZGM32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4816365937>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAARHQXARSPWDVUG7JV4Q6L5B6G7ZAVCNFSNUABFKJSXA33TNF2G64TZHM2TANZXGA4DGMBUHNEXG43VMU5TINZUHA4DEOJYG4YKC5QC> . You are receiving this because you authored the thread.Message ID: ***@***.***>