Bad interaction between kerning and non-latin1 characters in pdf output #18282

Bug report

Bug summary

Non-latin1 character occuring after strings that involve large kerning corrections are misplaced in the pdf output.

Code for reproduction

from pylab import *
figtext(.5, .5, "AVAVAVAVAVAVAVA€")
savefig("/tmp/test.pdf")

Actual outcome

(screenshot of the pdf)

Expected outcome

The € sign should come at the end.

This bisects to #14940, before which kerning was reduced by a factor of 64 and thus the effect would have been still present but unnoticeable except for extreeeemely long strings.

My analysis of the bug is as follows:
First, note that for Type3 output (the default), non-latin1 characters are written to the pdf file separately from the rest of the string. I believe(?) that's a fundamental limitation of the pdf spec (see pdf spec. section 9.4.3 Text Showing Operators). (We could generate a custom encoding to only cover the glyphs we want, and split the font into multiple independent Type3 subsets if the string has more than 255 different glyphs, I guess...) So we measure the widths of the latin1 substrings (using standard kerning) and position the other glyphs relative to these.
However, when subsetting the font as a Type3 font (the default), kerning information is lost. Therefore, the pdf software (okular/etc.) draws the string "AVAVAVAVA" without kerning; in this case it is much wider than expected by Matplotlib and overflows the position where the € sign is drawn.
I believe the solutions are either to

• Add an unkerned mode to _text_layout.layout and use that to measure the width of the latin1 substrings in this specific case. This isn't really worse than pre-Fix text kerning calculations and some FT2Font cleanup #14940, where we effectively never kerned.
• Use the showkern (TJ) operator instead of show (Tj) to include the kerning values in the outputted string in the pdf file (we already do this for usetex+pdf, in fact). I guess this is technically more correct.

Matplotlib version

• Operating system: fedora
• Matplotlib version: master, bisects to Fix text kerning calculations and some FT2Font cleanup #14940
• Matplotlib backend (print(matplotlib.get_backend())): pdf
• Python version: 38
• Jupyter version (if applicable):
• Other libraries: