[multiline] wordwrap for non-single-width characters (CJK)

Small description
Only english text works correctly when using multiline command (wrap text)

Expected result
Wrap text should work for all languages

Actual result with screenshot

sample.csv

Steps to reproduce
open a file -> execute multiline command (v or View -> toggle display -> multiline)

Configuration

Does this issue reproduce without any plugins or configuration (using the -N CLI flag)?
Yes
Does this issue reproduce with either the latest release, or with the develop branch?
Latest release v3.0.2

Additional context

What platform and version are you using (Linux, MacOS, Windows)?
Ubuntu 22.04
Which version of Python?
3.11.5
Which terminal are you using (for display and input issues)?
iTerm2 with macOS and use the ubuntu remotly

Can you provide sample data like that used in the screenshot?

yes i've updated issue

I looked into this a bit, for Python 3.10.12. The trouble seems to be the use of textwrap.TextWrapper._wrap_chunks(). The wrapping calculations there use len() for strings, where visidata would use their display width dispwidth(s). As a quick check, I seem to get proper wrapping when I use a modified version of _wrap_chunks() for sheets.py.

def dispwidth_wrap_chunks(self, chunks):
    len = dispwidth
    # the rest of the function is unmodified from the original _wrap_chunks()
    ...
textwrap.TextWrapper._wrap_chunks = dispwidth_wrap_chunks

So one solution would be to subclass textwrap.TextWrapper to override _wrap_chunks() with a version that uses dispwidth() instead of len(). Then replace textwrap.wrap() with wtw = WideTextWrapper(); wtw.wrap(). Overriding like that is a solution that textwrap puts forward:

If you want to completely replace the main wrapping algorithm,
you'll probably have to override _wrap_chunks().

Several other projects subclass TextWrapper to override _wrap_chunks(), in order to handle variable width characters:

Here is the Python PR (open since 2021) that could fix it: python/cpython#28136

It's not perfect, so it's not been merged. I suggest we throw our weight behind that, and/or help it become more perfect. In the meantime, if someone wants to take that code into VisiData immediately, in visidata/_textwrap.py, such that if/when the Python PR is merged, we can use the Python implementation for newer versions and then ultimately delete our interim implementation, then we can merge that to close this issue and do some field testing for the Python PR and prove its mettle.

It looks like the latest version of jquast's wcwidth solves this problem. wcwidth.wrap() appears to be a drop-in replacement for the textwrap.wrap() that we currently use, that wraps wide characters properly.

[cliptext] wrap text with jquast/wcwidth when installed #2416 #2685

[cliptext] wraplines for all text wrapping #2416 #2685

Navigation Menu

Repository navigation

[multiline] wordwrap for non-single-width characters (CJK) #2416

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[multiline] wordwrap for non-single-width characters (CJK) #2416

Description

Activity

saulpw commented on May 30, 2024

duofinal commented on May 30, 2024

midichef commented on Jun 2, 2024

saulpw commented on Jun 5, 2024

midichef commented on Mar 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions