Skip to content

Simpler ASCII printable length #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
clipperhouse merged 6 commits into from
Jan 25, 2026
+194 −301
Merged

Simpler ASCII printable length #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9f9d739
Simpler ASCII printable length
clipperhouse Jan 22, 2026
63eb885
coupla more tests
clipperhouse Jan 23, 2026
1c9606c
remove references to 8 byte boundaries
clipperhouse Jan 25, 2026
4ed112d
defensive check
clipperhouse Jan 25, 2026
515dd8a
typo
clipperhouse Jan 25, 2026
79443a7
typo
clipperhouse Jan 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 27 additions & 32 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,19 +97,14 @@ This package implements the Unicode East Asian Width standard
([UAX #11](https://www.unicode.org/reports/tr11/tr11-43.html)), and handles
[version selectors](https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)),
and [regional indicator pairs](https://en.wikipedia.org/wiki/Regional_indicator_symbol)
(flags). We implement [Unicode TR51](https://www.unicode.org/reports/tr51/tr51-27.html). We are keeping
an eye on [emerging standards](https://www.jeffquast.com/post/state-of-terminal-emulation-2025/).

(flags). We implement [Unicode TR51](https://www.unicode.org/reports/tr51/tr51-27.html)
for emojis. We are keeping an eye on
[emerging standards](https://www.jeffquast.com/post/state-of-terminal-emulation-2025/).

`clipperhouse/displaywidth`, `mattn/go-runewidth`, and `rivo/uniseg` will
give the same outputs for most real-world text. Extensive details are in the
[compatibility analysis](comparison/COMPATIBILITY_ANALYSIS.md).

If you wish to investigate the core logic, see the `lookupProperties` and `width`
functions in [width.go](width.go#L139). The essential trie generation logic is in
`buildPropertyBitmap` in [unicode.go](internal/gen/unicode.go#L316).


## Prior Art

[mattn/go-runewidth](https://github.com/mattn/go-runewidth)
Expand All @@ -133,39 +128,39 @@ goarch: arm64
pkg: github.com/clipperhouse/displaywidth/comparison
cpu: Apple M2

BenchmarkString_Mixed/clipperhouse/displaywidth-8 6685 ns/op 252.36 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 13952 ns/op 120.92 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19415 ns/op 86.89 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/clipperhouse/displaywidth-8 5460 ns/op 308.96 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 14301 ns/op 117.96 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19562 ns/op 86.24 MB/s 0 B/op 0 allocs/op

BenchmarkString_EastAsian/clipperhouse/displaywidth-8 6857 ns/op 246.02 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23316 ns/op 72.35 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19272 ns/op 87.54 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 5546 ns/op 304.20 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23801 ns/op 70.88 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19768 ns/op 85.34 MB/s 0 B/op 0 allocs/op

BenchmarkString_ASCII/clipperhouse/displaywidth-8 178.6 ns/op 716.77 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1164 ns/op 110.01 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1578 ns/op 81.13 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/clipperhouse/displaywidth-8 54.58 ns/op 2345.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1167 ns/op 109.73 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1577 ns/op 81.17 MB/s 0 B/op 0 allocs/op

BenchmarkString_Emoji/clipperhouse/displaywidth-8 3169 ns/op 228.43 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4664 ns/op 155.23 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6525 ns/op 110.95 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3127 ns/op 231.51 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4722 ns/op 153.31 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6562 ns/op 110.34 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3328 ns/op 506.96 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5327 ns/op 316.66 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3452 ns/op 488.68 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5367 ns/op 314.33 MB/s 0 B/op 0 allocs/op

BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3371 ns/op 500.37 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15306 ns/op 110.22 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3757 ns/op 449.06 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15390 ns/op 109.62 MB/s 0 B/op 0 allocs/op

BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.7 ns/op 498.66 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 262.5 ns/op 487.58 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.3 ns/op 499.40 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 262.3 ns/op 487.91 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1327 ns/op 545.59 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2212 ns/op 327.26 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1436 ns/op 504.16 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2267 ns/op 319.32 MB/s 0 B/op 0 allocs/op

BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 3804 ns/op 46.53 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 8097 ns/op 21.86 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 3120 ns/op 56.73 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 8134 ns/op 21.76 MB/s 192 B/op 14 allocs/op

BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 3426 ns/op 66.84 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 10441 ns/op 21.93 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 3427 ns/op 66.82 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 10410 ns/op 22.00 MB/s 0 B/op 0 allocs/op
```

Here are some notes on [how to make Unicode things fast](https://clipperhouse.com/go-unicode/).
57 changes: 28 additions & 29 deletions comparison/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
## Compatibility

In real-world text, you mostly should see the same outputs from
`clipperhouse/displaywidth`,`mattn/go-runewidth`, and `rivo/uniseg`. It's
mostly the same data and logic.
In real-world text, you should see the same outputs from
`clipperhouse/displaywidth`, `mattn/go-runewidth`, and `rivo/uniseg`.

The tests in this package exercise the behaviors of the three libraries.
Extensive details are available in the
The tests in this `comparison` package exercise the behaviors of the three
libraries. Extensive details are available in the
[compatibility analysis](COMPATIBILITY_ANALYSIS.md).

## Benchmarks
Expand All @@ -20,37 +19,37 @@ goarch: arm64
pkg: github.com/clipperhouse/displaywidth/comparison
cpu: Apple M2

BenchmarkString_Mixed/clipperhouse/displaywidth-8 10400 ns/op 162.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 14296 ns/op 118.00 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19770 ns/op 85.33 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/clipperhouse/displaywidth-8 5460 ns/op 308.96 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/mattn/go-runewidth-8 14301 ns/op 117.96 MB/s 0 B/op 0 allocs/op
BenchmarkString_Mixed/rivo/uniseg-8 19562 ns/op 86.24 MB/s 0 B/op 0 allocs/op

BenchmarkString_EastAsian/clipperhouse/displaywidth-8 10593 ns/op 159.26 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23980 ns/op 70.35 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19777 ns/op 85.30 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/clipperhouse/displaywidth-8 5546 ns/op 304.20 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/mattn/go-runewidth-8 23801 ns/op 70.88 MB/s 0 B/op 0 allocs/op
BenchmarkString_EastAsian/rivo/uniseg-8 19768 ns/op 85.34 MB/s 0 B/op 0 allocs/op

BenchmarkString_ASCII/clipperhouse/displaywidth-8 1032 ns/op 124.09 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1162 ns/op 110.16 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1586 ns/op 80.69 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/clipperhouse/displaywidth-8 54.58 ns/op 2345.21 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/mattn/go-runewidth-8 1167 ns/op 109.73 MB/s 0 B/op 0 allocs/op
BenchmarkString_ASCII/rivo/uniseg-8 1577 ns/op 81.17 MB/s 0 B/op 0 allocs/op

BenchmarkString_Emoji/clipperhouse/displaywidth-8 3017 ns/op 240.01 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4745 ns/op 152.58 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6745 ns/op 107.34 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/clipperhouse/displaywidth-8 3127 ns/op 231.51 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/mattn/go-runewidth-8 4722 ns/op 153.31 MB/s 0 B/op 0 allocs/op
BenchmarkString_Emoji/rivo/uniseg-8 6562 ns/op 110.34 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3381 ns/op 498.90 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5383 ns/op 313.41 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/clipperhouse/displaywidth-8 3452 ns/op 488.68 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Mixed/mattn/go-runewidth-8 5367 ns/op 314.33 MB/s 0 B/op 0 allocs/op

BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3395 ns/op 496.96 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15645 ns/op 107.83 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/clipperhouse/displaywidth-8 3757 ns/op 449.06 MB/s 0 B/op 0 allocs/op
BenchmarkRune_EastAsian/mattn/go-runewidth-8 15390 ns/op 109.62 MB/s 0 B/op 0 allocs/op

BenchmarkRune_ASCII/clipperhouse/displaywidth-8 257.8 ns/op 496.57 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 267.3 ns/op 478.89 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/clipperhouse/displaywidth-8 256.3 ns/op 499.40 MB/s 0 B/op 0 allocs/op
BenchmarkRune_ASCII/mattn/go-runewidth-8 262.3 ns/op 487.91 MB/s 0 B/op 0 allocs/op

BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1338 ns/op 541.24 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2287 ns/op 316.58 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/clipperhouse/displaywidth-8 1436 ns/op 504.16 MB/s 0 B/op 0 allocs/op
BenchmarkRune_Emoji/mattn/go-runewidth-8 2267 ns/op 319.32 MB/s 0 B/op 0 allocs/op

BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 3689 ns/op 47.98 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 8069 ns/op 21.93 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/clipperhouse/displaywidth-8 3120 ns/op 56.73 MB/s 192 B/op 14 allocs/op
BenchmarkTruncateWithTail/mattn/go-runewidth-8 8134 ns/op 21.76 MB/s 192 B/op 14 allocs/op

BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 3457 ns/op 66.24 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 10441 ns/op 21.93 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/clipperhouse/displaywidth-8 3427 ns/op 66.82 MB/s 0 B/op 0 allocs/op
BenchmarkTruncateWithoutTail/mattn/go-runewidth-8 10410 ns/op 22.00 MB/s 0 B/op 0 allocs/op
```
135 changes: 34 additions & 101 deletions width.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package displaywidth

import (
"unicode/utf8"
"unsafe"

"github.com/clipperhouse/stringish"
"github.com/clipperhouse/uax29/v2/graphemes"
Expand Down Expand Up @@ -34,33 +33,34 @@ func (options Options) String(s string) int {
pos := 0

for pos < len(s) {
// Try ASCII optimization (need >= 8 bytes for it to be worth it)
// Try ASCII optimization
asciiLen := printableASCIILength(s[pos:])
if asciiLen > 0 {
width += asciiLen
pos += asciiLen
continue
}

// Not ASCII (or < 8 bytes), use grapheme parsing
// Not ASCII, use grapheme parsing
g := graphemes.FromString(s[pos:])
start := pos

hitASCII := false
for g.Next() {
width += graphemeWidth(g.Value(), options)
absEnd := pos + g.End()
v := g.Value()
width += graphemeWidth(v, options)
pos += len(v)

// Quick check: if remaining might be an ASCII run, break to outer loop
if len(s)-absEnd >= 8 && s[absEnd] >= 0x20 && s[absEnd] <= 0x7E {
pos = absEnd
hitASCII = true
// Quick check: if remaining might have printable ASCII, break to outer loop
if pos < len(s) && s[pos] >= 0x20 && s[pos] <= 0x7E {
break
}
}

if !hitASCII {
// Consumed all remaining via graphemes
break
// Defensive, should not happen: if no progress was made,
// skip a byte to prevent infinite loop. Only applies if
// the grapheme parser misbehaves.
if pos == start {
pos++
}
}

Expand All @@ -81,33 +81,34 @@ func (options Options) Bytes(s []byte) int {
pos := 0

for pos < len(s) {
// Try ASCII optimization (need >= 8 bytes for it to be worth it)
asciiLen := printableASCIILengthBytes(s[pos:])
// Try ASCII optimization
asciiLen := printableASCIILength(s[pos:])
if asciiLen > 0 {
width += asciiLen
pos += asciiLen
continue
}

// Not ASCII (or < 8 bytes), use grapheme parsing
// Not ASCII, use grapheme parsing
g := graphemes.FromBytes(s[pos:])
start := pos

hitASCII := false
for g.Next() {
width += graphemeWidth(g.Value(), options)
absEnd := pos + g.End()
v := g.Value()
width += graphemeWidth(v, options)
pos += len(v)

// Quick check: if remaining might be an ASCII run, break to outer loop
if len(s)-absEnd >= 8 && s[absEnd] >= 0x20 && s[absEnd] <= 0x7E {
pos = absEnd
hitASCII = true
// Quick check: if remaining might have printable ASCII, break to outer loop
if pos < len(s) && s[pos] >= 0x20 && s[pos] <= 0x7E {
break
}
}

if !hitASCII {
// Consumed all remaining via graphemes
break
// Defensive, should not happen: if no progress was made,
// skip a byte to prevent infinite loop. Only applies if
// the grapheme parser misbehaves.
if pos == start {
pos++
}
}

Expand Down Expand Up @@ -263,90 +264,22 @@ func asciiWidth(b byte) int {
}

// printableASCIILength returns the length of consecutive printable ASCII bytes
// starting at the beginning of s. Returns -1 if fewer than 8 consecutive
// printable ASCII bytes are found (not worth optimizing). Uses SWAR to check
// 8 bytes at a time.
func printableASCIILength(s string) int {
if len(s) < 8 {
return -1
}

// starting at the beginning of s.
func printableASCIILength[T string | []byte](s T) int {
i := 0
for ; i+8 <= len(s); i += 8 {
x := *(*uint64)(unsafe.Add(unsafe.Pointer(unsafe.StringData(s)), i))
// Check for non-ASCII (high bit set)
if x&0x8080808080808080 != 0 {
break
}
// Check for control chars (< 0x20): add 0x60, printable bytes overflow to set high bit
if (x+0x6060606060606060)&0x8080808080808080 != 0x8080808080808080 {
break
}
// Check for DEL (0x7F) using zero-byte detection
xored := x ^ 0x7F7F7F7F7F7F7F7F
if ((xored - 0x0101010101010101) & ^xored & 0x8080808080808080) != 0 {
for ; i < len(s); i++ {
b := s[i]
// Printable ASCII is 0x20-0x7E (space through tilde)
if b < 0x20 || b > 0x7E {
break
}
}

// If we didn't get at least 8 bytes, not worth optimizing
if i == 0 {
return -1
}

// If the next byte is non-ASCII (>= 0x80), back off by 1. The grapheme
// parser may group the last ASCII byte with subsequent non-ASCII bytes,
// such as combining marks.
if i < len(s) && s[i] >= 0x80 {
if i > 0 && i < len(s) && s[i] >= 0x80 {
i--
if i < 8 {
return -1
}
}

return i
}

// printableASCIILengthBytes returns the length of consecutive printable ASCII bytes
// starting at the beginning of s. Returns -1 if fewer than 8 consecutive
// printable ASCII bytes are found (not worth optimizing). Uses SWAR to check
// 8 bytes at a time.
func printableASCIILengthBytes(s []byte) int {
if len(s) < 8 {
return -1
}

i := 0
for ; i+8 <= len(s); i += 8 {
x := *(*uint64)(unsafe.Pointer(&s[i]))
// Check for non-ASCII (high bit set)
if x&0x8080808080808080 != 0 {
break
}
// Check for control chars (< 0x20): add 0x60, printable bytes overflow to set high bit
if (x+0x6060606060606060)&0x8080808080808080 != 0x8080808080808080 {
break
}
// Check for DEL (0x7F) using zero-byte detection
xored := x ^ 0x7F7F7F7F7F7F7F7F
if ((xored - 0x0101010101010101) & ^xored & 0x8080808080808080) != 0 {
break
}
}

// If we didn't get at least 8 bytes, not worth optimizing
if i == 0 {
return -1
}

// If the next byte is non-ASCII (>= 0x80), back off by 1. The grapheme
// parser may group the last ASCII byte with subsequent non-ASCII bytes,
// such as combining marks.
if i < len(s) && s[i] >= 0x80 {
i--
if i < 8 {
return -1
}
}

return i
Expand Down
Loading