Hacker GitHub

← All repos

lemire/fastvalidate-utf-8

View on GitHub ↗ ★ 311 C

header-only library to validate utf-8 strings at high speeds (using SIMD instructions)

3 mentions on Hacker News by 2 people: 0 stories, 3 comments · first 2018-05-16 · latest 2019-11-16

Mentions per year 3 mentions · busiest year 2019 (2)
20082026
AllStoriesComments NewestTopOldest
  1. …A good starting point would be looking at the various implementations benchmarked here: https://github.com/lemire/fastvalidate-utf-8 It looks like the main naive validator there (validate_utf8) clocks in at 1.25 cycles/byte for ASCII and…
    comment by zwegner · 2019-11-16 · in Show HN: Faster UTF-8 validator
  2. …generated from the code in Daniel Lemire's fastvalidate-utf-8 repository, specifically this code: https://github.com/lemire/fastvalidate-utf-8/blob/ed53c0c64b... Looking closer at it, I think that the distribution of random UTF-8 is not as…
    comment by zwegner · 2019-11-16 · in Show HN: Faster UTF-8 validator
  3. …check the 8th bit though. On Skylake that would cut it to 0.33 cycles per 16 bytes (HSW 1 per 16). https://github.com/lemire/fastvalidate-utf-8/pull/2
    comment by zbjornson · 2018-05-16 · in Validating UTF-8 strings using as little as 0.7 cycles per byte