Activity

1. sobolevn

   added

   type-featureA feature request or enhancementA feature request or enhancement

   performancePerformance or resource usagePerformance or resource usage

   on Oct 12, 2022
2. mdboom commented on Oct 13, 2022

   

   mdboom

   on Oct 13, 2022

   Contributor

   More actions

   I'd generally be in favor of removing almost-duplicate code if there is no performance benefit.

   Have you done any analysis with the pyperformance benchmarks?
3. vstinner commented on Oct 13, 2022

   

   vstinner

   on Oct 13, 2022

   Member

   More actions

   ASCII-only strings have multiple efficient properties. Examples:

   • Looking for a non-ASCII character in an ASCII string has complexity of O(1): it's always false
   • _PyUnicode_FromASCII() doesn't have to scan the string to compute the maximum code path: it's known that the string is ASCII
   • Maximum code point is 127 (STRINGLIB_MAX_CHAR=127 for asciilib functions)

   Some optimizations are implemented outside stringlib. Like:

   static Py_ssize_t
   any_find_slice(PyObject* s1, PyObject* s2, ...)
   {
       ...
       kind1 = PyUnicode_KIND(s1);
       kind2 = PyUnicode_KIND(s2);
       if (kind1 < kind2)
           return -1;
       ...
   }
   
4. sobolevn commented on Oct 13, 2022

   

   sobolevn

   on Oct 13, 2022

   MemberAuthor

   More actions

   Have you done any analysis with the pyperformance benchmarks?

   Only very limited ones in #98025 (comment) and #98228 (comment)
5. serhiy-storchaka commented on Oct 14, 2022

   

   serhiy-storchaka

   on Oct 14, 2022

   Member

   More actions

   Please perform tests also with longer strings (100, 1000, 10000 characters or more). Also test with strings containing non-ASCII whitespace characters \x85 and \xa0.
6. rhpvorderman commented on Jan 24, 2023

   

   rhpvorderman

   on Jan 24, 2023

   Contributor

   More actions

   If you want to work with some real-world ASCII data you can download a FASTQ file named SRR15096500_1.fastq.gz here https://www.ebi.ac.uk/ena/browser/view/SRR15096500.
   FASTQ is a format to store DNA (or RNA) sequencing reads. You can find a description of the format here: https://en.wikipedia.org/wiki/FASTQ_format.

   The data can be quite big. Whole genome sequencing can yield FASTQ files that are 100GB when compressed with gzip.

   @marcelm has created a nice library called dnaio for parsing the format. To add to @vstinner ASCII has some really interesting other properties to (to complement @vstinner):

   • The last 5 bits of lowercase and uppercase characters are the same. Allowing quick case-independent comparing.
   • The last 4 bits of 0, 1, 2, 3, are 0000, 0001, 0010, 0011.. you get the idea
   • The fact that the highest code point is at most 127 allows you to create reasonably small lookup tables with only 128 entries.
   • The most significant bit of ASCII is always 0. So you can do a very quick check if strings are ascii by doing a Bitwise OR on all characters and checking if the result's significant bit is also zero. That trick is used in dnaio and runs at approximately 50GB/s (using SSE2 instructions) or 20GB/s (without). This could potentially be used in UTF-8 parsing.

   Anyway. I am curious to see if asciilib works well when using real-world data such as posted above. A lot of formats in my field are ASCII, so Python's performance in this area matters to me.

   👍React with 👍1Reacted by Marcel Martin
7. vstinner commented on Apr 5, 2023

   

   vstinner

   on Apr 5, 2023

   Member

   More actions

   This could potentially be used in UTF-8 parsing.

   The utf8_decode() function of the Python "stringlib" internal library uses this property to check if a string is ASCII-only in an efficient way. It even works on size_t words (ex: 8 bytes on 64-bit systems) per loop iteration. See: Objects/stringlib/codecs.h.
8. rhpvorderman commented on Apr 7, 2023

   

   rhpvorderman

   on Apr 7, 2023

   Contributor

   More actions

   The utf8_decode() function of the Python "stringlib" internal library uses this property to check if a string is ASCII-only in an efficient way.

   @vstinner I am aware that the current ASCII check exists and is quite efficient. However, it can be more efficient: #101289

   The current ASCII check is per string and involves a lot of code. A quick check on the input buffer is almost cost-free. The copying can then proceed using only PyUnicode_New(127) and memcpy. That is much faster. Although it will require some reworking of the ASCII/UTF-8 code.
9. iritkatriel

   added

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   on Nov 27, 2023