Repository navigation
uuid.uuid1() is too slow #50135
Description
Activity
wangchun commented
on Apr 30, 2009 wangchunmannequinMannequinAuthoruuid.uuid1() currently uses two different ways to generate a uuid. If
the system call "uuid_generate_time" is available, uuid1() uses the
system call via the ctypes interface, otherwise, it uses pure Python
code to generate a uuid. The problem is, the C interface
"uuid_generate_time" is even slower than the Python code. The ctypes
interface is too slow. According to my test, it took 55 microseconds to
generate a uuid via ctypes interface but only 45 microseconds via the
Python code. I also tried to test the performance of the
"uuid_generate_time" C API itself. It takes C code 12 microseconds. Most
of the time were wasted on ctypes. I believe we need to drop ctypes and
write a Python extensions in C for this job.wangchun commented
on Apr 30, 2009 wangchunmannequinMannequinAuthorThis is my test on another faster machine.
$ cat test.py import sys, time, uuid N = int(sys.argv[1]) t = time.time() for x in xrange(N): uuid.uuid1() print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / N)) $ cat test.c #include <stdio.h> #include <sys/time.h> #include <uuid/uuid.h> int main(int argc, char *argv[]) { int i, n; double t1, t2; uuid_t uuid; struct timeval t; struct timezone tz; sscanf(argv[1], "%d", &n); gettimeofday(&t, &tz); t1 = (double)t.tv_sec + (double)t.tv_usec / 1000000.0; for (i = 0; i < n; i++) { uuid_generate_time(uuid); } gettimeofday(&t, &tz); t2 = (double)t.tv_sec + (double)t.tv_usec / 1000000.0; printf("%.3f microseconds\n", (t2 - t1) * 1000000.0 / n); return 0; } $ gcc -l uuid -o test test.c $ python test.py 50000 25.944 microseconds $ python test.py 200000 25.810 microseconds $ python test.py 1000000 25.865 microseconds $ ./test 50000 0.214 microseconds $ ./test 200000 0.214 microseconds $ ./test 1000000 0.212 microseconds $
Can you provide a patch?
grooverdan commented
on Sep 1, 2009 grooverdanmannequinMannequinThis is a slightly crude module version. The speedups were only 10%
Python 3.2a0 (py3k:74612M, Sep 1 2009, 18:11:58)
[GCC 4.3.2] on linux2Using the same test from Wang Chun:
before:
uuid1(1000000)
101.759 microsecondsafter:
uuid1(1000000)
91.663 microsecondsThe delays are clearly in the _byte array copying as indicated by the test below: >>> import sys, time, uuid >>> def uu(n): ... t = time.time() ... for x in range(n): ... uuid._uuid_generate_time_fast() ... print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n)) ... [72265 refs] >>> uu(1000000) 13.157 microseconds [72267 refs]
I would expect fixing this for the ctypes version would have a similar
speedup.grooverdan commented
on Sep 1, 2009 grooverdanmannequinMannequinto prove it a bit more - ctype benchmark >>> import ctypes, ctypes.util >>> def uu1(n): ... t = time.time() ... _buffer = ctypes.create_string_buffer(16) ... for x in range(n): ... uuid._uuid_generate_time(_buffer) ... print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n)) ... >>> uu1(1000000) 15.819 microseconds
rosslagerwall commented
on Jan 22, 2011 rosslagerwallmannequinMannequinAttached is a patch based on the original patch, meant to have better performance.
On my PC, this:
import sys, time, uuid def uu(n): t = time.time() for x in range(n): uuid.uuid1() print('%.3f microseconds' % ((time.time() - t) * 1000000.0 / n)) uu(50000)
records a time of 38.5 microseconds unpatched (still using ctypes/libuuid) and a time of 16.5 microseconds afterwards.
uuid4() results in an improvement from 30 microseconds to 9 microseconds. From what I could see, what took the most time was the call to UUID() with a bytes object. That's why this patch passes in the uuid as a long.It also fixes setup.py to check for the uuid.h header.
Instead hexadecimals in _long_from_uuid_t you can use _PyLong_FromByteArray.
However adding new C implemented module has hight cost. I doubt that the speed up of UUID generation is worth this cost.
I agree that there is a maintenance cost associated with C extension modules. However, I would certainly be glad if it allowed us to eliminate uses of ctypes in this module because ctypes is quite unsafe and doesn't work well across platforms (though it is admittedly very convenient).
The original report says the ctypes call is slower than the python code used as a fallback. Would it not, then, be a performance improvement just to drop the ctypes call, without creating a new C module? Creating a C module would then be a separate enhancement issue if someone thought the performance improvement was enough to justify the module. Or maybe it could live in the os module?
And bpo-15206. Python implementation has a drawback.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: