Benchmark results

Encoder

Collection of bench results for various platforms and devices.
If you want to submit info about your device, simply run the bench tool or the extra/bench-all.sh and report the results in the comments below.

Suggestions for better summary of the results are welcome

CPU	OS	Config	Model	Th	Load	Enc.	Commit
MacBook M1 Pro	MacOS 13.0.1	NEON BLAS	tiny	8	71	102	`206fc93`
MacBook M1 Pro	MacOS 13.0.1	NEON BLAS	base	8	96	220	`206fc93`
MacBook M1 Pro	MacOS 13.0.1	NEON BLAS	small	8	233	685	`206fc93`
MacBook M1 Pro	MacOS 13.0.1	NEON BLAS	medium	8	603	1928	`206fc93`
MacBook M1 Pro	MacOS 13.0.1	NEON BLAS	large	8	1158	3350	`206fc93`
---
MacBook M1 Pro	MacOS 13.0.1	NEON BLAS	small	1	251	2605	`206fc93`
MacBook M1 Pro	MacOS 13.0.1	NEON BLAS	small	4	255	884	`206fc93`
---
Mac Mini M1	MacOS	NEON BLAS	tiny	4	62	194	`fcf515d`
Mac Mini M1	MacOS	NEON BLAS	base	4	81	380	`fcf515d`
Mac Mini M1	MacOS	NEON BLAS	small	4	204	1249	`fcf515d`
Mac Mini M1	MacOS	NEON BLAS	medium	4	876	3980	`fcf515d`
Mac Mini M1	MacOS	NEON BLAS	large	4	1876	7979	`fcf515d`
---
Ryzen 9 3900X	Ubuntu 20.04	AVX2	tiny	8	107	422	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2	base	8	137	880	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2	small	8	280	2874	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2	medium	8	692	9610	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2	large	8	1317	16917	`fcf515d`
---
Ryzen 9 3900X	Ubuntu 20.04	AVX2 BLAS	tiny	4	120	780	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2 BLAS	base	4	151	1173	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2 BLAS	small	4	289	3062	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2 BLAS	medium	4	711	9175	`fcf515d`
Ryzen 9 3900X	Ubuntu 20.04	AVX2 BLAS	large	4	1282	16050	`fcf515d`
---
Ryzen 9 5950X	Ubuntu 22.04	AVX2	tiny	8	135	197	`fcf515d`
Ryzen 9 5950X	Ubuntu 22.04	AVX2	base	8	176	421	`fcf515d`
Ryzen 9 5950X	Ubuntu 22.04	AVX2	small	8	357	1393	`fcf515d`
Ryzen 9 5950X	Ubuntu 22.04	AVX2	medium	8	855	4404	`fcf515d`
Ryzen 9 5950X	Ubuntu 22.04	AVX2	large	8	1576	8118	`fcf515d`
---
Raspberry Pi 4		NEON	tiny	4	1436	13839	`fcf515d`
Raspberry Pi 4		NEON	base	4	1894	30552	`fcf515d`
---
iPhone 13 Mini	iOS 16.0	NEON BLAS	base	4	97	1091	`fcf515d`
---
MacBook M1 Pro	Vivaldi	WASM	tiny	8	133	3785	`fcf515d`
MacBook M1 Pro	Vivaldi	WASM	base	8	172	8253	`fcf515d`
---
MacBook M1 Pro	Chrome	WASM	tiny	8	134	3776	`fcf515d`
MacBook M1 Pro	Chrome	WASM	base	8	168	8200	`fcf515d`
---
MacBook M1 Pro	Firefox	WASM	tiny	8	137	2626	`fcf515d`
MacBook M1 Pro	Firefox	WASM	base	8	183	6226	`fcf515d`

memcpy

MacBook M1 Pro

./bench -w 1 -t 1
memcpy: 37.59 GB/s

Ryzen 9 5950X

./bench -w 1 -t 1
memcpy: 16.74 GB/s

ggml_mul_mat

MacBook M1 Pro

./bench -w 2 -t 1
ggml_mul_mat:    64 x    64: F16    330.6 GFLOPS (128 runs) / F32    466.0 GFLOPS (128 runs)
ggml_mul_mat:   128 x   128: F16    737.5 GFLOPS (128 runs) / F32    838.9 GFLOPS (128 runs)
ggml_mul_mat:   256 x   256: F16    938.6 GFLOPS (128 runs) / F32   1062.3 GFLOPS (128 runs)
ggml_mul_mat:   512 x   512: F16   1312.5 GFLOPS (128 runs) / F32   1835.5 GFLOPS (128 runs)
ggml_mul_mat:  1024 x  1024: F16   1765.1 GFLOPS (128 runs) / F32   2041.4 GFLOPS (128 runs)
ggml_mul_mat:  2048 x  2048: F16   1784.3 GFLOPS (104 runs) / F32   1859.2 GFLOPS (109 runs)
ggml_mul_mat:  4096 x  4096: F16   1855.1 GFLOPS ( 14 runs) / F32   1873.3 GFLOPS ( 14 runs)

Ryzen 9 5950X

WHISPER_OPENBLAS=1 make -j bench && ./bench -w 2 -t 1
ggml_mul_mat:    64 x    64: F16     56.3 GFLOPS (128 runs) / F32     70.2 GFLOPS (128 runs)
ggml_mul_mat:   128 x   128: F16     47.8 GFLOPS (128 runs) / F32     67.0 GFLOPS (128 runs)
ggml_mul_mat:   256 x   256: F16    185.1 GFLOPS (128 runs) / F32    332.7 GFLOPS (128 runs)
ggml_mul_mat:   512 x   512: F16    386.4 GFLOPS (128 runs) / F32    658.6 GFLOPS (128 runs)
ggml_mul_mat:  1024 x  1024: F16    636.2 GFLOPS (128 runs) / F32   1012.0 GFLOPS (128 runs)
ggml_mul_mat:  2048 x  2048: F16    950.9 GFLOPS ( 56 runs) / F32   1296.8 GFLOPS ( 76 runs)
ggml_mul_mat:  4096 x  4096: F16   1168.6 GFLOPS (  9 runs) / F32   1403.1 GFLOPS ( 11 runs)

Results for Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz

CPU	OS	Model	Threads	Load [ms]	Encode [ms]
i7-4790K	Debian	tiny.en	4	165	808
i7-4790K	Debian	tiny.en	8	165	783
i7-4790K	Debian	base.en	4	212	1813
i7-4790K	Debian	base.en	8	214	1746

Results for Ryzen 5 4500U 6C/6T laptop CPU (I've just included one result for 8 threads as Encode time is much higher when threads > CPU cores).

CPU	OS	Model	Threads	Load [ms]	Encode [ms]
Ryzen 5 4500U (6C/6T)	Opensuse Leap	tiny.en	4	170.00	829.43
Ryzen 5 4500U (6C/6T)	Opensuse Leap	tiny.en	6	143.03	671.74
Ryzen 5 4500U (6C/6T)	Opensuse Leap	base.en	4	305.92	2,092.39
Ryzen 5 4500U (6C/6T)	Opensuse Leap	base.en	6	188.05	1,495.61
Ryzen 5 4500U (6C/6T)	Opensuse Leap	small.en	4	408.03	6,919.31
Ryzen 5 4500U (6C/6T)	Opensuse Leap	small.en	6	359.23	6,370.83
Ryzen 5 4500U (6C/6T)	Opensuse Leap	medium.en	4	2,238.11	25,863.28
Ryzen 5 4500U (6C/6T)	Opensuse Leap	medium.en	6	1,113.04	19,672.63
Ryzen 5 4500U (6C/6T)	Opensuse Leap	medium.en	8	973.65	39,619.20

CPU	OS	Config	Model	Threads	Load [ms]	Encode [ms]
i7-11800H	WSL2 Ubuntu	AVX2	tiny	2	164.35	1087.61
i7-11800H	WSL2 Ubuntu	AVX2	tiny	4	128.94	733.24
i7-11800H	WSL2 Ubuntu	AVX2	tiny	8	137.57	619.88
i7-11800H	WSL2 Ubuntu	AVX2 AVX512	tiny	2	143.02	1087.15
i7-11800H	WSL2 Ubuntu	AVX2 AVX512	tiny	4	127.60	730.57
i7-11800H	WSL2 Ubuntu	AVX2 AVX512	tiny	8	125.62	616.27
i7-11800H	WSL2 Ubuntu	AVX2 AVX512 BLAS	tiny	2	132.59	1511.38
i7-11800H	WSL2 Ubuntu	AVX2 AVX512 BLAS	tiny	4	132.48	1407.49
i7-11800H	WSL2 Ubuntu	AVX2 AVX512 BLAS	tiny	8	133.82	1458.27

CPU	OS	Config	Model	Threads	Load [ms]	Encode [ms]
i7-11800H	WSL2 Ubuntu	AVX2	base	2	174.34	2533.79
i7-11800H	WSL2 Ubuntu	AVX2	base	4	166.68	1830.67
i7-11800H	WSL2 Ubuntu	AVX2	base	8	165.53	1478.73
i7-11800H	WSL2 Ubuntu	AVX2	small	2	340.12	8714.24
i7-11800H	WSL2 Ubuntu	AVX2	small	4	394.32	6021.41
i7-11800H	WSL2 Ubuntu	AVX2	small	8	305.98	4828.84
i7-11800H	WSL2 Ubuntu	AVX2	large	2	3205.36	57109.10
i7-11800H	WSL2 Ubuntu	AVX2	large	4	2720.25	38519.89
i7-11800H	WSL2 Ubuntu	AVX2	large	8	3716.34	27739.99

CPU	OS	Config	Model	Threads	Load [ms]	Encode [ms]
i7-11800H	WSL2 Ubuntu	AVX2 AVX512	large	2	1954.21	54966.84
i7-11800H	WSL2 Ubuntu	AVX2 AVX512	large	4	1455.40	37320.62
i7-11800H	WSL2 Ubuntu	AVX2 AVX512	large	8	1372.58	27937.64

This performance is impressing!

M1 Pro | MacOS | | large | 8 | 1973 | 4208

This performance is impressing!

Yes, there is a huge performance boost due to using the built-in BLAS implementation on these devices. I will soon add OpenBLAS support for x86 architectures and see how this compares.

By the way, AVX-512 is not supported on master. I have added initial support here, but I am not sure if it works: #95

CPU	OS	Config	Model	Threads	Load[ms]	encode[ms]
Intel® Core™ i5-8250U	Win11 Home	AVX2	Large	8	2226.85	61547.61

compiled with MinGW64 gcc 11.3

Valve Jupiter (AMD Custom APU 0405, Zen 2 microarch, 4c8t, 16GB DDR5 @ 5200 MT/s)

CPU	OS	Config	Model	Threads	Load[ms]	encode[ms]
AMD Custom APU 0405	SteamOS 3.2	AVX2	Base	8	326.32	2592.96

Compiled with cc (GCC) 11.3.0

The performance gains on jfk.wav since last test (two weeks or so ago) are extremely impressive, ~10-20x speedup from 40 to 2-4 seconds.

CPU	OS	Config	Model	Threads	Load [ms]	Encode [ms]
MacBook M1 Max	macOS Ventura	BLAS	small	1	299.09	4166.00
MacBook M1 Max	macOS Ventura	BLAS	small	4	329.45	1304.32
MacBook M1 Max	macOS Ventura	BLAS	base	1	139.10	1302.17
MacBook M1 Max	macOS Ventura	BLAS	base	4	135.96	399.45

My command : C:\Users\Administrator\whisper.cpp>whisper-bench.exe -m ggml-large-v3-turbo-q5_0.bin -w 0 -t 22 -ng -fa
I copied the compiled DLL from the llama.cpp project to run the test. I'm wondering if we could adapt the RPC mechanism from llama.cpp to improve performance.

CPU Xeon Silver 4214 Dual Socket	OS	Config	Model	Th	FA	Load (ms)	Encode (ms)	Total Time (ms)	Commit
ggml-cpu-skylakex.dll (from llama.cpp)	Server 2016 Std	AVX512	ggml-large-v3-turbo-q5_0.bin	22/48	1	783.19	13928.96	25927.54	v1.8.0
ggml-cpu.dll	Server 2016 Std	AVX2	ggml-large-v3-turbo-q5_0.bin	22/48	1	832.5	14195.99	26384.29	v1.8.0
ggml-cpu-x64.dll (from llama.cpp)	Server 2016 Std	AVX2	ggml-large-v3-turbo-q5_0.bin	22/48	1	813.37	14177.88	27348.1	v1.8.0

A full output : C:\Users\Administrator\whisper.cpp>whisper-bench.exe -m ggml-large-v3-turbo-q5_0.bin -w 0 -t 22 -ng -fa load_backend: loaded RPC backend from C:\Users\Administrator\whisper.cpp\ggml-rpc.dll load_backend: loaded CPU backend from C:\Users\Administrator\whisper.cpp\ggml-cpu-skylakex.dll

system_info: n_threads = 22 / 48 | WHISPER : COREML = 0 | OPENVINO = 0 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | OPENMP = 1 | REPACK = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
whisper_init_from_file_with_params_no_state: loading model from 'ggml-large-v3-turbo-q5_0.bin'
whisper_init_with_params_no_state: use gpu = 0
whisper_init_with_params_no_state: flash attn = 1
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw = 0
whisper_init_with_params_no_state: devices = 3
whisper_init_with_params_no_state: backends = 4
whisper_model_load: loading model
whisper_model_load: n_vocab = 51866
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 1280
whisper_model_load: n_text_head = 20
whisper_model_load: n_text_layer = 4
whisper_model_load: n_mels = 128
whisper_model_load: ftype = 8
whisper_model_load: qntvr = 2
whisper_model_load: type = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs = 100
whisper_model_load: CPU total size = 573.40 MB
whisper_model_load: model size = 573.40 MB
whisper_backend_init_gpu: no GPU found
whisper_backend_init: using BLAS backend
whisper_init_state: kv self size = 10.49 MB
whisper_init_state: kv cross size = 31.46 MB
whisper_init_state: kv pad size = 7.86 MB
whisper_init_state: compute buffer (conv) = 36.15 MB
whisper_init_state: compute buffer (encode) = 55.35 MB
whisper_init_state: compute buffer (cross) = 9.27 MB
whisper_init_state: compute buffer (decode) = 99.12 MB

whisper_print_timings: load time = 790.12 ms
whisper_print_timings: fallbacks = 0 p / 0 h
whisper_print_timings: mel time = 0.00 ms
whisper_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per run)
whisper_print_timings: encode time = 14003.48 ms / 1 runs ( 14003.48 ms per run)
whisper_print_timings: decode time = 2336.72 ms / 256 runs ( 9.13 ms per run)
whisper_print_timings: batchd time = 1367.44 ms / 320 runs ( 4.27 ms per run)
whisper_print_timings: prompt time = 7844.55 ms / 4096 runs ( 1.92 ms per run)
whisper_print_timings: total time = 25554.03 ms

system_info: n_threads = 8 / 8 | WHISPER : COREML = 0 | OPENVINO = 0 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | OPENMP = 1 | REPACK = 1 |
whisper_init_from_file_with_params_no_state: loading model from 'Model\ggml-medium.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 1
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
whisper_init_with_params_no_state: devices    = 1
whisper_init_with_params_no_state: backends   = 1
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51865
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1024
whisper_model_load: n_audio_head  = 16
whisper_model_load: n_audio_layer = 24
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1024
whisper_model_load: n_text_head   = 16
whisper_model_load: n_text_layer  = 24
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 4 (medium)
whisper_model_load: adding 1608 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:          CPU total size =  1533.14 MB
whisper_model_load: model size    = 1533.14 MB
whisper_backend_init_gpu: no GPU found
whisper_init_state: kv self size  =   50.33 MB
whisper_init_state: kv cross size =  150.99 MB
whisper_init_state: kv pad  size  =    6.29 MB
whisper_init_state: compute buffer (conv)   =   28.57 MB
whisper_init_state: compute buffer (encode) =   44.60 MB
whisper_init_state: compute buffer (cross)  =    7.73 MB
whisper_init_state: compute buffer (decode) =   98.20 MB

whisper_print_timings:     load time =  1641.43 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (     0.00 ms per run)
whisper_print_timings:   encode time = 24890.42 ms /     1 runs ( 24890.42 ms per run)
whisper_print_timings:   decode time = 10957.87 ms /   256 runs (    42.80 ms per run)
whisper_print_timings:   batchd time =  8897.12 ms /   320 runs (    27.80 ms per run)
whisper_print_timings:   prompt time = 88702.21 ms /  4096 runs (    21.66 ms per run)
whisper_print_timings:    total time = 133449.70 ms

My command is :
whisper-bench.exe -m ./Model/ggml-medium.bin -t 8

Other information:

CPU: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz 2.42 GHz
OS: Windows 10 22H2 (19045.6322)
Complier: I don't know, I download it from this release

.\whisper-bench.exe -t 16

system_info: n_threads = 16 / 16 | WHISPER : COREML = 0 | OPENVINO = 0 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | OPENMP = 1 | REPACK = 1 |
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 1
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
whisper_init_with_params_no_state: devices    = 1
whisper_init_with_params_no_state: backends   = 1
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs       = 99
whisper_model_load:          CPU total size =   147.37 MB
whisper_model_load: model size    =  147.37 MB
whisper_backend_init_gpu: no GPU found
whisper_init_state: kv self size  =    6.29 MB
whisper_init_state: kv cross size =   18.87 MB
whisper_init_state: kv pad  size  =    3.15 MB
whisper_init_state: compute buffer (conv)   =   15.87 MB
whisper_init_state: compute buffer (encode) =   22.68 MB
whisper_init_state: compute buffer (cross)  =    4.25 MB
whisper_init_state: compute buffer (decode) =   95.96 MB

whisper_print_timings:     load time =   151.05 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (     0.00 ms per run)
whisper_print_timings:   encode time =  7786.09 ms /     1 runs (  7786.09 ms per run)
whisper_print_timings:   decode time =  3755.00 ms /   256 runs (    14.67 ms per run)
whisper_print_timings:   batchd time =  3210.51 ms /   320 runs (    10.03 ms per run)
whisper_print_timings:   prompt time = 34110.76 ms /  4096 runs (     8.33 ms per run)
whisper_print_timings:    total time = 48863.66 ms

Windows 11 Pro 26100
Ryzen 7 5700X 4.6GHz
64 GB DDR4

Bins downloaded from repo

@ggerganov

I am sharing my results, which I obtained with codex's help. The thing points out something interesting below (Vulkan looks fastest for the encoder microbench, but for real transcription (whisper-cli) the CPU AVX512 path is clearly faster) cc @ggerganov :

• **Using examples/bench on this WSL box, Vulkan looks fastest for the encoder microbench, but for real transcription (whisper-cli) the CPU AVX512 path is clearly faster on your HX 375 / 890M.**

  Environment (context)

  - CPU: AMD Ryzen AI 9 HX 375 w/ Radeon 890M (24 threads / 12 cores)
  - CPU ISA flags include: avx2, avx512f, avx512_vnni, avx512_bf16 (so you’re already on the “AVX/AVX512 fast path”)
  - Vulkan GPU path in WSL is via Mesa Dozen (dzn) and enumerates as: Microsoft Direct3D12 (AMD Radeon(TM) 890M Graphics)

  1) whisper-bench results (encoder-only)
  Command pattern:

    models/ggml-base.en.bin -t <N>

  - CPU (--no-gpu): best observed encode time was roughly ~389–442 ms (depending on threads / flash-attn)
  - Vulkan (Dozen): encode time was ~76–82 ms and basically insensitive to thread count

  So for the microbench’s “encoder on random audio” workload, Vulkan is ~5–6× faster.
  This matters because whisper-bench does not represent the whole Whisper pipeline; it’s encoder-focused. I ran the real CLI on samples/jfk.wav with the same model:

  - CPU-only (fastest observed):
      - ./build-vulkan/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav -t 16 --no-gpu
      - Total time: ~816 ms (best in my quick sweep)
        ggml-base.en.bin -f samples/jfk.wav -t 24
      - Total time: ~2373 ms

  Key detail: on Vulkan Dozen, decode time and “batchd” time jump a lot (hundreds of ms), so overall it’s ~3× slower even though the encoder microbench is faster.

  - For actual whisper-cli usage on this machine: CPU AVX512, not Vulkan.
  - Use 16 threads (matches physical cores) and keep flash-attn on (default).

  ———

  ## Environment

  ### Hardware

  - CPU: AMD Ryzen AI 9 HX 375 w/ Radeon 890M (12C / 24T)
  - GPU (WSL Vulkan via D3D12): AMD Radeon(TM) 890M (exposed as Microsoft Direct3D12 (AMD Radeon(TM) 890M Graphics))

  ### Software

  - Distro: Ubuntu 24.04.3 LTS (noble)
  - Kernel: 6.6.87.2-microsoft-standard-WSL2
  - WSL: 2.6.3.0
  - WSLg: 1.0.71
  - Windows: 10.0.26200.7462
  - Compiler: g++ 13.3.0 (Ubuntu), clang 18.1.3 (Ubuntu)

  ### whisper.cpp build

  - whisper.cpp commit: 6114e692136bea917dc88a5eb2e532c3d133d963
  - Build command used:

    cmake -S . -B build-vulkan -DGGML_VULKAN=1 -DCMAKE_BUILD_TYPE=Release
    cmake --build build-vulkan -j --config Release
  - Model used: models/ggml-base.en.bin

  ### Vulkan in WSL (Dozen / dzn)

  - Mesa Dozen (microsoft-experimental) built from Mesa 25.0.7 and installed user-local
  - ICD JSON used:
      - ~/.local/share/vulkan/icd.d/dzn_icd.x86_64.json
  - For Vulkan runs, I forced the ICD + WSL D3D12 libs:

    export LD_LIBRARY_PATH=/usr/lib/wsl/lib:${LD_LIBRARY_PATH:-}
    export VK_ICD_FILENAMES=$HOME/.local/share/vulkan/icd.d/dzn_icd.x86_64.json

  Note: with Dozen installed, even CPU --no-gpu runs may still print Vulkan device enumeration lines, but use gpu = 0 confirms CPU-only execution.

  ———

  ## Results

  ### 1) CPU-only (AVX/AVX2/AVX512 enabled), --no-gpu, threads=16

  Command:

  ./build-vulkan/bin/whisper-bench -m models/ggml-base.en.bin -t 16 --no-gpu

  Output:

  WARNING: dzn is not a conformant Vulkan implementation, testing use only.
  Dropped Escape call with ulEscapeCode : 0x03007703
  ggml_vulkan: Found 1 Vulkan devices:
  ggml_vulkan: 0 = Microsoft Direct3D12 (AMD Radeon(TM) 890M Graphics) (Dozen) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix
  cores: none
  system_info: n_threads = 16 / 24 | WHISPER : COREML = 0 | OPENVINO = 0 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 =
  1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | OPENMP = 1 | REPACK = 1 |
  whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
  whisper_init_with_params_no_state: use gpu    = 0
  whisper_init_with_params_no_state: flash attn = 1
  ...
  whisper_print_timings:     load time =   179.77 ms
  whisper_print_timings:   encode time =   436.79 ms /     1 runs (   436.79 ms per run)
  whisper_print_timings:   decode time =   736.06 ms /   256 runs (     2.88 ms per run)
  whisper_print_timings:   batchd time =   287.53 ms /   320 runs (     0.90 ms per run)
  whisper_print_timings:   prompt time =  1960.76 ms /  4096 runs (     0.48 ms per run)
  whisper_print_timings:    total time =  3422.60 ms

  ### 2) CPU-only (AVX/AVX2/AVX512 enabled), --no-gpu, threads=24

  Command:

  ./build-vulkan/bin/whisper-bench -m models/ggml-base.en.bin -t 24 --no-gpu

  Output:

  WARNING: dzn is not a conformant Vulkan implementation, testing use only.
  Dropped Escape call with ulEscapeCode : 0x03007703
  ggml_vulkan: Found 1 Vulkan devices:
  ggml_vulkan: 0 = Microsoft Direct3D12 (AMD Radeon(TM) 890M Graphics) (Dozen) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix
  cores: none
  system_info: n_threads = 24 / 24 | WHISPER : COREML = 0 | OPENVINO = 0 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 =
  1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | OPENMP = 1 | REPACK = 1 |
  whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
  whisper_init_with_params_no_state: flash attn = 1
  ...
  whisper_print_timings:     load time =    63.27 ms
  whisper_print_timings:   encode time =   394.63 ms /     1 runs (   394.63 ms per run)
  whisper_print_timings:   prompt time =  1967.22 ms /  4096 runs (     0.48 ms per run)
  whisper_print_timings:    total time =  4653.42 ms

  ### 3) GPU (Vulkan / Dozen dzn), threads=8

  Command:

  LD_LIBRARY_PATH=/usr/lib/wsl/lib:${LD_LIBRARY_PATH:-} \
  VK_ICD_FILENAMES=$HOME/.local/share/vulkan/icd.d/dzn_icd.x86_64.json \
  ./build-vulkan/bin/whisper-bench -m models/ggml-base.en.bin -t 8

  Output:

  WARNING: dzn is not a conformant Vulkan implementation, testing use only.
  Dropped Escape call with ulEscapeCode : 0x03007703
  ggml_vulkan: Found 1 Vulkan devices:
  ggml_vulkan: 0 = Microsoft Direct3D12 (AMD Radeon(TM) 890M Graphics) (Dozen) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix
  cores: none
  system_info: n_threads = 8 / 24 | WHISPER : COREML = 0 | OPENVINO = 0 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1
  | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | OPENMP = 1 | REPACK = 1 |
  whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
  whisper_init_with_params_no_state: use gpu    = 1
  whisper_init_with_params_no_state: flash attn = 1
  ...
  whisper_print_timings:     load time =   295.44 ms
  whisper_print_timings:   encode time =    86.28 ms /     1 runs (    86.28 ms per run)
  whisper_print_timings:   decode time =  1499.77 ms /   256 runs (     5.86 ms per run)
  whisper_print_timings:   batchd time =   465.48 ms /   320 runs (     1.45 ms per run)
  whisper_print_timings:   prompt time =   440.70 ms /  4096 runs (     0.11 ms per run)
  whisper_print_timings:    total time =  2493.36 ms

  ———

Followup info in case it's useful:

• The short version is: whisper-bench is not an end-to-end transcription benchmark, and it intentionally exercises parts of the model in a way that can be very
different from what whisper-cli does on a real audio file. On this WSL setup, the rest of the pipeline is dominated by work where the Vulkan backend (Dozen/dzn) is
currently slower, so the end-to-end result flips in favor of CPU.

Below is the detailed breakdown.

Why whisper-bench and whisper-cli disagree so much

1) whisper-bench is a synthetic “kernel benchmark”, not “transcribe this file”

If you look at examples/bench/bench.cpp, the what=0 (“whisper”) benchmark does:

Initializes a Whisper context.

Calls whisper_set_mel(ctx, nullptr, 0, n_mels) (i.e., it does not decode audio, does not compute a real mel spectrogram from a WAV/MP3).

“Heats” the encoder once and the decoder a bit, then resets timings.

Then the “actual run” is:

1x encode

256x decode of 1 token each (text-generation loop)

64x decode of 5 tokens each (batched decoding loop)

16x decode of 256 tokens (prompt processing loop)

So whisper-bench spends a ton of time in decode-style calls and prompt-style calls, on a fixed synthetic setup. It’s designed to give a stable, comparable compute
workload, not to mimic a typical whisper-cli run on an 11s clip.

That’s also why the “total time” numbers from whisper-bench can look “too big” compared to real transcription: it’s doing lots of repeated decode work on purpose.

2) whisper-cli does real audio preprocessing + segmentation + real decoding strategy

For a real transcription run, the pipeline is closer to:

Audio I/O + decoding

Read WAV/MP3/OGG/FLAC

Decode + resample

Convert to mono float PCM

Feature extraction

Compute the log-mel spectrogram (“mel time”, “sample time” in timings)

Segmentation / chunking

Split audio into windows (typically ~30s chunks internally, plus overlap/offset logic)

Possibly VAD if enabled, etc.

Encoder pass per chunk

Run the Whisper encoder on each chunk (scales roughly with audio duration / number of chunks)

Decoder loop per chunk

Autoregressive token generation with:

beam search + best-of (defaults are not “cheap”)

logits filtering, suppression rules, timestamps logic if enabled, etc.

This part scales with how many tokens you generate, which depends on:

audio length

speaking rate

language

decoding settings (beam size, best-of, temperature fallback, timestamps…)

So in end-to-end transcription, the decoder work pattern is fundamentally different than the bench’s “256 single-token decodes + big prompt loops” pattern.

3) On your WSL Vulkan path, decode is the big problem (not encoder)

From the timings we saw:

CPU (--no-gpu) decode time on samples/jfk.wav was tiny (single-digit ms in one run, low tens of ms in another).

Vulkan (Dozen) decode time was hundreds of ms and “batchd” was also much larger.

So even if Vulkan accelerates some matrix-math in isolation (and it can look good in the microbench), the end-to-end run gets dragged down by:

slower decoder kernels and/or

extra synchronization and data movement between CPU↔GPU backend buffers

lack of specialized features on Dozen for this workload (matrix cores: none, int dot: 0 in the device printout)

plus the fact that Dozen is a Vulkan-over-D3D12 translation layer in WSL (more overhead than a native RADV/AMDVLK path on bare metal Linux)

In other words: your CPU is extremely strong (AVX512/VNNI/BF16), and Dozen is “good enough to run” but not “fast for this particular full pipeline.”

What’s “in the rest of the pipeline” beyond encoder?

Concretely, the parts that can dominate outside the encoder-only view are:

Autoregressive decoding (per token, repeatedly)

attention ops at small batch sizes

softmax/logits processing

KV-cache reads/writes

beam search and best-of multiply the amount of decoder compute

Prompt processing / context resets

repeated “decode a longer token sequence” operations

Data movement / backend glue

if some ops are on Vulkan and some remain on CPU (or fall back), you can get expensive sync/copies

even if everything is “on Vulkan”, you still have CPU-side orchestration overhead

Pre/post-processing

mel spectrogram (CPU)

tokenization / decoding text (CPU)

timestamp logic (if enabled)

Does this hold for much larger files?

It can, but it depends on what is getting larger and what settings you use.

If the file is longer (more audio minutes)

Encoder cost scales roughly with duration (more chunks → more encoder calls).

Decoder cost scales with the amount of text produced (more speech → more tokens).

So for long files, whichever side (CPU or Vulkan) is faster at the dominant part wins.

On your setup, two key observations suggest the CPU will often stay ahead for longer files too:

In real whisper-cli timings, Vulkan decode was much slower than CPU decode.

Vulkan encode was not obviously beating CPU encode in the real transcription path (despite the microbench suggesting it might).

That said, it’s not a law of physics. Two scenarios where results could change:

Much larger models (small/medium/large)

Larger models increase compute per encoder and per decoder step.

Sometimes GPUs start winning only once the workload is large enough to amortize overhead.

Dozen still may not be great, but it’s the most plausible place you’d see Vulkan catch up.

Cheaper decoding settings
If you reduce decoder work, Vulkan’s slow decode hurts less. For example:

set beam size to 1 and best-of to 1 (greedy-ish)

avoid timestamp-heavy settings if they increase token overhead

Those changes can make encoder time dominate more, which is where Vulkan might have a chance.

CoreML benchmarks

Mac Studio
Apple M1 Ultra (16+4 CPU cores, 48 GPU cores, 64Gb RAM)
32 core Apple Neural Engine
macOS Tahoe 26.2 (25C56)

cmake -B build -DWHISPER_COREML=1

cmake --build build -j --config Release

Flash attention: OFF

CPU	OS	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
M1U	macOS	NEON COREML METAL	tiny	8	18.88	1.49	0.37	0.02	`f53dc74`
M1U	macOS	NEON COREML METAL	base	8	34.28	2.11	0.55	0.03	`f53dc74`
M1U	macOS	NEON COREML METAL	small	8	114.99	4.06	1.19	0.08	`f53dc74`
M1U	macOS	NEON COREML METAL	medium	8	346.97	8.18	2.65	0.18	`f53dc74`
M1U	macOS	NEON COREML METAL	large-v3-turbo Q8_0	8	618.23	2.01	0.60	0.06	`f53dc74`

Flash attention: ON

CPU	OS	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
M1U	macOS	NEON COREML METAL	tiny	8	1	18.10	1.27	0.28	0.01	`f53dc74`
M1U	macOS	NEON COREML METAL	base	8	1	35.21	1.74	0.42	0.02	`f53dc74`
M1U	macOS	NEON COREML METAL	small	8	1	114.12	3.13	0.80	0.06	`f53dc74`
M1U	macOS	NEON COREML METAL	medium	8	1	352.35	6.22	1.75	0.13	`f53dc74`
M1U	macOS	NEON COREML METAL	large-v3-turbo Q8_0	8	1	621.38	1.63	0.43	0.04	`f53dc74`

Modified benchmark script ( with --no-gpu option)

CPU	OS	Config	Model	Th	Enc.	Dec.	Bch5	PP	Commit
M1U	macOS	NEON COREML METAL	tiny	8	28.51	1.17	0.41	0.15	`f53dc74`
M1U	macOS	NEON COREML METAL	base	8	48.46	1.90	0.68	0.30	`f53dc74`
M1U	macOS	NEON COREML METAL	small	8	148.01	4.53	1.71	0.76	`f53dc74`
M1U	macOS	NEON COREML METAL	medium	8	429.67	11.48	4.72	1.85	`f53dc74`
M1U	macOS	NEON COREML METAL	large-v3-turbo Q8_0	8	635.54	2.70	0.86	0.36	`f53dc74`

CPU, GPU & ANE utilization for default benchmark:

CPU, GPU & ANE utilization for benchmark with --no-gpu option:

OS: Windows 11 Pro x64 build 26200
Software: whisper.cpp 1.8.3 built with MSVC 19.50.35723.0 and Vulkan SDK 1.4.341.0
Hardware: AMD Radeon RX 9070 XT 16GB (Powercolor Hellhound)
Model: Large v3


> .\whisper-bench.exe -m Models\large.bin
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 9070 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat

system_info: n_threads = 4 / 16 | WHISPER : COREML = 0 | OPENVINO = 0 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | OPENMP = 1 | REPACK = 1 |
whisper_init_from_file_with_params_no_state: loading model from 'Models\large.bin'
whisper_init_with_params_no_state: use gpu    = 1
whisper_init_with_params_no_state: flash attn = 1
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw        = 0
whisper_init_with_params_no_state: devices    = 2
whisper_init_with_params_no_state: backends   = 2
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51866
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1280
whisper_model_load: n_audio_head  = 20
whisper_model_load: n_audio_layer = 32
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1280
whisper_model_load: n_text_head   = 20
whisper_model_load: n_text_layer  = 32
whisper_model_load: n_mels        = 128
whisper_model_load: ftype         = 1
whisper_model_load: qntvr         = 0
whisper_model_load: type          = 5 (large v3)
whisper_model_load: adding 1609 extra tokens
whisper_model_load: n_langs       = 100
whisper_model_load:      Vulkan0 total size =  3094.36 MB
whisper_model_load: model size    = 3094.36 MB
whisper_backend_init_gpu: device 0: Vulkan0 (type: 1)
whisper_backend_init_gpu: found GPU device 0: Vulkan0 (type: 1, cnt: 0)
whisper_backend_init_gpu: using Vulkan0 backend
whisper_init_state: kv self size  =   83.89 MB
whisper_init_state: kv cross size =  251.66 MB
whisper_init_state: kv pad  size  =    7.86 MB
whisper_init_state: compute buffer (conv)   =   37.69 MB
whisper_init_state: compute buffer (encode) =   55.35 MB
whisper_init_state: compute buffer (cross)  =   47.67 MB
whisper_init_state: compute buffer (decode) =  100.04 MB

whisper_print_timings:     load time =  2441.52 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =     0.00 ms
whisper_print_timings:   sample time =     0.00 ms /     1 runs (     0.00 ms per run)
whisper_print_timings:   encode time =    88.51 ms /     1 runs (    88.51 ms per run)
whisper_print_timings:   decode time =  1424.01 ms /   256 runs (     5.56 ms per run)
whisper_print_timings:   batchd time =   748.78 ms /   320 runs (     2.34 ms per run)
whisper_print_timings:   prompt time =   456.94 ms /  4096 runs (     0.11 ms per run)
whisper_print_timings:    total time =  2718.84 ms

./scripts/bench-all.sh 16
Usage: ./scripts/bench-all.sh [n_threads] [encoder-only] [flash-attn]

Running memcpy benchmark

memcpy:   16.37 GB/s (heat-up)
memcpy:   16.38 GB/s ( 1 thread)
memcpy:   16.36 GB/s ( 1 thread)
memcpy:   17.94 GB/s ( 2 thread)
memcpy:   18.75 GB/s ( 3 thread)
memcpy:   19.00 GB/s ( 4 thread)
memcpy:   18.55 GB/s ( 5 thread)
memcpy:   18.83 GB/s ( 6 thread)
memcpy:   18.75 GB/s ( 7 thread)
memcpy:   18.52 GB/s ( 8 thread)
memcpy:   18.41 GB/s ( 9 thread)
memcpy:   18.32 GB/s (10 thread)
memcpy:   18.23 GB/s (11 thread)
memcpy:   18.22 GB/s (12 thread)
memcpy:   18.12 GB/s (13 thread)
memcpy:   18.16 GB/s (14 thread)
memcpy:   17.91 GB/s (15 thread)
memcpy:   17.88 GB/s (16 thread)
sum:    -9215995752.000000

Running ggml_mul_mat benchmark with 16 threads

WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 8 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 2 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 3 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 4 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 5 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 6 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 7 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
  64 x   64: Q4_0    23.6 GFLOPS (128 runs) | Q4_1    23.5 GFLOPS (128 runs)
  64 x   64: Q5_0    24.1 GFLOPS (128 runs) | Q5_1    23.4 GFLOPS (128 runs) | Q8_0    24.8 GFLOPS (128 runs)
  64 x   64: F16     13.9 GFLOPS (128 runs) | F32     27.2 GFLOPS (128 runs)
 128 x  128: Q4_0    82.5 GFLOPS (128 runs) | Q4_1    85.9 GFLOPS (128 runs)
 128 x  128: Q5_0    90.5 GFLOPS (128 runs) | Q5_1    81.9 GFLOPS (128 runs) | Q8_0    92.5 GFLOPS (128 runs)
 128 x  128: F16     76.1 GFLOPS (128 runs) | F32    102.3 GFLOPS (128 runs)
 256 x  256: Q4_0   239.8 GFLOPS (128 runs) | Q4_1   243.0 GFLOPS (128 runs)
 256 x  256: Q5_0   210.4 GFLOPS (128 runs) | Q5_1   214.3 GFLOPS (128 runs) | Q8_0   254.8 GFLOPS (128 runs)
 256 x  256: F16    203.0 GFLOPS (128 runs) | F32    275.4 GFLOPS (128 runs)
 512 x  512: Q4_0   386.3 GFLOPS (128 runs) | Q4_1   382.0 GFLOPS (128 runs)
 512 x  512: Q5_0   332.6 GFLOPS (128 runs) | Q5_1   324.9 GFLOPS (128 runs) | Q8_0   438.0 GFLOPS (128 runs)
 512 x  512: F16    274.4 GFLOPS (128 runs) | F32    388.0 GFLOPS (128 runs)
1024 x 1024: Q4_0   480.0 GFLOPS (128 runs) | Q4_1   453.6 GFLOPS (128 runs)
1024 x 1024: Q5_0   393.9 GFLOPS (128 runs) | Q5_1   387.3 GFLOPS (128 runs) | Q8_0   529.3 GFLOPS (128 runs)
1024 x 1024: F16    305.3 GFLOPS (128 runs) | F32    451.8 GFLOPS (128 runs)
2048 x 2048: Q4_0   509.6 GFLOPS ( 30 runs) | Q4_1   486.4 GFLOPS ( 29 runs)
2048 x 2048: Q5_0   420.6 GFLOPS ( 25 runs) | Q5_1   417.7 GFLOPS ( 25 runs) | Q8_0   556.6 GFLOPS ( 33 runs)
2048 x 2048: F16    312.6 GFLOPS ( 19 runs) | F32    420.5 GFLOPS ( 25 runs)
4096 x 4096: Q4_0   545.2 GFLOPS (  4 runs) | Q4_1   509.6 GFLOPS (  4 runs)
4096 x 4096: Q5_0   439.3 GFLOPS (  4 runs) | Q5_1   438.0 GFLOPS (  4 runs) | Q8_0   591.1 GFLOPS (  5 runs)
4096 x 4096: F16    317.1 GFLOPS (  3 runs) | F32    238.4 GFLOPS (  3 runs)

Running benchmark for all models
This can take a while!

CPU	OS	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
AMD EPYC 7532	Debian 12	AVX2	small	16	0	59.35	2.85	1.47	0.09	`764482c`

./scripts/bench-all.sh 16 0 1
Usage: ./scripts/bench-all.sh [n_threads] [encoder-only] [flash-attn]

Running memcpy benchmark

memcpy:   15.69 GB/s (heat-up)
memcpy:   15.69 GB/s ( 1 thread)
memcpy:   15.69 GB/s ( 1 thread)
memcpy:   17.03 GB/s ( 2 thread)
memcpy:   18.31 GB/s ( 3 thread)
memcpy:   18.96 GB/s ( 4 thread)
memcpy:   18.57 GB/s ( 5 thread)
memcpy:   18.86 GB/s ( 6 thread)
memcpy:   18.71 GB/s ( 7 thread)
memcpy:   18.39 GB/s ( 8 thread)
memcpy:   18.45 GB/s ( 9 thread)
memcpy:   18.22 GB/s (10 thread)
memcpy:   18.33 GB/s (11 thread)
memcpy:   18.20 GB/s (12 thread)
memcpy:   18.19 GB/s (13 thread)
memcpy:   18.10 GB/s (14 thread)
memcpy:   18.00 GB/s (15 thread)
memcpy:   17.90 GB/s (16 thread)
sum:    -9215992579.000000

Running ggml_mul_mat benchmark with 16 threads

WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
WARNING: radv is not a conformant Vulkan implementation, testing use only.
ggml_vulkan: Found 8 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 2 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 3 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 4 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 5 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 6 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 7 = AMD Radeon AI PRO R9700 (RADV GFX1201) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
  64 x   64: Q4_0    23.7 GFLOPS (128 runs) | Q4_1    23.0 GFLOPS (128 runs)
  64 x   64: Q5_0    24.0 GFLOPS (128 runs) | Q5_1    22.7 GFLOPS (128 runs) | Q8_0    24.3 GFLOPS (128 runs)
  64 x   64: F16     14.1 GFLOPS (128 runs) | F32     26.2 GFLOPS (128 runs)
 128 x  128: Q4_0    79.9 GFLOPS (128 runs) | Q4_1    82.1 GFLOPS (128 runs)
 128 x  128: Q5_0    86.9 GFLOPS (128 runs) | Q5_1    81.7 GFLOPS (128 runs) | Q8_0    90.5 GFLOPS (128 runs)
 128 x  128: F16     76.9 GFLOPS (128 runs) | F32    103.7 GFLOPS (128 runs)
 256 x  256: Q4_0   237.8 GFLOPS (128 runs) | Q4_1   242.0 GFLOPS (128 runs)
 256 x  256: Q5_0   211.9 GFLOPS (128 runs) | Q5_1   213.9 GFLOPS (128 runs) | Q8_0   258.7 GFLOPS (128 runs)
 256 x  256: F16    213.6 GFLOPS (128 runs) | F32    276.4 GFLOPS (128 runs)
 512 x  512: Q4_0   387.2 GFLOPS (128 runs) | Q4_1   384.0 GFLOPS (128 runs)
 512 x  512: Q5_0   331.9 GFLOPS (128 runs) | Q5_1   327.1 GFLOPS (128 runs) | Q8_0   440.7 GFLOPS (128 runs)
 512 x  512: F16    278.2 GFLOPS (128 runs) | F32    372.2 GFLOPS (128 runs)
1024 x 1024: Q4_0   482.4 GFLOPS (128 runs) | Q4_1   454.1 GFLOPS (128 runs)
1024 x 1024: Q5_0   394.3 GFLOPS (128 runs) | Q5_1   388.0 GFLOPS (128 runs) | Q8_0   531.1 GFLOPS (128 runs)
1024 x 1024: F16    306.0 GFLOPS (128 runs) | F32    451.3 GFLOPS (128 runs)
2048 x 2048: Q4_0   512.9 GFLOPS ( 30 runs) | Q4_1   486.5 GFLOPS ( 29 runs)
2048 x 2048: Q5_0   421.0 GFLOPS ( 25 runs) | Q5_1   417.7 GFLOPS ( 25 runs) | Q8_0   556.8 GFLOPS ( 33 runs)
2048 x 2048: F16    312.0 GFLOPS ( 19 runs) | F32    421.4 GFLOPS ( 25 runs)
4096 x 4096: Q4_0   543.7 GFLOPS (  4 runs) | Q4_1   509.0 GFLOPS (  4 runs)
4096 x 4096: Q5_0   440.9 GFLOPS (  4 runs) | Q5_1   438.2 GFLOPS (  4 runs) | Q8_0   591.2 GFLOPS (  5 runs)
4096 x 4096: F16    317.5 GFLOPS (  3 runs) | F32    239.4 GFLOPS (  3 runs)

Running benchmark for all models
This can take a while!

CPU	OS	Config	Model	Th	FA	Enc.	Dec.	Bch5	PP	Commit
AMD EPYC 7532	Debian 12	AVX2	small	16	1	53.91	2.47	0.74	0.07	`764482c`

bench : sync submit-results URL to ggml-org

bench : sync submit-results URL to ggml-org (#3769)

bench : sync submit-results URL to ggml-org (ggml-org#3769)

CPU	OS	Config	Model	Threads	load time	encode time
AMD Ryzen 7 6800H	Kubuntu 26.04	AVX2	base.en	8	146.01 ms	661.18 ms / 1 runs

This issue was marked as stale because it has been inactive for 30 days.

Navigation Menu

Repository navigation

Benchmark results #89

Encoder

memcpy

MacBook M1 Pro

Ryzen 9 5950X

ggml_mul_mat

MacBook M1 Pro

Ryzen 9 5950X

174 remaining items

Why whisper-bench and whisper-cli disagree so much

1) whisper-bench is a synthetic “kernel benchmark”, not “transcribe this file”

2) whisper-cli does real audio preprocessing + segmentation + real decoding strategy

3) On your WSL Vulkan path, decode is the big problem (not encoder)

What’s “in the rest of the pipeline” beyond encoder?

Does this hold for much larger files?

If the file is longer (more audio minutes)

CoreML benchmarks

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Benchmark results #89

Description

Encoder

memcpy

MacBook M1 Pro

Ryzen 9 5950X

ggml_mul_mat

MacBook M1 Pro

Ryzen 9 5950X

Activity

cdosoftei commented on Oct 25, 2022

rjwilmsi commented on Oct 26, 2022

ArtyomZemlyak commented on Oct 26, 2022

ArtyomZemlyak commented on Oct 26, 2022

ArtyomZemlyak commented on Oct 26, 2022

ArtyomZemlyak commented on Oct 26, 2022

ggerganov commented on Oct 26, 2022

cristianglezm commented on Oct 28, 2022

tazz4843 commented on Oct 29, 2022

yujinqiu commented on Oct 30, 2022

trholding commented on Oct 31, 2022

trholding commented on Oct 31, 2022

trholding commented on Oct 31, 2022

174 remaining items

Kotali-2019 commented on Oct 11, 2025

Xionghaizi001 commented on Oct 27, 2025

DimucaTheDev commented on Dec 3, 2025

santiago-afonso commented on Dec 29, 2025

santiago-afonso commented on Dec 29, 2025

Why whisper-bench and whisper-cli disagree so much

1) whisper-bench is a synthetic “kernel benchmark”, not “transcribe this file”

2) whisper-cli does real audio preprocessing + segmentation + real decoding strategy

3) On your WSL Vulkan path, decode is the big problem (not encoder)

What’s “in the rest of the pipeline” beyond encoder?

Does this hold for much larger files?

If the file is longer (more audio minutes)

TheCluster commented on Jan 17, 2026

CoreML benchmarks

kz26 commented on Feb 4, 2026

dimon777 commented on Feb 15, 2026

dimon777 commented on Feb 15, 2026

tabletuser-blogspot commented on Aug 12, 2026

github-actions commented on Sep 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions