The native host memory manager makes spill decisions without a shared limit on active spills. Multiple consumers can spill simultaneously, potentially increasing I/O contention. Automatic pressure checks also skip a consumer’s first memory reservation.
Integrate the host memory manager with the shared spill coordinator, using native memory reservations, JVM direct memory, and process-memory pressure.
Introduce configurable spill watermarks and concurrency:
- Below the low watermark, avoid automatic spills.
- Between the watermarks, select the largest spillable consumer.
- At or above the high watermark, allow reporting consumers to spill within the concurrency limit.
Use defaults of 0.8 and 0.9 for the watermarks and 2 for concurrent spills. Preserve the existing operator API and the 16 MiB minimum for automatic spill checks. Check the first growing reservation and maintain correct accounting during forced spills and pinning.
The native host memory manager makes spill decisions without a shared limit on active spills. Multiple consumers can spill simultaneously, potentially increasing I/O contention. Automatic pressure checks also skip a consumer’s first memory reservation.
Integrate the host memory manager with the shared spill coordinator, using native memory reservations, JVM direct memory, and process-memory pressure.
Introduce configurable spill watermarks and concurrency:
Use defaults of 0.8 and 0.9 for the watermarks and 2 for concurrent spills. Preserve the existing operator API and the 16 MiB minimum for automatic spill checks. Check the first growing reservation and maintain correct accounting during forced spills and pinning.