Skip to content

Commit 733f80e

Browse files
petyaslavovapetyaslavova
committed
Updates in default connection and retry settings (#4082)
* Updating default socket keep alive config * Updating default socket read size to 32 kb * Updating default socket_timeout and socket_connect_timeout values to 5s * Updating the default retry configuration * Updating max_connections allowed in ConnectionPool * Unifying SENTINEL marker usage throughout library method args * Fixing failing maint notifications tests - updated the expected original timeout to be 5s * Fixing backing up socket_timeout and socket_connect_timeout for maint notifications handling * Fixing linters * Fixing failing tests - after changing default retry counts for cluster and max_connections in pool some test configs needed to be updated * Fixing linters * Fixing failing tests * Using socket timeout constants for all default socket timeout configs * Applying review comments * Fixing linters * Extracting retry count,base and cap configs as constants
1 parent a68a16c commit 733f80e

29 files changed

Lines changed: 753 additions & 331 deletions

‎docs/geographic_failover.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -185,8 +185,8 @@ MultiDbConfig
185185
186186
# Global command retry policy (applied at multi-db layer)
187187
command_retry=Retry(
188-
retries=3,
189-
backoff=ExponentialWithJitterBackoff(base=1, cap=10),
188+
retries=10,
189+
backoff=ExponentialWithJitterBackoff(base=0.01, cap=1),
190190
),
191191
192192
# Health checks

‎docs/retry.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,8 +48,8 @@ Retry in Redis Cluster
4848

4949
Retry behaviour in Redis Cluster is a little bit different from Standalone:
5050

51-
* ``retry``: :class:`~.Retry` instance with a :ref:`backoff-label` strategy and the max number of retries, default value is ``Retry(ExponentialWithJitterBackoff(base=1, cap=10), cluster_error_retry_attempts)``
52-
* ``cluster_error_retry_attempts``: number of times to retry before raising an error when :class:`~.TimeoutError`, :class:`~.ConnectionError`, :class:`~.ClusterDownError` or :class:`~.SlotNotCoveredError` are encountered, default value is ``3``
51+
* ``retry``: :class:`~.Retry` instance with a :ref:`backoff-label` strategy and the max number of retries, default value is ``Retry(ExponentialWithJitterBackoff(base=0.01, cap=1), cluster_error_retry_attempts)``
52+
* ``cluster_error_retry_attempts``: number of times to retry before raising an error when :class:`~.TimeoutError`, :class:`~.ConnectionError`, :class:`~.ClusterDownError` or :class:`~.SlotNotCoveredError` are encountered, default value is ``10``
5353
* This argument is deprecated - it is used to initialize the number of retries for the retry object,
5454
only in the case when the ``retry`` object is not provided.
5555
When the ``retry`` argument is provided, the ``cluster_error_retry_attempts`` argument is ignored!
@@ -70,4 +70,4 @@ Let's consider the following example:
7070
#. given the hash slot, it then determines which node to connect to, in order to execute the command.
7171
#. during the connection, a :class:`~.ConnectionError` is raised.
7272
#. because we set ``retry=Retry(ExponentialBackoff(), 6)``, the cluster client starts a cluster update, removes the failed node from the startup nodes, and re-initializes the cluster.
73-
#. the cluster client retries the command until it either succeeds or the max number of retries is reached.
73+
#. the cluster client retries the command until it either succeeds or the max number of retries is reached.

‎redis/_defaults.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
"""Internal default helper functions shared across redis-py modules."""
2+
3+
import socket
4+
5+
# Connection defaults
6+
7+
DEFAULT_SOCKET_TIMEOUT = 5 # 5s
8+
DEFAULT_SOCKET_CONNECT_TIMEOUT = DEFAULT_SOCKET_TIMEOUT
9+
DEFAULT_SOCKET_READ_SIZE = 32768 # 32KB
10+
11+
12+
def get_default_socket_keepalive_options() -> dict[int, int]:
13+
options = {}
14+
15+
# Linux exposes TCP_KEEPIDLE; macOS exposes the equivalent TCP_KEEPALIVE.
16+
# Some platforms expose neither and only support SO_KEEPALIVE itself.
17+
tcp_keepidle = getattr(socket, "TCP_KEEPIDLE", None)
18+
if tcp_keepidle is None:
19+
tcp_keepidle = getattr(socket, "TCP_KEEPALIVE", None)
20+
if tcp_keepidle is not None:
21+
options[tcp_keepidle] = 30
22+
23+
# Not every platform exposes interval/probe tuning constants.
24+
tcp_keepintvl = getattr(socket, "TCP_KEEPINTVL", None)
25+
if tcp_keepintvl is not None:
26+
options[tcp_keepintvl] = 5
27+
28+
# Not every platform exposes interval/probe tuning constants.
29+
tcp_keepcnt = getattr(socket, "TCP_KEEPCNT", None)
30+
if tcp_keepcnt is not None:
31+
options[tcp_keepcnt] = 3
32+
33+
return options
34+
35+
36+
# Retry defaults
37+
DEFAULT_RETRY_COUNT = 10
38+
DEFAULT_RETRY_BASE = 0.01 # 10ms
39+
DEFAULT_RETRY_CAP = 1 # 1s

‎redis/_parsers/hiredis.py‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55

66
from ..exceptions import ConnectionError, InvalidResponse, RedisError, TimeoutError
77
from ..typing import EncodableT
8-
from ..utils import HIREDIS_AVAILABLE, deprecated_function
8+
from ..utils import HIREDIS_AVAILABLE, SENTINEL, deprecated_function
99
from .base import (
1010
AsyncBaseParser,
1111
AsyncPushNotificationsParser,
@@ -15,7 +15,6 @@
1515
from .socket import (
1616
NONBLOCKING_EXCEPTION_ERROR_NUMBERS,
1717
NONBLOCKING_EXCEPTIONS,
18-
SENTINEL,
1918
SERVER_CLOSED_CONNECTION_ERROR,
2019
)
2120

‎redis/_parsers/resp2.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@
22

33
from ..exceptions import ConnectionError, InvalidResponse, ResponseError
44
from ..typing import EncodableT
5+
from ..utils import SENTINEL
56
from .base import _AsyncRESPBase, _RESPBase
6-
from .socket import SENTINEL, SERVER_CLOSED_CONNECTION_ERROR
7+
from .socket import SERVER_CLOSED_CONNECTION_ERROR
78

89

910
class _RESP2Parser(_RESPBase):

‎redis/_parsers/resp3.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,14 @@
33

44
from ..exceptions import ConnectionError, InvalidResponse, ResponseError
55
from ..typing import EncodableT
6+
from ..utils import SENTINEL
67
from .base import (
78
AsyncPushNotificationsParser,
89
PushNotificationsParser,
910
_AsyncRESPBase,
1011
_RESPBase,
1112
)
12-
from .socket import SENTINEL, SERVER_CLOSED_CONNECTION_ERROR
13+
from .socket import SERVER_CLOSED_CONNECTION_ERROR
1314

1415

1516
class _RESP3Parser(_RESPBase, PushNotificationsParser):

‎redis/_parsers/socket.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,6 @@
77
from ..exceptions import ConnectionError, TimeoutError
88
from ..utils import SENTINEL, SSL_AVAILABLE
99

10-
# Re-export SENTINEL from this historical location for compatibility.
11-
1210
NONBLOCKING_EXCEPTION_ERROR_NUMBERS = {BlockingIOError: errno.EWOULDBLOCK}
1311

1412
if SSL_AVAILABLE:

‎redis/asyncio/client.py‎

Lines changed: 58 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@
2727
cast,
2828
)
2929

30+
from redis._defaults import (
31+
DEFAULT_RETRY_BASE,
32+
DEFAULT_RETRY_CAP,
33+
DEFAULT_RETRY_COUNT,
34+
DEFAULT_SOCKET_CONNECT_TIMEOUT,
35+
DEFAULT_SOCKET_READ_SIZE,
36+
DEFAULT_SOCKET_TIMEOUT,
37+
)
3038
from redis._parsers.helpers import bool_ok, get_response_callbacks
3139
from redis.asyncio.connection import (
3240
Connection,
@@ -232,49 +240,53 @@ def __init__(
232240
*,
233241
host: str = "localhost",
234242
port: int = 6379,
235-
db: Union[str, int] = 0,
236-
password: Optional[str] = None,
237-
socket_timeout: Optional[float] = None,
238-
socket_connect_timeout: Optional[float] = None,
239-
socket_keepalive: Optional[bool] = None,
240-
socket_keepalive_options: Optional[Mapping[int, Union[int, bytes]]] = None,
241-
connection_pool: Optional[ConnectionPool] = None,
242-
unix_socket_path: Optional[str] = None,
243+
db: str | int = 0,
244+
password: str | None = None,
245+
socket_timeout: float | None = DEFAULT_SOCKET_TIMEOUT,
246+
socket_connect_timeout: float | None = DEFAULT_SOCKET_CONNECT_TIMEOUT,
247+
socket_read_size: int = DEFAULT_SOCKET_READ_SIZE,
248+
socket_keepalive: bool | None = True,
249+
socket_keepalive_options: Mapping[int, int | bytes] | object | None = SENTINEL,
250+
connection_pool: ConnectionPool | None = None,
251+
unix_socket_path: str | None = None,
243252
encoding: str = "utf-8",
244253
encoding_errors: str = "strict",
245254
decode_responses: bool = False,
246255
retry_on_timeout: bool = False,
247256
retry: Retry = Retry(
248-
backoff=ExponentialWithJitterBackoff(base=1, cap=10), retries=3
257+
backoff=ExponentialWithJitterBackoff(
258+
base=DEFAULT_RETRY_BASE, cap=DEFAULT_RETRY_CAP
259+
),
260+
retries=DEFAULT_RETRY_COUNT,
249261
),
250-
retry_on_error: Optional[list] = None,
262+
retry_on_error: list | None = None,
251263
ssl: bool = False,
252-
ssl_keyfile: Optional[str] = None,
253-
ssl_certfile: Optional[str] = None,
254-
ssl_cert_reqs: Union[str, VerifyMode] = "required",
255-
ssl_include_verify_flags: Optional[List[VerifyFlags]] = None,
256-
ssl_exclude_verify_flags: Optional[List[VerifyFlags]] = None,
257-
ssl_ca_certs: Optional[str] = None,
258-
ssl_ca_data: Optional[str] = None,
259-
ssl_ca_path: Optional[str] = None,
264+
ssl_keyfile: str | None = None,
265+
ssl_certfile: str | None = None,
266+
ssl_cert_reqs: "str | VerifyMode" = "required",
267+
ssl_include_verify_flags: List["VerifyFlags"] | None = None,
268+
ssl_exclude_verify_flags: List["VerifyFlags"] | None = None,
269+
ssl_ca_certs: str | None = None,
270+
ssl_ca_data: str | None = None,
271+
ssl_ca_path: str | None = None,
260272
ssl_check_hostname: bool = True,
261-
ssl_min_version: Optional[TLSVersion] = None,
262-
ssl_ciphers: Optional[str] = None,
263-
ssl_password: Optional[str] = None,
264-
max_connections: Optional[int] = None,
273+
ssl_min_version: "TLSVersion | None" = None,
274+
ssl_ciphers: str | None = None,
275+
ssl_password: str | None = None,
276+
max_connections: int | None = None,
265277
single_connection_client: bool = False,
266278
health_check_interval: int = 0,
267-
client_name: Optional[str] = None,
268-
lib_name: Union[Optional[str], object] = SENTINEL,
269-
lib_version: Union[Optional[str], object] = SENTINEL,
270-
driver_info: Union[Optional["DriverInfo"], object] = SENTINEL,
271-
username: Optional[str] = None,
272-
auto_close_connection_pool: Optional[bool] = None,
279+
client_name: str | None = None,
280+
lib_name: str | object | None = SENTINEL,
281+
lib_version: str | object | None = SENTINEL,
282+
driver_info: DriverInfo | object | None = SENTINEL,
283+
username: str | None = None,
284+
auto_close_connection_pool: bool | None = None,
273285
redis_connect_func=None,
274-
credential_provider: Optional[CredentialProvider] = None,
275-
protocol: Optional[int] = None,
286+
credential_provider: CredentialProvider | None = None,
287+
protocol: int | None = None,
276288
legacy_responses: bool = True,
277-
event_dispatcher: Optional[EventDispatcher] = None,
289+
event_dispatcher: EventDispatcher | None = None,
278290
):
279291
"""
280292
Initialize a new Redis client.
@@ -296,6 +308,20 @@ def __init__(
296308
297309
When 'connection_pool' is provided - the retry configuration of the
298310
provided pool will be used.
311+
312+
Args:
313+
314+
socket_keepalive:
315+
if `True`, TCP keepalive is enabled for TCP socket connections.
316+
Argument is ignored when connection_pool is provided.
317+
socket_keepalive_options:
318+
mapping of TCP keepalive socket option constants to values, for
319+
example `{socket.TCP_KEEPIDLE: 30}`. If left unspecified, redis-py
320+
uses TCP keepalive defaults when `socket_keepalive` is enabled:
321+
idle 30 seconds, interval 5 seconds, and 3 probes. Platform-specific
322+
options that are not available are skipped. Pass `None` or `{}` to
323+
avoid setting additional TCP keepalive options. Argument is ignored
324+
when connection_pool is provided.
299325
"""
300326
kwargs: Dict[str, Any]
301327
if event_dispatcher is None:
@@ -333,6 +359,7 @@ def __init__(
333359
"password": password,
334360
"credential_provider": credential_provider,
335361
"socket_timeout": socket_timeout,
362+
"socket_read_size": socket_read_size,
336363
"encoding": encoding,
337364
"encoding_errors": encoding_errors,
338365
"decode_responses": decode_responses,

0 commit comments

Comments
 (0)