Skip to content

cluster_update_slot failing if new master added #2740

Description

@vadimonus

Summary: failing to get key from slot migrated to new master which not existed on connection time.

Steps to reproduce:

  • create 5 redis nodes but add only 3 to cluster
redis-server --daemonize yes --port 7000 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7000.conf
redis-server --daemonize yes --port 7001 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7001.conf
redis-server --daemonize yes --port 7002 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7002.conf
redis-server --daemonize yes --port 7003 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7003.conf
redis-server --daemonize yes --port 7004 --appendonly yes --cluster-enabled yes --cluster-config-file nodes-7004.conf
redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 --cluster-replicas 0 --cluster-yes
  • start script that connects to cluster and get random key
// note - here we pass only one node address, but phpredis will resolve all other masters, when connected to this node
$redisCluster = new RedisCluster(null, ['127.0.0.1:7000']);
while (true) {
            $masters = $redisCluster->_masters(); // so we will get 3 master here
            foreach ($masters as $master) {
                $randomKey = $redisCluster->randomKey($master);
                $redisCluster->get($randomKey);
            }
           sleep(1);
}
  • add 2 nodes to cluster, and rebalance it, causing to move some slots to new masters.
redis-cli --cluster add-node 172.27.0.199:7003 172.27.0.199:7000 
redis-cli --cluster add-node 172.27.0.199:7004 172.27.0.199:7000
sleep 3
redis-cli --cluster rebalance 172.27.0.199:7000 --cluster-use-empty-masters
redis-cli -h 172.27.0.199 -p 7000 cluster slots

connection will keep info only about 3 masters, that was known on connection time.
when trying to get key from moved slot, it will search new master, but this master is not in array.

Before 6.3.0RC1 (fixed in b0ba827) there was a segfault in this case.
In 6.3.0 it will cause Redis Cluster's master data is incomplete in this case. But cluster is in valid state.
https://github.com/phpredis/phpredis/blob/develop/cluster_library.c#L1419 here should check, if new master is in masters list, and add it' if it is not there.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions