Question: `MaxNumOutboundPeers` default value #9956

rootulp · 2022-12-22T14:55:48Z

rootulp
Dec 22, 2022

Context

tendermint/spec/p2p/v0.34/configuration.md

Line 15 in a9feb1c

    
           |	[MaxNumOutboundPeers](peer_manager.md#ensure-peers)         |  10 |  Maximum number of outbound peers to connect to, excluding persistent peers |

Question

Why was the default value 10 chosen? What should one consider before increasing or decreasing this value? Tagging @Bidon15 for awareness

Bidon15 · 2022-12-22T15:09:23Z

Bidon15
Dec 22, 2022

Results so far

Increasing the MaxNumOutboundPeers to 20 made me wonder even more how p2p stack works
Here are some logs snapshot from 4 validators out of a 100 validators' set:

Logs

Validator 1

Ensure peers module=pex numDialing=0 numInPeers=35 numOutPeers=20 numToDial=0

Validator 2

Ensure peers module=pex numDialing=0 numInPeers=34 numOutPeers=20 numToDial=0

Validator 3

Ensure peers module=pex numDialing=0 numInPeers=3 numOutPeers=6 numToDial=14

Validator 4

Ensure peers module=pex numDialing=0 numInPeers=8 numOutPeers=20 numToDial=0

Questions:

Why having 5 seed nodes(with 100/100 inbound/outbound peers and being in the config of each validator) some validators fail to dial all 5 seeds? This doesn't look like a normal behaviour to me
Point 1 results in some validators lacking peers, which is not ok. Is there any other mechanism to mitigate? Or the only option is to spin up even more seed nodes?
Point 2 causes the Q: what is the practical ratio of seeds to validators? How much is too much?

2 replies

lasarojc Dec 22, 2022

Can you share more context on what happened to Validator3? Is this snapshot of the logs taken soon after Validator3 started?

Wrt 1, why do you say a validator could not dial the 5 seeds? Could you share the logs for such validator?

Bidon15 Dec 23, 2022

Can you share more context on what happened to Validator3? Is this snapshot of the logs taken soon after Validator3 started?

Validator3(and other 98) starts after 5 seed nodes and 1 validator(it serves the genesis.json for seeds)
My expectation is that all 5 should be fine serving seeds asap to validator 3 and others.

For context, 99 validators do these pre-requisites, too:

wait until 5 seed nodes start
add 5 seed nodes' node-id@ip into seeds section in config.toml
add random 10 validators from the set into the address book (addrbook.json)
have no persistent_peers in the config.toml
start

Here is the snapshot of unsuccessful

11:20AM INF Ensure peers module=pex numDialing=0 numInPeers=10 numOutPeers=9 numToDial=21
11:20AM INF We need more addresses. Sending pexRequest to random peer module=pex peer={"Data":{},"Logger":{}}
11:20AM INF No addresses to dial. Falling back to seeds module=pex
11:20AM INF Dialing peer address={"id":"79d185ce179630e2bedf4c17af91aeffdf37120c","ip":"10.32.1.76","port":26656} module=p2p
11:20AM ERR Error dialing seed err="dial tcp 10.32.1.76:26656: i/o timeout" module=p2p seed={"id":"79d185ce179630e2bedf4c17af91aeffdf37120c","ip":"10.32.1.76","port":26656}
11:20AM INF Dialing peer address={"id":"13a0c933fcd609cf8fae1ae53e429f7877b5f0a8","ip":"10.32.1.4","port":26656} module=p2p
11:20AM INF Timed out dur=14945.055498 height=4 module=consensus round=0 step=1
11:20AM ERR Error dialing seed err="dial tcp 10.32.1.4:26656: i/o timeout" module=p2p seed={"id":"13a0c933fcd609cf8fae1ae53e429f7877b5f0a8","ip":"10.32.1.4","port":26656}
11:20AM INF Dialing peer address={"id":"2a5e64dc050ed75b507b114cae6c18266442833d","ip":"10.32.1.70","port":26656} module=p2p
11:20AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"302E59ACF250BCD749C458A89948648DEBCE96A4212C62BDFFEEF1370F91EAFE","parts":{"hash":"7A53456B4EC923EFD7BF82C3362A1EB4E8888FF278FD46BFAD485B5D42477D1E","total":93}},"height":4,"pol_round":-1,"round":0,"signature":"WYEOFu9WJ83dWmeYrcTPwTVQhS7qTl9N/nvooBu7rWaWlQGdcpbC5zV+3gwz6Ft+2NtYyiNPbsRipO6ObKurAg==","timestamp":"2022-12-23T11:20:32.432308955Z"}
11:20AM ERR Error dialing seed err="dial tcp 10.32.1.70:26656: i/o timeout" module=p2p seed={"id":"2a5e64dc050ed75b507b114cae6c18266442833d","ip":"10.32.1.70","port":26656}
11:20AM INF Dialing peer address={"id":"0bdff5a311d846cca2aa3a6688f758109740e3de","ip":"10.32.1.98","port":26656} module=p2p
11:20AM ERR Error dialing seed err="dial tcp 10.32.1.98:26656: i/o timeout" module=p2p seed={"id":"0bdff5a311d846cca2aa3a6688f758109740e3de","ip":"10.32.1.98","port":26656}
11:20AM INF Dialing peer address={"id":"35260112e5632e0302ecb4926e58cc2ba5a97919","ip":"10.32.1.17","port":26656} module=p2p
11:20AM INF received complete proposal block hash=302E59ACF250BCD749C458A89948648DEBCE96A4212C62BDFFEEF1370F91EAFE height=4 module=consensus
11:20AM ERR Error dialing seed err="dial tcp 10.32.1.17:26656: i/o timeout" module=p2p seed={"id":"35260112e5632e0302ecb4926e58cc2ba5a97919","ip":"10.32.1.17","port":26656}
11:20AM ERR Couldn't connect to any seeds module=p2p
11:20AM INF finalizing commit of block hash={} height=4 module=consensus num_txs=94 root=CE3FD3E97057B7E293E9C33AC2C856426665751F4C404F5F0322E9F53F7EA29C
11:20AM INF minted coins from module account amount=20597271405utia from=mint module=x/bank
11:20AM INF executed block height=4 module=state num_invalid_txs=0 num_valid_txs=94
11:20AM INF commit synced commit=436F6D6D697449447B5B323035203935203431203133372032343720313539203231392031313620323239203231203231382031343020323231203139362031343520362031363420313637203831203637203233352038322035302032343220313035203135203230342035382034352031383720323434203234385D3A347D
11:20AM INF committed state app_hash=CD5F2989F79FDB74E515DA8CDDC49106A4A75143EB5232F2690FCC3A2DBBF4F8 height=4 module=state num_txs=94
{"height":"4","txhash":"938C5F69395BD21092F6BE605D91DDBEDC74DEDD53F99CCD62A360C4518A175E","codespace":"","code":0,"data":"12200A1E2F7061796D656E742E4D7367506179466F7244617461526573706F6E7365","raw_log":"[{\"msg_index\":0,\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"/payment.MsgPayForData\"}]},{\"type\":\"payfordata\",\"attributes\":[{\"key\":\"signer\",\"value\":\"celestia1vdjkcetnw35kzvt2v3ukwwtsd5c8wur9wpaxgcmrw4n8jdr6vech573nxfc8wump09nkvd3jd4usfgv028\"},{\"key\":\"size\",\"value\":\"64000\"}]}]}]","logs":[{"msg_index":0,"log":"","events":[{"type":"message","attributes":[{"key":"action","value":"/payment.MsgPayForData"}]},{"type":"payfordata","attributes":[{"key":"signer","value":"celestia1vdjkcetnw35kzvt2v3ukwwtsd5c8wur9wpaxgcmrw4n8jdr6vech573nxfc8wump09nkvd3jd4usfgv028"},{"key":"size","value":"64000"}]}]}],"info":"","gas_wanted":"1000000000","gas_used":"123098","tx":null,"timestamp":"","events":[{"type":"coin_spent","attributes":[{"key":"c3BlbmRlcg==","value":"Y2VsZXN0aWExamR5ZzlwbTB3cGVwemRjY3VmeTR6ZnF6ejMycHdzYXlnZjYybXk=","index":true},{"key":"YW1vdW50","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"coin_received","attributes":[{"key":"cmVjZWl2ZXI=","value":"Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=","index":true},{"key":"YW1vdW50","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"transfer","attributes":[{"key":"cmVjaXBpZW50","value":"Y2VsZXN0aWExN3hwZnZha20yYW1nOTYyeWxzNmY4NHoza2VsbDhjNWxwbmpzM3M=","index":true},{"key":"c2VuZGVy","value":"Y2VsZXN0aWExamR5ZzlwbTB3cGVwemRjY3VmeTR6ZnF6ejMycHdzYXlnZjYybXk=","index":true},{"key":"YW1vdW50","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"message","attributes":[{"key":"c2VuZGVy","value":"Y2VsZXN0aWExamR5ZzlwbTB3cGVwemRjY3VmeTR6ZnF6ejMycHdzYXlnZjYybXk=","index":true}]},{"type":"tx","attributes":[{"key":"ZmVl","value":"MTAwMDAwMDAwMDAwdXRpYQ==","index":true}]},{"type":"tx","attributes":[{"key":"YWNjX3NlcQ==","value":"Y2VsZXN0aWExamR5ZzlwbTB3cGVwemRjY3VmeTR6ZnF6ejMycHdzYXlnZjYybXkvMQ==","index":true}]},{"type":"tx","attributes":[{"key":"c2lnbmF0dXJl","value":"UW95Sm5OYXVSZWhyWUo1WWJmVkQ2YXV6MGJrbWo1UTdyMnVBYVVNeWR5UVlaaHZIRkN4THJmcXp2c3ZPOG9nc2ZlZXowVlJaQSt4VUlhb1pmc0hzNVE9PQ==","index":true}]},{"type":"message","attributes":[{"key":"YWN0aW9u","value":"L3BheW1lbnQuTXNnUGF5Rm9yRGF0YQ==","index":true}]},{"type":"payfordata","attributes":[{"key":"c2lnbmVy","value":"Y2VsZXN0aWExdmRqa2NldG53MzVrenZ0MnYzdWt3d3RzZDVjOHd1cjl3cGF4Z2Ntcnc0bjhqZHI2dmVjaDU3M254ZmM4d3VtcDA5bmt2ZDNqZDR1c2ZndjAyOA==","index":true},{"key":"c2l6ZQ==","value":"NjQwMDA=","index":true}]}]}
11:20AM INF indexed block height=4 module=txindex

Wrt 1, why do you say a validator could not dial the 5 seeds? Could you share the logs for such validator?

Sure. the above logs is what I usually see during first blocks. This fluctuates for different validators as further block production continues on.
We are working out a log analyzer as manually going through 105 instances(100 validators and 5 seeds) is a heavy human labour 😅

In addition, I see that we can't dial 2/3/4 seeds, but I digress.

Navigation Menu

Question: `MaxNumOutboundPeers` default value #9956

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Question: MaxNumOutboundPeers default value #9956

Uh oh!

rootulp Dec 22, 2022

Context

Question

Replies: 1 comment · 2 replies

Uh oh!

Bidon15 Dec 22, 2022

Results so far

Logs

Validator 1

Validator 2

Validator 3

Validator 4

Questions:

Uh oh!

lasarojc Dec 22, 2022

Uh oh!

Bidon15 Dec 23, 2022

Question: `MaxNumOutboundPeers` default value #9956

rootulp
Dec 22, 2022

Replies: 1 comment 2 replies

Bidon15
Dec 22, 2022