Repository navigation
IoT: connection drop after around 24 hours #2603
Description
Our raspi connection to google iot via mqtt fails to persist past the 24hour mark. I have the JWT refreshing about every 20 mins or so.
Running on a raspi 3 with mqtt and code is identical from the examples posted here just changed my device id and project id and other vars to match my instance.
Activity
[-]connection drop after around 24 hours[/-][+]IoT: connection drop after around 24 hours[/+]gguuss commentedon Dec 11, 2019
Thanks for reporting this issue, I'll see if I can figure out what's going on.
kilabyte commentedon Dec 12, 2019
If i use the pubsub client library as also on your documentation this works flawlessly but then i get no information within the IoT console for things like last seen, heartbeat, config sent etc
gguuss commentedon Dec 12, 2019
There are a lot of potential causes for a disconnect. If you go to the console and look up your device are you seeing any error details?
kilabyte commentedon Dec 13, 2019
I get either "The connection was closed or broken by the client" or i get "The authorization token expired" Which either doesnt make sense. The raspi is also hardwired to my router and im refreshing the token every 20 minutes.
kilabyte commentedon Dec 13, 2019
Or if i use the client library instead is there anyway to specify the device id so i can maintain status of each iot device?
gguuss commentedon Dec 13, 2019
I'm able to reproduce the console error issue now. There's a chance the device is not correctly refreshing the JWT, but it looks like to me like it should work as the JWT is refreshed in the call to
get_client.There's a chance on the reconnect we need to call:
To cleanly disconnect after this line to avoid the error. But I believe I have tested this before and it worked, despite the console errors. If you're doing a lot of calculations between the calls to
mqtt.loop()the client loses its connection and behavior after that could get inconsistent.kilabyte commentedon Dec 13, 2019
ok i can update and try that. what is the difference between client.loop(), client.stop_loop() and client.loop_start() ?
gguuss commentedon Dec 13, 2019
client.loopis synchronous,client.loop_stop/client.loop_startare asynchronous helpers that create a background thread. The asynchronous methods run smoother in practice but can be difficult to test.Don't mix
client.loop()withclient.loop_stop()andclient.loop_start().gguuss commentedon Dec 13, 2019
Of note, adding the disconnect seems to be the right thing to do, I'm working on a PR to set it up.
kilabyte commentedon Dec 14, 2019
Also can evey ioT device use the same public key? Or should each device have a unique one? and does client.loop() can be called without calling client.loop_stop()?
kilabyte commentedon Dec 14, 2019
Ok so i added the disconnect here and i get this just repeating:
on_disconnect 1: Out of memory. Connection: Connection Accepted. on_disconnect 1: Out of memory. Connection: Connection Accepted.Im using client.loop_start() every time the loop is entered as i get better performance with the async call. Should i be calling client.loop_stop() somewhere?
If i just use client.loop() i see performance issues but it looks like the token refresh works but now my 30 second loop takes 60 seconds
kilabyte commentedon Dec 14, 2019
OK i think i figured it out. I kept with the async loop calls. I put a
client.loop_start()within mygetClientmethod right before the return. Then when the JWT refresh is called i callclient.loop_stop(). Once the connection is recreated it starts the loop again since i have it abstracted into a connect method. Im going to run this for a bit and see what i getgguuss commentedon Dec 16, 2019
Sounds good, @kilabyte - I have created a new branch where we cleanly disconnect the MQTT sample. Feel free to reopen the issue if you're still seeing the client disconnecting or crashing.
kilabyte commentedon Dec 17, 2019
@gguuss can you just explain why you used .loop() and .loop_stop() in the same file? and i think for performance your PR would be better off with using the async calls instead that way you dont have to implicitly disconnect as i have found this has issues with the backoff in the disconnect callback
gguuss commentedon Dec 17, 2019
@kilabyte Which file are you referring to? This should be a bug if it's going on. In my anecdotal tests, there is no harm in calling loop in the mix with loop_start() and loop_stop() but if you call loop_start without ever calling loop_stop later on, the program will hang.
If you're referring to the removed lines from
manager_test.pythose were a correction of the issue I'm referring to. In thecloud_iot_mqtt.pyexample I'm using the synchronous / explicit methods because they are easier to debug. To replace them with the loop_start / loop_stop methods would be the first thing I would do in real-world code.kilabyte commentedon Dec 18, 2019
Gotcha! Thanks for the explanation :) much appreciated. For handling like network offline and disconnects that is where the backoff comes in right? And then any offline message that was "published" is stored locally until the network is back in which case the messages will all dump out? (sorry i know this is off topic to the issue)
gguuss commentedon Dec 18, 2019
@kilabyte Yes, you got it. The back off is to ensure devices behave well under a number of circumstances. The sample/demo/example code itself does not buffer messages or do anything intelligent in terms of online / offline. There are other solutions built on top of Cloud IoT Core that are more robust.
kilabyte commentedon Dec 19, 2019
do you have links to said examples?
gguuss commentedon Dec 19, 2019
A few come to mind:
There's also the device SDK written in C.
kilabyte commentedon Dec 19, 2019
thank you!