devstack's uWSGI configuration of lazy-apps=true should be close-on-exec=true

Bug #2147304 reported by Doug Goldstein
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
devstack
In Progress
Undecided
Unassigned

Bug Description

The setting originated with https://launchpad.net/bugs/1600394 in https://review.opendev.org/c/openstack/devstack/+/363940 but this was introduced as a workaround and wasn't the correct fix. The proper behavior is to close any file descriptors when forking and reconnecting any connections that were previously used. Which is what SQLAlchemy does with os.register_at_fork() as well as Kombu (for oslo.messaging) do. While pymemcache will reconnect on error which is what close-on-exec will provide and redis-py (for oslo.cache) checks the PID every time the connection is grabbed for any operation and will reconnect if the PID has changed.

While effective, this fix has a downside as it defers application loading until after fork, meaning we cannot know that the application will run successfully. Additionally we are defeating the improvement that uWSGI's Copy-on-Write from its master process to its workers gives production workloads. While
this is devstack so it is for development and testing, the various projects use the devstack config as their recommended and tested configuration for production. The semantics would also potentially
change the behavior between dev/testing and production so I'm suggesting the changes here.

The correct solution is close-on-exec=true and close-on-exec2=true, which instruct uWSGI to close inherited file descriptors in each worker process after forking. This directly addresses the socket-sharing problem without sacrificing CoW benefits or incurring the per-worker app load overhead.

Also add need-app=true, a production-appropriate setting that causes uWSGI to exit if the application fails to load. Without this, the master process can remain running and appear healthy while workers are unable to serve requests, making failures harder to detect and recover from via the process manager. lazy-apps=true is sometimes used as a dev convenience but is inconsistent with recommended production values and is better left unset.

References:
- https://uwsgi.readthedocs.io/en/latest/ThingsToKnow.html
- https://blog.ionelmc.ro/2022/03/14/how-to-run-uwsgi/
- https://www.bloomberg.com/company/stories/configuring-uwsgi-production-deployment/

Changed in devstack:
status: New → In Progress
Revision history for this message
Doug Goldstein (cardoe) wrote :
Revision history for this message
Doug Goldstein (cardoe) wrote :

Originally the lazy-apps=true was set to fix https://bugs.launchpad.net/keystone/+bug/1600394 in https://review.opendev.org/c/openstack/devstack/+/363940 but from the change and the description close-on-exec is what was really needed and lazy-apps=true had a similar result by accident and there was not enough validation of that change before it was made.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.