DevStack master branch single-node deployment on Ubuntu 24.04 fails frequently due to MySQL default max_connections (1024) being exceeded by concurrent service connections.

Bug #2145067 reported by 刘骐豪
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
devstack
In Progress
Undecided
Unassigned

Bug Description

Environment

    OS: Ubuntu 24.04
    Deployment: DevStack master branch (single-node deployment)
    Database: MySQL with default connection limit (max_connections = 1024)
    Networking: Neutron OVS provider network (full local.conf attached below)
    Hardware: 128-core CPU, 256GB memory, 22TB RAID disk (sufficient resources for single-node deployment)

Actual Result
During DevStack deployment (via ./stack.sh), all OpenStack core services (Keystone/Neutron/Nova/Cinder/Glance etc.) establish concurrent, non-pooled connections to the single local MySQL instance. The connection count rapidly exceeds MySQL’s default limit of 1024 (verified via Max_used_connections = 1025 in MySQL status), leading to:

1.Glance service initialization failure (cannot connect to MySQL for metadata setup);
2.Image upload stage (05:22:34) triggers critical errors:
prove chain:
    Failed to contact the endpoint at http://10.126.41.229/image for discovery. Fallback to using that endpoint as the base url.
result:"
    Deployment aborts with Error on exit at the image upload step, resulting in a high failure rate for single-node DevStack master branch deployment.

Expected Result
DevStack master branch should natively adapt to single-node deployment scenarios by:

1.Optimizing service connection strategies (e.g., enabling connection pooling) to avoid exhausting MySQL connections;
2.Or dynamically adjusting MySQL’s max_connections limit for single-node deployments;
3.Ensuring Glance (and all core services) initializes successfully and image upload completes without errors.

Deployment Timeline & Key Error Details
1. Deployment Sequence (Critical Stages)
plaintext

2026-03-21 05:13:22 | Configuring and starting MySQL (max_connections=1024 by default)
2026-03-21 05:13:59 - 05:22:05 | Starting Keystone/Neutron/Nova/Cinder (connections to MySQL accumulate)
2026-03-21 05:22:25 | Starting Glance (MySQL connections already exhausted)
2026-03-21 05:22:34 | Uploading images (deployment fails at this step)

2. MySQL Connection Verification
sql

mysql> show status like 'max_used_connections';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 1025 |
+----------------------+-------+

Confirmed: MySQL connections exceed the default 1024 limit during deployment.
3. Mitigation Validation
Increasing MySQL’s max_connections (e.g., to 2048) completely resolves the issue—DevStack deployment completes successfully, Glance initializes normally, and image upload works without errors.

Local.conf (Full Configuration)

[[local|localrc]]
# Git configuration
GIT_BASE="https://github.com"

# Network & Path configuration
HOST_IP=10.126.41.229
DEST=/opt/stack
LOGDIR=$DEST/logs
LOGFILE=$LOGDIR/stack.sh.log

# Credentials (shared for all services)
ADMIN_PASSWORD=suma123456
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD

# Service endpoints
SERVICE_HOST=10.126.41.229
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292

# Neutron (OVS provider network)
Q_USE_SECGROUP=True
FLOATING_RANGE="10.126.40.0/21"
Q_FLOATING_ALLOCATION_POOL=start=10.126.47.240,end=10.126.47.250
PUBLIC_NETWORK_GATEWAY="10.126.40.1"
PUBLIC_INTERFACE=eno1

Q_USE_PROVIDERNET_FOR_PUBLIC=True
OVS_PHYSICAL_BRIDGE=br-ex
PUBLIC_PHYSICAL_NETWORK=public
PUBLIC_BRIDGE=br-ex
OVS_BRIDGE_MAPPINGS=public:br-ex

Revision history for this message
刘骐豪 (lqh-bjtu) wrote :

This is my secondary deployment when i change the max_connections in /lib/databases/mysql
mysql> show status like 'max_used_connections';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 1115 |
+----------------------+-------+
1 row in set (0.00 sec)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to devstack (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/devstack/+/981632

Changed in devstack:
status: New → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.