Skip to content

Commit e90b6de

Browse files
authored
Python: Improve python package management operations (microsoft#7274)
* improve package mgmt timings * Address Python release validation review feedback
1 parent d98ac29 commit e90b6de

7 files changed

Lines changed: 935 additions & 27 deletions

File tree

‎python/.github/skills/agent-framework-py-release/SKILL.md‎

Lines changed: 42 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ For release work, derive the live tier map at release time from `python/PACKAGE_
2828

2929
## Inputs to confirm before bumping
3030

31-
1. **The changeset**: explicit commits/PRs the release covers, OR derive from `git log ${LAST_RELEASED_TAG}..origin/main -- python/`.
31+
1. **The changeset**: explicit commits/PRs the release covers, OR derive from `git log ${LAST_RELEASED_TAG}..${RELEASE_BASE} -- python/`.
3232
2. **Per-package CHANGELOG entries**: which packages will get a line in the new release section. This list IS the bump list.
3333
3. **Per-released-package semver bump**: for each released-tier package that has a CHANGELOG entry, decide PATCH / MINOR / MAJOR.
3434
4. **Date stamp** (only if any alpha/beta is being bumped): default from the `python-package-management`
@@ -55,12 +55,20 @@ If the user states target versions or a date explicitly, use exactly what they s
5555
git fetch origin main --tags --quiet
5656
git fetch upstream main --tags --quiet 2>/dev/null || true
5757
git status
58+
59+
# Fork clones use upstream/main as the authoritative release base; direct clones use origin/main.
60+
if git show-ref --verify --quiet refs/remotes/upstream/main; then
61+
RELEASE_BASE=upstream/main
62+
else
63+
RELEASE_BASE=origin/main
64+
fi
65+
git log -1 --oneline "$RELEASE_BASE"
5866
```
5967

6068
If the user already has a `bump-py-ver-release-*` branch checked out, use it. Otherwise:
6169

6270
```bash
63-
git checkout -b bump-py-ver-release-YYMMDD origin/main
71+
git checkout -b bump-py-ver-release-YYMMDD "$RELEASE_BASE"
6472
```
6573

6674
### 2. Build the live tier map
@@ -84,20 +92,20 @@ echo "Compare base: $LAST_RELEASED_TAG"
8492
List commits and packages touched:
8593

8694
```bash
87-
git log --oneline ${LAST_RELEASED_TAG}..origin/main -- python/ ':!python/CHANGELOG.md'
95+
git log --oneline ${LAST_RELEASED_TAG}..${RELEASE_BASE} -- python/ ':!python/CHANGELOG.md'
8896

8997
# Per-commit package footprint
90-
for sha in $(git log --format='%H' ${LAST_RELEASED_TAG}..origin/main -- python/); do
98+
for sha in $(git log --format='%H' ${LAST_RELEASED_TAG}..${RELEASE_BASE} -- python/); do
9199
echo "--- $(git show -s --format='%h %s' $sha) ---"
92100
git show --name-only --format='' $sha | grep '^python/packages/' | \
93101
sed 's|^python/packages/||' | awk -F/ '{print $1}' | sort -u
94102
done
95103
```
96104

97-
If the release ultimately tags from `upstream/main` but `origin/main` is behind, also run:
105+
When both remotes exist, record whether the fork is behind the authoritative base:
98106

99107
```bash
100-
git log --oneline ${LAST_RELEASED_TAG}..upstream/main -- python/ ':!python/CHANGELOG.md'
108+
git rev-list --left-right --count origin/main...upstream/main
101109
```
102110

103111
If user provides an explicit commit/PR list, treat THAT as authoritative.
@@ -108,14 +116,14 @@ Aggregate the per-commit footprint into a single union across the whole range. T
108116

109117
```bash
110118
# Union of all touched package directories across the range
111-
git log --name-only --format='' ${LAST_RELEASED_TAG}..origin/main -- python/packages/ \
119+
git log --name-only --format='' ${LAST_RELEASED_TAG}..${RELEASE_BASE} -- python/packages/ \
112120
| grep '^python/packages/' \
113121
| sed 's|^python/packages/||' \
114122
| awk -F/ '{print $1}' \
115123
| sort -u
116124

117125
# Root-level files (drive a root agent-framework entry if substantive)
118-
git log --name-only --format='' ${LAST_RELEASED_TAG}..origin/main \
126+
git log --name-only --format='' ${LAST_RELEASED_TAG}..${RELEASE_BASE} \
119127
-- python/pyproject.toml python/agent_framework_meta/ python/README.md \
120128
2>/dev/null | grep -v '^$' | sort -u
121129
```
@@ -175,13 +183,13 @@ Before moving on, prove that every ship-affecting touched package has at least o
175183

176184
```bash
177185
# 1. Touched ship-affecting packages and root package files (from step 3a)
178-
TOUCHED_PACKAGES=$(git log --name-only --format='' ${LAST_RELEASED_TAG}..origin/main -- python/packages/ \
186+
TOUCHED_PACKAGES=$(git log --name-only --format='' ${LAST_RELEASED_TAG}..${RELEASE_BASE} -- python/packages/ \
179187
| grep '^python/packages/' \
180188
| sed 's|^python/packages/||' \
181189
| awk -F/ '{print $1}' \
182190
| sort -u)
183191

184-
ROOT_TOUCHED=$(git log --name-only --format='' ${LAST_RELEASED_TAG}..origin/main \
192+
ROOT_TOUCHED=$(git log --name-only --format='' ${LAST_RELEASED_TAG}..${RELEASE_BASE} \
185193
-- python/pyproject.toml python/agent_framework_meta/ python/README.md \
186194
2>/dev/null | grep -v '^$' | sort -u)
187195

@@ -279,7 +287,7 @@ Spot-check with `grep '^version' python/pyproject.toml python/packages/*/pyproje
279287
Only relevant when `core` itself bumped this cycle. Two policies, pick one explicitly with the user:
280288

281289
- **Conservative (default)**: raise `agent-framework-core>=X.Y.Z` to the new core version on every non-core package that is ALSO bumping this cycle. Leaves packages-not-bumped at their existing floor.
282-
- **Strict per-upstream-doc**: only raise the floor on packages that actually consume a new core API introduced in the bump. This requires per-package code inspection. Use only when the user is comfortable letting `validate-dependency-bounds-test` (lower-resolution pass) catch any mistakes.
290+
- **Strict per-upstream-doc**: only raise the floor on packages that actually consume a new core API introduced in the bump. This requires per-package code inspection because release probes use the co-released local core and cannot prove compatibility with an older published core floor.
283291

284292
When raising a core floor, replace only the `>=OLD` half of the bound you intend to change:
285293

@@ -294,12 +302,29 @@ If `core` did not bump this cycle, do not touch floors.
294302
### 7. Validate
295303

296304
```bash
297-
cd python && uv run poe validate-dependency-bounds-test
305+
cd python && uv run poe validate-python-release --base-ref "$RELEASE_BASE"
298306
```
299307

300-
Must exit 0. This is the safety net for selective bumping: the lower-resolution pass catches floors set too low for code that depends on new APIs, and the upper pass catches caps that exclude installable versions. If it fails, the output names the offending bound — fix and re-run before committing. This step also regenerates `uv.lock` to match new bounds.
308+
Use the same freshly fetched main ref that the release branch was based on (`upstream/main` above; use `origin/main`
309+
when that is the authoritative release base). Must exit 0. This task first regenerates `uv.lock`, then discovers the
310+
package `pyproject.toml` files changed from that base and runs their published runtime dependencies and
311+
non-development extras through lock-independent `lowest-direct` and `highest` import probes. The probes run in
312+
parallel, derive the minimum supported Python minor from each package's internal editable closure, and share a hard
313+
300-second deadline. Use `--python` only when the release requires an explicit interpreter override.
314+
315+
This is the release safety net for selective bumping: the lower probe catches unresolvable or unimportable external
316+
floors, internal constraints that reject co-released package versions, and the upper probe catches caps that exclude
317+
an installable package set. The JSON report records the concrete versions resolved in both scenarios. It does not
318+
replace the package-by-package code inspection required by the strict core-floor policy. If it fails, fix the named
319+
package/bound and re-run before committing.
320+
321+
Do not substitute the workspace-wide `validate-dependency-bounds-test` command here. That command runs every
322+
package's full tests and Pyright in separate isolated environments and is intentionally reserved for CI or an
323+
explicit dependency-range audit. If the release itself changes an external dependency range, also run
324+
`validate-dependency-bounds-project --mode both --package <pkg> --dependency <name>` for that dependency.
301325

302-
If only prereleases changed (no `core` bump, no floor changes), this validation is still required — `uv.lock` regeneration alone justifies the run.
326+
If only prereleases changed (no `core` bump, no floor changes), release validation is still required because the
327+
lockfile and both ends of each changed package's published dependency metadata must remain installable.
303328

304329
### 8. Commit (expect hook retry)
305330

@@ -349,13 +374,14 @@ The push output includes a `Create a pull request for '<branch>' on GitHub by vi
349374
do not infer a local timezone from the user's current shell.
350375
- **`Co-Authored-By` trailer.** Never add it. Rewrite/amend if it slipped in.
351376
- **Stale inventory in this skill.** Always read `python/PACKAGE_STATUS.md` for the live tier map. Do not trust a hardcoded list.
352-
- **Divergent origin vs upstream.** If the release tags from `upstream/main` but `origin/main` is behind, check both — warn if they differ and offer to sync.
377+
- **Divergent origin vs upstream.** In fork clones, use freshly fetched `upstream/main` consistently for branch creation, changeset discovery, and release validation. A stale `origin/main` must never become the implicit compare base.
353378
- **`--pre` README cleanup on promotion.** When a package is promoted to `released` in this cycle, grep for `pip install agent-framework-<pkg> --pre` in READMEs and drop the `--pre` flag.
354379
- **RC counter inflation.** Do not increment `1.0.0rcN` without a CHANGELOG entry for that package. The counter tracks iterations, not calendar.
355380

356381
## References
357382

358383
- Package lifecycle and versioning source of truth: `python/.github/skills/python-package-management/SKILL.md`
359384
- Lifecycle source of truth: `python/PACKAGE_STATUS.md`
360-
- Validator: `python/scripts/dependencies/validate_dependency_bounds.py` (runs `lowest-direct` and `highest` resolution smoke tests; catches floors/caps that don't match the code)
385+
- Release validator: `python/scripts/dependencies/validate_dependency_bounds.py --mode release` (changed-package,
386+
lock-independent `lowest-direct` and `highest` import probes under a five-minute deadline)
361387
- Poe task definitions: `python/pyproject.toml` `[tool.poe.tasks]`

‎python/.github/skills/python-package-management/SKILL.md‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,13 @@ uv lock --upgrade-package <dependency-name> && uv run poe install
4545
# Refresh exact development dependency-group pins, lockfile, and validation in one run
4646
uv run poe upgrade-dev-dependencies
4747

48-
# First, run workspace-wide lower/upper compatibility gates
48+
# Release cuts: refresh uv.lock and probe changed packages at both bound extremes.
49+
# The release probe has a shared five-minute deadline.
50+
uv run poe validate-python-release --base-ref upstream/main
51+
52+
# Exhaustive test+typing matrix (slow; use for deliberate dependency-range work or CI)
4953
uv run poe validate-dependency-bounds-test
50-
# Defaults to --package "*"; pass a package to scope test mode
54+
# Defaults to --package "*"; scope locally whenever possible.
5155
uv run poe validate-dependency-bounds-test --package core
5256

5357
# Then expand bounds for one dependency in the target package
@@ -66,7 +70,16 @@ uv run poe add-dependency-and-validate-bounds --package core --dependency "<depe
6670
- Prerelease (`dev`/`a`/`b`/`rc`) and `<1.0` dependencies should use hard bounds with an explicit upper cap (avoid open-ended ranges).
6771
- For `<1.0` dependencies, prefer the broadest validated range the package can really support. That may be a patch line, a minor line, or multiple minor lines when checks/tests show the broader lane is compatible.
6872
- Prefer supporting multiple majors when practical; if APIs diverge across supported majors, use version-conditional imports/paths.
69-
- For dependency changes, run workspace-wide bound gates first, then `validate-dependency-bounds-project --mode both` for the target package/dependency to keep minimum and maximum constraints current. The same task can also drive repo-wide upper-bound automation by using `--package "*"` and omitting `--dependency`.
73+
- For release-only version, lifecycle, pin, and internal-floor edits, use `validate-python-release`. It refreshes
74+
`uv.lock`, finds changed package metadata relative to the selected main ref, and runs the changed packages'
75+
published runtime dependencies and non-development extras through lock-independent `lowest-direct` and `highest`
76+
import probes on the minimum Python minor supported by each package's internal editable closure. The probes run
77+
concurrently under one 300-second deadline; pass `--python` only when an explicit interpreter override is needed.
78+
- For deliberate external dependency-range changes, use
79+
`validate-dependency-bounds-project --mode both` for the target package/dependency to find and validate the actual
80+
minimum and maximum constraints. Scope the exhaustive `validate-dependency-bounds-test` matrix to affected
81+
packages during local iteration; reserve the workspace-wide form for CI or an intentional full audit. The same
82+
project task can drive repo-wide upper-bound automation by using `--package "*"` and omitting `--dependency`.
7083
- Prefer targeted lock updates with `uv lock --upgrade-package <dependency-name>` to reduce `uv.lock` merge conflicts.
7184
- Use `add-dependency-and-validate-bounds` for package-scoped dependency additions plus bound validation in one command.
7285
- Keep shared tooling and source/type-check support in the root or package `dev` group. Put package-specific test

‎python/pyproject.toml‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -395,10 +395,32 @@ args = [
395395
]
396396

397397
[tool.poe.tasks.validate-dependency-bounds-test]
398-
help = "Run workspace dependency-bound validation in test mode, optionally scoped with -P/--package short names such as `core`."
398+
help = "Run the exhaustive workspace dependency-bound test+typing matrix, optionally scoped with -P/--package short names such as `core`."
399399
shell = "python -m scripts.dependencies.validate_dependency_bounds --mode test --package \"$project\""
400400
args = [{ name = "project", default = "*", options = ["-P", "--package"] }]
401401

402+
[tool.poe.tasks.validate-python-release]
403+
help = "Refresh uv.lock, then run lower/upper import probes for changed package metadata on each package closure's minimum Python."
404+
executor = "simple"
405+
shell = """
406+
command=(
407+
python -m scripts.dependencies.validate_dependency_bounds
408+
--mode release
409+
--base-ref "${base_ref}"
410+
--release-timeout-seconds "${timeout}"
411+
)
412+
if [ -n "${python}" ]; then
413+
command+=(--python "${python}")
414+
fi
415+
"${command[@]}"
416+
"""
417+
interpreter = "bash"
418+
args = [
419+
{ name = "base_ref", options = ["-B", "--base-ref"] },
420+
{ name = "python", default = "", options = ["--python"] },
421+
{ name = "timeout", default = "300", options = ["--timeout-seconds"] },
422+
]
423+
402424
[tool.poe.tasks.validate-dependency-bounds-project]
403425
help = "Validate lower and upper dependency bounds for a -P/--package workspace package, optionally narrowed with -M/--mode and -D/--dependency."
404426
shell = """

‎python/scripts/dependencies/README.md‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,19 @@ Run the commands below from the `python/` directory.
1212

1313
- `validate_dependency_bounds.py`
1414
- Main entrypoint for dependency-bound workflows.
15-
- Supports `test`, `lower`, `upper`, and `both` modes.
16-
- `test` runs workspace-wide smoke validation at the lower and upper ends of the currently allowed ranges.
15+
- Supports `release`, `test`, `lower`, `upper`, and `both` modes.
16+
- `release` refreshes `uv.lock`, then runs changed packages through fast lock-independent lower/upper import probes.
17+
- `test` runs the exhaustive workspace test+typing compatibility matrix.
1718
- `lower`, `upper`, and `both` dispatch to the lower/upper optimizer implementations for one package.
1819

20+
- `_dependency_bounds_release_impl.py`
21+
- Discovers package metadata changed from the selected release base.
22+
- Resolves published runtime dependencies and non-development extras independently of `uv.lock` with both
23+
`lowest-direct` and `highest` strategies.
24+
- Derives the minimum supported Python minor from each changed package's internal editable dependency closure.
25+
- Imports each changed package and records resolved dependency versions in a JSON report.
26+
- Runs probes concurrently under one five-minute deadline.
27+
1928
- `upgrade_dev_dependencies.py`
2029
- Refreshes exact dev dependency pins across the root `pyproject.toml` and package `pyproject.toml` files.
2130
- Reuses the same version-selection logic as the upper-bound tooling so direct dev-tooling refreshes and dependency-range expansion stay consistent.
@@ -45,14 +54,18 @@ These are the normal user-facing entrypoints:
4554
```bash
4655
uv run poe upgrade-dev-dependency-pins
4756
uv run poe upgrade-dev-dependencies
57+
uv run poe validate-python-release --base-ref upstream/main
4858
uv run poe validate-dependency-bounds-test
4959
uv run poe validate-dependency-bounds-test --package core
5060
uv run poe validate-dependency-bounds-project --mode both --package core --dependency "<dependency-name>"
5161
```
5262

5363
- `upgrade-dev-dependency-pins` only refreshes exact dev pins in `pyproject.toml` files.
5464
- `upgrade-dev-dependencies` refreshes dev pins (using task above), runs `uv lock --upgrade`, reinstalls from the frozen lockfile, then runs `check`, `typing`, and `test`.
55-
- `validate-dependency-bounds-test` runs the repo-wide lower/upper smoke gate.
65+
- `validate-python-release` is the bounded release gate: it refreshes `uv.lock`, finds changed package metadata,
66+
and probes both dependency-bound extremes without reusing the lockfile.
67+
- `validate-dependency-bounds-test` runs the exhaustive package test+typing matrix and is intentionally not part of
68+
the routine release path.
5669
- `validate-dependency-bounds-project` is the single package-scoped task; use `--mode lower`, `--mode upper`, or `--mode both` for the target package/dependency pair. Its `--package` argument defaults to `*`, and `--dependency` is optional, so automation can also use it for repo-wide upper-bound runs.
5770

5871
### GitHub Actions workflows
@@ -76,6 +89,7 @@ These are useful for debugging or targeted manual runs:
7689

7790
```bash
7891
python -m scripts.dependencies.upgrade_dev_dependencies --dry-run --version-source lock
92+
python -m scripts.dependencies.validate_dependency_bounds --mode release --base-ref upstream/main --dry-run
7993
python -m scripts.dependencies.validate_dependency_bounds --mode test --package core --dry-run
8094
python -m scripts.dependencies.validate_dependency_bounds --mode both --package core --dependencies openai --dry-run
8195
python -m scripts.dependencies._dependency_bounds_lower_impl --packages core --dependencies openai --dry-run
@@ -89,6 +103,7 @@ Use the direct lower/upper implementation modules mainly for debugging or develo
89103
The validators write JSON reports into this folder:
90104

91105
- `dependency-bounds-test-results.json`
106+
- `dependency-bounds-release-results.json`
92107
- `dependency-lower-bound-results.json`
93108
- `dependency-range-results.json`
94109

0 commit comments

Comments
 (0)