Skip to content

Automated performance tests #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Kookster310 wants to merge 34 commits into
base: main
Choose a base branch
from
Open

Automated performance tests #17

Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
293afea
add node group creating and deletion scripts
Kookster310 Mar 13, 2026
b1b9cdb
update ref on performanceTests job
Kookster310 Mar 13, 2026
b779628
fix branch
Kookster310 Mar 13, 2026
3641c8e
fix scheduling and db storageclass values
Kookster310 Mar 13, 2026
a0d393c
re-add deploy-perf-instance.sh
Kookster310 Mar 13, 2026
02f6a74
yea longhorn for mysql
Kookster310 Mar 13, 2026
bd585ce
use local-path for mysql
Kookster310 Mar 13, 2026
ad0042e
install helm release into default ns
Kookster310 Mar 13, 2026
d44f8bf
fix nfs security group to be able to mount efs to new node group
Kookster310 Mar 14, 2026
2b9d428
fix cleanup
Kookster310 Mar 14, 2026
ccb6fea
remove original plan .md
Kookster310 Mar 14, 2026
2d45297
set name tag on ec2 instance
Kookster310 Mar 14, 2026
f33c810
change baseline from hardcoded develop to the base branch of PR
Kookster310 Mar 16, 2026
15df115
add git token to checkout automated performance tests
Kookster310 Mar 16, 2026
de96606
wait for tls cert
Kookster310 Mar 16, 2026
1c5c310
append random suffix to performance URLS to avoid letsencrypt
Kookster310 Mar 16, 2026
c9a8797
clean up performance summary comment and add comparison
Kookster310 Mar 16, 2026
a21ba6a
base image build on ci:performance-tests
Kookster310 Mar 16, 2026
e1b8ecc
add cert deletion if tls not ready
Kookster310 Mar 17, 2026
db2bd31
generate tokens on web pod
Kookster310 Mar 17, 2026
913f0c1
discover tests and iterate any found
Kookster310 Mar 17, 2026
45ee6ea
add googleApiToken place holder
Kookster310 Mar 17, 2026
3a52b66
deploy separate node groups for baseline and update.
Kookster310 Mar 17, 2026
815ec93
remove tests timeout
Kookster310 Mar 17, 2026
8f4ecac
fix indentation
Kookster310 Mar 17, 2026
f909656
make deploy-perf-instance.sh executable in deploy update job
Kookster310 Mar 17, 2026
635d9a9
update security group change
Kookster310 Mar 17, 2026
1a27222
add force cleanup for stuck pods
Kookster310 Mar 17, 2026
4cadf53
update permissions on perfComment
Kookster310 Mar 17, 2026
304b263
fix body reporting
Kookster310 Mar 17, 2026
677efd9
remove debug info
Kookster310 Mar 17, 2026
591f9e2
Update bearer token
nolanpro Mar 19, 2026
0384b91
add performance metrics tag validation script. only run tests that ma…
Kookster310 Mar 25, 2026
4e196a7
fix pr tag to look for ci:performance-tests:*
Kookster310 Mar 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add force cleanup for stuck pods
  • Loading branch information
@Kookster310
Kookster310 committed Mar 17, 2026
commit 1a27222dd200dce1f082e1470de6b5cb0bb781ea
21 changes: 20 additions & 1 deletion .github/workflows/deploy-pm4.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -631,6 +631,13 @@ jobs:
APP_NS="${RELEASE_NAME}-ns-pm4"
helm uninstall "${RELEASE_NAME}" --namespace default 2>/dev/null || true
kubectl delete namespace "${APP_NS}" --timeout=120s --ignore-not-found=true || true
# Force-cleanup: remove stuck pods so namespace can terminate and node group delete can complete
for pod in $(kubectl get pods -n "${APP_NS}" -o name 2>/dev/null); do
kubectl delete "${pod}" -n "${APP_NS}" --grace-period=0 --force 2>/dev/null || true
done
if kubectl get namespace "${APP_NS}" &>/dev/null; then
kubectl get namespace "${APP_NS}" -o json | jq '.metadata.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/${APP_NS}/finalize" -f - 2>/dev/null || true
fi

- name: Delete performance node group (baseline)
if: always()
Expand Down Expand Up @@ -888,6 +895,13 @@ jobs:
APP_NS="${RELEASE_NAME}-ns-pm4"
helm uninstall "${RELEASE_NAME}" --namespace default 2>/dev/null || true
kubectl delete namespace "${APP_NS}" --timeout=120s --ignore-not-found=true || true
# Force-cleanup: remove stuck pods so namespace can terminate and node group delete can complete
for pod in $(kubectl get pods -n "${APP_NS}" -o name 2>/dev/null); do
kubectl delete "${pod}" -n "${APP_NS}" --grace-period=0 --force 2>/dev/null || true
done
if kubectl get namespace "${APP_NS}" &>/dev/null; then
kubectl get namespace "${APP_NS}" -o json | jq '.metadata.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/${APP_NS}/finalize" -f - 2>/dev/null || true
fi

- name: Delete performance node group (update)
if: always()
Expand Down Expand Up @@ -915,7 +929,12 @@ jobs:

- name: Prepare performance comment body
run: |
cp baseline-results/perf-test-scripts.txt . 2>/dev/null || cp update-results/perf-test-scripts.txt .
# Merge script lists from both artifacts so we compare every test that ran in either job
(cat baseline-results/perf-test-scripts.txt 2>/dev/null; cat update-results/perf-test-scripts.txt 2>/dev/null) | sort -u > perf-test-scripts.txt
if [ ! -s perf-test-scripts.txt ]; then
touch perf-test-scripts.txt
fi
# Copy result files into current dir (same names as baseline/update jobs produced)
cp baseline-results/baseline-*-k6-results.txt . 2>/dev/null || true
cp update-results/update-*-k6-results.txt . 2>/dev/null || true

Expand Down
Loading