Skip to content

CSTACKEX-270: Automation for the template cache workflows #99

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rajiv-jain-netapp merged 6 commits into from
Oct 5, 2026
Merged

CSTACKEX-270: Automation for the template cache workflows #99

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8829e0e
CSTACKEX-270: Automation for the template cache workflows
rajiv-jain-netapp Aug 27, 2026
1e02c80
Merge feature/CSTACKEX-259 into feature/CSTACKEX-270
rajiv-jain-netapp Sep 9, 2026
ee9bbf1
Merge remote-tracking branch 'origin/main' into feature/CSTACKEX-270
rajiv-jain-netapp Sep 10, 2026
52e4815
CSTACKEX-270: added negative test also
rajiv-jain-netapp Sep 11, 2026
ca31c00
CSTACKEX-270: incorporate review comments
rajiv-jain-netapp Sep 24, 2026
f3628d5
CSTACKEX-270: incorporating review comments
rajiv-jain-netapp Oct 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
CSTACKEX-270: incorporate review comments
  • Loading branch information
@rajiv-jain-netapp
rajiv-jain-netapp committed Sep 24, 2026
commit ca31c00d9dd7bfefd5764aad60b24b003a820da1
29 changes: 14 additions & 15 deletions test/integration/plugins/ontap/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ CI wiring:
test/integration/plugins/ontap/
├── ontap.cfg # Environment config (IPs, credentials, zone info)
├── ontap_test_base.py # Shared base class and ONTAP REST client
├── TEST_CASES.md # Full test case reference table (62 tests)
├── TEST_CASES.md # Full test case reference table (72 tests)
├── README.md # This file
│
├── nfs3/
Expand All @@ -42,8 +42,10 @@ test/integration/plugins/ontap/
│ │ └── test_zone_scoped_pool.py # Zone-scoped pool (attachZone)
│ ├── volume/
│ │ └── test_volume_lifecycle.py # Volume create/delete/negative-delete
│ └── instance/
│ └── test_vm_volume_attach.py # Pool + volume + VM + attach/detach
│ ├── instance/
│ │ └── test_vm_volume_attach.py # Pool + volume + VM + attach/detach + template cache
│ └── template/
│ └── test_template_cache_negative.py # Template-cache boundary cases
│
└── iscsi/
├── pool/
Expand All @@ -52,8 +54,10 @@ test/integration/plugins/ontap/
│ └── test_zone_scoped_pool.py # Zone-scoped iSCSI pool
├── volume/
│ └── test_volume_lifecycle.py # LUN create/delete/negative-delete
└── instance/
└── test_vm_volume_attach.py # Pool + LUN + VM + attach/LUN-map lifecycle
├── instance/
│ └── test_vm_volume_attach.py # Pool + LUN + VM + attach/LUN-map lifecycle + template cache
└── template/
└── test_template_cache_negative.py # Template-cache boundary cases
```

---
Expand Down Expand Up @@ -149,14 +153,13 @@ bash test/integration/plugins/ontap/run_tests.sh both
bash test/integration/plugins/ontap/run_tests.sh
bash test/integration/plugins/ontap/run_tests.sh all

# Template-cache suites only
bash test/integration/plugins/ontap/run_tests.sh nfs3_template_cache
bash test/integration/plugins/ontap/run_tests.sh iscsi_template_cache
# Template-cache negative / boundary suites only
# (the template-cache happy path runs inside the VM attach suites)
bash test/integration/plugins/ontap/run_tests.sh nfs3_template_cache_negative
bash test/integration/plugins/ontap/run_tests.sh iscsi_template_cache_negative
```

Each protocol batch runs suites in this order: pool lifecycle → pool with volumes → volume lifecycle → zone-scoped pool → VM attach → template cache (last).
Each protocol batch runs suites in this order: pool lifecycle → pool with volumes → volume lifecycle → zone-scoped pool → VM attach (includes template cache seed / reuse / survive) → template cache negative (last).

| Command | What it runs |
|---------|--------------|
Expand All @@ -165,8 +168,6 @@ Each protocol batch runs suites in this order: pool lifecycle → pool with volu
| `run_tests.sh both` | iSCSI batch, then NFS3 batch + combined report |
| `run_tests.sh all` | `setup_zone`, then `both` (iSCSI before NFS3) |
| `run_tests.sh nfs3_workflow` | Single suite by tag (unchanged) |
| `run_tests.sh nfs3_template_cache` | NFS3 template-cache suite only |
| `run_tests.sh iscsi_template_cache` | iSCSI template-cache suite only |
| `run_tests.sh nfs3_template_cache_negative` | NFS3 template-cache boundary/negative suite |
| `run_tests.sh iscsi_template_cache_negative` | iSCSI template-cache boundary/negative suite |
| `run_tests.sh setup_zone` | Zone setup only |
Expand Down Expand Up @@ -318,15 +319,13 @@ self.assertEqual(result.state, "Maintenance")
| NFS3 Pool with Volumes | `nfs3/pool/test_pool_with_volumes.py` | 7 | Same + live volume present; negative delete guard |
| NFS3 Zone-Scoped Pool | `nfs3/pool/test_zone_scoped_pool.py` | 4 | Zone scope — all hosts connected via `attachZone` |
| NFS3 Volume Lifecycle | `nfs3/volume/test_volume_lifecycle.py` | 5 | Volume is metadata-only; FlexVol unchanged on delete |
| NFS3 VM + Volume Attach | `nfs3/instance/test_vm_volume_attach.py` | 8 | Full VM lifecycle with hot-plug/detach |
| NFS3 Template Cache | `nfs3/template/test_template_cache.py` | 6 | ROOT on tagged pool; seed/reuse cache; survive VM delete |
| NFS3 VM + Volume Attach | `nfs3/instance/test_vm_volume_attach.py` | 10 | Full VM lifecycle with hot-plug/detach; ROOT on tagged pool seeds/reuses template cache, which survives VM delete |
| NFS3 Template Cache Negative | `nfs3/template/test_template_cache_negative.py` | 3 | Tag mismatch; undersized pool; out-of-band cache delete |
| iSCSI Pool Lifecycle | `iscsi/pool/test_pool_lifecycle.py` | 8 | Create, disable, enable, maintenance, delete + igroups |
| iSCSI Pool with Volumes | `iscsi/pool/test_pool_with_volumes.py` | 7 | Same + live LUN present; negative delete guard |
| iSCSI Zone-Scoped Pool | `iscsi/pool/test_zone_scoped_pool.py` | 4 | Zone scope |
| iSCSI Volume Lifecycle | `iscsi/volume/test_volume_lifecycle.py` | 5 | LUN created per CS volume; LUN removed on delete |
| iSCSI VM + Volume Attach | `iscsi/instance/test_vm_volume_attach.py` | 8 | Full VM lifecycle; LUN-maps on VM start/stop/detach |
| iSCSI Template Cache | `iscsi/template/test_template_cache.py` | 6 | ROOT on tagged pool; `cs_tmpl_*` LUN cache seed/reuse |
| iSCSI VM + Volume Attach | `iscsi/instance/test_vm_volume_attach.py` | 10 | Full VM lifecycle; LUN-maps on VM start/stop/detach; ROOT on tagged pool seeds/reuses `cs_tmpl_*` LUN cache |
| iSCSI Template Cache Negative | `iscsi/template/test_template_cache_negative.py` | 3 | Tag mismatch; undersized pool; out-of-band cache delete |

For the goal, dependencies, and exact success criteria of every individual test, see [TEST_CASES.md](TEST_CASES.md).
Expand Down
70 changes: 18 additions & 52 deletions test/integration/plugins/ontap/TEST_CASES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@

# ONTAP Integration Test Cases

Complete reference for all 74 test cases across 12 test suites.
Complete reference for all 72 test cases across 12 test suites.
Each suite is sequential — tests must run in numbered order; each step builds on state created by the previous step.

---
Expand Down Expand Up @@ -114,18 +114,20 @@ Each suite is sequential — tests must run in numbered order; each step builds
**File:** `nfs3/instance/test_vm_volume_attach.py`
**Class:** `TestOntapVMVolumeAttach`
**Tag:** `vm_volume_workflow`
**Total:** 8 tests | **Scope:** end-to-end — NFS3 pool, data volume, running VM, attach/detach lifecycle
**Total:** 10 tests | **Scope:** end-to-end — NFS3 pool, data volume, running VM (ROOT on the ONTAP pool via a tagged compute offering), primary template cache seed / reuse / survive VM delete, attach/detach lifecycle

| # | Test method | Goal | Depends on | CloudStack success criteria | ONTAP success criteria | Type |
|---|-------------|------|------------|-----------------------------|------------------------|------|
| 01 | `test_01_create_nfs3_pool` | Create NFS3 ONTAP primary storage pool | setUpClass (zone, cluster, template) | `pool.state == "Up"` | FlexVol `online`; export policy present | positive |
| 01 | `test_01_create_nfs3_pool` | Create NFS3 ONTAP primary storage pool tagged with `templateCacheTags` | setUpClass (zone, cluster, template, tagged SO) | `pool.state == "Up"` | FlexVol `online`; export policy present | positive |
| 02 | `test_02_create_ontap_data_volume` | Allocate a CloudStack data volume on the ONTAP pool | test_01 (`pool`) | Volume non-None and listed in `listVolumes` | FlexVol still `online` | positive |
| 03 | `test_03_deploy_vm` | Deploy a VM using the first available ready KVM template | test_02 (`pool`, `volume`) | `vm.state == "Running"`; template auto-selected from `listTemplates` | n/a | positive |
| 03 | `test_03_deploy_vm` | Deploy a VM with the tagged SO — ROOT on ONTAP; seeds template cache | test_02 (`pool`, `volume`) | `vm.state == "Running"`; ROOT `storageid` = pool; `template_spool_ref` Ready/DOWNLOADED | Cache file present at spool `install_path` | positive |
| 03a | `test_03a_deploy_second_vm_reuses_template_cache` | Deploy VM-2 — reuses cache | test_03 | VM-2 Running; ROOT on pool; still exactly one `template_spool_ref` | Same cache file (no second cache) | positive |
| 03b | `test_03b_expunge_second_vm_template_cache_survives` | Expunge VM-2 — cache must remain (lazy GC) | test_03a | spool_ref still Ready | Cache file still present | positive |
| 04 | `test_04_attach_volume_to_vm` | Attach the ONTAP data volume to the running VM (hot-plug) | test_03 (`vm`, `volume`) | `volume.virtualmachineid == vm.id`; `attachVolume` job succeeds | FlexVol `online`; after attach, a data file matching volume UUID present in FlexVol (`list_files_in_volume`) | positive |
| 05 | `test_05_stop_vm_export_retained` | Stop the running VM with volume attached | test_04 | `vm.state == "Stopped"` | FlexVol still `online`; NFS export policy still present | positive |
| 06 | `test_06_start_vm_volume_accessible` | Start the stopped VM | test_05 | `vm.state == "Running"` | FlexVol still `online` | positive |
| 07 | `test_07_detach_volume_from_vm` | Hot-detach the ONTAP volume from the running VM (TDS Detach NFS3) | test_06 (`vm`, `volume`) | `volume.virtualmachineid` cleared; `volume.state == "Ready"` | FlexVol still `online`; data file **still present** (NFS3: file persists until `deleteVolume`, not on detach) | positive |
| 08 | `test_08_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, delete pool | test_07 | VM no longer listed; volume no longer listed; pool no longer listed | FlexVol deleted; export policy deleted | cleanup |
| 08 | `test_08_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, force-delete pool | test_07 | VM no longer listed; spool_ref still Ready after VM expunge; volume no longer listed; pool no longer listed | Cache file present after VM expunge; FlexVol deleted; export policy deleted | cleanup |

---

Expand Down Expand Up @@ -206,60 +208,26 @@ Each suite is sequential — tests must run in numbered order; each step builds
**File:** `iscsi/instance/test_vm_volume_attach.py`
**Class:** `TestOntapVMVolumeAttachISCSI`
**Tag:** `iscsi_vm_workflow`
**Total:** 8 tests | **Scope:** end-to-end — iSCSI pool, data volume (LUN), running VM, attach/stop/start/detach lifecycle
**Total:** 10 tests | **Scope:** end-to-end — iSCSI pool, data volume (LUN), running VM (ROOT on the ONTAP pool via a tagged compute offering), `cs_tmpl_<templateId>` LUN cache seed / reuse / survive VM delete, attach/stop/start/detach lifecycle

| # | Test method | Goal | Depends on | CloudStack success criteria | ONTAP success criteria | Type |
|---|-------------|------|------------|-----------------------------|------------------------|------|
| 01 | `test_01_create_iscsi_pool` | Create iSCSI ONTAP primary storage pool | setUpClass | `pool.state == "Up"`, `pool.type == "OntapiSCSI"` | FlexVol `online`; igroup per cluster host with host IQN | positive |
| 01 | `test_01_create_iscsi_pool` | Create iSCSI ONTAP primary storage pool tagged with `templateCacheTags` | setUpClass (tagged SO) | `pool.state == "Up"`, `pool.type == "OntapiSCSI"` | FlexVol `online`; igroup per cluster host with host IQN | positive |
| 02 | `test_02_create_ontap_data_volume` | Allocate a CloudStack data volume (creates a LUN in the FlexVol) | test_01 (`pool`) | Volume non-None | ≥1 LUN in FlexVol | positive |
| 03 | `test_03_deploy_vm` | Deploy VM using first ready KVM template; verify 0 LUN-maps exist before attach | test_02 (`volume`) | `vm.state == "Running"`; 0 LUN-maps on ONTAP | 0 LUN-maps (`list_lun_maps_for_volume` returns empty) | positive |
| 03 | `test_03_deploy_vm` | Deploy VM with the tagged SO — ROOT on ONTAP; seeds `cs_tmpl_*`; verify 0 data-volume LUN-maps before attach | test_02 (`volume`) | `vm.state == "Running"`; ROOT on pool; spool_ref Ready (`local_path` = LUN uuid) | Exactly one `/vol/<flex>/cs_tmpl_<id>` LUN; 0 data-volume LUN-maps | positive |
| 03a | `test_03a_deploy_second_vm_reuses_template_cache` | Deploy VM-2 — reuse cache | test_03 | VM-2 Running; ROOT on pool; still one spool_ref | Still one `cs_tmpl_*`; non-cache LUN count +1 | positive |
| 03b | `test_03b_expunge_second_vm_template_cache_survives` | Expunge VM-2 — cache LUN remains | test_03a | spool_ref still Ready | VM-2 ROOT LUN gone (non-cache count back to baseline); `cs_tmpl_*` still present | positive |
| 04 | `test_04_attach_volume_to_vm` | Hot-attach the ONTAP iSCSI volume to the running VM — a LUN-map is created (TDS SN 27) | test_03 (`vm`, `volume`) | `volume.virtualmachineid == vm.id` | ≥1 LUN-map linking the LUN to the host's igroup | positive |
| 05 | `test_05_stop_vm_lun_unmapped` | Stop VM — LUN-maps must be removed (TDS VM Stop iSCSI) | test_04 | `vm.state == "Stopped"` | 0 LUN-maps; LUN itself **still present** in FlexVol | positive |
| 06 | `test_06_start_vm_lun_remapped` | Start VM — LUN-maps must be re-created (TDS VM Start iSCSI) | test_05 | `vm.state == "Running"` | ≥1 LUN-map re-created | positive |
| 07 | `test_07_detach_volume_from_vm` | Hot-detach the iSCSI volume from the running VM (TDS Detach iSCSI) | test_06 (`vm`, `volume`) | `volume.virtualmachineid` cleared | 0 LUN-maps; LUN still in FlexVol | positive ⚠️ |
| 08 | `test_08_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, delete pool | test_07 | VM gone; volume gone; pool gone | FlexVol deleted; all LUNs and igroups deleted | cleanup |
| 08 | `test_08_destroy_vm_and_cleanup` | Destroy VM (expunge), delete volume, enter maintenance, delete pool | test_07 | VM gone; spool_ref still Ready after VM expunge; volume gone; pool gone | `cs_tmpl_*` present after VM expunge; FlexVol deleted; all LUNs and igroups deleted | cleanup |

> ⚠️ **test_07 known status:** iSCSI hot-detach from a running VM relies on the KVM guest acknowledging the SCSI device removal. On this environment the guest does not acknowledge in time, causing CloudStack error 530. This is a KVM-host-level or guest-template limitation, not a test code defect. All other 61 tests pass.

---

## Suite 11 — NFS3 Template Cache

**File:** `nfs3/template/test_template_cache.py`
**Class:** `TestOntapNfs3TemplateCache`
**Tag:** `nfs3_template_cache`
**Total:** 6 tests | **Scope:** ROOT on tagged NFS3 ONTAP pool; primary template cache seed / reuse / survive VM delete

| # | Test method | Goal | Depends on | CloudStack success criteria | ONTAP success criteria | Type |
|---|-------------|------|------------|-----------------------------|------------------------|------|
| 01 | `test_01_create_tagged_pool_and_service_offering` | Create NFS3 pool + SO sharing `templateCacheTags` | setUpClass | Pool `Up`; SO created | FlexVol `online` | positive |
| 02 | `test_02_deploy_vm1_seeds_template_cache` | Deploy VM-1 — ROOT on ONTAP; seeds cache | test_01 | VM Running; ROOT `storageid` = pool; `template_spool_ref` Ready/DOWNLOADED | Cache file present at spool `install_path` | positive |
| 03 | `test_03_assert_single_spool_ref_and_cache` | Exactly one spool_ref + cache object | test_02 | One `template_spool_ref` row | Cache file still present | positive |
| 04 | `test_04_deploy_vm2_reuses_cache` | Deploy VM-2 — reuses cache | test_03 | Still one spool_ref; VM-2 Running; ROOT on pool | Same cache file (no second cache) | positive |
| 05 | `test_05_destroy_vms_cache_survives` | Expunge VMs — cache must remain (lazy GC) | test_04 | spool_ref still Ready | Cache file still present | positive |
| 06 | `test_06_cleanup_pool_and_offering` | Delete SO; force-delete pool | test_05 | Pool gone | FlexVol deleted | cleanup |

---

## Suite 12 — iSCSI Template Cache

**File:** `iscsi/template/test_template_cache.py`
**Class:** `TestOntapIscsiTemplateCache`
**Tag:** `iscsi_template_cache`
**Total:** 6 tests | **Scope:** ROOT on tagged iSCSI ONTAP pool; `cs_tmpl_<templateId>` LUN cache

| # | Test method | Goal | Depends on | CloudStack success criteria | ONTAP success criteria | Type |
|---|-------------|------|------------|-----------------------------|------------------------|------|
| 01 | `test_01_create_tagged_pool_and_service_offering` | Create iSCSI pool + tagged SO | setUpClass | Pool `Up`; SO created | FlexVol `online` | positive |
| 02 | `test_02_deploy_vm1_seeds_template_cache` | Deploy VM-1 — seeds `cs_tmpl_*` + ROOT LUN | test_01 | VM Running; ROOT on pool; spool_ref Ready (`local_path` = LUN uuid) | LUN `/vol/<flex>/cs_tmpl_<id>` exists; ≥1 volume LUN | positive |
| 03 | `test_03_assert_single_spool_ref_and_cache` | Exactly one spool_ref + one cache LUN | test_02 | One spool_ref | Exactly one `cs_tmpl_*` LUN | positive |
| 04 | `test_04_deploy_vm2_reuses_cache` | Deploy VM-2 — reuse cache | test_03 | Still one spool_ref | Still one `cs_tmpl_*`; volume LUN count +1 | positive |
| 05 | `test_05_destroy_vms_cache_survives` | Expunge VMs — cache LUN remains | test_04 | spool_ref still Ready | Volume LUNs gone; `cs_tmpl_*` still present | positive |
| 06 | `test_06_cleanup_pool_and_offering` | Delete SO; force-delete pool | test_05 | Pool gone | FlexVol deleted | cleanup |

---

## Suite 13 — NFS3 Template Cache Negative / Boundary
## Suite 11 — NFS3 Template Cache Negative / Boundary

**File:** `nfs3/template/test_template_cache_negative.py`
**Class:** `TestOntapNfs3TemplateCacheNegative`
Expand All @@ -274,7 +242,7 @@ Each suite is sequential — tests must run in numbered order; each step builds

---

## Suite 14 — iSCSI Template Cache Negative / Boundary
## Suite 12 — iSCSI Template Cache Negative / Boundary

**File:** `iscsi/template/test_template_cache_negative.py`
**Class:** `TestOntapIscsiTemplateCacheNegative`
Expand All @@ -297,14 +265,12 @@ Each suite is sequential — tests must run in numbered order; each step builds
| NFS3 Pool with Volumes | NFS3 | Cluster | 7 | ✅ |
| NFS3 Zone-Scoped Pool | NFS3 | Zone | 4 | ✅ |
| NFS3 Volume Lifecycle | NFS3 | Cluster | 5 | ✅ |
| NFS3 VM + Volume Attach | NFS3 | Cluster | 8 | ✅ |
| NFS3 Template Cache | NFS3 | Cluster | 6 | 🆕 |
| NFS3 VM + Volume Attach | NFS3 | Cluster | 10 | 🆕 +2 template cache |
| NFS3 Template Cache Negative | NFS3 | Cluster | 3 | 🆕 |
| iSCSI Pool Lifecycle | iSCSI | Cluster | 8 | ✅ |
| iSCSI Pool with Volumes | iSCSI | Cluster | 7 | ✅ |
| iSCSI Zone-Scoped Pool | iSCSI | Zone | 4 | ✅ |
| iSCSI Volume Lifecycle | iSCSI | Cluster | 5 | ✅ |
| iSCSI VM + Volume Attach | iSCSI | Cluster | 8 | ⚠️ 7/8 |
| iSCSI Template Cache | iSCSI | Cluster | 6 | 🆕 |
| iSCSI VM + Volume Attach | iSCSI | Cluster | 10 | ⚠️ 7/8 + 🆕 2 template cache |
| iSCSI Template Cache Negative | iSCSI | Cluster | 3 | 🆕 |
| **Total** | | | **80** | |
| **Total** | | | **72** | |
Loading
Loading