google-cloud-storage: BucketMetadataCache (ACO) triggers storage.buckets.get on object-only operations; no supported opt-out #17650

Environment details

• Programming language: Python 3.11
• Package: google-cloud-storage
• Versions observed: 3.11.0 and 3.12.0 (the BucketMetadataCache / ACO code path is present in both)

Summary

Client.__init__ unconditionally instantiates a BucketMetadataCache ("App-centric Observability" / ACO). As a side effect, ordinary object-level operations (list_blobs, download_*, upload_*, etc.) now trigger a background Client.get_bucket() call, i.e. the REST storage.buckets.get permission — even though the application code never requests bucket metadata and the operation itself only needs storage.objects.*.

For principals granted object-only roles (roles/storage.objectViewer, roles/storage.objectAdmin) — which intentionally exclude storage.buckets.get — this produces a continuous stream of denied storage.buckets.get entries in Cloud Audit Logs (data_access, severity ERROR), one per bucket per client instance.

The failure is swallowed internally (data operations still succeed), so it is not a functional outage — but it is:

• Alarming, high-volume audit-log noise (ERROR severity, "lots per second" under fan-out workloads that create many short-lived clients), and
• Pressure to over-grant IAM — teams are asked to add storage.buckets.get (e.g. via roles/storage.bucketViewer/legacyBucketReader) purely to satisfy a telemetry probe the application does not use, across every bucket the service account ever touches. This directly conflicts with least-privilege.

There is no supported way to disable it — no client option and no environment variable (the only observability env var, ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES, is a separate opt-in for OpenTelemetry tracing). The only workaround is mutating the private attribute client._bucket_metadata_cache = None, which is fragile and fails open on library upgrades.

Steps to reproduce

Object-only operations cause a background get_bucket() (→ storage.buckets.get). The repro below is fully offline and needs no credentials or real bucket — it monkeypatches get_bucket only to observe that it is invoked and by whom:

import time
import traceback

from google.auth.credentials import AnonymousCredentials
from google.cloud import storage

calls = []
def traced_get_bucket(self, bucket_name, *a, **k):
    if not calls:
        print("get_bucket() caller:\n" + "".join(traceback.format_stack()[-3:-1]))
    calls.append(bucket_name)
    raise RuntimeError("short-circuit (offline repro)")  # swallowed by ACO bg thread
storage.Client.get_bucket = traced_get_bucket

client = storage.Client(project="proof", credentials=AnonymousCredentials())
# Optional workaround under test: client._bucket_metadata_cache = None
try:
    list(client.list_blobs("some-bucket", prefix="x/", max_results=1))
except Exception:
    pass                       # objects.list fails offline; irrelevant to the point
time.sleep(1.0)                # let the ACO daemon thread run

print(f"google-cloud-storage {storage.__version__}: "
      f"get_bucket invoked {len(calls)} time(s) by one list_blobs() call")

Actual output

get_bucket() caller:
  File ".../google/cloud/storage/_bucket_metadata_cache.py", line 98, in _fetch_background
    bucket = self._client.get_bucket(bucket_name, timeout=10.0)
google-cloud-storage 3.11.0: get_bucket invoked 1 time(s) by one list_blobs() call

With client._bucket_metadata_cache = None uncommented, the count is 0.

Root cause (call chain)

1. _helpers.create_trace_span_helper wraps every operation. For any span not in {Storage.Client.getBucket, Storage.Client.lookupBucket, Storage.Bucket.reload, Storage.Bucket.exists} (i.e. object ops like listBlobs), it calls client._bucket_metadata_cache.get_or_queue_fetch(bucket_name).
2. On a cache miss, get_or_queue_fetch spawns a daemon thread → _bucket_metadata_cache._fetch_background.
3. _fetch_background calls self._client.get_bucket(bucket_name, timeout=10.0) → REST storage.buckets.get.
4. On Forbidden, it caches a fallback and returns; the object op proceeds. Every fresh client re-probes on first touch of each bucket.

Expected behavior

Object-level operations should not require storage.buckets.get. This is a long-standing, explicitly-recognized contract for this library — cf. googleapis/python-storage #528 ("Redesign of API so it does not need storage.buckets.get access for every single operation") and #1225 ("Need to be able to download objects without storage.buckets.get permission same as gsutil"). gsutil/gcloud storage operate on objects without buckets.get, and object-only IAM roles deliberately omit it.

At minimum, ACO's background metadata probe should be opt-out via a supported mechanism, and ideally should not fire for principals/operations that don't need bucket metadata.

Requested change

A supported opt-out, e.g. one or more of:

1. A Client constructor option (e.g. enable_bucket_metadata_cache: bool = True), and/or
2. An environment variable mirroring the existing ENABLE_GCS_PYTHON_CLIENT_OTEL_TRACES pattern (e.g. DISABLE_GCS_PYTHON_CLIENT_BUCKET_METADATA_CACHE), and/or
3. Making the probe lazy/suppressible so a Forbidden result is cached once per client without re-emitting audit-log errors, and documenting that ACO requires storage.buckets.get.

Any of these lets callers using object-only roles avoid the audit-log noise without granting storage.buckets.get, and gives a stable alternative to mutating the private _bucket_metadata_cache attribute.

Happy to open a PR for option 1 or 2 if a maintainer confirms the preferred shape.