Skip to content

Commit 599aea9

Browse files
sushanbsushanb
authored
fix(bigtable): real per-resource pool teardown on sessionTable.Close + cache close-race gate (#20264)
Fixes two related latent bugs in the session data plane: 1. **sessionTable.Close was a no-op** — the interface godoc at `bigtable/internal/session/api.go:45-48` promises to release the resource's read + write session pools; the implementation returned nil. Pools were reclaimed only by `sessionClient.Close`, so `bigtable.Client`'s TTL cache could evict a handle without freeing the underlying pool + streams + goroutines. 2. **sessionTableCache close-race (audit finding #5)** — a slow-path openFn straddling `sessionTableCache.close()` would install a fresh handle into a cache the sweeper had already stopped clearing. Zero-impact while sessionTable.Close was a no-op; a per-race pool leak once teardown becomes real. ## Design Real teardown routed through symmetric release closures: - `sessionClient.releaseSessionPool(key)` — under `sessionPoolsMu`, delete the entry then `unregister()` + `pool.Close()` outside the lock (matches the snapshot-under-lock pattern in `Close`). - `buildLazyReleaser(key)` — sibling to `buildLazyOpener`; returns a `func() error` closure for a specific poolKey. - `sessionTable.Close` — invokes closeRead + closeWrite, joining errors via `errors.Join`. Nil-safe for materialized views' missing write side. No refcount inside sessionClient. Rationale: `bigtable.Client`'s sessionTableCache dedupes handles per fully-qualified resource name, so at-most-one sessionTable per resource per Client at any moment — the cache is the 'refcount of at-most-1'. Doc on `sessionTable.Close` names the invariant; a future caller bypassing the cache would need to add a refcount then. Paired cache guard: `sessionTableCache` gains a `closed bool` under `c.mu`, flipped by `close()`. `getOrOpen`'s slow path re-checks it before insert and, if set, releases the freshly-opened api and returns nil so `TableShim` falls back to classic. Prevents the finding-#5 leak. ## Tests New unit tests (all pass under `-race`): - `TestSessionTable_Close_CallsBothReleasers` - `TestSessionTable_Close_NilWriteReleaserOK` (materialized view) - `TestSessionTable_Close_JoinsErrors` - `TestSessionTable_Close_ReleasersIdempotent` - `TestReleaseSessionPool_AfterClientClose_NoOp` - `TestReleaseSessionPool_MissingKeyNoOp` - `TestReleaseSessionPool_RemovesEntryAndInvokesUnregister` - `TestSessionTableCache_ClosedGate_SlowPathInsertNoLeak` ## Drive-by The pre-existing `closeCountingTable` test helper races between the sweeper goroutine's `*counter++` and the test-goroutine's read. Switched `*int` to `*atomic.Int32` so the full-package `-race` sweep is green (fires on the base branch too — this was blocking my race-stress verification). ## Test plan - [x] `go build ./...` and `go vet ./...` clean - [x] `go test -race -count=1 -short -timeout=120s ./ ./internal/session/ ./internal/transport/` — all green - [x] `TestSessionTable_Close*` and `TestReleaseSessionPool*` — all pass - [x] `TestSessionTableCache_ClosedGate_SlowPathInsertNoLeak` — reproduces the race deterministically, passes with the fix - [x] Sandbox smoke against sushanb-uc1 via `CBT_RUN_SANDBOX=1 CBT_FORCE_SESSION=true`: Table + AV round-trip on both classic and session paths ## Stack Stacked on #20263 (session_table_cache) which is stacked on #20262 (session.Client wiring into bigtable.Client Open*). Both must land first, or this PR must be rebased onto main after they merge.
1 parent d630a35 commit 599aea9

8 files changed

Lines changed: 573 additions & 46 deletions

File tree

‎bigtable/client.go‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -275,8 +275,15 @@ func NewClientWithConfig(ctx context.Context, project, instance string, config C
275275
// traffic through this Client's Diverter — a control-plane update
276276
// then shifts traffic across every open TableShim without a client
277277
// restart.
278+
// Inspect the merged option list (o), not the raw caller opts.
279+
// BIGTABLE_EMULATOR_HOST injects option.WithGRPCConn inside
280+
// btopt.DefaultClientOptions (option.go:113), so it lives in o —
281+
// callers never pass it explicitly. Reading only from opts misses
282+
// the emulator conn and lets session.NewClient dial an empty
283+
// resolver target (fails with "passthrough: received empty target
284+
// in Build()"), breaking every emulator-based test.
278285
preDialed := false
279-
if uResolver, resErr := internaloption.NewUnsafeResolver(opts...); resErr == nil {
286+
if uResolver, resErr := internaloption.NewUnsafeResolver(o...); resErr == nil {
280287
preDialed = uResolver.ResolvedGRPCConnIsCustom()
281288
}
282289
if !preDialed {

‎bigtable/internal/session/api.go‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,14 @@ type TableAPI interface {
4343
MutateRow(ctx context.Context, req *btpb.SessionMutateRowRequest) (*btpb.SessionMutateRowResponse, error)
4444

4545
// Close releases this resource's underlying read + write session
46-
// pools. Independent from Client.Close — closing an
47-
// individual resource does not close the shared channel pool.
46+
// pools from the sessionClient's per-resource keyed map. Idempotent.
47+
// Independent from Client.Close — closing an individual resource
48+
// does not close the shared channel pool.
49+
//
50+
// Per-handle pool teardown is safe because callers reach this method
51+
// through bigtable.Client's sessionTableCache, which guarantees
52+
// at-most-one TableAPI per resource per Client. A future caller that
53+
// bypasses that cache must add a refcount before calling Close.
4854
Close() error
4955
}
5056

‎bigtable/internal/session/client.go‎

Lines changed: 72 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -466,43 +466,53 @@ func (sc *sessionClient) OpenTable(tableID string) TableAPI {
466466
fullName := sc.fullTableName(tableID)
467467
streamFactory := func(ctx context.Context) (btransport.Stream, error) { return sc.stub.OpenTable(ctx) }
468468
resource := "table:" + tableID
469+
readKey := poolKey{resource, permissionRead}
470+
writeKey := poolKey{resource, permissionWrite}
469471
openRead := sc.buildLazyOpener(fullName, btransport.TABLE_SESSION, streamFactory,
470472
&btpb.OpenTableRequest{TableName: fullName, AppProfileId: sc.cfg.AppProfile, Permission: btpb.OpenTableRequest_PERMISSION_READ},
471-
poolKey{resource, permissionRead})
473+
readKey)
472474
openWrite := sc.buildLazyOpener(fullName, btransport.TABLE_SESSION, streamFactory,
473475
&btpb.OpenTableRequest{TableName: fullName, AppProfileId: sc.cfg.AppProfile, Permission: btpb.OpenTableRequest_PERMISSION_WRITE},
474-
poolKey{resource, permissionWrite})
475-
return newSessionTable(tableID, openRead, openWrite, btransport.READ_ROW, btransport.MUTATE_ROW, sc.perResourceMetadata(fullName, "table_name", fullName), sc.metricsFactory)
476+
writeKey)
477+
closeRead := sc.buildLazyReleaser(readKey)
478+
closeWrite := sc.buildLazyReleaser(writeKey)
479+
return newSessionTable(tableID, openRead, openWrite, closeRead, closeWrite, btransport.READ_ROW, btransport.MUTATE_ROW, sc.perResourceMetadata(fullName, "table_name", fullName), sc.metricsFactory)
476480
}
477481

478482
// OpenAuthorizedView returns a TableAPI for an authorized view.
479483
func (sc *sessionClient) OpenAuthorizedView(table, view string) TableAPI {
480484
fullName := sc.fullAuthorizedViewName(table, view)
481485
streamFactory := func(ctx context.Context) (btransport.Stream, error) { return sc.stub.OpenAuthorizedView(ctx) }
482486
resource := fmt.Sprintf("av:%s:%s", table, view)
487+
readKey := poolKey{resource, permissionRead}
488+
writeKey := poolKey{resource, permissionWrite}
483489
openRead := sc.buildLazyOpener(fullName, btransport.AUTHORIZED_VIEW_SESSION, streamFactory,
484490
&btpb.OpenAuthorizedViewRequest{AuthorizedViewName: fullName, AppProfileId: sc.cfg.AppProfile, Permission: btpb.OpenAuthorizedViewRequest_PERMISSION_READ},
485-
poolKey{resource, permissionRead})
491+
readKey)
486492
openWrite := sc.buildLazyOpener(fullName, btransport.AUTHORIZED_VIEW_SESSION, streamFactory,
487493
&btpb.OpenAuthorizedViewRequest{AuthorizedViewName: fullName, AppProfileId: sc.cfg.AppProfile, Permission: btpb.OpenAuthorizedViewRequest_PERMISSION_WRITE},
488-
poolKey{resource, permissionWrite})
489-
return newSessionTable(table, openRead, openWrite, btransport.READ_ROW_AUTH_VIEW, btransport.MUTATE_ROW_AUTH_VIEW, sc.perResourceMetadata(fullName, "authorized_view_name", fullName), sc.metricsFactory)
494+
writeKey)
495+
closeRead := sc.buildLazyReleaser(readKey)
496+
closeWrite := sc.buildLazyReleaser(writeKey)
497+
return newSessionTable(table, openRead, openWrite, closeRead, closeWrite, btransport.READ_ROW_AUTH_VIEW, btransport.MUTATE_ROW_AUTH_VIEW, sc.perResourceMetadata(fullName, "authorized_view_name", fullName), sc.metricsFactory)
490498
}
491499

492500
// OpenMaterializedView returns a read-only TableAPI for a
493501
// materialized view. Only a read pool is opened; MutateRow errors
494502
// cleanly via the nil openWrite passed to newSessionTable.
495503
func (sc *sessionClient) OpenMaterializedView(view string) TableAPI {
496504
fullName := sc.fullMaterializedViewName(view)
505+
readKey := poolKey{"mv:" + view, permissionRead}
497506
openRead := sc.buildLazyOpener(fullName, btransport.MATERIALIZED_VIEW_SESSION,
498507
func(ctx context.Context) (btransport.Stream, error) { return sc.stub.OpenMaterializedView(ctx) },
499508
&btpb.OpenMaterializedViewRequest{
500509
MaterializedViewName: fullName,
501510
AppProfileId: sc.cfg.AppProfile,
502511
Permission: btpb.OpenMaterializedViewRequest_PERMISSION_READ,
503512
},
504-
poolKey{"mv:" + view, permissionRead})
505-
return newSessionTable("", openRead, nil, btransport.READ_ROW_MAT_VIEW, nil, sc.perResourceMetadata(fullName, "materialized_view_name", fullName), sc.metricsFactory)
513+
readKey)
514+
closeRead := sc.buildLazyReleaser(readKey)
515+
return newSessionTable("", openRead, nil, closeRead, nil, btransport.READ_ROW_MAT_VIEW, nil, sc.perResourceMetadata(fullName, "materialized_view_name", fullName), sc.metricsFactory)
506516
}
507517

508518
// Close tears down in a phased order that keeps late callbacks from
@@ -592,6 +602,60 @@ func (sc *sessionClient) buildLazyOpener(
592602
}
593603
}
594604

605+
// buildLazyReleaser returns a closure that releases the pool entry
606+
// for the given key from sessionPools + closes the underlying pool.
607+
// Returned closure is idempotent (second call finds entry absent and
608+
// no-ops) and nil-safe when key is the zero value (the caller can
609+
// pass nil for resources without a write side — e.g. materialized
610+
// views — mirroring buildLazyOpener's payload==nil convention).
611+
//
612+
// Symmetric with buildLazyOpener so sessionTable can drive real
613+
// per-resource teardown from its Close() without needing back-refs
614+
// or knowing about poolKey / sessionPools internals.
615+
func (sc *sessionClient) buildLazyReleaser(key poolKey) func() error {
616+
return func() error {
617+
return sc.releaseSessionPool(key)
618+
}
619+
}
620+
621+
// releaseSessionPool removes key from sessionPools and closes the
622+
// underlying pool. No-op when:
623+
// - the client has already Closed (sessionPools == nil), so any late
624+
// release from a cache handle draining on the way out doesn't
625+
// racily double-close a pool that Client.Close is already
626+
// tearing down.
627+
// - the entry is absent (already released — makes second calls safe).
628+
//
629+
// Assumes the caller (currently bigtable.Client via sessionTableCache)
630+
// guarantees at-most-one sessionTable per resource at any moment. That
631+
// invariant means no co-owner can be relying on the pool we're closing.
632+
// If session.Client is ever exposed to callers that bypass that cache,
633+
// this must gain a refcount.
634+
//
635+
// Teardown runs OUTSIDE sessionPoolsMu so a graceful pool drain (which
636+
// can block on in-flight vRPCs) doesn't deadlock any snapshotter
637+
// waiting on the lock. Matches the snapshot-under-lock / teardown-
638+
// outside pattern in Client.Close.
639+
func (sc *sessionClient) releaseSessionPool(key poolKey) error {
640+
sc.sessionPoolsMu.Lock()
641+
if sc.sessionPools == nil {
642+
sc.sessionPoolsMu.Unlock()
643+
return nil
644+
}
645+
mp, ok := sc.sessionPools[key]
646+
if !ok {
647+
sc.sessionPoolsMu.Unlock()
648+
return nil
649+
}
650+
delete(sc.sessionPools, key)
651+
sc.sessionPoolsMu.Unlock()
652+
653+
if mp.unregister != nil {
654+
mp.unregister()
655+
}
656+
return mp.pool.Close()
657+
}
658+
595659
// createSessionPoolForPayload marshals the resource-typed OpenXxxRequest
596660
// into the transport-level OpenSessionRequest envelope, builds routing
597661
// metadata via the descriptor's MetadataFn, and delegates to

‎bigtable/internal/session/client_test.go‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import (
2323

2424
btpb "cloud.google.com/go/bigtable/apiv2/bigtablepb"
2525
metrics "cloud.google.com/go/bigtable/internal/metrics"
26+
btransport "cloud.google.com/go/bigtable/internal/transport"
2627
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
2728
"google.golang.org/grpc/metadata"
2829
"google.golang.org/protobuf/proto"
@@ -456,3 +457,67 @@ func TestPoolKey_DisplayName(t *testing.T) {
456457
})
457458
}
458459
}
460+
461+
// --- releaseSessionPool ------------------------------------------------------
462+
463+
// TestReleaseSessionPool_AfterClientClose_NoOp: once Close nils out
464+
// sessionPools, any late release from a cache handle draining on the
465+
// way out must be a benign no-op rather than a nil-map panic.
466+
func TestReleaseSessionPool_AfterClientClose_NoOp(t *testing.T) {
467+
sc := newTestClient(t, &fakeChannelPool{}, Config{})
468+
sc.sessionPools = nil // simulate Close having snapshotted+nil'd
469+
if err := sc.releaseSessionPool(poolKey{"table:foo", permissionRead}); err != nil {
470+
t.Errorf("releaseSessionPool on nil map = %v, want nil (post-Close no-op)", err)
471+
}
472+
}
473+
474+
// TestReleaseSessionPool_MissingKeyNoOp: releasing a key that isn't in
475+
// the map returns nil and leaves other entries untouched. This is what
476+
// makes double-release safe (winner deletes; loser sees absent).
477+
func TestReleaseSessionPool_MissingKeyNoOp(t *testing.T) {
478+
sc := newTestClient(t, &fakeChannelPool{}, Config{})
479+
present := poolKey{"table:present", permissionRead}
480+
sc.sessionPools[present] = &managedSessionPool{} // sentinel, not touched
481+
if err := sc.releaseSessionPool(poolKey{"table:absent", permissionRead}); err != nil {
482+
t.Errorf("releaseSessionPool on absent key = %v, want nil", err)
483+
}
484+
if _, still := sc.sessionPools[present]; !still {
485+
t.Error("releaseSessionPool on absent key deleted a different entry")
486+
}
487+
}
488+
489+
// TestReleaseSessionPool_RemovesEntryAndInvokesUnregister covers the
490+
// happy path with a real (unstarted) SessionPoolImpl. The pool never
491+
// dialed a stream so Close returns cleanly; we assert the map entry
492+
// is gone AND the config-listener unregister thunk fired.
493+
func TestReleaseSessionPool_RemovesEntryAndInvokesUnregister(t *testing.T) {
494+
sc := newTestClient(t, &fakeChannelPool{}, Config{})
495+
key := poolKey{"table:foo", permissionRead}
496+
pool := btransport.NewSessionPoolImpl(
497+
1, "test-pool", 0, 0,
498+
func(ctx context.Context) (btransport.Stream, error) { return nil, errors.New("test never dials") },
499+
&btpb.OpenSessionRequest{}, nil,
500+
btransport.SessionTypeTable, false,
501+
)
502+
var unregistered int
503+
sc.sessionPools[key] = &managedSessionPool{
504+
pool: pool,
505+
unregister: func() { unregistered++ },
506+
}
507+
if err := sc.releaseSessionPool(key); err != nil {
508+
t.Fatalf("releaseSessionPool = %v, want nil", err)
509+
}
510+
if _, still := sc.sessionPools[key]; still {
511+
t.Error("sessionPools still holds the released key")
512+
}
513+
if unregistered != 1 {
514+
t.Errorf("unregister fired %d times, want 1", unregistered)
515+
}
516+
// Second release is a clean no-op (winner deleted, loser sees absent).
517+
if err := sc.releaseSessionPool(key); err != nil {
518+
t.Errorf("second releaseSessionPool = %v, want nil", err)
519+
}
520+
if unregistered != 1 {
521+
t.Errorf("unregister fired %d times after second release, want still 1", unregistered)
522+
}
523+
}

‎bigtable/internal/session/table.go‎

Lines changed: 103 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ import (
1818
"context"
1919
"errors"
2020
"fmt"
21+
"sync/atomic"
2122

2223
btpb "cloud.google.com/go/bigtable/apiv2/bigtablepb"
2324
metrics "cloud.google.com/go/bigtable/internal/metrics"
@@ -64,29 +65,90 @@ type sessionTable struct {
6465
writeVRpcDesc btransport.VRpcDescriptor
6566
md metadata.MD
6667
metricsFactory *metrics.Factory
68+
// closeRead / closeWrite release the per-resource pool entries on
69+
// this sessionTable's Close(). Both are supplied by sessionClient
70+
// via buildLazyReleaser and no-op cleanly when the pool was never
71+
// opened or the client already tore itself down. closeWrite is nil
72+
// for materialized views (read-only resources), mirroring the nil
73+
// write side of the openRead/openWrite pair.
74+
closeRead func() error
75+
closeWrite func() error
76+
// closed is set by Close() BEFORE the releasers run, and re-checked
77+
// inside the lazy-open wrapper (guardOpen) around openRead/openWrite
78+
// so an opener whose slow path straddles Close cleans up its own
79+
// insert instead of leaking a fresh pool that the releaser missed.
80+
// See guardOpen for the interleaving rules.
81+
closed atomic.Bool
6782
}
6883

6984
// newSessionTable is the internal constructor. Callers (sessionClient)
70-
// build the lazyPool open closures + supply the vRPC descriptors and
71-
// resource-scoped metadata. metricsFactory may be nil to disable
72-
// per-attempt metrics.
85+
// build the lazyPool open + release closures, supply the vRPC
86+
// descriptors, and resource-scoped metadata. metricsFactory may be nil
87+
// to disable per-attempt metrics. closeWrite may be nil for
88+
// materialized views (no write side).
7389
func newSessionTable(
7490
tableID string,
7591
openRead func() (Invoker, error),
7692
openWrite func() (Invoker, error),
93+
closeRead func() error,
94+
closeWrite func() error,
7795
readVRpcDesc btransport.VRpcDescriptor,
7896
writeVRpcDesc btransport.VRpcDescriptor,
7997
md metadata.MD,
8098
metricsFactory *metrics.Factory,
8199
) *sessionTable {
82-
return &sessionTable{
100+
t := &sessionTable{
83101
tableID: tableID,
84-
readPool: &lazyPool{open: openRead},
85-
writePool: &lazyPool{open: openWrite},
86102
readVRpcDesc: readVRpcDesc,
87103
writeVRpcDesc: writeVRpcDesc,
88104
md: md,
89105
metricsFactory: metricsFactory,
106+
closeRead: closeRead,
107+
closeWrite: closeWrite,
108+
}
109+
// Wrap the raw openers with a closed-bit guard so an open racing
110+
// with Close either bails early or cleans up its own insert. Nil
111+
// openers stay nil to preserve the MV-write-side "no pool" contract.
112+
t.readPool = &lazyPool{open: t.guardOpen(openRead, closeRead)}
113+
t.writePool = &lazyPool{open: t.guardOpen(openWrite, closeWrite)}
114+
return t
115+
}
116+
117+
// guardOpen wraps a lazy pool opener with a check-before + check-after
118+
// against t.closed. Interleaving cases:
119+
//
120+
// - Close ran first: early check trips, opener never runs. No leak.
121+
// - Close runs while opener's slow path holds sessionPoolsMu: post-check
122+
// trips after opener returns; guardOpen invokes release itself to tear
123+
// down the pool the opener just inserted. releaseSessionPool is
124+
// idempotent, so Close's parallel call to release harmlessly no-ops.
125+
// - Close runs after guardOpen returns: normal path — Close's releaser
126+
// finds the pool in sessionPools and closes it. In-flight Invoke on
127+
// the returned pool may fail with a "pool closed" error; the client
128+
// was concurrently being torn down, so that's expected.
129+
//
130+
// A nil opener passes through as nil so the MV write side (no write pool)
131+
// stays a lazyPool-with-nil-open ("no session support"), not a
132+
// guarded-nil that would try to Load t.closed on every get.
133+
func (t *sessionTable) guardOpen(open func() (Invoker, error), release func() error) func() (Invoker, error) {
134+
if open == nil {
135+
return nil
136+
}
137+
return func() (Invoker, error) {
138+
if t.closed.Load() {
139+
return nil, ErrClientClosed
140+
}
141+
inv, err := open()
142+
if err != nil {
143+
return nil, err
144+
}
145+
if t.closed.Load() {
146+
if release != nil {
147+
_ = release()
148+
}
149+
return nil, ErrClientClosed
150+
}
151+
return inv, nil
90152
}
91153
}
92154

@@ -219,12 +281,42 @@ func dispatch[Args any, R any, Resp interface {
219281
return resp, nil
220282
}
221283

222-
// Close is a no-op today — the underlying pools are shared across
223-
// resources via sessionClient.pools and torn down by sessionClient.Close.
224-
// Retained on the interface so callers get a symmetric Open/Close
225-
// pattern and future implementations can add per-resource teardown.
284+
// Close releases this sessionTable's underlying read + write session
285+
// pools from the sessionClient's per-resource keyed map. Both release
286+
// closures are idempotent (second call finds the map entry absent
287+
// and no-ops) so double-close from the cache sweeper + explicit
288+
// caller is safe.
289+
//
290+
// Ordering matters: closed is Store'd BEFORE the releasers run so an
291+
// opener whose slow path is straddling this Close (already past its
292+
// early check, still dialing) sees closed=true on its post-insert
293+
// re-check inside guardOpen and cleans up its own insert. See guardOpen.
294+
// Otherwise the releaser would no-op on the empty map and the
295+
// post-insert would leak a fresh pool.
296+
//
297+
// Safety of per-handle pool teardown also rests on a caller-side
298+
// invariant: bigtable.Client's sessionTableCache guarantees at-most-one
299+
// sessionTable per resource at any moment, so no co-owner sharing a
300+
// pool with us can be mid-flight when we tear it down. If session.Client
301+
// ever gains a caller that bypasses that cache, this method must gain
302+
// a refcount on the sessionClient side (or the caller must add its own).
303+
//
304+
// Returns the joined read + write teardown errors (nil for the
305+
// materialized-view case where closeWrite is nil).
226306
func (t *sessionTable) Close() error {
227-
return nil
307+
t.closed.Store(true)
308+
var errs []error
309+
if t.closeRead != nil {
310+
if err := t.closeRead(); err != nil {
311+
errs = append(errs, err)
312+
}
313+
}
314+
if t.closeWrite != nil {
315+
if err := t.closeWrite(); err != nil {
316+
errs = append(errs, err)
317+
}
318+
}
319+
return errors.Join(errs...)
228320
}
229321

230322
// ensureTracer returns a Tracer stashed on ctx (via metrics.NewContext

0 commit comments

Comments
 (0)