Skip to content

Commit 2ec917f

Browse files
Feature: store tool artifact on host (dodopizza#18)
* feat(cli): define flag for store on host artifact & implement cli part Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(cli): refactor output handling Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(*): extract constant usage Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(*): extract constant usage Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(*): extract constant usage Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(watchdog): eliminate code duplication Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * feat(cli): pass additional args to dumper Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * feat(dumper): implement storing dump on host Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(cli): convert var to field Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(tests): add test cases for store on host feature Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(test): eliminate test code duplication Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(tests): eliminate code duplication Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(dumper): resolve linting issue Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(tests): fix unit test Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(cli): rename flag shorthand Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(*): update cli docs Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(tests): update targets to run integration tests Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(*): format imports Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(dumper): fix dumper flags usage Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(*): move arch mapping in tests to shell scripts Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(tests): make test logging with t.Log & fix test case usage Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(docs): fix typos in docs Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * feat(watchdog): starts operators only on output flags Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(tests): split integration tests to two stages Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(*): fix flags generation Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * fix(dumper): fix output file location for store-on-host Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(tests): use parallel in setup method Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(test): run tests in parallel in ci mode Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com> * chore(flags): extract helper type to build cli args & update usages Signed-off-by: ArtemTrofimushkin <artemtrofimushkin@gmail.com>
1 parent 81c18cd commit 2ec917f

37 files changed

Lines changed: 677 additions & 498 deletions

‎Makefile‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ GREEN := $(shell tput -Txterm setaf 2)
22
YELLOW := $(shell tput -Txterm setaf 3)
33
WHITE := $(shell tput -Txterm setaf 7)
44
RESET := $(shell tput -Txterm sgr0)
5+
ARCH := $(shell uname -m)
56

67
.PHONY: all
78
all: help
@@ -41,13 +42,14 @@ test: test-unit test-integration
4142
test-unit:
4243
TEST_RUN_ARGS="$(TEST_RUN_ARGS)" TEST_DIR="$(TEST_DIR)" ./hacks/run-unit-tests.sh
4344

45+
.PHONY: test-integration-prepare
46+
test-integration-prepare:
47+
./hacks/prepare-integration-tests.sh "$(ARCH)"
48+
4449
.PHONY: test-integration
4550
test-integration:
46-
./hacks/run-integration-tests.sh amd64
47-
48-
.PHONY: test-integration-arm64
49-
test-integration-arm64:
50-
./hacks/run-integration-tests.sh arm64
51+
./hacks/prepare-integration-tests.sh "$(ARCH)"
52+
./hacks/run-integration-tests.sh
5153

5254
.PHONY: prepare
5355
prepare: tidy lint doc build-cli build-dumper test
@@ -59,15 +61,15 @@ help:
5961
@echo ' ${YELLOW}make${RESET} ${GREEN}<target>${RESET}'
6062
@echo ''
6163
@echo 'Targets:'
62-
@echo " ${YELLOW}cover ${RESET} Open html coverage report in browser"
63-
@echo " ${YELLOW}doc ${RESET} Run doc generation"
64-
@echo " ${YELLOW}lint ${RESET} Run linters via golangci-lint"
65-
@echo " ${YELLOW}tidy ${RESET} Run tidy for go module to remove unused dependencies"
66-
@echo " ${YELLOW}build-cli ${RESET} Build cli component of shovel"
67-
@echo " ${YELLOW}build-dumper ${RESET} Build dumper component of shovel"
68-
@echo " ${YELLOW}setup ${RESET} Setup local environment. Create kind cluster"
69-
@echo " ${YELLOW}test ${RESET} Run all available tests"
70-
@echo " ${YELLOW}test-unit ${RESET} Run all unit tests"
71-
@echo " ${YELLOW}test-integration ${RESET} Run all integration tests (for amd64 arch)"
72-
@echo " ${YELLOW}test-integration-arm64 ${RESET} Run all integration tests (for arm64 arch)"
73-
@echo " ${YELLOW}prepare ${RESET} Run all available checks and generators"
64+
@echo " ${YELLOW}cover ${RESET} Open html coverage report in browser"
65+
@echo " ${YELLOW}doc ${RESET} Run doc generation"
66+
@echo " ${YELLOW}lint ${RESET} Run linters via golangci-lint"
67+
@echo " ${YELLOW}tidy ${RESET} Run tidy for go module to remove unused dependencies"
68+
@echo " ${YELLOW}build-cli ${RESET} Build cli component of shovel"
69+
@echo " ${YELLOW}build-dumper ${RESET} Build dumper component of shovel"
70+
@echo " ${YELLOW}setup ${RESET} Setup local environment. Create kind cluster"
71+
@echo " ${YELLOW}test ${RESET} Run all available tests"
72+
@echo " ${YELLOW}test-unit ${RESET} Run all unit tests"
73+
@echo " ${YELLOW}test-integration ${RESET} Run all integration tests"
74+
@echo " ${YELLOW}test-integration-prepare ${RESET} Prepare integration tests (build dumper and load to kind cluster)"
75+
@echo " ${YELLOW}prepare ${RESET} Run all available checks and generators"

‎cli/cmd/command_builder.go‎

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,27 +2,33 @@ package cmd
22

33
import (
44
"fmt"
5-
"github.com/dodopizza/kubectl-shovel/internal/flags"
6-
"github.com/dodopizza/kubectl-shovel/internal/globals"
5+
76
"github.com/spf13/cobra"
87
"github.com/spf13/pflag"
8+
99
"k8s.io/cli-runtime/pkg/genericclioptions"
10+
11+
"github.com/dodopizza/kubectl-shovel/internal/flags"
12+
"github.com/dodopizza/kubectl-shovel/internal/globals"
13+
"github.com/dodopizza/kubectl-shovel/internal/kubernetes"
1014
)
1115

12-
// CommonOptions contains generic arguments for cli
16+
// CommonOptions contains shared arguments for cli commands
1317
type CommonOptions struct {
14-
Container string
15-
Image string
16-
Pod string
17-
Output string
18+
Container string
19+
Image string
20+
Pod string
21+
Output string
22+
StoreOutputOnHost bool
1823

19-
kube *genericclioptions.ConfigFlags
24+
kubeConfig *genericclioptions.ConfigFlags
2025
}
2126

2227
// CommandBuilder represents logic for building and running tools
2328
type CommandBuilder struct {
2429
CommonOptions *CommonOptions
2530
tool flags.DotnetTool
31+
kube *kubernetes.Client
2632
}
2733

2834
// GetFlags return FlagSet that describes generic options
@@ -55,9 +61,15 @@ func (options *CommonOptions) GetFlags(tool string) *pflag.FlagSet {
5561
fmt.Sprintf("./output.%s", tool),
5662
"Output file",
5763
)
64+
fs.BoolVarP(
65+
&options.StoreOutputOnHost,
66+
"store-output-on-host",
67+
"t",
68+
options.StoreOutputOnHost,
69+
"Flag, indicating that output should be stored on host /tmp folder")
5870

59-
options.kube = genericclioptions.NewConfigFlags(false)
60-
options.kube.AddFlags(fs)
71+
options.kubeConfig = genericclioptions.NewConfigFlags(false)
72+
options.kubeConfig.AddFlags(fs)
6173

6274
return fs
6375
}

‎cli/cmd/commands.go‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,10 @@ package cmd
22

33
import (
44
"fmt"
5-
"github.com/dodopizza/kubectl-shovel/internal/flags"
5+
66
"github.com/spf13/cobra"
7+
8+
"github.com/dodopizza/kubectl-shovel/internal/flags"
79
)
810

911
var (
@@ -20,7 +22,7 @@ func NewGCDumpCommand() *cobra.Command {
2022
builder := NewCommandBuilder(flags.NewDotnetGCDump)
2123
return builder.Build(
2224
"Get dotnet-gcdump results",
23-
"This subcommand will run dotnet-gcdump tool for running in k8s appplication.\n"+
25+
"This subcommand will run dotnet-gcdump tool for running in k8s application.\n"+
2426
"Result will be saved locally so you'll be able to analyze it with appropriate tools.\n"+
2527
"You can find more info about dotnet-gcdump tool by the following links:\n\n"+
2628
"\t* https://devblogs.microsoft.com/dotnet/collecting-and-analyzing-memory-dumps/\n"+
@@ -34,7 +36,7 @@ func NewTraceCommand() *cobra.Command {
3436
builder := NewCommandBuilder(flags.NewDotnetTrace)
3537
return builder.Build(
3638
"Get dotnet-trace results",
37-
"This subcommand will capture runtime events with dotnet-trace tool for running in k8s appplication.\n"+
39+
"This subcommand will capture runtime events with dotnet-trace tool for running in k8s application.\n"+
3840
"Result will be saved locally in nettrace format so you'll be able to convert it and analyze with appropriate tools.\n"+
3941
"You can find more info about dotnet-trace tool by the following links:\n\n"+
4042
"\t* https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-trace-instructions.md\n"+

‎cli/cmd/launch.go‎

Lines changed: 72 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -3,30 +3,75 @@ package cmd
33
import (
44
"context"
55
"fmt"
6-
"github.com/dodopizza/kubectl-shovel/internal/events"
7-
"github.com/dodopizza/kubectl-shovel/internal/globals"
86
"strings"
97

108
"github.com/pkg/errors"
119

10+
"github.com/dodopizza/kubectl-shovel/internal/events"
11+
"github.com/dodopizza/kubectl-shovel/internal/flags"
12+
"github.com/dodopizza/kubectl-shovel/internal/globals"
1213
"github.com/dodopizza/kubectl-shovel/internal/kubernetes"
1314
"github.com/dodopizza/kubectl-shovel/internal/watchdog"
1415
)
1516

16-
func (cb *CommandBuilder) args(info *kubernetes.ContainerInfo) []string {
17-
args := []string{"--container-id", info.ID, "--container-runtime", info.Runtime}
18-
args = append(args, cb.tool.ToolName())
19-
args = append(args, cb.tool.FormatArgs()...)
20-
return args
17+
func (cb *CommandBuilder) newKubeClient() error {
18+
kube, err := kubernetes.NewClient(cb.CommonOptions.kubeConfig)
19+
if err != nil {
20+
return err
21+
}
22+
23+
cb.kube = kube
24+
return nil
25+
}
26+
27+
func (cb *CommandBuilder) args(pod *kubernetes.PodInfo, container *kubernetes.ContainerInfo) []string {
28+
args := flags.NewArgs().
29+
Append("container-id", container.ID).
30+
Append("container-runtime", container.Runtime)
31+
32+
if cb.CommonOptions.StoreOutputOnHost {
33+
args.AppendKey("store-output-on-host")
34+
}
35+
36+
return args.
37+
Append("container-name", container.Name).
38+
Append("pod-name", pod.Name).
39+
Append("pod-namespace", pod.Namespace).
40+
AppendCommand(cb.tool.ToolName()).
41+
AppendFrom(cb.tool).
42+
Get()
43+
}
44+
45+
func (cb *CommandBuilder) copyOutput(pod *kubernetes.PodInfo, output string) error {
46+
ctx, cancel := context.WithCancel(context.Background())
47+
defer cancel()
48+
49+
pinger := watchdog.NewPinger(cb.kube, pod.Name)
50+
go func() {
51+
if err := pinger.Run(ctx); err != nil {
52+
fmt.Println(err)
53+
}
54+
}()
55+
56+
fmt.Println("Retrieve output from diagnostics job")
57+
if err := cb.kube.CopyFromPod(pod.Name, output, cb.CommonOptions.Output); err != nil {
58+
return errors.Wrap(err, "Error while retrieving diagnostics job output")
59+
}
60+
fmt.Printf("Result successfully written to %s\n", cb.CommonOptions.Output)
61+
return nil
62+
}
63+
64+
func (cb *CommandBuilder) storeOutputOnHost(pod *kubernetes.PodInfo, output string) error {
65+
fmt.Printf("Output located on host: %s, at path: %s\n", pod.Node, output)
66+
return nil
2167
}
2268

2369
func (cb *CommandBuilder) launch() error {
24-
k8s, err := kubernetes.NewClient(cb.CommonOptions.kube)
25-
if err != nil {
70+
if err := cb.newKubeClient(); err != nil {
2671
return errors.Wrap(err, "Failed to init kubernetes client")
2772
}
2873

29-
targetPod, err := k8s.GetPodInfo(cb.CommonOptions.Pod)
74+
targetPod, err := cb.kube.GetPodInfo(cb.CommonOptions.Pod)
3075
if err != nil {
3176
return errors.Wrap(err, "Failed to get info about target pod")
3277
}
@@ -42,34 +87,29 @@ func (cb *CommandBuilder) launch() error {
4287
}
4388

4489
jobSpec := kubernetes.
45-
NewJobRunSpec(cb.args(targetContainer), cb.CommonOptions.Image, targetPod).
90+
NewJobRunSpec(cb.args(targetPod, targetContainer), cb.CommonOptions.Image, targetPod).
4691
WithContainerFSVolume(targetContainer)
4792

4893
if targetPod.ContainsMountedTmp(targetContainerName) {
4994
jobSpec.WithContainerMountsVolume(targetContainer)
5095
}
5196

97+
if cb.CommonOptions.StoreOutputOnHost {
98+
jobSpec.WithHostTmpVolume()
99+
}
100+
52101
fmt.Printf("Spawning diagnostics job with command:\n%s\n", strings.Join(jobSpec.Args, " "))
53-
if err := k8s.RunJob(jobSpec); err != nil {
102+
if err := cb.kube.RunJob(jobSpec); err != nil {
54103
return errors.Wrap(err, "Failed to spawn diagnostics job")
55104
}
56105

57106
fmt.Println("Waiting for a diagnostics job to start")
58-
jobPod, err := k8s.WaitPod(jobSpec.Selectors)
107+
jobPod, err := cb.kube.WaitPod(jobSpec.Selectors)
59108
if err != nil {
60109
return errors.Wrap(err, "Failed to wait diagnostics job execution")
61110
}
62111

63-
op := watchdog.NewOperator(k8s, jobPod.Name)
64-
ctx, cancel := context.WithCancel(context.Background())
65-
defer cancel()
66-
go func() {
67-
if err := op.Run(ctx); err != nil {
68-
fmt.Println(err)
69-
}
70-
}()
71-
72-
jobPodLogs, err := k8s.ReadPodLogs(jobPod.Name, globals.PluginName)
112+
jobPodLogs, err := cb.kube.ReadPodLogs(jobPod.Name, globals.PluginName)
73113
if err != nil {
74114
return errors.Wrap(err, "Failed to read logs from diagnostics job targetPod")
75115
}
@@ -81,13 +121,17 @@ func (cb *CommandBuilder) launch() error {
81121
return err
82122
}
83123

84-
fmt.Println("Retrieve output from diagnostics job")
85-
if err := k8s.CopyFromPod(jobPod.Name, output, cb.CommonOptions.Output); err != nil {
86-
return errors.Wrap(err, "Error while retrieving diagnostics job output")
124+
// dealing with output
125+
outputHandler := cb.copyOutput
126+
if cb.CommonOptions.StoreOutputOnHost {
127+
outputHandler = cb.storeOutputOnHost
128+
}
129+
if err := outputHandler(jobPod, output); err != nil {
130+
return err
87131
}
88132

89-
fmt.Printf("Result successfully written to %s\nCleanup diagnostics job", cb.CommonOptions.Output)
90-
if err := k8s.DeleteJob(jobSpec.Name); err != nil {
133+
fmt.Println("Cleanup diagnostics job")
134+
if err := cb.kube.DeleteJob(jobSpec.Name); err != nil {
91135
return errors.Wrap(err, "Error while deleting job")
92136
}
93137

‎cli/docs/kubectl-shovel_dump.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ Also use `-n`/`--namespace` if your pod is not in current context's namespace:
5656
-p, --process-id int The process ID to collect the trace from (default 1)
5757
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
5858
-s, --server string The address and port of the Kubernetes API server
59+
-t, --store-output-on-host Flag, indicating that output should be stored on host /tmp folder
5960
--tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
6061
--token string Bearer token for authentication to the API server
6162
--type type The kinds of information that are collected from process. Supported types:

‎cli/docs/kubectl-shovel_gcdump.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Get dotnet-gcdump results
44

55
### Synopsis
66

7-
This subcommand will run dotnet-gcdump tool for running in k8s appplication.
7+
This subcommand will run dotnet-gcdump tool for running in k8s application.
88
Result will be saved locally so you'll be able to analyze it with appropriate tools.
99
You can find more info about dotnet-gcdump tool by the following links:
1010

@@ -54,6 +54,7 @@ Also use `-n`/`--namespace` if your pod is not in current context's namespace:
5454
-p, --process-id int The process ID to collect the trace from (default 1)
5555
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
5656
-s, --server string The address and port of the Kubernetes API server
57+
-t, --store-output-on-host Flag, indicating that output should be stored on host /tmp folder
5758
--timeout timeout Give up on collecting the GC dump if it takes longer than this many seconds.
5859
Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
5960
Will be rounded to seconds. If no unit provided defaults to seconds.

‎cli/docs/kubectl-shovel_trace.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Get dotnet-trace results
44

55
### Synopsis
66

7-
This subcommand will capture runtime events with dotnet-trace tool for running in k8s appplication.
7+
This subcommand will capture runtime events with dotnet-trace tool for running in k8s application.
88
Result will be saved locally in nettrace format so you'll be able to convert it and analyze with appropriate tools.
99
You can find more info about dotnet-trace tool by the following links:
1010

@@ -92,6 +92,7 @@ Or convert any other format to speedscope format with:
9292
https://docs.microsoft.com/en-us/dotnet/core/diagnostics/well-known-event-providers
9393
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
9494
-s, --server string The address and port of the Kubernetes API server
95+
-t, --store-output-on-host Flag, indicating that output should be stored on host /tmp folder
9596
--tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
9697
--token string Bearer token for authentication to the API server
9798
--user string The name of the kubeconfig user to use

0 commit comments

Comments
 (0)