Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apiserver tracing #94942

Merged
merged 1 commit into from Jun 25, 2021
Merged

Conversation

dashpole
Copy link
Contributor

@dashpole dashpole commented Sep 21, 2020

What type of PR is this?

/kind feature

What this PR does / why we need it:

Enhancement issue: kubernetes/enhancements#647
Implementation of kubernetes/enhancements#1458.

Add tracing for API Server requests using opentelemetry-go, and the OTLP exporter. This has two primary components:

  1. Use otelhttp "wrapper" library in api server's http server
  2. Add file-based configuration for enabling and configuring tracing using the OTLP exporter.

The following will be done as follow-ups:

Dependencies added

Does this PR introduce a user-facing change?:

Add distributed tracing to the kube-apiserver.  It is can be enabled with the feature gate: APIServerTracing=true

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/0034-distributed-tracing-kep.md

@k8s-ci-robot
Copy link
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. kind/feature Categorizes issue or PR as related to a new feature. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Sep 21, 2020
@dashpole dashpole changed the title [WIP} Apiserver tracing [WIP] Apiserver tracing Sep 21, 2020
@dashpole
Copy link
Contributor Author

/sig instrumentation
/sig apimachinery

@k8s-ci-robot
Copy link
Contributor

@dashpole: The label(s) sig/apimachinery cannot be applied, because the repository doesn't have them

In response to this:

/sig instrumentation
/sig apimachinery

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 21, 2020
@dashpole
Copy link
Contributor Author

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Sep 21, 2020
@dashpole
Copy link
Contributor Author

/test all

@k8s-ci-robot k8s-ci-robot added area/apiserver area/cloudprovider area/code-generation area/dependency Issues or PRs related to dependency changes area/kubectl sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. labels Sep 21, 2020
@k8s-ci-robot k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Sep 21, 2020
@dashpole
Copy link
Contributor Author

/retest

@liggitt
Copy link
Member

liggitt commented Jun 25, 2021

lgtm, looks like there might be a few commits that should be squashed down?

@dashpole
Copy link
Contributor Author

squashed

}
// Even if there is no TracerProvider, the otelhttp still handles context propagation.
// See https://github.com/open-telemetry/opentelemetry-go/tree/main/example/passthrough
return otelhttp.NewHandler(handler, "KubernetesAPI", opts...)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just realized this is in k8s.io/apiserver, not specific to kube-apiserver... if there are multiple servers built on this library, does each need to have a unique id, or is it ok if they all report KubernetesAPI

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This controls the name of the span, so it should be whatever would best identify the server, and it doesn't have to be unique. https://github.com/kubernetes/kubernetes/pull/94942/files#diff-0e1453d49880a4f88f0353b9902c7e914513dd1274e3115100ea42c2d116136bR101-R106 is where we set the resource tags that should distinguish different instances of the apiserver. Right now, we tag it with the service name, "kube-apiserver", and the API Server ID, which has the pattern kube-apiserver-.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok... looks like APIServerID is overrideable by consumers of the k8s.io/apiserver library, so that's a reasonable thing to use... hard-coding kube-apiserver isn't good long-term, but can be adjusted in a follow-up

"k8s.io/utils/path"
)

const apiserverService = "kube-apiserver"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

things other than kube-apiserver use k8s.io/apiserver, this will need a follow-up to be able to be set by the particular binary consuming the k8s.io/apiserver library

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will rename to "apiserver".

@liggitt
Copy link
Member

liggitt commented Jun 25, 2021

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 25, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, dashpole, dims, liggitt, lilic

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 25, 2021
@liggitt
Copy link
Member

liggitt commented Jun 25, 2021

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 25, 2021
@dashpole
Copy link
Contributor Author

/retest

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jun 25, 2021

@dashpole: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-conformance-kind-ipv6-parallel 797197d link /test pull-kubernetes-conformance-kind-ipv6-parallel

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@dashpole
Copy link
Contributor Author

/retest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver area/cloudprovider area/code-generation area/dependency Issues or PRs related to dependency changes area/kubectl area/provider/gcp Issues or PRs related to gcp provider area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
Archived in project
SIG Node PR Triage
Waiting on Author
Development

Successfully merging this pull request may close these issues.

None yet