Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.10 hyperkube kubelet is losing some parameters from help #62009

Closed
gyliu513 opened this issue Apr 2, 2018 · 28 comments
Closed

1.10 hyperkube kubelet is losing some parameters from help #62009

gyliu513 opened this issue Apr 2, 2018 · 28 comments
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/cli Categorizes an issue or PR as relevant to SIG CLI.

Comments

@gyliu513
Copy link
Contributor

gyliu513 commented Apr 2, 2018

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

root@gyliu-ubuntu-1:~# /opt/kubernetes/hyperkube kubectl -s 127.0.0.1:8888 version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
root@gyliu-ubuntu-1:~# /opt/kubernetes/hyperkube kubelet --help
The kubelet is the primary "node agent" that runs on each
node. The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object
that describes a pod. The kubelet takes a set of PodSpecs that are provided through
various mechanisms (primarily through the apiserver) and ensures that the containers
described in those PodSpecs are running and healthy. The kubelet doesn't manage
containers which were not created by Kubernetes.

Other than from an PodSpec from the apiserver, there are three ways that a container
manifest can be provided to the Kubelet.

File: Path passed as a flag on the command line. Files under this path will be monitored
periodically for updates. The monitoring period is 20s by default and is configurable
via a flag.

HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This endpoint
is checked every 20 seconds (also configurable with a flag).

HTTP server: The kubelet can also listen for HTTP and respond to a simple API
(underspec'd currently) to submit a new manifest.

Usage:
  hyperkube kubelet [flags]

Flags:
      --alsologtostderr                                     log to standard error as well as files
      --azure-container-registry-config string              Path to the file containing Azure container registry configuration information.
      --bootstrap-checkpoint-path string                    <Warning: Alpha feature> Path to to the directory where the checkpoints are stored
      --bootstrap-kubeconfig string                         Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.
      --cert-dir string                                     The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "/var/lib/kubelet/pki")
      --chaos-chance float                                  If > 0.0, introduce random client errors and latency. Intended for testing.
      --cloud-config string                                 The path to the cloud provider configuration file.  Empty string for no configuration file.
      --cloud-provider string                               The provider for cloud services. Specify empty string for running with no cloud provider.
      --cni-bin-dir string                                  <Warning: Alpha feature> The full path of the directory in which to search for CNI plugin binaries. Default: /opt/cni/bin
      --cni-conf-dir string                                 <Warning: Alpha feature> The full path of the directory in which to search for CNI config files. Default: /etc/cni/net.d
      --config string                                       The Kubelet will load its initial configuration from this file. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Omit this flag to use the built-in default configuration values. Command-line flags override configuration from this file.
      --container-runtime string                            The container runtime to use. Possible values: 'docker', 'remote', 'rkt (deprecated)'. (default "docker")
      --container-runtime-endpoint string                   [Experimental] The endpoint of remote runtime service. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735' (default "unix:///var/run/dockershim.sock")
      --containerized                                       Running kubelet in a container.
      --docker-endpoint string                              Use this for the docker endpoint to communicate with (default "unix:///var/run/docker.sock")
      --docker-root string                                  DEPRECATED: docker root is read from docker info (this is a fallback, default: /var/lib/docker) (default "/var/lib/docker")
      --dynamic-config-dir string                           The Kubelet will use this directory for checkpointing downloaded configurations and tracking configuration health. The Kubelet will create this directory if it does not already exist. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Providing this flag enables dynamic Kubelet configuration. Presently, you must also enable the DynamicKubeletConfig feature gate to pass this flag.
      --enable-server                                       Enable the Kubelet's server (default true)
      --exit-on-lock-contention                             Whether kubelet should exit upon lock-file contention.
      --experimental-allocatable-ignore-eviction            When set to 'true', Hard Eviction Thresholds will be ignored while calculating Node Allocatable. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. [default=false]
      --experimental-allowed-unsafe-sysctls strings         Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk.
      --experimental-bootstrap-kubeconfig string            deprecated: use --bootstrap-kubeconfig
      --experimental-check-node-capabilities-before-mount   [Experimental] if set true, the kubelet will check the underlying node for required components (binaries, etc.) before performing the mount
      --experimental-kernel-memcg-notification              If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling.
      --experimental-mounter-path string                    [Experimental] Path of mounter binary. Leave empty to use the default mount.
      --experimental-qos-reserved mapStringString           A set of ResourceName=Percentage (e.g. memory=50%) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. [default=none]
  -h, --help                                                help for kubelet
      --hostname-override string                            If non-empty, will use this string as identification instead of the actual hostname.
      --housekeeping-interval duration                      Interval between container housekeepings (default 10s)
      --image-pull-progress-deadline duration               If no pulling progress is made before this deadline, the image pulling will be cancelled. (default 1m0s)
      --image-service-endpoint string                       [Experimental] The endpoint of remote image service. If not specified, it will be the same with container-runtime-endpoint by default. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735'
      --kubeconfig string                                   Path to a kubeconfig file, specifying how to connect to the API server. Providing --kubeconfig enables API server mode, omitting --kubeconfig enables standalone mode.
      --lock-file string                                    <Warning: Alpha feature> The path to file for kubelet to use as a lock file.
      --log-backtrace-at traceLocation                      when logging hits line file:N, emit a stack trace (default :0)
      --log-dir string                                      If non-empty, write log files in this directory
      --log-flush-frequency duration                        Maximum number of seconds between log flushes (default 5s)
      --logtostderr                                         log to standard error instead of files (default true)
      --network-plugin string                               <Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle
      --network-plugin-mtu int32                            <Warning: Alpha feature> The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU.
      --node-ip string                                      IP address of the node. If set, kubelet will use this IP address for the node
      --node-labels mapStringString                         <Warning: Alpha feature> Labels to add when registering the node in the cluster.  Labels must be key=value pairs separated by ','.
      --pod-infra-container-image string                    The image whose network/ipc namespaces containers in each pod will use. (default "k8s.gcr.io/pause-amd64:3.1")
      --provider-id string                                  Unique identifier for identifying the node in a machine database, i.e cloudprovider
      --really-crash-for-testing                            If true, when panics occur crash. Intended for testing.
      --register-node                                       Register the node with the apiserver. If --kubeconfig is not provided, this flag is irrelevant, as the Kubelet won't have an apiserver to register with. Default=true. (default true)
      --register-with-taints []api.Taint                    Register the node with the given list of taints (comma separated "<key>=<value>:<effect>"). No-op if register-node is false.
      --root-dir string                                     Directory path for managing kubelet files (volume mounts,etc). (default "/var/lib/kubelet")
      --rotate-certificates                                 <Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.
      --runonce                                             If true, exit after spawning pods from static pod files or remote urls. Exclusive with --enable-server
      --runtime-cgroups string                              Optional absolute name of cgroups to create and run the runtime in.
      --seccomp-profile-root string                         <Warning: Alpha feature> Directory path for seccomp profiles. (default "/var/lib/kubelet/seccomp")
      --stderrthreshold severity                            logs at or above this threshold go to stderr (default 2)
  -v, --v Level                                             log level for V logs
      --version version[=true]                              Print version information and quit
      --vmodule moduleSpec                                  comma-separated list of pattern=N settings for file-filtered logging
      --volume-plugin-dir string                            The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")

The above help do not include some parameters such as max-pods, max-open-files etc.

/sig cli

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. sig/cli Categorizes an issue or PR as relevant to SIG CLI. labels Apr 2, 2018
@gyliu513
Copy link
Contributor Author

gyliu513 commented Apr 2, 2018

This is kubelet help info from Kubernetes 1.9.

root@master2:~# /opt/kubernetes/hyperkube kubelet --help
The kubelet binary is responsible for maintaining a set of containers on a

  particular node. It syncs data from a variety of sources including a
  Kubernetes API server, an etcd cluster, HTTP endpoint or local file. It then
  queries Docker to see what is currently running.  It synchronizes the
  configuration data, with the running set of containers by starting or stopping
  Docker containers.
Usage:
  kubelet [flags]

Available Flags:
      --address 0.0.0.0                                                                                           The IP address for the Kubelet to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 0.0.0.0)
      --allow-privileged                                                                                          If true, allow containers to request privileged mode.
      --allow-verification-with-non-compliant-keys                                                                Allow a SignatureVerifier to use keys which are technically non-compliant with RFC6962.
      --alsologtostderr                                                                                           log to standard error as well as files
      --anonymous-auth                                                                                            Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. (default true)
      --application-metrics-count-limit int                                                                       Max number of application metrics to store (per container) (default 100)
      --authentication-token-webhook                                                                              Use the TokenReview API to determine authentication for bearer tokens.
      --authentication-token-webhook-cache-ttl duration                                                           The duration to cache responses from the webhook token authenticator. (default 2m0s)
      --authorization-mode string                                                                                 Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (default "AlwaysAllow")
      --authorization-webhook-cache-authorized-ttl duration                                                       The duration to cache 'authorized' responses from the webhook authorizer. (default 5m0s)
      --authorization-webhook-cache-unauthorized-ttl duration                                                     The duration to cache 'unauthorized' responses from the webhook authorizer. (default 30s)
      --azure-container-registry-config string                                                                    Path to the file container Azure container registry configuration information.
      --boot-id-file string                                                                                       Comma-separated list of files to check for boot-id. Use the first one that exists. (default "/proc/sys/kernel/random/boot_id")
      --bootstrap-checkpoint-path string                                                                          <Warning: Alpha feature> Path to to the directory where the checkpoints are stored
      --bootstrap-kubeconfig string                                                                               Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.
      --cadvisor-port int32                                                                                       The port of the localhost cAdvisor endpoint (set to 0 to disable) (default 4194)
      --cert-dir string                                                                                           The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "/var/lib/kubelet/pki")
      --cgroup-driver string                                                                                      Driver that the kubelet uses to manipulate cgroups on the host.  Possible values: 'cgroupfs', 'systemd' (default "cgroupfs")
      --cgroup-root string                                                                                        Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default.
      --cgroups-per-qos                                                                                           Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (default true)
      --chaos-chance float                                                                                        If > 0.0, introduce random client errors and latency. Intended for testing.
      --client-ca-file string                                                                                     If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      --cloud-config string                                                                                       The path to the cloud provider configuration file.  Empty string for no configuration file.
      --cloud-provider string                                                                                     The provider for cloud services. Specify empty string for running with no cloud provider.
      --cloud-provider-gce-lb-src-cidrs cidrs                                                                     CIDRs opened in GCE firewall for LB traffic proxy & health checks (default 130.211.0.0/22,35.191.0.0/16,209.85.152.0/22,209.85.204.0/22)
      --cluster-dns stringSlice                                                                                   Comma-separated list of DNS server IP address.  This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst". Note: all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution.
      --cluster-domain string                                                                                     Domain for this cluster.  If set, kubelet will configure all containers to search this domain in addition to the host's search domains
      --cni-bin-dir string                                                                                        <Warning: Alpha feature> The full path of the directory in which to search for CNI plugin binaries. Default: /opt/cni/bin
      --cni-conf-dir string                                                                                       <Warning: Alpha feature> The full path of the directory in which to search for CNI config files. Default: /etc/cni/net.d
      --container-hints string                                                                                    location of the container hints file (default "/etc/cadvisor/container_hints.json")
      --container-runtime string                                                                                  The container runtime to use. Possible values: 'docker', 'rkt'. (default "docker")
      --container-runtime-endpoint string                                                                         [Experimental] The endpoint of remote runtime service. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735' (default "unix:///var/run/dockershim.sock")
      --containerd string                                                                                         containerd endpoint (default "unix:///var/run/containerd.sock")
      --containerized                                                                                             Experimental support for running kubelet in a container.  Intended for testing.
      --contention-profiling                                                                                      Enable lock contention profiling, if profiling is enabled
      --cpu-cfs-quota                                                                                             Enable CPU CFS quota enforcement for containers that specify CPU limits (default true)
      --cpu-manager-policy string                                                                                 <Warning: Alpha feature> CPU Manager policy to use. Possible values: 'none', 'static'. Default: 'none' (default "none")
      --cpu-manager-reconcile-period NodeStatusUpdateFrequency                                                    <Warning: Alpha feature> CPU Manager reconciliation period. Examples: '10s', or '1m'. If not supplied, defaults to NodeStatusUpdateFrequency (default 10s)
      --default-not-ready-toleration-seconds int                                                                  Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300)
      --default-unreachable-toleration-seconds int                                                                Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration. (default 300)
      --docker string                                                                                             docker endpoint (default "unix:///var/run/docker.sock")
      --docker-disable-shared-pid                                                                                 The Container Runtime Interface (CRI) defaults to using a shared PID namespace for containers in a pod when running with Docker 1.13.1 or higher. Setting this flag reverts to the previous behavior of isolated PID namespaces. This ability will be removed in a future Kubernetes release. (default true)
      --docker-endpoint string                                                                                    Use this for the docker endpoint to communicate with (default "unix:///var/run/docker.sock")
      --docker-env-metadata-whitelist string                                                                      a comma-separated list of environment variable keys that needs to be collected for docker containers
      --docker-only                                                                                               Only report docker containers in addition to root stats
      --docker-root string                                                                                        DEPRECATED: docker root is read from docker info (this is a fallback, default: /var/lib/docker) (default "/var/lib/docker")
      --docker-tls                                                                                                use TLS to connect to docker
      --docker-tls-ca string                                                                                      path to trusted CA (default "ca.pem")
      --docker-tls-cert string                                                                                    path to client certificate (default "cert.pem")
      --docker-tls-key string                                                                                     path to private key (default "key.pem")
      --dynamic-config-dir string                                                                                 The Kubelet will use this directory for checkpointing downloaded configurations and tracking configuration health. The Kubelet will create this directory if it does not already exist. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Providing this flag enables dynamic Kubelet configuration. Presently, you must also enable the DynamicKubeletConfig feature gate to pass this flag.
      --enable-controller-attach-detach                                                                           Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations (default true)
      --enable-debugging-handlers                                                                                 Enables server endpoints for log collection and local running of containers and commands (default true)
      --enable-load-reader                                                                                        Whether to enable cpu load reader
      --enable-server                                                                                             Enable the Kubelet's server (default true)
      --enforce-node-allocatable stringSlice                                                                      A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptible options are 'pods', 'system-reserved' & 'kube-reserved'. If the latter two options are specified, '--system-reserved-cgroup' & '--kube-reserved-cgroup' must also be set respectively. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. (default [pods])
      --event-burst int32                                                                                         Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding event-qps. Only used if --event-qps > 0 (default 10)
      --event-qps int32                                                                                           If > 0, limit event creations per second to this value. If 0, unlimited. (default 5)
      --event-storage-age-limit string                                                                            Max length of time for which to store events (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is a duration. Default is applied to all non-specified event types (default "default=0")
      --event-storage-event-limit string                                                                          Max number of events to store (per type). Value is a comma separated list of key values, where the keys are event types (e.g.: creation, oom) or "default" and the value is an integer. Default is applied to all non-specified event types (default "default=0")
      --eviction-hard mapStringString                                                                             A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction. (default imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%)
      --eviction-max-pod-grace-period int32                                                                       Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met.  If negative, defer to pod specified value.
      --eviction-minimum-reclaim mapStringString                                                                  A set of minimum reclaims (e.g. imagefs.available=2Gi) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure.
      --eviction-pressure-transition-period duration                                                              Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (default 5m0s)
      --eviction-soft mapStringString                                                                             A set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a pod eviction.
      --eviction-soft-grace-period mapStringString                                                                A set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction.
      --exit-on-lock-contention                                                                                   Whether kubelet should exit upon lock-file contention.
      --experimental-allocatable-ignore-eviction                                                                  When set to 'true', Hard Eviction Thresholds will be ignored while calculating Node Allocatable. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. [default=false]
      --experimental-allowed-unsafe-sysctls stringSlice                                                           Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk.
      --experimental-bootstrap-kubeconfig string                                                                  deprecated: use --bootstrap-kubeconfig
      --experimental-check-node-capabilities-before-mount                                                         [Experimental] if set true, the kubelet will check the underlying node for required componenets (binaries, etc.) before performing the mount
      --experimental-kernel-memcg-notification                                                                    If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling.
      --experimental-mounter-path string                                                                          [Experimental] Path of mounter binary. Leave empty to use the default mount.
      --experimental-qos-reserved mapStringString                                                                 A set of ResourceName=Percentage (e.g. memory=50%) pairs that describe how pod resource requests are reserved at the QoS level. Currently only memory is supported. [default=none]
      --fail-swap-on                                                                                              Makes the Kubelet fail to start if swap is enabled on the node.  (default true)
      --feature-gates mapStringBool                                                                               A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIResponseCompression=true|false (ALPHA - default=false)
Accelerators=true|false (ALPHA - default=false)
AdvancedAuditing=true|false (BETA - default=true)
AllAlpha=true|false (ALPHA - default=false)
AllowExtTrafficLocalEndpoints=true|false (default=true)
AppArmor=true|false (BETA - default=true)
BlockVolume=true|false (ALPHA - default=false)
CPUManager=true|false (ALPHA - default=false)
CSIPersistentVolume=true|false (ALPHA - default=false)
CustomPodDNS=true|false (ALPHA - default=false)
CustomResourceValidation=true|false (BETA - default=true)
DebugContainers=true|false (ALPHA - default=false)
DevicePlugins=true|false (ALPHA - default=false)
DynamicKubeletConfig=true|false (ALPHA - default=false)
EnableEquivalenceClassCache=true|false (ALPHA - default=false)
ExpandPersistentVolumes=true|false (ALPHA - default=false)
ExperimentalCriticalPodAnnotation=true|false (ALPHA - default=false)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
HugePages=true|false (ALPHA - default=false)
Initializers=true|false (ALPHA - default=false)
KubeletConfigFile=true|false (ALPHA - default=false)
LocalStorageCapacityIsolation=true|false (ALPHA - default=false)
MountContainers=true|false (ALPHA - default=false)
MountPropagation=true|false (ALPHA - default=false)
PVCProtection=true|false (ALPHA - default=false)
PersistentLocalVolumes=true|false (ALPHA - default=false)
PodPriority=true|false (ALPHA - default=false)
ResourceLimitsPriorityFunction=true|false (ALPHA - default=false)
RotateKubeletClientCertificate=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (ALPHA - default=false)
ServiceNodeExclusion=true|false (ALPHA - default=false)
StreamingProxyRedirects=true|false (BETA - default=true)
SupportIPVSProxyMode=true|false (BETA - default=false)
TaintBasedEvictions=true|false (ALPHA - default=false)
TaintNodesByCondition=true|false (ALPHA - default=false)
VolumeScheduling=true|false (ALPHA - default=false)
      --file-check-frequency duration                                                                             Duration between checking config files for new data (default 20s)
      --global-housekeeping-interval duration                                                                     Interval between global housekeepings (default 1m0s)
      --google-json-key string                                                                                    The Google Cloud Platform Service Account JSON Key to use for authentication.
      --hairpin-mode string                                                                                       How should the kubelet setup hairpin NAT. This allows endpoints of a Service to loadbalance back to themselves if they should try to access their own Service. Valid values are "promiscuous-bridge", "hairpin-veth" and "none". (default "promiscuous-bridge")
      --healthz-bind-address 0.0.0.0                                                                              The IP address for the healthz server to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 127.0.0.1)
      --healthz-port int32                                                                                        The port of the localhost healthz endpoint (set to 0 to disable) (default 10248)
  -h, --help                                                                                                      help for hyperkube
      --host-ipc-sources stringSlice                                                                              Comma-separated list of sources from which the Kubelet allows pods to use the host ipc namespace. (default [*])
      --host-network-sources stringSlice                                                                          Comma-separated list of sources from which the Kubelet allows pods to use of host network. (default [*])
      --host-pid-sources stringSlice                                                                              Comma-separated list of sources from which the Kubelet allows pods to use the host pid namespace. (default [*])
      --hostname-override string                                                                                  If non-empty, will use this string as identification instead of the actual hostname.
      --housekeeping-interval duration                                                                            Interval between container housekeepings (default 10s)
      --http-check-frequency duration                                                                             Duration between checking http for new data (default 20s)
      --image-gc-high-threshold int32                                                                             The percent of disk usage after which image garbage collection is always run. (default 85)
      --image-gc-low-threshold int32                                                                              The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. (default 80)
      --image-pull-progress-deadline duration                                                                     If no pulling progress is made before this deadline, the image pulling will be cancelled. (default 1m0s)
      --image-service-endpoint string                                                                             [Experimental] The endpoint of remote image service. If not specified, it will be the same with container-runtime-endpoint by default. Currently unix socket is supported on Linux, and tcp is supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'tcp://localhost:3735'
      --init-config-dir string                                                                                    The Kubelet will look in this directory for the init configuration. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Omit this argument to use the built-in default configuration values. Presently, you must also enable the KubeletConfigFile feature gate to pass this flag.
      --iptables-drop-bit int32                                                                                   The bit of the fwmark space to mark packets for dropping. Must be within the range [0, 31]. (default 15)
      --iptables-masquerade-bit int32                                                                             The bit of the fwmark space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in kube-proxy. (default 14)
      --ir-data-source string                                                                                     Data source used by InitialResources. Supported options: influxdb, gcm. (default "influxdb")
      --ir-dbname string                                                                                          InfluxDB database name which contains metrics required by InitialResources (default "k8s")
      --ir-hawkular string                                                                                        Hawkular configuration URL
      --ir-influxdb-host string                                                                                   Address of InfluxDB which contains metrics required by InitialResources (default "localhost:8080/api/v1/namespaces/kube-system/services/monitoring-influxdb:api/proxy")
      --ir-namespace-only                                                                                         Whether the estimation should be made only based on data from the same namespace.
      --ir-password string                                                                                        Password used for connecting to InfluxDB (default "root")
      --ir-percentile int                                                                                         Which percentile of samples should InitialResources use when estimating resources. For experiment purposes. (default 90)
      --ir-user string                                                                                            User used for connecting to InfluxDB (default "root")
      --kube-api-burst int32                                                                                      Burst to use while talking with kubernetes apiserver (default 10)
      --kube-api-content-type string                                                                              Content type of requests sent to apiserver. (default "application/vnd.kubernetes.protobuf")
      --kube-api-qps int32                                                                                        QPS to use while talking with kubernetes apiserver (default 5)
      --kube-reserved mapStringString                                                                             A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi) pairs that describe resources reserved for kubernetes system components. Currently cpu, memory and local ephemeral storage for root file system are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. [default=none]
      --kube-reserved-cgroup string                                                                               Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via '--kube-reserved' flag. Ex. '/kube-reserved'. [default='']
      --kubeconfig string                                                                                         Path to a kubeconfig file, specifying how to connect to the API server. (default "/var/lib/kubelet/kubeconfig")
      --kubelet-cgroups string                                                                                    Optional absolute name of cgroups to create and run the Kubelet in.
      --lock-file string                                                                                          <Warning: Alpha feature> The path to file for kubelet to use as a lock file.
      --log-backtrace-at traceLocation                                                                            when logging hits line file:N, emit a stack trace (default :0)
      --log-cadvisor-usage                                                                                        Whether to log the usage of the cAdvisor container
      --log-dir string                                                                                            If non-empty, write log files in this directory
      --log-flush-frequency duration                                                                              Maximum number of seconds between log flushes (default 5s)
      --loglevel int                                                                                              Log level (0 = DEBUG, 5 = FATAL) (default 1)
      --logtostderr                                                                                               log to standard error instead of files (default true)
      --machine-id-file string                                                                                    Comma-separated list of files to check for machine-id. Use the first one that exists. (default "/etc/machine-id,/var/lib/dbus/machine-id")
      --make-iptables-util-chains                                                                                 If true, kubelet will ensure iptables utility rules are present on host. (default true)
      --manifest-url string                                                                                       URL for accessing the container manifest
      --manifest-url-header --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful'   Comma-separated list of HTTP headers to use when accessing the manifest URL. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful'
      --max-open-files int                                                                                        Number of files that can be opened by Kubelet process. (default 1000000)
      --max-pods int32                                                                                            Number of Pods that can run on this Kubelet. (default 110)
      --minimum-image-ttl-duration duration                                                                       Minimum age for an unused image before it is garbage collected.  Examples: '300ms', '10s' or '2h45m'. (default 2m0s)
      --network-plugin string                                                                                     <Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle
      --network-plugin-mtu int32                                                                                  <Warning: Alpha feature> The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU.
      --node-ip string                                                                                            IP address of the node. If set, kubelet will use this IP address for the node
      --node-labels mapStringString                                                                               <Warning: Alpha feature> Labels to add when registering the node in the cluster.  Labels must be key=value pairs separated by ','.
      --node-status-update-frequency duration                                                                     Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with nodeMonitorGracePeriod in nodecontroller. (default 10s)
      --oom-score-adj int32                                                                                       The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000] (default -999)
      --pod-cidr string                                                                                           The CIDR to use for pod IP addresses, only used in standalone mode.  In cluster mode, this is obtained from the master.
      --pod-infra-container-image string                                                                          The image whose network/ipc namespaces containers in each pod will use. (default "gcr.io/google_containers/pause-amd64:3.0")
      --pod-manifest-path string                                                                                  Path to the directory containing pod manifest files to run, or the path to a single pod manifest file. Files starting with dots will be ignored.
      --pods-per-core int32                                                                                       Number of Pods per core that can run on this Kubelet. The total number of Pods on this Kubelet cannot exceed max-pods, so max-pods will be used if this calculation results in a larger number of Pods allowed on the Kubelet. A value of 0 disables this limit.
      --port int32                                                                                                The port for the Kubelet to serve on. (default 10250)
      --protect-kernel-defaults                                                                                   Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults.
      --provider-id string                                                                                        Unique identifier for identifying the node in a machine database, i.e cloudprovider
      --read-only-port int32                                                                                      The read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable) (default 10255)
      --really-crash-for-testing                                                                                  If true, when panics occur crash. Intended for testing.
      --register-node                                                                                             Register the node with the apiserver. If --kubeconfig is not provided, this flag is irrelevant, as the Kubelet won't have an apiserver to register with. Default=true. (default true)
      --register-with-taints []api.Taint                                                                          Register the node with the given list of taints (comma separated "<key>=<value>:<effect>"). No-op if register-node is false.
      --registry-burst int32                                                                                      Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry-qps. Only used if --registry-qps > 0 (default 10)
      --registry-qps int32                                                                                        If > 0, limit registry pull QPS to this value.  If 0, unlimited. (default 5)
      --resolv-conf string                                                                                        Resolver configuration file used as the basis for the container DNS resolution configuration. (default "/etc/resolv.conf")
      --rkt-api-endpoint string                                                                                   The endpoint of the rkt API service to communicate with. Only used if --container-runtime='rkt'. (default "localhost:15441")
      --rkt-path string                                                                                           Path of rkt binary. Leave empty to use the first rkt in $PATH.  Only used if --container-runtime='rkt'.
      --root-dir string                                                                                           Directory path for managing kubelet files (volume mounts,etc). (default "/var/lib/kubelet")
      --rotate-certificates                                                                                       <Warning: Beta feature> Auto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches.
      --runonce                                                                                                   If true, exit after spawning pods from local manifests or remote urls. Exclusive with --enable-server
      --runtime-cgroups string                                                                                    Optional absolute name of cgroups to create and run the runtime in.
      --runtime-request-timeout duration                                                                          Timeout of all runtime requests except long running request - pull, logs, exec and attach. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (default 2m0s)
      --seccomp-profile-root string                                                                               <Warning: Alpha feature> Directory path for seccomp profiles. (default "/var/lib/kubelet/seccomp")
      --serialize-image-pulls                                                                                     Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. (default true)
      --stderrthreshold severity                                                                                  logs at or above this threshold go to stderr (default 2)
      --storage-driver-buffer-duration duration                                                                   Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction (default 1m0s)
      --storage-driver-db string                                                                                  database name (default "cadvisor")
      --storage-driver-host string                                                                                database host:port (default "localhost:8086")
      --storage-driver-password string                                                                            database password (default "root")
      --storage-driver-secure                                                                                     use secure connection with database
      --storage-driver-table string                                                                               table name (default "stats")
      --storage-driver-user string                                                                                database username (default "root")
      --streaming-connection-idle-timeout duration                                                                Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: '5m' (default 4h0m0s)
      --sync-frequency duration                                                                                   Max period between synchronizing running containers and config (default 1m0s)
      --system-cgroups /                                                                                          Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under /. Empty for no container. Rolling back the flag requires a reboot.
      --system-reserved mapStringString                                                                           A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See http://kubernetes.io/docs/user-guide/compute-resources for more detail. [default=none]
      --system-reserved-cgroup string                                                                             Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via '--system-reserved' flag. Ex. '/system-reserved'. [default='']
      --tls-cert-file string                                                                                      File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir.
      --tls-private-key-file string                                                                               File containing x509 private key matching --tls-cert-file.
  -v, --v Level                                                                                                   log level for V logs
      --version version[=true]                                                                                    Print version information and quit
      --vmodule moduleSpec                                                                                        comma-separated list of pattern=N settings for file-filtered logging
      --volume-plugin-dir string                                                                                  <Warning: Alpha feature> The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
      --volume-stats-agg-period duration                                                                          Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to 0. (default 1m0s)

@liggitt
Copy link
Member

liggitt commented Apr 2, 2018

/assign @mtaufen

@hzxuzhonghu
Copy link
Member

hzxuzhonghu commented Apr 2, 2018

All KubeletConfiguration flags are now deprecated, see this
func AddKubeletConfigFlags(mainfs *pflag.FlagSet, c *kubeletconfig.KubeletConfiguration) {

@gyliu513
Copy link
Contributor Author

gyliu513 commented Apr 2, 2018

@hzxuzhonghu what do you mean of All KubeletConfiguration flags are now deprecated? How can I get all of the parameters supported by kubelet as Kubernetes 1.9?

@hzxuzhonghu
Copy link
Member

Sorry, did not look into flag parsing

@liggitt
Copy link
Member

liggitt commented Apr 2, 2018

I think the behavior of hiding deprecated flags needs to change. That has caused problems with masking important information about deprecated behaviors like --insecure-port in the apiserver, and confusion for these flags as well.

@gyliu513
Copy link
Contributor Author

gyliu513 commented Apr 3, 2018

Based on my test, not only deprecated parameters are being hidden, but also non-deprecated parameters, such as max-pods, max-open-files etc. @liggitt

@liggitt
Copy link
Member

liggitt commented Apr 3, 2018

Based on my test, not only deprecated parameters are being hidden, but also non-deprecated parameters, such as max-pods, max-open-files etc. @liggitt

All values that can be passed via a kubelet config file are marked as deprecated (this included max-pods, etc). See https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/options/options.go#L430-L442

@gyliu513
Copy link
Contributor Author

gyliu513 commented Apr 3, 2018

@liggitt so where can we get a list of parameters supported by kubelet?

@liggitt
Copy link
Member

liggitt commented Apr 3, 2018

unfortunately, it looks as though looking at the code is the only way to see flags marked as deprecated. @mtaufen, that's a fairly significant issue for these... can you look into restoring visibility to those flags?

@mtaufen
Copy link
Contributor

mtaufen commented Apr 3, 2018 via email

@mtaufen
Copy link
Contributor

mtaufen commented Apr 3, 2018 via email

@mtaufen
Copy link
Contributor

mtaufen commented Apr 3, 2018

unfortunately, it looks as though looking at the code is the only way to see flags marked as deprecated.

Anyone using deprecated flags should be getting a warning in their log, but it's a crapshoot whether people actually pay attention to the logs, so I agree that this is an issue.

@mtaufen
Copy link
Contributor

mtaufen commented Apr 4, 2018

I'd like to just make a custom copy of pflag.FlagSet.FlagUsagesWrapped that includes deprecated flags. Not sure how to handle the licensing with this, since pflag is third party code.

It looks like we do third_party/forked for this sort of thing, but since other third party code (like Cobra) would be type-incompatible with a forked pflag, I don't think just forking and updating imports is viable.

Would it suffice to put a single file in third_party/forked with a modified version of pflag.FlagSet.FlagUsagesWrapped, but carrying the BSD license from pflag?

I'm not a lawyer, I don't know how to handle these things. @thockin who should I talk to?

@thockin
Copy link
Member

thockin commented Apr 5, 2018

Can we try to send a patch to pflag? I know a maintainer :) @eparis thoughts?

@eparis
Copy link
Contributor

eparis commented Apr 5, 2018

We don't want to fork. I don't think it's the right answer, but we'd probably set FlagSet.Usage or edit cobra.Command.UsageTemplate if we were going to do it ourselves. Not fork.

< rant >But any of those options sound like us making the horrible mistake of no working with other communities. NIH syndrome is oddly high recently. Weird that we'd talk about forking pflag before even filing an issue or sending a patch to change it. < /rant >

If I'm understanding the issue it seems like we have a weird case. Where for some commands (kubelet) these flags really are deprecated and we really don't want users to find them anymore. We want users to use the kubelet config. But in other commands (hyperkube) that same flagset has flags which aren't really deprecated and shouldn't really be hidden from users. Is that right?

Or are they deprecated for both but we really want to put them right in the user's face so they use them anyway? If they really are deprecated I'm not sure I understand @liggitt assertion that this is a 'significant issue'. I'm not sure I understand the use case of continuing to show users flags they shouldn't ever be setting.

Let's assume I'm convinced that Jordan's position is correct. Would we rather have a whole new section in the help text just with deprecated flags? Would we rather have them in the Local Flags or Inherited Flags sections, but with some disclaimer? Would we use the Deprecated flag text somehow? How would we designate them deprecated?

It's relatively easy to set our own cobra.Command.UsageTemplate and add our own section which prints the deprecated flags however we want.

But if someone can convince me that showing deprecated flags is a win, and describe how they think we should show them for that use case, I might get off my lazy butt and make changes to cobra and pflag.

@mtaufen
Copy link
Contributor

mtaufen commented Apr 5, 2018 via email

@eparis
Copy link
Contributor

eparis commented Apr 5, 2018

I'm not convinced this is the right thing to do. The help page should tell the users the right thing to do, not expose the wrong thing to do. But, just in case, people really really think it's important to keep telling users the wrong way to do things, I wrote spf13/pflag#163

@mtaufen
Copy link
Contributor

mtaufen commented Apr 5, 2018

I think your comment on that PR outlines the problem well; some people (like me 😉) want a deprecation policy that keeps deprecated flags visible with a warning until they are actually removed, while other people want the flags to be hidden as soon as they are marked deprecated.

@mtaufen
Copy link
Contributor

mtaufen commented Apr 5, 2018

The help page should tell the users the right thing to do, not expose the wrong thing to do.

We should be able to treat help as an opinionated recommendation, or simply as reference documentation that explains available options, depending on our target audience.

In this case, we surprised at least a few people by defaulting to the former...

@eparis
Copy link
Contributor

eparis commented Apr 5, 2018

I really do try to listen, even when I don't agree :)

If jordan and tim tell me I'm wrong and we should do this, I'll merge to pflag.

@mtaufen
Copy link
Contributor

mtaufen commented Apr 6, 2018

@liggitt @thockin :)

and @eparis, thank you very much for being open to discussion here

@liggitt
Copy link
Member

liggitt commented Apr 10, 2018

If jordan and tim tell me I'm wrong and we should do this, I'll merge to pflag.

I definitely want the option to show help for a deprecated flag until it is removed.

@eparis
Copy link
Contributor

eparis commented Apr 11, 2018

I 'definitely' think Jordan's wrong, but I merged he pflag change. So we can rebase and unhide as much as we'd like. :)

@mtaufen
Copy link
Contributor

mtaufen commented Apr 11, 2018

Thanks! I blocked out some time to make the corresponding k8s changes tomorrow.

@eparis
Copy link
Contributor

eparis commented Apr 11, 2018

@mtaufen awesome, thank you. Let me know if it doesn't do what we need!

@mtaufen
Copy link
Contributor

mtaufen commented Apr 13, 2018

#62505

k8s-github-robot pushed a commit that referenced this issue Apr 14, 2018
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Show help for deprecated Kubelet flags

We recently deprecated a bunch of Kubelet flags, which caused them to disappear from `--help` output. This PR unhides these flags, so that the deprecation notice is clearly visible in `--help`.

Fixes: #62009

```release-note
NONE
```

/cc @eparis
@jakewarr8
Copy link

Does anyone know how I can pass the cadviors metric flags to hyperkube? i.e disable_metrics
https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#metrics

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/cli Categorizes an issue or PR as relevant to SIG CLI.
Projects
None yet
Development

No branches or pull requests

8 participants