Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(kubernetes_logs): use kube-apiserver cache for list requests #17095

Merged
merged 4 commits into from Apr 13, 2023
Merged

feat(kubernetes_logs): use kube-apiserver cache for list requests #17095

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9e4449b
feat(kubernetes_logs): use kube-apiserver cache for list requests
nabokihms Apr 10, 2023
327c6c4
Satisfy clippy
nabokihms Apr 11, 2023
bfcd886
Update src/sources/kubernetes_logs/mod.rs
nabokihms Apr 13, 2023
c7abeb3
Regenerate docs
nabokihms Apr 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
112 changes: 53 additions & 59 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions Cargo.toml
View file
Open in desktop
Expand Up @@ -271,8 +271,8 @@ indexmap = { version = "~1.9.3", default-features = false, features = ["serde"]
infer = { version = "0.13.0", default-features = false, optional = true}
indoc = { version = "2.0.1", default-features = false }
inventory = { version = "0.3.5", default-features = false }
k8s-openapi = { version = "0.16.0", default-features = false, features = ["api", "v1_19"], optional = true }
kube = { version = "0.75.0", default-features = false, features = ["client", "native-tls", "runtime"], optional = true }
k8s-openapi = { version = "0.18.0", default-features = false, features = ["api", "v1_26"], optional = true }
spencergilbert marked this conversation as resolved.
Show resolved Hide resolved
kube = { version = "0.82.0", default-features = false, features = ["client", "openssl-tls", "runtime"], optional = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting to see @jszwedko - that feature was removed by this PR. Worth noting given our previous conversations around openssl vs native.

listenfd = { version = "1.0.1", default-features = false, optional = true }
logfmt = { version = "0.0.2", default-features = false, optional = true }
lru = { version = "0.10.0", default-features = false, optional = true }
Expand Down
24 changes: 20 additions & 4 deletions src/sources/kubernetes_logs/mod.rs
View file
Open in desktop
Expand Up @@ -19,7 +19,7 @@ use futures_util::Stream;
use k8s_openapi::api::core::v1::{Namespace, Node, Pod};
use k8s_paths_provider::K8sPathsProvider;
use kube::{
api::{Api, ListParams},
api::Api,
config::{self, KubeConfigOptions},
runtime::{
reflector::{self},
Expand Down Expand Up @@ -219,6 +219,9 @@ pub struct Config {
#[configurable(metadata(docs::examples = "/path/to/.kube/config"))]
kube_config_file: Option<PathBuf>,

/// Determines if requests to the kube-apiserver can be served by a cache.
use_apiserver_cache: bool,

/// How long to delay removing metadata entries from the cache when a pod deletion event
/// event is received from the watch stream.
///
Expand Down Expand Up @@ -271,6 +274,7 @@ impl Default for Config {
ingestion_timestamp_field: None,
timezone: None,
kube_config_file: None,
use_apiserver_cache: false,
delay_deletion_ms: default_delay_deletion_ms(),
log_namespace: None,
}
Expand Down Expand Up @@ -519,6 +523,7 @@ struct Source {
max_line_bytes: usize,
fingerprint_lines: usize,
glob_minimum_cooldown: Duration,
use_apiserver_cache: bool,
ingestion_timestamp_field: Option<OwnedTargetPath>,
delay_deletion: Duration,
}
Expand Down Expand Up @@ -595,6 +600,7 @@ impl Source {
max_line_bytes: config.max_line_bytes,
fingerprint_lines: config.fingerprint_lines,
glob_minimum_cooldown,
use_apiserver_cache: config.use_apiserver_cache,
ingestion_timestamp_field,
delay_deletion,
})
Expand Down Expand Up @@ -625,6 +631,7 @@ impl Source {
max_line_bytes,
fingerprint_lines,
glob_minimum_cooldown,
use_apiserver_cache,
ingestion_timestamp_field,
delay_deletion,
} = self;
Expand All @@ -633,11 +640,18 @@ impl Source {

let pods = Api::<Pod>::all(client.clone());

let list_semantic = if use_apiserver_cache {
watcher::ListSemantic::Any
} else {
watcher::ListSemantic::MostRecent
};

let pod_watcher = watcher(
pods,
ListParams {
watcher::Config {
field_selector: Some(field_selector),
label_selector: Some(label_selector),
list_semantic: list_semantic.clone(),
..Default::default()
},
)
Expand All @@ -658,8 +672,9 @@ impl Source {
let namespaces = Api::<Namespace>::all(client.clone());
let ns_watcher = watcher(
namespaces,
ListParams {
watcher::Config {
label_selector: Some(namespace_label_selector),
list_semantic: list_semantic.clone(),
..Default::default()
},
)
Expand All @@ -680,8 +695,9 @@ impl Source {
let nodes = Api::<Node>::all(client);
let node_watcher = watcher(
nodes,
ListParams {
watcher::Config {
field_selector: Some(node_selector),
list_semantic,
..Default::default()
},
)
Expand Down