Performance tuning

[jimmidyson]

@repeatedly Thought it best to discuss this in a separate issue so we can stay focused.

This plugin can retrieve data from the Kubernetes API server to enrich records with namespace, pod name, container name, labels, etc. This requires making REST requests to the API server. The relevant REST calls are @client.get_namespace(namespace_name) (see http://kubernetes.io/docs/api-reference/v1/operations/#_read_the_specified_namespace) and @client.get_pod(pod_name, namespace_name) (see http://kubernetes.io/docs/api-reference/v1/operations/#_read_the_specified_pod).

You can see examples for both of those responses in the vcr cassettes.

Metadata is cached to remove the need to continually query the API server: once a new pod is found, the API server is queried & metadata cached for attaching to subsequent records. The cache size is configurable & entries are evicted once a pod or namespace is deleted.

To complicate things slightly, some metadata from the API server is mutable: e.g. you can edit pod labels. To keep this in sync, the plugin configures watches for both namespaces & pods & updates the cached metadata when the metadata changes.

Is that enough info to get started testing the performance? Thanks for helping out with this!

[jcantrill]

Closing with merge of #347