Grafana Agent Flow feedback

[rfratto]

Use this discussion to provide feedback about Grafana Agent Flow to its developers.

[mtrin]

Hey there, I'm testing this and really like how it streamlines and helps understand prometheus configuration.
I'm wondering if there's a way to import existing prometheus.yaml configs and convert to a river file or just reference it? I can see the scrape component uses prometheus model.

[rfratto]

Thanks for the feedback!

I'm wondering if there's a way to import existing prometheus.yaml configs and convert to a river file or just reference it? I can see the scrape component uses prometheus model.

Not at the moment, but we're definitely interested in introducing tooling to do things like this once Flow is more stable.

[akselleirv]

Hello @rfratto,

Does it exists any plugin for River for either vscode or jetbrains? Even syntax highlighting would be greatly appreciated. I assume autocompletion would be considered out of scope for now?

[rfratto]

Yeah, autocompletion would take a bit of work I'm afraid, but definitely possible in the future.

I created a VSCode extension a few months back, but haven't gotten around to publishing it yet. Ideally we move it to the Grafana org before it's published. In the meantime, the VSIX can be installed manually.

Jetbrains is also technically possible, but it's much harder to write a syntax highlighting extension for compared to Vim or VSCode, so I'm not sure when I'll have time to get around to writing one.

[akselleirv]

Thank you! I’ll try out the VSCode plug-in. It’s definitely not a must to support both IDEs.

[jzck]

Hello @rfratto,

I've been using the grafana agent operator to deploy the agent on k8s. I'm interested in using the flow UI to debug scrape jobs, but it's obvious I can't just set AGENT=flow because the config language is different. Is it planned for the grafana agent operator to generate river configs for flow from the operator CRDs instead of the yaml configs?

[rfratto]

There's no plans at the moment for having the operator generate Flow mode configs. Instead, there's plans to move operator functionality into Flow: #2466

[jzck]

That makes more sense, so what you're saying is:

1. v1alpha1/PodLogs -> becomes v1alpha2/PodLogs which is consumed by the loki.source.podlogs flow component.
2. v1alpha1/LogsInstance -> not used anymore, replaced with:
   1. the PodMonitor loki CRD and
   2. the prometheus.kubernetes.pod_monitors flow component

I'd like to understand the full vision, because currently operator doesn't just configure logs but also deploys agents pods thanks to the v1alpha1/GrafanaAgent, will this be replaced by a v1alpha2/GrafanaAgent that deploys a flow agent?

[rfratto]

The full vision is in flux, so grain of salt:

Right now we suspect that the combination of the Grafana Agent Helm chart plus Grafana Agent Flow (with support for PodLogs/ServiceMonitor/PodMonitor/Probe) would be all users would need.

In that scenario, there would be no need for a GrafanaAgent resource, since Grafana Agent Flow would be deployed via the Helm chart. There would also be no need for MetricsInstances, LogsInstances, or Integrations, since they would be replaced by Flow components which discover the other resources you want to operate on (PodLogs, PodMonitors, etc).

We're still exploring this space right now; if it turns out it doesn't work in practice we'll consider what we need to do to have the operator work with Flow. But for the moment, it seems like the operator might not be necessary long term; the hope is that it's practical to consolidate everything down to a single solution (Flow) to make things simpler for everyone.

[jzck]

Right now we suspect that the combination of the Grafana Agent Helm chart plus Grafana Agent Flow (with support for PodLogs/ServiceMonitor/PodMonitor/Probe) would be all users would need.

That makes sense. I see that the grafana agent helm chart deploys flow by default, do you think you'll be maintaining both modes or deprecating the yaml config mode? if so, will you also deprecate the operator?

found the ansnwer:

But for the moment, it seems like the operator might not be necessary long term; the hope is that it's practical to consolidate everything down to a single solution (Flow) to make things simpler for everyone.

In that scenario, there would be no need for a GrafanaAgent resource, since Grafana Agent Flow would be deployed via the Helm chart. There would also be no need for MetricsInstances, LogsInstances, or Integrations, since they would be replaced by Flow components which discover the other resources you want to operate on (PodLogs, PodMonitors, etc).

FYI: I've tried flow over the weekend, loki.source.kubernetes does the job, I don't see a need for podlogs.

[virtualtam]

Hi @rfratto 👋

We are deploying Grafana Agent in Flow mode in Kubernetes via the official Helm chart, and have ported the YAML configuration for Grafana Cloud to Flow to gather:

• Kubernetes event logs
• Kubernetes pod logs (via the Kubernetes API)
• Kubelet and cAdvisor metrics
• Kubernetes control plane metrics (via kube-state-metrics)
• Kubernetes node metrics (via node_exporter)
• Business metrics from pods and services

The Web interface for the Agent is quite useful to diagnose issues with service discovery, scrape jobs and relabeling/filtering 👍

We worked our way through trial and error for:

• validating River configuration before deploying it:
  • agentctl config-check does not seem to support Flow mode, or it is not clearly documented (tested with Agent v0.32.1)
  • instead we ran the Agent locally to identify the most obvious configuration issues
  • a working config-check for the River configuration would be very useful for CI/CD
• passing secrets for authentication to Grafana Cloud services (Mimir, Loki):
  • when generating configuration from Grafana Cloud, credentials are present in plain text
  • to work around this, we declare the secrets as volumes at the controller level, and mount them for the Agent to read
  • this corresponds to Flow proposal: remote.kubernetes.configmap and remote.kubernetes.secret  #2976
• defining whether we should use exporter integrations or deploy extra exporters to scrape Kubernetes metrics:
  • the Agent provides several exporter components that may be useful to avoid deploying extra exporters (eg node_exporter, https://github.com/grafana/agent/issues/3260)
• getting dashboards to work; the Agent configuration generated from Grafana Cloud contains metrics relabeling rules to reduce the number of collected metrics, and filters out the up metrics for kube-state-metrics and node_exporter, that is required for the Kubernetes dashboards provided by the Grafana Cloud integration

If there is interest, I could share (a curated version of) our current Grafana Agent configuration 📜

[tpaschalis]

Hey, apologies for the super belated message! How has Flow worked for you in the meantime? We'd love to hear more of your experience!

If there is interest, I could share (a curated version of) our current Grafana Agent configuration 📜

That could be a good idea. We now have a repository to share reusable pieces of Flow configuration as Modules, maybe we can add a new module there?

https://github.com/grafana/agent-modules

[johndo6776]

Possible problem with loki.source.docker ?

Hi @rfratto

I'm setting up Grafana Agent in flow mode on a set of VMs in order to get metrics from the VM and running docker containers (forwarded to mimir), and also the logs from the running docker containers (forwarded to loki).
(Really liking the way flow mode is configured and the concepts used by it 👍 , just verry hard to find good examples of configurations for now)

For the metrics I made it work, but for the container logs I stumbled on a problem on container restarts.

For the container logs I have the following components set up : dicovery.docker -> dicovery.relabel -> loki.source.docker -> loki.write

/////////////////////////////////////////////////////////
// LOGGING (from containers to loki)
/////////////////////////////////////////////////////////

discovery.docker "container_logs" {
	host             = "unix:///var/run/docker.sock"
	// refresh_interval = "5s"

// only for containers with a label allow_scraping_logs_for_monitoring=True 
	filter {
		name   = "label"
		values = ["allow_scraping_logs_for_monitoring=True"]
	}
}


// add label docker_container_name
discovery.relabel "container_logs" {
	rule {
		source_labels = ["__meta_docker_container_name"]
		regex         = "/(.*)"
		target_label  = "docker_container_name"
	}

	targets = discovery.docker.container_logs.targets
}

// use this to get logs from containers for loki 
loki.source.docker "container_logs" {
  host       = "unix:///var/run/docker.sock"
  targets    = discovery.relabel.container_logs.output

  forward_to = [loki.write.default.receiver]
}

// write to loki
loki.write "default" {
	endpoint {
		url       = env("LOKI_URL")
		tenant_id = env("MIMIR_AND_LOKI_TENANT_ID")
	}
	external_labels = {}
}

In the beginning everything works fine, but after a docker restart of a container, logs are no longer tailed for that container, and in the grafana agent UI for loki.source.docker, I have the part : Debug info / targets_info -> is_running: "false" for the restarted container.
And in the logs of grafana agent I can find a line with "... finished transferring logs ..."

I went trough the source code for the loki.source.docker component, and what I think is happening is the following : (I'm by no means an expert in Golang, so it's possible I misinterpreted some steps) :

• On restart of container : when it stops :
  • in func (t *Target) processLoop(ctx context.Context)
  • the stream logs created at https://github.com/grafana/agent/blob/main/component/loki/source/docker/internal/dockertarget/target.go#L110C3-L110C3 closes
  • io.Copy(wstdout, logs) https://github.com/grafana/agent/blob/main/component/loki/source/docker/internal/dockertarget/target.go#L131C27-L131C34 or stdcopy.StdCopy(wstdout, wstderr, logs) https://github.com/grafana/agent/blob/main/component/loki/source/docker/internal/dockertarget/target.go#L133 finishes without erroring
  • the agent logs "... finished transferring logs ..." https://github.com/grafana/agent/blob/main/component/loki/source/docker/internal/dockertarget/target.go#L138
  • defer t.running.Store(false) sets running of the target at false => in UI we can see is_running: "false" for this target
• Until here no problem
• If we wait long enough before restarting the container, there is also no problem, because
  • the targets given by dicovery.docker won't include the stopped container
  • loki.source.docker component Update(args component.Arguments) is called without the target for the stopped container https://github.com/grafana/agent/blob/main/component/loki/source/docker/docker.go#L140C21-L140C53
  • which calls c.manager.syncTargets(context.Background(), targets) https://github.com/grafana/agent/blob/main/component/loki/source/docker/docker.go#L206
  • which calls m.runner.ApplyTasks(ctx, tasks) https://github.com/grafana/agent/blob/main/component/loki/source/docker/runner.go#L136
  • ApplyTasks() searches for tasks to remove by comparing if it has already a worker for a task that has no corresponding task in the newTasks Hashmap : if newTasks.Has(w.(*workerTask).Task) { ... https://github.com/grafana/agent/blob/main/pkg/runner/runner.go#L120
  • as in this case we waited long enough to have the newTasks hashMap to not include a task for the stopped container, it will call w.Worker.Cancel() on the running task and wait correctly for the task to finish https://github.com/grafana/agent/blob/main/pkg/runner/runner.go#L130C4-L130C21
• If we start the container again after this process everything is ok, because prvious task and associated worker have been cleaned up.
• The problem occurs when the container restarts before the previously explained cleanup : (this happens for example with a simple $ docker restart ... command, even when refresh_interval of discovery.docker is set to 1s) : because this time :
  • the targets given by dicovery.docker WILL include the target definition of the restarted container, and is exactly the same as the target definition of the container before it was stopped
  • loki.source.docker component Update(args component.Arguments) is called WITH the target for the restarted container https://github.com/grafana/agent/blob/main/component/loki/source/docker/docker.go#L140C21-L140C53
  • which calls c.manager.syncTargets(context.Background(), targets) https://github.com/grafana/agent/blob/main/component/loki/source/docker/docker.go#L206
  • which calls m.runner.ApplyTasks(ctx, tasks) https://github.com/grafana/agent/blob/main/component/loki/source/docker/runner.go#L136
  • ApplyTasks() searches for tasks to remove by comparing if it has already a worker for a task that has no corresponding task in the newTasks Hashmap : if newTasks.Has(w.(*workerTask).Task) { ...
  • In this case have the newTasks hashMap INCLUDING a task for the restarted container, and already a worker assiciated with an identically declared task => it will NOT call w.Worker.Cancel() on the running task and wait correctly for the task to finish https://github.com/grafana/agent/blob/main/pkg/runner/runner.go#L130C4-L130C21 => and it will NOT add a new task and call run on it to tail the logs of the restarted container.

My questions are the following :

• Did I misconfigured something that causes this ?
• If not, were you already aware of this problem ? (I didn't find anything about this ...)
• Is there a workaround / quick fix ?
• Is there a fix in development for this ?
  • A first idea that could fix it :
    • by adding a property to the target , something like finished or finishedTailing, defaulting at false,
      • but would be set at true when io.Copy(wstdout, logs) https://github.com/grafana/agent/blob/main/component/loki/source/docker/internal/dockertarget/target.go#L131C27-L131C34 or stdcopy.StdCopy(wstdout, wstderr, logs) https://github.com/grafana/agent/blob/main/component/loki/source/docker/internal/dockertarget/target.go#L133 finishes
    • and in the Equals function of the tailerTask https://github.com/grafana/agent/blob/main/component/loki/source/docker/runner.go#L65 , this could be used to differenciate targets finished tailing logs (e.g. because of docker stop or docker restart ) and ones that have not started yet, from the newTasks map (but should be because they're still given by e.g. discovery.docker ) -> as they are different (if finishedTailing set to true), the old task/worker would be removed, and a new one would be started (as finishedTailing defaults to false, one in a newTasks map would be "Equal" to one running but not finished yet, so no new task/worker would be added in this case)
    • inconvenient : it needs targets of loki.source.docker to be updated updated in order to trigger the launch of a new task/worker
    • advantage : if discovery.docker is used, we have already the refresh_interval to parameter the frequency of the checks for this
  • A second idea could be to try continuously to launch the tailing in func (t *Target) processLoop(ctx context.Context) as long as it is not explicitely stopped by removing the task from the taskmanager.
    • I suppose this can be implemented by passing the context ctx of processLoop to the go routine that copies from logs to the streams, and the go routines that process the streams (func (t *Target) process(r io.Reader, logStreamLset model.LabelSet)), and making them theck on ctx.Err() (= in case the context is effectively cancelled) in a for loop, and adding some sleep time (e.g. 1s) when we have to try again because of some other error or closing of streams occured (e.g. because of a docker restart)
    • advantage : no need for new update of targets of loki.source.docker + this is probably most likely what people expect what the component will do (continually try to tail the logs of what is specified in the targets of the component)
    • inconvenient : if targets of loki.source.docker are not updated , it will try forever (only inconvenient if bad config / entry for targets of component)

[mlcdf]

Hello @rfratto!

I'm having a hard time with river lang.

prometheus.remote_write "metrics_default" {

	endpoint {
		name = "Mimir endpoint"
		url  = json_decode(local.file.mimir_secrets.content)["url"] + "/api/v1/push"

		basic_auth {
			username = json_decode(local.file.mimir_secrets.content)["basic_auth"]["username"]
			password = json_decode(local.file.mimir_secrets.content)["basic_auth"]["password"]
		}

	}
}

local.file "mimir_secrets" {
  filename = "/path/to/file"
}

Is there a way for me to factorize this json_decode(local.file.mimir_secrets.content) ?

I tried this, but it didn't work and I couldn't find any example in the docs.

xxx = json_decode(local.file.mimir_secrets.content)
xxx["url"]

[tpaschalis]

No, not yet at least unfortunately. There is a proposal to implement aliases but it's not been done yet.

You could maybe use a module where you pass in the filename and it outputs the field that you'd like.

[mlcdf]

Oh I see. I'll keep it that way then, it's not that big of a deal for this specific case! Thanks.

Though, I do like the idea of aliases for more complex use cases :)

[wally007]

Use this discussion to provide feedback about Grafana Agent Flow to its developers.

Hello,
I'm confused about the flow mode and discovery.file local.file_match and components in general.
Essentially what I'm trying to find out, is it possible to split configuration into several files ?
(each component in a separate file, like one can do with telegraf for example)

Reason I ask, is that we have multiple systems and ansible playbooks deploying applications to the server.
Preferably each playbook (say deploy mysql server) would just dump a new file (mysql flow component and reference main default remote write component in separate file) into grafana agents configuration directory and restart or force re-read of the configuration.

If grafana agent forces to use single file, this is not possible to do reliably.
We already use this approach with telegraf but we're evaluating new agent or to use native prometheus exporters directly.

Thanks

[tpaschalis]

Hey there @wally007, apologies for the belated response.

is it possible to split configuration into several files

Yes, since the last release, you can pass a directory to the run command which will concatenate and run all files. In this case you have to pay attention so that the component references map correctly.

Otherwise, you could make use of Flow modules as another way of building modular, reusable configurations (that you can even version control in Git and load from the Git repo directly).

Whenever the configuration is updated, you'd have to hit the /-/reload endpoint, as currently we don't watch for changes in files (although it's probable we might in the future).

[rfratto]

I'm going to close and lock this discussion; the original purpose was to get feedback on Flow when it was first introduced to know whether we should keep investing effort into it. That ship has long sailed, and Flow has come a long way since this discussion was first opened.

Thank you to everyone who provided feedback on this. If you have questions about Flow, please feel free to open new discussions or chat with us on #agent in the Grafana Slack.