Stackdriver Output When not on GCP/GCE

[paulleroyza]

Bug Report

The stackdriver plugin appears to only want to use the google metadata service even if running off cloud. Reopening #5563

To Reproduce

• deploy fluentbit:3.0.3 daemonset on K3S cluster
• set fluentbit.conf as

[SERVICE]
    Flush         5
    Grace         120
    Log_Level     trace
    #Log_File      /var/log/fluentbit.log
    Daemon        off
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_PORT     2020
    storage.backlog.mem_limit 5M

[INPUT]
    Name cpu
    Tag  my_cpu

[OUTPUT]
    Name        stackdriver
    Match       *

Error message

[2024/05/08 16:47:04] [ info]      stackdriver.0
[2024/05/08 16:47:04] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2024/05/08 16:47:05] [debug] [stackdriver:stackdriver.0] created event channels: read=850 write=896
[2024/05/08 16:47:05] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] can't fetch token from the metadata server
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] token retrieval failed
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] can't fetch project id from the metadata server
[2024/05/08 16:47:05] [error] [output] failed to initialize 'stackdriver' plugin

Expected behavior
The plugin should honour the GOOGLE_APPLICATION_CREDENTIALS environment variable and use the service account impersonation for workload identity federation from https://cloud.google.com/iam/docs/workload-identity-federation#oidc-credential-security

Your Environment

• Version used: fluentbit:3.0.3 (arm)
• Configuration: daemonset with Workload Identity Federation configured
• Environment name and version (e.g. Kubernetes? What version?): Raspberry Pi cluster running K3S 1.29
• Filters and plugins: stackdriver plugin

Additional context
I am setting up fluentbit logging to Google Cloud using workload identity federation. This would be for non-GCP non-GKE clusters to use GCP as a centralized log sink. I have tried adding the google_service_credentials, project_id_key and export_to_project_id keys in all variations and they have been ignored as the metadata service seems to be the only way the plugin gets the credentials.

The credential configuration file has the correct details for the KSA/GSA federation and I have changed the container from fluentbit:3.0.3 to gcloud-sdk:alpine to test that the pod has connection to cloud logging and I can fetch the logs (I gave the SA logs writer and logs viewer in order to test). This is not a network or credential/federation issue but seems to be the plugin ignoring that it isn't in the cloud.