Docker compose example from official docs broken? #391

stemabo · 2022-02-21T09:12:31Z

Describe the bug

I am (to the best of my knowledge) strictly following the official docker-compose example (https://docs.fluentd.org/container-deployment/docker-compose) but the Fluentd image runs into an error installing fluent-plugin-elasticsearch:

fluentd_1 | 2022-02-21 08:44:31 +0000 [info]: gem 'fluentd' version '1.12.0'
fluentd_1 | /usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in 'require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)

Is also experienced by others (@https://stackoverflow.com/users/18044852/henry-kao):
https://stackoverflow.com/questions/71120621/efk-system-is-build-on-docker-but-fluentd-cant-start-up/71126320

To Reproduce

follow https://docs.fluentd.org/container-deployment/docker-compose

Expected behavior

Fluentd container to start up with no errors...

Your Environment

- Fluentd version: fluent/fluentd:v1.12.0-debian-1.0 (as per the docs, newer version seems to make no difference) 
- Operating system: Ubuntu 20.04.4 LTS
- Kernel version: 5.4.0-89-generic

Your Configuration

as in [https://docs.fluentd.org/container-deployment/docker-compose](url)

Your Error Log

/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in 'require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)

Additional context

I would assume this is me getting something wrong, but as others confirm this behavior it might actually be an issue...

The text was updated successfully, but these errors were encountered:

BrianDouglas · 2022-04-07T22:57:15Z

Can confirm as I'm having the exact same issue. Been having a bunch of trouble trying to get the EFK stack working. Seems like a lot of articles and tutorials out there are out of date. A real shame that even the official documentation doesn't work.

Xernosch · 2022-04-14T11:22:22Z

Hi!

I came across the same Problem and I got it to work. Not sure if this solution fits for your situation.

While following the official docs ( https://docs.fluentd.org/container-deployment/docker-compose ) I changed the config to use the newest versions up to date.

In the fluentd Dockerfile change the version from "5.0.3" to the newest version

RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.2.2"]

Since this only seems to work with elastic >= 8, you have to update elastic and kibana in the docker-compose.yml as well. I also needed to disable TLS for elastic as well.

version: "3"
services:
[...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.1.2
    container_name: elasticsearch
    environment:
      - "discovery.type=single-node"
      - xpack.security.enabled=false
    expose:
      - "9200"
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.1.2
[...]

This works for me. Of course, disabling security (xpack.security.enabled=false) is just for proof of concept. Otherwise elastic will complain about the http connections where it expects https. Right now I am not aware how to configure fluentd to do this.

Hope this helps!
Cheers

SEJSCloud · 2022-05-15T10:31:43Z

Thank you @Xernosch
I have suffered for 1 week beacuse of version compatibility of fluentd ...
It works perfectely!!

iiienapg · 2022-06-07T07:43:55Z

In the fluentd Dockerfile change the version from "5.0.3" to the newest version
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.2.2"]
Since this only seems to work with elastic >= 8, you have to update elastic and kibana in the docker-compose.yml as well. I also needed to disable TLS for elastic as well.

If you don't use ES8 for any reasons install elasticsearch gem version < 8 just before fluent-plugin-elasticsearch:

RUN ["gem", "install", "elasticsearch", "--no-document", "--version", "< 8"]
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.2.2"]

and don't switch to ES8 in docker-compose.yml

valentinacala · 2023-07-19T15:27:46Z

Hi, I tried to apply the suggested approach. Now I receive a new error:

2023-07-19 15:24:59 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ dir_s_mkdir - /var/log/td-agent"
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/fileutils.rb:247:in `mkdir'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/fileutils.rb:247:in `fu_mkdir'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/fileutils.rb:228:in `block (2 levels) in mkdir_p'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/fileutils.rb:226:in `reverse_each'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/fileutils.rb:226:in `block in mkdir_p'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/fileutils.rb:211:in `each'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/fileutils.rb:211:in `mkdir_p'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/plugin/in_tail.rb:243:in `start'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:203:in `block in start'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `each'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:191:in `block in lifecycle'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `each'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:178:in `lifecycle'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/root_agent.rb:202:in `start'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:248:in `start'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/engine.rb:147:in `run'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:720:in `block in run_worker'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:971:in `main_process'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:711:in `run_worker'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:376:in `<top (required)>'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/gems/fluentd-1.14.6/bin/fluentd:15:in `<top (required)>'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `load'
  2023-07-19 15:24:59 +0000 [error]: #0 /usr/local/bundle/bin/fluentd:23:in `<main>'
2023-07-19 15:24:59 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ dir_s_mkdir - /var/log/td-agent"
  2023-07-19 15:24:59 +0000 [error]: #0 suppressed same stacktrace
2023-07-19 15:24:59 +0000 [error]: Worker 0 finished unexpectedly with status 1

elasticsearch/kibana version: 7.10.2
fluentd version: v1.12.0-debian-1.0

Any suggestion on why having this error and a possible solution?

Thank you in advance!

paulsjohnson91 · 2023-08-17T13:52:14Z

For those having problems, it all seems to come down to version incompatibility. This worked for me:

version: "3"
services:
  web:
    container_name: web
    image: httpd
    ports:
      - "80:80"
    links:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: httpd.access

  fluentd:
    container_name: fluentd
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    ports:
      - "24224:24224"
      - "24224:24224/udp"

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.1.2
    container_name: elasticsearch
    environment:
      - "discovery.type=single-node"
      - "xpack.security.enabled=false"
    expose:
      - "9200"
    ports:
      - "9200:9200"

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:8.1.2
    links:
      - "elasticsearch"
    ports:
      - "5601:5601"

With the Dockerfile being

# fluentd/Dockerfile

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem", "install", "elasticsearch", "--no-document", "--version", "8.1.2"]
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.2.2"]
USER fluent

You might need to restart the fluentd pod if it fails before the elasticsearch pod has finished initialising

reason211 · 2023-08-23T12:20:14Z

mark

willzhang · 2023-10-12T08:41:20Z

elasticsearch 8.x dockerfile

FROM fluent/fluentd:v1.16.2-debian-1.1
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.3.0"]
USER fluent

docker-compose.yaml

version: "3"
services:
  web:
    image: httpd
    ports:
      - "80:80"
    links:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: httpd.access

  fluentd:
    image: registry.cn-shenzhen.aliyuncs.com/cnregistry/fluentd-es:v1.16.2-debian-1.1
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    ports:
      - "24224:24224"
      - "24224:24224/udp"

  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.2
    ports:
      - 9200:9200
    environment:
      - http.host=0.0.0.0
      - transport.host=127.0.0.1
      - xpack.security.enabled=false

  kibana:
    image: docker.elastic.co/kibana/kibana:8.10.2
    links:
      - "elasticsearch"
    ports:
      - "5601:5601"

bestwebua · 2023-10-24T16:21:43Z

@paulsjohnson91 Thanks for your notes! It works! ❤️ ❤️ ❤️

ashie transferred this issue from fluent/fluentd Feb 22, 2022

g3kr mentioned this issue May 27, 2022

fluentd container exiting immediately digikin/fluentd-elastic-kibana#7

Closed

kumashun8 mentioned this issue Jan 18, 2023

docker-composeでEFKを動作させる kumashun8/fluentd-plugin-sample#1

Merged

ashie mentioned this issue Apr 19, 2023

fluentd docker-compose error: `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError) fluent/fluentd#4039

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker compose example from official docs broken? #391

Docker compose example from official docs broken? #391

stemabo commented Feb 21, 2022 •

edited

BrianDouglas commented Apr 7, 2022

Xernosch commented Apr 14, 2022 •

edited

SEJSCloud commented May 15, 2022

iiienapg commented Jun 7, 2022

valentinacala commented Jul 19, 2023

paulsjohnson91 commented Aug 17, 2023

reason211 commented Aug 23, 2023

willzhang commented Oct 12, 2023

bestwebua commented Oct 24, 2023

Docker compose example from official docs broken? #391

Docker compose example from official docs broken? #391

Comments

stemabo commented Feb 21, 2022 • edited

Describe the bug

To Reproduce

Expected behavior

Your Environment

Your Configuration

Your Error Log

Additional context

BrianDouglas commented Apr 7, 2022

Xernosch commented Apr 14, 2022 • edited

SEJSCloud commented May 15, 2022

iiienapg commented Jun 7, 2022

valentinacala commented Jul 19, 2023

paulsjohnson91 commented Aug 17, 2023

reason211 commented Aug 23, 2023

willzhang commented Oct 12, 2023

bestwebua commented Oct 24, 2023

stemabo commented Feb 21, 2022 •

edited

Xernosch commented Apr 14, 2022 •

edited