Elastic stack (ELK) + Filebeat on Docker

Project fork from deviantony/docker-elk.

This fork show how to implements ELK stack + filebeat on Docker + docker-compose.

Structure

About

We have one (or more) application (python for example) what write some logs on application/logs/.log, and the filebeat will read this file and send to logstash that will do regex filter and after also send to elasticsearch and finally we'll build some graphics on kibana.

App example

This simple python example will write some logs for us.

import os
import uuid
import time

import log


def hello_log():
    while True:
        log.registry(log.LOGGING_ENUM.INFO, f'Info {uuid.uuid4()}')
        log.registry(log.LOGGING_ENUM.WARNING, f'Warning {uuid.uuid4()}')
        log.registry(log.LOGGING_ENUM.ERROR, f'Error {uuid.uuid4()}')
        time.sleep(15)


if __name__ == '__main__':
    if not os.path.exists(os.path.join(os.getcwd(), 'logs', '.log')):
        os.mkdir(f'{os.getcwd()}/logs3')

    hello_log()

ps: I split the logs in a specific module: log.py

Logs example

28/05/2021 20:33:19 INFO     || 3e860697f14e || my_module || Company Name || Info 2473c107-aff8-4b48-85d6-a7ef6cdf89a6
28/05/2021 20:33:19 WARNING  || 3e860697f14e || my_module || Company Name || Warning 9a2d445a-5859-4f6a-8496-059a04c461d3
28/05/2021 20:33:19 ERROR    || 3e860697f14e || my_module || Company Name || Error d336fe31-86f9-4abe-b0c1-3738f5965351
28/05/2021 20:33:24 INFO     || 3e860697f14e || my_module || Company Name || Info a469d96b-564e-4af9-a874-ef2237ae3e71
28/05/2021 20:33:24 WARNING  || 3e860697f14e || my_module || Company Name || Warning a8a123b3-3379-4000-8c8a-83629a4ecd05
28/05/2021 20:33:24 ERROR    || 3e860697f14e || my_module || Company Name || Error 6e07dbeb-8969-4b95-97df-c1253c934d19
28/05/2021 20:33:29 INFO     || 3e860697f14e || my_module || Company Name || Info 9c6c5e09-2001-48ba-a33a-f2f2729b68a8

Click here to see the regex from this.

Running

$ make build

This step takes a few minutes to start ELK and all services to connect.

See what's happening

$ docker-compose logs -f  # all containers
$ docker-compose logs -f <name>  # [app, elasticsearch, logstash, filebeat, kibana]

Kibana

Go to http://localhost:5601/ to access kibana.

Default credentials

username: elastic
password: changeme

Search for index patterns

Create index patterns

We set the index to my_index_patterns on logstash.conf

ps: don't care about my old tests

Select timestamp or not and CREATE INDEX PATTERNS

Select my_index_patterns and see the logs comming

Let's create one visualization

Go to sidebar > dashboard > Create Dashboard

Click on Create Panel > Aggregation Based

Select bar chart and choose my_index_patterns

Let's create X-axis

Click to update to see changes

Create Split Series too, and select terms > status.keyword > update > save return

Dashboard created 🥳

just save to persist

References

• deviantony/docker-elk
• waldemarnt/elk-compose