This is a project intended to aggregate different sources of information from Cloud Deployments (OpenStack, Kubernetes, Containers, Operating System ...) using tools from the Big Data ecosystem.
The origin of this project comes from a prototype done by Red Hat's Strategic Design Team and KEEDIO to investigate the different ways to perform log aggregation.
The initial target of the prototype is to aggregate logs from an OpenStack deployment for the OpenStack Operator Persona, howevernt he following data types are considered to be aggregated:
- Logs
- Metrics
- Events
- Configuration
To understand the concept you may go throught there blog posts:
- Ops Happiness: The Quest for Operations Intelligence
- Ops Happiness – Harness Data for Operations Intelligence
- Events and Monitoring Supercharging your Operational Intelligence
There is a running prototype set up on a test lab with half the services manually deployed in VMs on OpenStack and the other half on Containers on OpenShift.
The toolchain selected is the following
rsyslog --> nifi --> kafka --> flink --> cassandra --> patternfly
All the content is available in the main repository.
All the configuration files used in the prototype are uploaded to the configfiles repository.
There is initial work being done to package all components. You may want to check Eidenworks COPR repo