Diploma Thesis

A platform for monitoring cloud infrastructures by the infrastructure administrator.

Created to run in ICBnet Lab of NTUA.Completed on June, 2016.

The platform that works in the Cloud Infrastructure, gathers Logs from Openstack & Zabbix elements and stores them to Apache Cassandra which is a NoSQL database and Apache Spark on top of Cassandra does batch analysis.

This repository has been created to show code samples of the platform.

Programming Languages, Libraries, Api's and Frameworks were used:

• Python 3.4

• Scala 2.10.4

• Java 1.8.0_72-b15

• ApacheSpark 1.5.2

• Apache Cassandra 3.2

• Zabbix 3.0

• Spark Cassandra Connector 1.5.0 DataStax Python Driver for Apache Cassandra 3.0.0

• Requests 2.9.1

Architecture

• Monitoring Service

This service is responisble for the collection (through Zabbix Rest Api) and forwarding of Zabbix Data from VM's to Cassandra

• Log Service

This service parses Openstack log files to Cassandra.

log-cassandra_connector.py Custom script that reads the below nova-api.log.1 and stores it to Cassandra Cluster.

nova-api.log.1 Sample of nova-api.log.1 file

• Data Analysis Service

This service uses Spark on top of Cassandra to analyze data.
The scripts analyze data from Cassandra & stores results back to it.

spark-script_1.scala Simple script that analyzes data from Cassandra & stores results back to it

spark-script_2.scala Simple script that analyzes data from Cassandra & stores results back to it

• Administration Interconnection Service

This services is responsible to represent the requested queries about the Cloud infrastracture to the administrator of the infrastructure.

view-data_1.py A script that shows grouped past results about CPU utilization