-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Poll Monster is a distributed data collection framework that spans over several hosts to extend the capability to collect many millions bits of information in a short time frame.
The goal of the project is to be able to collect over 60 million SNMP OIDs every 5 minutes on a very small set of hardware.
System spec: HP xw4600 workstation, Core 2 Duo E8400 3GHz
For a single (A->B) one way RPC connection it produced 120.000 small size req/s, 80.000 medium size req/s and 40.000 large size req/s.
For a full setup communication (OrderProcessor -> WorkDispatcher -> Worker -> WorkDispatcher -> OrderProcessor) it produced ~14.000 small size req/s and ~6000 large size req/s. The large req was a 15 SNMP OIDs to 40 hosts so if this can be sent to workers capable of fetching all those OIDs the sum would be 990.000.000 OIDs per 5 minutes. This of course is not including saving all that data into a database but even so 60.000.000 goal should be achievable.
The framework is built in Perl and C/C++. It support different data types to gather starting with SNMP, Telnet/Rancid and IPDR. The data collected can be stored in any DBI-enabled database. The API will be available in SOAP, XML-RPC and JSON-RPC.
It is only a framework meaning you yourself have to create the modules that decide what type of data you collect and how often.
Poll Monster is not a replacement for Cacti, MRTG, OpenNMS or any other kind of data analyze/aggregation software. It is also not a data warehouse product. How you analyze, aggregate or warehouse your data after it's been gathered is purely up to you.