Accept and write logs to ClickHouse in Loghouse compatible format.
Daemon accepts log only in syslog RFC5424 format from rsyslog. Later may be will be added some additinal protocols support.
Configuration is possible with environment variables
ClickHouse address
CLICKHOUSE_ADDR=127.0.0.1:9000
Period in seconds to send data to ClickHouse if batch size is not reached.
ACC_PERIOD=60
Batch size to send to ClickHouse immediately.
ACC_BATCH=10000
Loghouse table type. In most cases you need basic table type - default to Loghouse. But you can use other type - extended, with some additional vendored columns.
ACC_TABLE_TYPE=basic
Configurable partitions periods - daily or hourly. Loghouse supports both types.
ACC_PARTITION_TYPE=hourly
Sharding type. If you didn't use shards and replicas set this to default value "basic". In other case see "Sharding and replication in ClickHouse" section.
ACC_SHARD_TYPE=basic
Cluster name in ClickHouse. See "Sharding and replication in ClickHouse" section.
ACC_TABLE_CLUSTER=my-cluster
Configure rsyslog to resend logs to daemon
action(type="omfwd" Target="127.0.0.1" Port="3333" Protocol="tcp" KeepAlive="on" Template="RSYSLOG_SyslogProtocol23Format")
docker run --rm -it -p 3333:3333 -e ClickHouse_ADDR=127.0.0.1:9000 kaktuss/loghouse-acceptor
It is possible to use and create sharded and replicated partitions. See replication and distributed in ClickHouse docs.
To use this feature with loghouse-acceptor you need to configure two macros in ClickHouse.
<macros>
...
<shard>02</shard>
<replica>example05-02-1.yandex.ru</replica>
...
</macros>
Then configure cluster in ClickHouse
<remote_servers>
<my-cluster>
...
</my-cluster>
<remote_servers>
Then set ACC_SHARD_TYPE variable to basic_sharded and ACC_TABLE_CLUSTER to your cluster name.
ACC_SHARD_TYPE=basic_sharded
ACC_TABLE_CLUSTER=my-cluster