Hi all,
I am setting up a new ELK system to centralize Windows Event logs and in the future syslogs etc. The shipping of the logs using winlogbeat is working fine however when I look at my index for logstash-ddmmyyy I find I have hundreds of shards being created and the performance of the system makes it unusable. I would be expecting 5 shards per index and a new index each day.
I have cleared all of my indexes this morning to start fresh and currently have about 30 Windows Servers sending logs into the system. It has currently been running for 15 minutes and my cluster health is:
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1111,
"active_shards" : 1111,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1111,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
My config file for logstash to Elastic Search is:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
Filter syslog events from filebeat
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
I am clearly making a mistake and would be grateful if someone could help me out.