So I have a problem with logstash:
some messages are being sent to default index:(below is the message that went into 'default' index.
{ "_index": "default-logs-2018.10.10", "_type": "doc", "_id": "MioyXmYBMkEJhrYr2PRg", "_score": 1, "_source": { "source": "/var/log/messages", "host": { "name": "serverx" }, "beat": { "version": "6.4.2", "name": "serverx", "hostname": "serverx" }, "@version": "1", "offset": 2767233, "fields": { "document_type": "syslog" }, "@timestamp": "2018-10-10T13:37:56.000Z", "system": { "syslog": { "program": "root", "timestamp": "Oct 10 15:37:56", "hostname": "serverx", "message": "172.30.11.18 - - [10/Oct/2018:15:37:56 +0200] \"GET /something HTTP/1.1\" 200 3839 \"-\" \"Java/1.8.0_152\"" } }, "tags": [ "syslog", "prod", "beats_input_codec_plain_applied" ] }, "fields": { "@timestamp": [ "2018-10-10T13:37:56.000Z" ] } }
It happens only with syslog logs.
This is relevant part of logstash
if "syslog" in [tags] { grok { match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] } pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" } remove_field => "message" } date { match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } output { if "syslog" in [tags] { elasticsearch { user => logstash password => logstash ssl => true hosts => ["https://localhost:9200"] cacert => '/etc/logstash/esnode.pem' sniffing => false manage_template => false index => "system-logs-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } else { elasticsearch { user => logstash password => logstash ssl => true hosts => ["https://localhost:9200"] cacert => '/etc/logstash/esnode.pem' sniffing => false manage_template => false index => "default-logs-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } }
This is filebeats conf:
- input_type: log paths: - /var/log/messages* - /var/log/syslog* fields: document_type: syslog multiline.pattern: "^\\s" multiline.match: after exclude_files: [".gz$"] tags: ["syslog", "prod"]
Same amount of messages goes to defaults-logs index and system-logs index.