Hello,
I have a Filebeat instance shipping logs from a single source to two different servers with Logstash running on both.
I have copied and pasted the same following .conf file on both servers:
input {
beats {
ssl => false
port => 5044
}
}
filter {
if [message] =~ "^#" {
drop {}
}
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:serviceName} %{WORD:serverName} %{IP:serverIP} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:protocolVersion} %{NOTSPACE:userAgent} %{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:requestHost} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response} %{NUMBER:bytesSent} %{NUMBER:bytesReceived} %{NUMBER:timetaken}"]
}
if [serviceName] == "W3SVC2" {
mutate {
replace => [ "type", "iis" ]
remove_field => [ "serviceName", "timestamp", "[beat]", "[prospector]", "[tags]", "[host]", "[input]"]
convert => {
"timetaken" => "float"
"bytesReceived" => "float"
"bytesSent" => "float"
}
}
}
date {
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ]
timezone => "Etc/UTC"
}
}
output {
elasticsearch {
hosts => ["xxx.xxx.xxx.xxx:9200", "xxx.xxx.xxx.xxx:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
stdout {codec => rubydebug}
}
...so I was expecting to find my Elasticsearch documents stored in the same way, instead of:
As you can see, the first field of the first document (which comes from one logstash server) is the "offset" field, while the first field of the second document (which comes from the other one) is the "method", and that makes it a bit confusing to look and compare documents.
I also noticed that everytime I restart Logstash, the sorting of the fields changes.
Does anybody knows how to fix this?
Thanks in advance.