Howdy.
I have setup ELK stack on one machine, and feed logs from filebeat from another machine, the one that hosts the actual logs, to logstash.
Log entries are quite diverse, however they conform to the following structure:
2017-11-14T11:52:44.386-0500 INFO 8646 com.l7tech.traffic: APITransaction - ;service=oauth/clients;RequestUrl=https://oauth2dns:8443/pub/oauth/clientstore/getAll?client_ident=&client_key=da95da9c-2ab1-42a9-8878-d4457c322036&format=&name=&org=®istered_by=&queryOffset=0&filterStatus=&environment=&master=;RequestMethod=GET;RequestSize=0;ResponseSize=;TotalTime=7;RoutingTime=;HttpStatus=0;Request=XXXXXXXXXXXX;Response=YYYYYYYYYYYYYYYY;
Request and Response may be pretty much anything and span multiple lines, therefore I am using multiline in filebeat.yml like this:
pattern: ^(19|20)\\d\\d-(0[1-9]|1[012])-([012]\\d|3[01])T([01]\\d|2[0-3]):([0-5]\\d):([0-5]\\d)$
negate: true
match: after
logstash in turn is configured as follows:
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{SPACE} %{WORD:length} %{IPORHOST:dns}: %{WORD:transaction_text} - ;%{GREEDYDATA:kvpairs};Request=%{GREEDYDATA:request};Response=%{GREEDYDATA:response};" }
}
kv {
source => "kvpairs"
value_split => "="
field_split => ";"
# remove_field => [ "kvpairs" ] #Delete the field afterwards
}
}
output {
elasticsearch {hosts => ["localhost:9200"] }
stdout {codec => rubydebug }
}
This usually works, unless the log events are written within the same second or so, when they are "merged" into one single event. In other words, I am sending to requests, a GET and a POST 3-4 seconds apart, everything is just fine, logstash output shows me a GET event followed by a POST. Sending exactly the same 2 requests quickly, sometimes I se one event only GET, POST in logstash output.
I would assume there is something to be tuned up, but whatever I tried, didn't really help.
Thanks with anticipation