5 days now ive been playing around, still havent figured out how to get it to work ...
I have 20G of logs that need to be backfilled into ELK stack ... now upgraded to 4.4 kibana 2.2 elastricsearch etc.
heres another roundup of the config setup :
I use a 10- to 49-*.conf input setup 50-output.conf
10-.conf is working as intended. so I have copied it to 11-.conf and played around with it .. no succes
heres 11-.conf
input { file { path => ["/var/log/fortigate/fg.log"] start_position => "beginning" sincedb_path => "/tmp/sucemamarde1" type => "fgt-backfill" } }
filter{
#grok { # match => [ # "message", # "%{GREEDYDATA:kv}" # ] # remove_field => ["message"] #}
kv { source => "message" field_split => " " value_split => "=" }
#date { # match => ["itime", "UNIX_MS"] # locale => "en" #}
geoip{ source =>"dstip" database =>"/opt/logstash/GeoLiteCity.dat" }
}
Tried different setup, using grok to match only %{GREEDYDATA:kv} --- according to grok debugger I should match the log !
one exemple of logs into fg.log :
"itime=1453486381 date=2016-01-22 time=13:13:01 devid=FG200D3913801116 logid=0001000014 type=traffic subtype=local level=notice vd=root srcip=192.168.0.7 srcport=137 srcintf="port1" dstip=192.168.0.255 dstport=137 dstintf="root" sessionid=781856124 status=deny policyid=0 dstcountry="Reserved" srccountry="Reserved" trandisp=noop service=137/udp proto=17 app=137/udp duration=0 sentbyte=0 rcvdbyte=0"
As Magnus Baeck Said, absolutely everything should be parsed with the KV filter by it self. its only a key value chain!, still it dont...
Nothing relevent is output to logstash.log .err or stdout
heres my 50-output.conf
output { #if [type] == "fgt-backfill" { if [path] == "/var/log/fortigate/fg.log" { elasticsearch { hosts => ["localhost:9200"] index => "fgt-backfill-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } }
else { elasticsearch { hosts => ["localhost:9200"] } } #DEBUG TOUT #stdout { codec => rubydebug } }
indices fgt-backfill-* simply doesnt get created at all ... tried with if [path] == XXXX if [type] == "fgt-backfill"
still, a no go. the else work correctly ( meaning my other input are working as intended.
Please. help, I dont know where else, I could ask for help nor. how am i suppose to figure it out by my self...
According to the doc, the index doesnt have to be created PRIOR to logstash indexation.. so im lost.