Latest version of Elastic Stack and Suricata examples

[AndrewW (Andrew Welham)]

I've got the latest version of suricata 3.2.1 working with EVE json logs
I followed the woefully out of date instruction on
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

The version of Elastic stack i am use is
ElasticSearch 5.4.1
Kibana 5.4.1
LogStash 5.4.1

I've tried several samples config files from this forum and other sites but none work, and i'm totally new to this area.
my current logstash.conf file looks like

input {
file {
path => [ "/var/log/suricata/eve.json" ]
codec => json
}
}

filter {
if [src_ip] {
geoip {
source => "src_ip"
target => "geoip"
# set database variable to full path to geoip database if you've got
# the message: "You must specify 'database => ...' in your geoip filter"
#database => "/path/to/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}

output {
stdout { codec => rubydebug }
elasticsearch { embedded => true }
}

What ever i try i keep getting errors like

[2017-06-04T12:10:50,343][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, ", ', -, [, {, ] at line 33, column 22 (byte 778) after input {\n file { \n path => ["/var/log/suricata/eve.json"]\n
sincedb_path => ["}

The line shown (33 in this example changes when i change the conf file

Looking for some guidance on how to get this working.

[AndrewW (Andrew Welham)]

Trying a different tact i have wipes the reporting apps and installed older version according to https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-16-04

everything looks better now except i have no indeces in elastic search

curl http://localhost:9200/_cat/indices

yellow open .kibana 5 1 225 1 408.7kb 408.7kb

mu conf file now looks like this
input {
file {
path => ["/tmp/eve.json"]
codec => json
type => "SuricataIDPS"
start_position => "beginning"
}

}

filter {
if [type] == "SuricataIDPS" {
date {
match => [ "timestamp", "ISO8601" ]
}
ruby {
code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;"
}
}

if [src_ip] {
geoip {
source => "src_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
if ![geoip.ip] {
if [dest_ip] {
geoip {
source => "dest_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
#protocol => http
}
}

what have i missed

[warkolm (Mark Walkom)]

@AndrewW

[2017-06-04T12:10:50,343][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, ", ', -, [, {, ] at line 33, column 22 (byte 778) after input {\n file { \n path => ["/var/log/suricata/eve.json"]\n
sincedb_path => ["}

There doesn't look to be a line 33 in the config, are there other config files?

@AndrewW

add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

You don't need this, LS creates that automatically.