Need help with Logstash

Elastic Stack Logstash
1

I am new to Logstash, i dont have a background of building a logstash before. I built Suricata (IDS) and on the same machine i built logstash. I was successful in installing it and bringing up the webpage. However, i am stuck at this point " Unable to fetch mapping. Do you have incdices?"

Status after starting the service

log [00:01:43.995] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready
log [00:01:44.712] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [00:01:44.980] [info][status][plugin:kbn_vislib_vis_types@1.0.0] Status changed from uninitialized to green - Ready
log [00:01:45.100] [info][status][plugin:markdown_vis@1.0.0] Status changed from uninitialized to green - Ready
log [00:01:45.291] [info][status][plugin:metric_vis@1.0.0] Status changed from uninitialized to green - Ready
log [00:01:45.371] [info][status][plugin:spyModes@1.0.0] Status changed from uninitialized to green - Ready
log [00:01:45.470] [info][status][plugin:statusPage@1.0.0] Status changed from uninitialized to green - Ready
log [00:01:45.559] [info][status][plugin:table_vis@1.0.0] Status changed from uninitialized to green - Ready
log [00:01:45.744] [info][listening] Server running at http://0.0.0.0:5601
log [00:01:47.833] [error][status][plugin:elasticsearch@1.0.0] Status changed from yellow to red - Request Timeout after 3000ms
log [00:02:03.857] [info][status][plugin:elasticsearch@1.0.0] Status changed from red to green - Kibana index ready
log [00:02:43.557] [error][status][plugin:elasticsearch@1.0.0] Status changed from green to red - Request Timeout after 3000ms
log [00:02:51.693] [info][status][plugin:elasticsearch@1.0.0] Status changed from red to green - Kibana index ready
log [00:31:56.885] [error][status][plugin:elasticsearch@1.0.0] Status changed from green to red - Request Timeout after 3000ms
log [00:32:04.388] [info][status][plugin:elasticsearch@1.0.0] Status changed from red to green - Kibana index ready
log [00:34:39.853] [error][status][plugin:elasticsearch@1.0.0] Status changed from green to red - Request Timeout after 3000ms
log [00:34:57.915] [info][status][plugin:elasticsearch@1.0.0] Status changed from red to green - Kibana index ready

Kindly help me on how to fix it. Please do not assume i know everything already. Kindly help me, i am doing this as my school project.

2

What does your Logstash configuration look like? How do you know Logstash is processing input events? Have you checked with a stdout { codec => rubydebug } output? If that seems to work, is there anything in the logs indicating whether Elasticsearch is accepting any of the events? If that seems okay, have you configured the index pattern in Kibana correctly?

3

Hi Magnus,

Thank you very much for your reply.

I exactly followed the steps given in this link below,
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/_Logstash_Kibana_and_Suricata_JSON_output

however, i am not getting the output as expected.

This is the conf file that i created using that link

input {
file {
path => ["/var/log/suricata/eve.json"]
sincedb_path => ["/var/lib/logstash/"]
codec => json
type => "SuricataIDPS"
}

}

filter {
if [type] == "SuricataIDPS" {
date {
match => [ "timestamp", "ISO8601" ]
}
ruby {
code => "if event['event_type'] == 'fileinfo'; event['fileinfo']['type']=event['fileinfo']['magic'].to_s.split(',')[0]; end;"
}
}

if [src_ip] {
geoip {
source => "src_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
if ![geoip.ip] {
if [dest_ip] {
geoip {
source => "dest_ip"
target => "geoip"
#database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [ "[geoip][coordinates]", "float" ]
}
}
}
}
}

output {
elasticsearch {
host => localhost
#protocol => http
}
}

Also, now i ran into a new problem

FATAL { [Error: listen EADDRINUSE 0.0.0.0:5601]
cause:
{ [Error: listen EADDRINUSE 0.0.0.0:5601]
code: 'EADDRINUSE',
errno: 'EADDRINUSE',
syscall: 'listen',
address: '0.0.0.0',
port: 5601 },
isOperational: true,
code: 'EADDRINUSE',
errno: 'EADDRINUSE',
syscall: 'listen',
address: '0.0.0.0',
port: 5601 }

I checked online, it say if i try to start the process (bin/kibana) which is already running i may end up with this error. However, the fact is, that is the only process that i am trying to start.

I am totally new to this and very confused on how to move forward. Kindly let me know if you need any log files or config files. I can upload it. Any help is much appreciated.

4

Hi Magnus,

I kind a got the kibana up now, but now i have to create indices to make the logstash to see suricata logs. all the materials that i refered are confusing it was not clear, can you help me explain how to create indices (Like syntax, and where those syntax must be applied). I think that is the final piece of my puzzle. kindly help me at the earliest.