After insert document_id at logstash, elastic stops receiveing data

Elastic Stack Elasticsearch
1

HI guys

Using debian + logstash+kibana+logstash version 5.6.0

If I insert document_id at output elastic , I do not receive any new input at elasticsearch.
Part of logstash file:

input {
file {
path => "/var/log/firewall/firewall.log"
type => "firewall"
}
}
....
output {

elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-firewall-%{+YYYY.MM.dd}"
document_id => "%{type}"
}
}
and letting logs at debug in logstash.yml I can see logs whithout error... What is wrong?

I have already try to clean all indices and restart elasticsearch but no difference.....8(

If I remove document_id from logstash and restart it all goes right...

part og logstash logs:

[2017-09-20T11:18:10,674][DEBUG][logstash.instrument.periodicpoller.cgroup] Error, cannot retrieve cgroups i
nformation {:exception=>"Errno::ENOENT", :message=>"No such file or directory - sys/fs/cgroup/cpuacct/cpu.cfs_period_us"}
[2017-09-20T11:18:11,094][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-09-20T11:18:14,037][DEBUG][logstash.inputs.file ] each: file grew: /var/log/firewall/firewall.log:
old size 1618301, new size 1618595
[2017-09-20T11:18:14,038][DEBUG][logstash.inputs.file ] Received line {:path=>"/var/log/firewall/firewal
l.log", :text=>"Sep 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:51:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 "}
[2017-09-20T11:18:14,048][DEBUG][logstash.pipeline ] filter received {"event"=>{"@version"=>"1", "hos
t"=>"zeus", "path"=>"/var/log/firewall/firewall.log", "@timestamp"=>2017-09-20T14:18:14.038Z, "message"=>"Se
p 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:5
1:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 ", "type"=>"firewall"}}
[2017-09-20T11:18:14,048][DEBUG][logstash.filters.grok ] Running grok filter {:event=>2017-09-20T14:18:14
.038Z zeus Sep 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:51:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 }
[2017-09-20T11:18:14,049][DEBUG][logstash.filters.grok ] Event now: {:event=>2017-09-20T14:18:14.038Z ze
us Sep 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:51:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 }

Any ideas?

2

This means that all documents will get exactly the same ID, resulting in the same document being overwritten repeatedly. This will be very slow, but also looks incorrect to me - what is it you are looking to achieve with this setting?

3

Hi Christian

The main reason is that comparing to metricbeat, I supose that is the way I can see at kibana an easy url.

I would like to see a dasboard like below:
http://xxx:5601/app/kibana#/dashboard/firewall?_g=()

Is that correct ? or I am using a wrong way?

4

Leave out the document_id parameter and Elasticsearch will generate an id for each document.

5

ok thanks Christian .

Do you know how can I construct a dashboard with name instead indice? or should I ask at kibana forum?

Something like http://xxx:5601/app/kibana#/dashboard/firewall?_g=()

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.