After insert document_id at logstash, elastic stops receiveing data

HI guys

Using debian + logstash+kibana+logstash version 5.6.0

If I insert document_id at output elastic , I do not receive any new input at elasticsearch.
Part of logstash file:

input {
file {
path => "/var/log/firewall/firewall.log"
type => "firewall"
}
}
....
output {

elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-firewall-%{+YYYY.MM.dd}"
document_id => "%{type}"
}
}
and letting logs at debug in logstash.yml I can see logs whithout error... What is wrong?

I have already try to clean all indices and restart elasticsearch but no difference.....8(

If I remove document_id from logstash and restart it all goes right...

part og logstash logs:

[2017-09-20T11:18:10,674][DEBUG][logstash.instrument.periodicpoller.cgroup] Error, cannot retrieve cgroups i
nformation {:exception=>"Errno::ENOENT", :message=>"No such file or directory - sys/fs/cgroup/cpuacct/cpu.cfs_period_us"}
[2017-09-20T11:18:11,094][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2017-09-20T11:18:14,037][DEBUG][logstash.inputs.file ] each: file grew: /var/log/firewall/firewall.log:
old size 1618301, new size 1618595
[2017-09-20T11:18:14,038][DEBUG][logstash.inputs.file ] Received line {:path=>"/var/log/firewall/firewal
l.log", :text=>"Sep 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:51:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 "}
[2017-09-20T11:18:14,048][DEBUG][logstash.pipeline ] filter received {"event"=>{"@version"=>"1", "hos
t"=>"zeus", "path"=>"/var/log/firewall/firewall.log", "@timestamp"=>2017-09-20T14:18:14.038Z, "message"=>"Se
p 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:5
1:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 ", "type"=>"firewall"}}
[2017-09-20T11:18:14,048][DEBUG][logstash.filters.grok ] Running grok filter {:event=>2017-09-20T14:18:14
.038Z zeus Sep 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:51:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 }
[2017-09-20T11:18:14,049][DEBUG][logstash.filters.grok ] Event now: {:event=>2017-09-20T14:18:14.038Z ze
us Sep 20 11:18:13 zeus kernel: [72287.762928] fir:block_input_tcp_wan2 IN=wan2 OUT= PHYSIN=enp6s1 MAC=00:e0:4c:51:01:66:00:01:5c:80:60:46:08:00 SRC=78.120.52.31 DST=201.6.110.223 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39664 DF PROTO=TCP SPT=42003 DPT=51413 WINDOW=7300 RES=0x00 SYN URGP=0 MARK=0x2 }

Any ideas?

This means that all documents will get exactly the same ID, resulting in the same document being overwritten repeatedly. This will be very slow, but also looks incorrect to me - what is it you are looking to achieve with this setting?

Hi Christian

The main reason is that comparing to metricbeat, I supose that is the way I can see at kibana an easy url.

I would like to see a dasboard like below:
http://xxx:5601/app/kibana#/dashboard/firewall?_g=()

Is that correct ? or I am using a wrong way?

Leave out the document_id parameter and Elasticsearch will generate an id for each document.

ok thanks Christian .

Do you know how can I construct a dashboard with name instead indice? or should I ask at kibana forum?

Something like http://xxx:5601/app/kibana#/dashboard/firewall?_g=()

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.