Errors when Logstash is writing to InfluxDB

Elastic Stack Logstash
1

Hi, I am going to send logs from Logstash to both Elasticsearch and InfluxDB.

This is logstash.conf:

input {
  beats {
    port => 5044
    ssl => false
    ssl_certificate => "..."
    ssl_key => "..."
  }
}

filter {
}

output {
  influxdb {
    data_points => {}
    host => "influxdb"
    db => "logstash"
  }
}

The problem is that I get errors when Logstash tries to write data to InfluxDB:

[2017-05-31T20:53:41,118][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-05-31T20:53:54,133][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}
[2017-05-31T20:53:54,805][WARN ][logstash.outputs.influxdb] Non recoverable exception while writing to InfluxDB {:exception=>nil}

InfluxDB log shows:

[httpd] xxx.xx.x.xxx - - [31/May/2017:20:53:54 +0000] "POST /write?db=logstash&precision=ms&rp=autogen HTTP/1.1" 400 76 "-" "Ruby" 41f1e554-4643-11e7-8002-000000000000 2188
[httpd] xxx.xx.x.xxx - - [31/May/2017:20:53:54 +0000] "POST /write?db=logstash&precision=ms&rp=autogen HTTP/1.1" 400 6511 "-" "Ruby" 42591a26-4643-11e7-8003-000000000000 1363

Can anybody explain what could be a reason of such errors?

2

@theuntergeek @magnusbaeck @Christian_Dahlqvist
Any recommendations? Could the reason be in Logstash InfluxDB plugin and InfluxDB versions incompatibility?

3

I don't have any domain knowledge of influxdb, to be honest. The influxdb plugin was contributed by someone not on the Logstash team. While it could be some incompatibility, I would have to do a lot of digging to find out for sure.

I would raise an issue at https://github.com/logstash-plugins/logstash-output-influxdb

4

I got more info about the issue. Logstash logs say there is an error due to invalid field format.
I send Metricbeat data to Logstash and then to InfluxDB in JSON format.

Any ideas what is that format issue about?

This is error log and Metricbeat data in Logstash.

"2017-06-06T17:29:41.676000+0000", :message=>"Error writing to InfluxDB", :response=>#<Manticore::Response:0x2dea8269 
@message="Bad Request", @callback_result="{\"error\":\"unable to parse 'logstash  1496770179090': 
invalid field format\\nunable to parse 'logstash  1496770179090': invalid field format\\nunable to parse 'logstash  1496770179091': 
invalid field format\\nunable to parse 'logstash  1496770179091': invalid field format\\nunable to parse 'logstash  1496770179091': 
invalid field format\\nunable to parse 'logstash  1496770179091': invalid field format\\nunable to parse 'logstash  1496770179091': 
...

{"@timestamp":"2017-06-06T17:29:39.091Z","type":"metricsets","system":{"process":{"ppid":2,"memory":{"size":0,"rss":{"bytes":0,"pct":0.000000},"share":0},
"state":"sleeping","fd":{"open":0,"limit":{"hard":4096,"soft":1024}},"pid":809,"username":"root","cpu":{"total":{"pct":0.000000},
"start_time":"2016-02-28T21:38:32.000Z"},"pgid":0,"name":"jbd2/dm-2-8"}},"metricset":{"module":"system","name":"process","rtt":106444},
"beat":{"name":"host_name","hostname":"host_name","version":"5.3.0"},"@version":"1",
"host":"host_name","tags":["beats_input_raw_event"]}{"type":"metricsets","system":{"process":{"name":"ext4-dio-unwrit","pgid":0,
"cpu":{"total":{"pct":0.000000},"start_time":"2016-02-28T21:38:32.000Z"},"fd":{"open":0,"limit":{"soft":1024,"hard":4096}},"ppid":2,"state":"sleeping",
"memory":{"size":0,"rss":{"bytes":0,"pct":0.000000},"share":0},"username":"root","pid":810}},"metricset":{"module":"system","name":"process","rtt":106444},
"beat":{"version":"5.3.0","name":"host_name","hostname":"host_name"},"@timestamp":"2017-06-06T17:29:39.091Z",
"@version":"1","host":"host_name","tags":["beats_input_raw_event"]}{"@timestamp":"2017-06-06T17:29:39.091Z","type":"metricsets",
"system":{"process":{"username":"root","memory":{"share":0,"size":0,"rss":{"bytes":0,"pct":0.000000}},"cpu":{"total":{"pct":0.000000},
"start_time":"2016-02-28T21:38:32.000Z"},"pid":849,"ppid":2,"fd":{"open":0,"limit":{"hard":4096,"soft":1024}},"pgid":0,"name":"kauditd","state":"sleeping"}},
"metricset":{"rtt":106444,"module":"system","name":"process"},"beat":{"name":"host_name","hostname":"host_name","version":"5.3.0"},
"@version":"1","host":"host_name","tags":["beats_input_raw_event"]}{"beat":{"version":"5.3.0","name":"host_name",
"hostname":"host_name"},"@timestamp":"2017-06-06T17:29:39.091Z","type":"metricsets","system":{"process":{"state":"sleeping","memory":
...
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.