Hi there!
I am trying to get vmstat output (in CSV format) and load the data into influxdb using logstash and logstash-output-plugin, but am getting error "invalid field format" when writing data to influxdb. See details below.
Any assistance would be much appreciated.
Thanks
OS: Windows 7 Enterprise
InfluxDB: 1.5.2-1
logstash: 6.2.4
logstash-output-influxdb plugin
CSV file:
2018-06-14 12:09:30.123,13,0,0
2018-06-14 12:09:40.123,27,0,0
2018-06-14 12:09:50.123,98,0,0
logstash config file:
input {
stdin {}
file {
path => "C:/Temp/*.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["vmtimestamp","procsR","procsB","procsW"]
convert => {
"procsR" => "integer"
"procsB" => "integer"
"procsW" => "integer"
}
}
}
output {
influxdb {
id => "influxdb1"
host => "localhost"
db => "monitoring"
allow_time_override => true
exclude_fields => ["@timestamp", "@version", "sequence", "message", "type", "path"]
codec => json
}
stdout { codec => json}
}
logstash stdout:
The stdin plugin is now waiting for input:
[2018-06-14T14:55:10,886][INFO ][logstash.agent ] Pipelines running {:
count=>1, :pipelines=>["main"]}
[2018-06-14T14:55:11,870][WARN ][logstash.outputs.influxdb] Non recoverable exce
ption while writing to InfluxDB {:exception=>#<InfluxDB::Error: {"error":"unable
to parse 'logstash 1528952111126': invalid field format"}
}
{"procsR":13,"procsW":0,"@timestamp":"2018-06-14T04:55:11.126Z","@version":"1","
path":"C:/Temp/oncillat_vmstat_201806141454.csv","message":"2018-06-14 12:09:30.
123,13,0,0","procsB":0,"vmtimestamp":"2018-06-14 12:09:30.123","host":"NIGSA9370
04"}{"procsR":98,"procsW":0,"@timestamp":"2018-06-14T04:55:11.151Z","@version":"
1","path":"C:/Temp/oncillat_vmstat_201806141454.csv","message":"2018-06-14 12:09
:50.123,98,0,0","procsB":0,"vmtimestamp":"2018-06-14 12:09:50.123","host":"NIGSA
937004"}{"procsR":27,"procsW":0,"@timestamp":"2018-06-14T04:55:11.150Z","@versio
n":"1","path":"C:/Temp/oncillat_vmstat_201806141454.csv","message":"2018-06-14 1
2:09:40.123,27,0,0","procsB":0,"vmtimestamp":"2018-06-14 12:09:40.123","host":"N
IGSA937004"}[2018-06-14T14:55:12,093][WARN ][logstash.outputs.influxdb] Non reco
verable exception while writing to InfluxDB {:exception=>#<InfluxDB::Error: {"er
ror":"unable to parse 'logstash 1528952111150': invalid field format\nunable to
parse 'logstash 1528952111151': invalid field format"}
}
influxdb log:
[httpd] 127.0.0.1 - - [14/Jun/2018:14:55:11 +1000] "POST /write?db=monitoring&pr
ecision=ms&rp=autogen HTTP/1.1" 400 76 "-" "Ruby" 1e7cd7d8-6f8f-11e8-8029-000000
000000 2000
[httpd] 127.0.0.1 - - [14/Jun/2018:14:55:12 +1000] "POST /write?db=monitoring&pr
ecision=ms&rp=autogen HTTP/1.1" 400 141 "-" "Ruby" 1eb32a18-6f8f-11e8-802a-00000
0000000 1000