Hello,
I'm having issues when feeding CSV data into elastic.
My logstash .conf file monitors a results folder and injects new .csv data into elastic with the file input plugin.
The issue is that the .csv file contains different columns from time to time, depending on the test being performed.
The output header can be any combination depending on the actual results returned by the URL.
Only the first 15 lines are the same.
standard:
avg_ct,avg_lt,avg_rt,concurrency,fail,stdev_rt,succ,throughput,perc_95.0,perc_0.0,perc_99.9,perc_90.0,perc_100.0,perc_99.0,perc_50.0,label
variation
avg_ct,avg_lt,avg_rt,concurrency,fail,stdev_rt,succ,throughput,perc_95.0,perc_0.0,perc_99.9,perc_90.0,perc_100.0,perc_99.0,perc_50.0,rc_200,label
SSL variation
avg_ct,avg_lt,avg_rt,concurrency,fail,stdev_rt,succ,throughput,perc_95.0,perc_0.0,perc_99.9,perc_90.0,perc_100.0,perc_99.0,perc_50.0,rc_200,rc_SSLHandshakeException,rc_503,label
SSL variation 2
avg_ct,avg_lt,avg_rt,concurrency,fail,stdev_rt,succ,throughput,perc_95.0,perc_0.0,perc_99.9,perc_90.0,perc_100.0,perc_99.0,perc_50.0,rc_200,rc_SSLHandshakeException,rc_503,rc_ConnectException,rc_SocketException,label
My .conf file is below:
input {
file {
path => "/var/lib/pbench-agent/user-benchmark_ose3_test_*/1/taurus.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ','
columns => ["avg_ct","avg_lt","avg_rt","concurrency",
"fail","stdev_rt","succ","throughput",
"perc_95.0","perc_0.0","perc_99.9","perc_90.0",
"perc_100.0","perc_99.0","perc_50.0","rc_200",
"rc_503","rc_SSLHandshakeException","rc_ConnectException","rc_SocketException",
"label"]
}
de_dot {
fields => ["perc_95.0","perc_0.0","perc_99.9","perc_90.0",
"perc_100.0","perc_99.0","perc_50.0"]
}
output {
elasticsearch {
hosts => "gprfc076:9200"
manage_template => false
index => "bzt_pbench-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
}
How could I create different types and send them to elastic according to the csv header line?
Wouldn't this mean changing the doc_type/mapping/.conf file (columns) and eventually restart logstash?
Any suggestions on how to handle this would be greatly appreciated!
Thank you,
Ricardo