Hello,
I wasn't sure if this is more an Elasticsearch or Logstash issue, hope this is the right place. As you can probably guess I am rather new to Elasticsearch.
This is Elasticsearch 6.8 on CentOS 7.7.
I have a CSV that I'm parsing in logstash using the file input plugin, the csv filter, and elasticsearch output. It works out okay, I'm seeing the parsed values in Kibana. But the problem is, they all have the text data type when in reality, it's mostly numbers. This means I cannot create any meaningful visualizations. It was my understanding that through auto-mapping, Logstash (or Elasticsearch?) would recognize that a number is a number, but in my case all fields are text.
My data file looks like this:
datetime,timestamp,processname,pid,cpu_time,mem_vms,mem_rss
2020-04-01 00:00:00.473727,1585699200.47,housekeeper,24835,20,8830976,53592064
(and obviously many more lines)
You see, there is a date (which I can parse using Logstash's date filter), and one text field (processname) - the rest are numbers.
How can I tell Logstash (or Elasticsearch?) that these are numbers?
Bonus question: there are other CSV files with lots and lots of columns, whose name or order may change over time. How can I handle this as elegantly as possible? (I was really hoping the auto-mapping would work...)
Relevant parts of my logstash config (can post in full if necessary):
input {
file {
path => ["/tmp/*mpmon*"]
mode => "read"
file_completed_action => "log"
file_completed_log_path => "/dev/null"
sincedb_path => "/dev/null"
start_position => "beginning"
}
}
filter {
csv {
autodetect_column_names => true
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "procmon-%{+YYYY.MM.dd}"
}
}