How to speed up indexing of csv file via logstash

Elastic Stack Logstash
1

Hello,
ES 7.3 cluster on k8s, 6 data nodes
csv file: 4500000 rows, 90 fields
Indexing takes an hour and a half approximately.
</> logstash conf file:
input { stdin {
type => "stdin-type"
}
}
filter {
csv {
separator => ","
skip_header => "true"
columns => [ ..... ]
}
date {
match => [ "my_date", "YYYYMMddHHmmss" ]
target => "my_date"
}
mutate {
convert => {
"Speed_Min" => "integer"
"Speed_Max" => "integer"
"Speed_Avg" => "integer"
}
}
}
output {
elasticsearch {
hosts => [ "elasticsearch:9200"]
index => "index-%{+YYYY-MM-dd}"
}
}
</>
What can be done to improve indexing time?
Thanks in advance

2

To find that out you need to identify where the bottleneck is. It could be logstash, it could be elasticsearch. It could be CPU, it could (in elasticsearch) be disk throughput. It could be several other things. You need to monitor the components of your system and see which one is limiting throughput. Nobody can predict that without visibility into your servers.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.