We are trying to load data from oracle database to elasticsearch using logstash. We have created a config file with input section having the driver info, connection to db etc. The data is extracted using select query and the output points to the data nodes of the elastic cluster.
After doing the config updates, we are now trying to get maximum data loading speed. But the max we could get was 25 GB per hour.
Do you have any advise on what to be tuned, where to look at for finding the bottleneck?
We are using elastic and logstash 7.1.1.
Logstash workers = 12 (16 cpus) and pipeline batch size is 64000. JVM is 64G on 128 G server.
Elastic has 7 data nodes. Each having jvm 32G off 64 G machine.
Thank You in Advance