I am using ELK stack. I have to load Oracle data to ElasticSearch. There are around 5 million data to load at initial.
Problem: The logstash is very very slow to load data, it loads around 50k in 30 mins. this speed is not at all acceptable. Prior to ELK, I had setup on Solr for same set of full data used to be loaded in 1 hr. I am using same query in both places. Also the machines are same, so cpu/memory are all same.
I tried to tune the logstash by params like worker:4, pipeline:default,jdbc-paging-enabled=true. when I enable jdbc-fetch-size= 50,000 then its slower than default. jdbc_page_size= 200000 (while default is 100000 which is good enough).
Please suggest what else can I try ? Do we have any other plugin ?
ELK version: 7.3.1