Hello,
Currently our logstash consists of 7 pipelines that launch queries against a DB accessed by JDBC. With logstash we enrich and normalize this data and index it directly to Elasticsearch. In total 20 jdbc queries are made with a schedule every 5minutes.
Some of these queries collect up to 120,000 documents/minute and logstash is not able to process it and takes more than the 5 minutes it should, so we are chaining delays at the time of indexing.
From our local VDI we have been testing and obtaining the expected results, a timely indexing, and it has the following characteristics:
- 4 CPUs
- 16GB of memory
The integration container, where logstash has to run, consists of:
- 1 CPU
- 2GB of memory
Our intention was to upgrade the integration container to 4CPUs, to match the VDI environment, but in principle this option is not valued by the customer, but we think it is necessary due to the number of traces we work with.
Is there any alternative to increase the number of CPUs to have more workers?
Is it possible to configure different threads for different pipelines?
Is there any way to monitor the performance of the queries?