There is a confusion regarding the usage of pipeline.batch.size parameter in Logstash.
Can anyone please clarify?
From the documentations, got to understand that pipeline.batch.size is used to control the number of events passed from input to filters and output section of Logstash.
If the Logstash output is sent to AWS Opensearch, will the setting in pipeline.batch.size have an impact on the _bulk request? Or will it consider the default bulk limit of the AWS instance type?
Because of less queue size in Filebeat and less batch size in Logstash, some of the loglines are not processed before the file gets deleted.
(FileBeat slow - Improve performance)
Increased queue size and batch size based on this to handle the missing logs
I would like to get a common solution to handle both of the above issues.
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.
(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )
If you are talking about the opensearch output then yes, it does affect the size of the _bulk request because a pipeline batch is sent as a single _bulk request. See here and follow the code down...
An elasticsearch output works very differently. It has a maximum size for a bulk request and will break up a pipeline batch into multiple requests if necessary. It also no longer endlessly retries a request that fails with a 413 error. You would need the opensearch folks to add similar functionality.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
)