How can I make same logstash instance run on 2 nodes. For example I have a normal configuration input a csv file, csv filter and output to elastic search. Can I make the 2 nodes process the same file for faster indexing ?
Not with putting the data in some sort of a queue.
have read here, its about scaling.
I read it before and well im still lost can you provide an example
How have you come to the conclusion that Logstash is the bottleneck? If your configuration is as simple as you said I think that sounds unlikely.
You have 1 file which you want to process with 2 logstash servers and you want each line only processed once. You can split the file in 2 and give each server there own part. while this could work for 1 or 2 times it is not very efficient.
What you can also do is setup for example an redis server, you push your data in there line by line and have both logstashes read from that queue.
Paul.
I changed the number of shards into 2 after making the first node master into a data node after that the performance is now slower than before by half. and logstash is taking minimum 90% cpu
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.