In Logstash under the pipeline directory there will be multiple .conf files that set specific grok filters. The format of the files are as follows:
Input
Grok Filter
Output [list of nodes]
For that Output section do you need to put every node that is listed in your Cluster? Or instead, would you only list the dedicated Ingest nodes as the Output?
If you have dedicated ingest nodes, then list them.
Awesome, thank you! If those dedicated ingest nodes are listed on the Output on logstash. Would I then need to make any additional changes for the data to go from the ingest nodes to the data nodes? Or would elastic take care of that on its own?
Elasticsearch will take care of that.
Thanks, that's a huge help. I really appreciate it. Only quick follow up. In the environment, when we do the API command "GET /_ingest/pipeline" we see grok filters are running as pipeline processors on the Ingest nodes. If we have the complete ELK stack would a more default/generic solution be to just let these run on Logstash instead? Ive been researching and it shows that usually its either Logstash or Ingest and not very often do people have both of them combined. Is that correct?
There's flexibility to do both, so it depends on what you want to do and what in the stack can do it for you.
Just pick whatever is easiest 
OK cool, options are always a good thing. This was a super fast and helpful reply. Definitely going to start another thread about something else lol. Thanks again.
Sorry, a question popped up at work. What would happen if you list a node in that Logstash output section, but it is NOT an ingest node. Any loss of data? Would some data not get run though the pipeline processors? Etc (we have deployments that come with defaults so just want to cover our basis if anything could go wrong if that output node listing is wrong).
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.