Sorry for the rambling topic, but I am falling down a rabbit hole. Once I feel like a get a handle on ELK terminology and infrastructure the floor drops.
I have never completely understood why you would ship a beat directly to ES? Is it not responsible for the storage and indexing of data for querying? I thought for parsing of the logs events you had to use logstash. Does that mean that parsing can take place at beats level now before it is shipped out and in some cases logstash is not even needed?
And pipelines? What are these and how are they different than shipping via the beat to logstash? Why would I use one over the other? I did try to pull this information from the documentation. To my knowledge I have never setup a pipeline and have tons of data shipping. Am I missing out on something?
Finally modules. I am not totally clear on what they accomplish. I thought they are predefined parsing methods for known products, but if that is the case then why are they tied to beats and not tied to logstash which I thought did all of the parsing? Andy why are they tied to pipelines?
Is there any documentation that gives a broad overview of this? I can only fine details for each in the documentation but not why or how they all act together.
Thanks for everyone's time.