Using filebeat 6.1.1 to gather apache logs.
It all started when I tried to change the index name for filebeat, and this seems to be a damn complicated thing despite seemingly an easy task in my mind. BTW I eventually reached the point where the ES index was created with the name I wanted, but when I started filebeat and data went into the index, each record only contained basic data (@timestamp, _id and a couple other fields) but nothing about my logs. Commenting out the lines to change index name and setup.template.* reverts to the standard index name, filebeat-6.1.1, and the data is complete so the input pipeline is fine
I thought it was some problem with the template, but found no documentation to explain how to work with it, just "set it up" type docs. A very basic howto would be very handy, I'll write one once I'll be fine with the setup.
So I switched to using logstash since I had to modify the logs anyhow. I setup my listener and the output was set to both ES with an index name of my choice, and the ruby colored output.
Starting filebeat I can see the console output containing all my fields coming from filebeat, but the data in ES only has the same basic fields as in my first tentative.
After two days of googling I suppose it is not something obvious as I thought. Any suggestion on:
- where to find a comprehensive docs that says how to change the index name and setup the templates accordingly, something finer than "you have to setup templates".
- why my data that seems ok in the console output does not go into ES in a complete way with all the fields?