You have multiple inputs, but no filtering on conditions for actual source in your outputs. The weird index might have been created due to gelf or events comming from the dead letter queue.
How many outputs do you have?
You can protect the Elasticsearch output by adding a condition, checking for the presence of [@metadata][beat].
The output might process fields from the gelf input for example. You should protect the output to process events from beats only. The filter looks like a noop for gelf based events.
Have you had a look at Logstash pipelines? A pipeline consists of inputs, filters and outputs. Using multiple pipelines, you can more easily separate data-flows, without having to use conditionals.
all events from gelf input has container_id key, which I check for in output, therefor events are ending up in right index.
events that coming in through beats input on another hand, looks like are ending up in right index as well, as I see them in %{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd} index, unless I'm wrong, that if statement in output works as expected. The issue here is @metadata is missing, now I can add yet another condition to check [@metadata][beat], however in best case scenario event would end up in same index or due to @metadatais missing after all would not even end up there anymore...
I'm reading and trying to understand Multiple Pipelines...
@rcowart thanks again for your reply, however this configuration isn't exactly what I'm looking for.
in my case (per your configuration), indices that would get created: web0-6.2.3-2018.04.10, that would break visualization and/or dashboard in Kibana as beats pattern are: filebeat-* or metricbeat-*.
You are right! I forgot that I set name to be the name of the beat in yml config file. Looking at your data, you have the option of using %{[beat][version]} or %{[@metadata"][version]}, and you would use %{[@metadata][beat]} for the beat name.
The event looks correct. You can use any field for creating the index name. You can even us a filter plugin to create a custom index name and add it to @metadata. The @metadata field is not index in Elasticsearch.
@steffens, even though I still not really sure what cause this not to work the first time, after configure logstash with multiple pipelines, that seems to address issue. I'm not seeing anything in Dashboard (yet), but it looks like I'm missing more Logstash configuration to parse logs correctly.
I was reading about multi-pipline and then you mentioned in one of your comment as well, so I marked that comment as "Solution".
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.