%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}

I'm trying to follow Configure the Logstash output | Metricbeat Reference (Accessing metadata fields)

# curl --silent --request GET $ELASTICSEARCH_URI/_cat/indices | grep metadata
green open %{[@metadata][beat]}-%{[@metadata][version]}-2018.04.10 V3M9LavYTSa_7zcKEOh1yQ 5 1          6         0    272kb  139.3kb
# 

elasticsearch's log:

esm1 | [2018-04-10T00:00:20,446][INFO ][o.e.c.m.MetaDataMappingService] [esm1] [%{[@metadata][beat]}-%{[@metadata][version]}-2018.04.10/1OIr7aShTB-9eBIhJYncKA] update_mapping [doc]

logstash's (input/output):

# cat 10-input-beats.conf 
input {
	beats {
		port => 5044
	}
}
# grep -v password 30-output-elasticsearch.conf
output {
	if [container_id] {
		elasticsearch {
			hosts => "elasticsearch:9200"
			user => "elastic"
		}
	} else {
		elasticsearch {
			hosts => "elasticsearch:9200"
			user => "elastic"
			manage_template => false
			index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
			#document_type => "%{[@metadata][type]}"
		}
	}
}
#

Beats input plugin | Logstash Reference

Logstash will create this index name if the fields are not present in the event.

  1. Which versions of Beats and Logstash have you installed?

  2. There might be some filter in your configs removing/changing some of the events metadata.

  1. 6.2.3
  2. default configuration

Beats always send @metadata to Logstash. Grep all your logstash configs/directories for:

  • input: another input might receive inputs without @metadata
  • metadata: a filter might modify @metadata

following is my entire logstash configuration (minus my own comments and my password):

# grep -Ev '^#|password' *.conf
10-input-beats.conf:
10-input-beats.conf:input {
10-input-beats.conf:	beats {
10-input-beats.conf:		port => 5044
10-input-beats.conf:	}
10-input-beats.conf:}
10-input-dead_letter_queue.conf:input {
10-input-dead_letter_queue.conf:	dead_letter_queue {
10-input-dead_letter_queue.conf:		path => "/usr/share/logstash/data/dead_letter_queue"
10-input-dead_letter_queue.conf:	}
10-input-dead_letter_queue.conf:}
10-input-gelf.conf:
10-input-gelf.conf:input {
10-input-gelf.conf:	gelf {
10-input-gelf.conf:		codec => "json"
10-input-gelf.conf:	}
10-input-gelf.conf:}
20-filter.conf:
20-filter.conf:filter {
20-filter.conf:	json {
20-filter.conf:		skip_on_invalid_json => true
20-filter.conf:		source => "message"
20-filter.conf:	}
20-filter.conf:}
20-filter.conf:
30-output-elasticsearch.conf:
30-output-elasticsearch.conf:output {
30-output-elasticsearch.conf:	if [container_id] {
30-output-elasticsearch.conf:		elasticsearch {
30-output-elasticsearch.conf:			hosts => "elasticsearch:9200"
30-output-elasticsearch.conf:			user => "elastic"
30-output-elasticsearch.conf:		}
30-output-elasticsearch.conf:	} else {
30-output-elasticsearch.conf:		elasticsearch {
30-output-elasticsearch.conf:			hosts => "elasticsearch:9200"
30-output-elasticsearch.conf:			user => "elastic"
30-output-elasticsearch.conf:			manage_template => false
30-output-elasticsearch.conf:			index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
30-output-elasticsearch.conf:			#document_type => "%{[@metadata][type]}"
30-output-elasticsearch.conf:		}
30-output-elasticsearch.conf:	}
30-output-elasticsearch.conf:}
#

You have multiple inputs, but no filtering on conditions for actual source in your outputs. The weird index might have been created due to gelf or events comming from the dead letter queue.

How many outputs do you have?

You can protect the Elasticsearch output by adding a condition, checking for the presence of [@metadata][beat].

sorry, I'm bit confused... you're saying to add a condition checking to output or to filter?

I mean, we can clearly see that events are ending up in that weirdly named index already..

The output might process fields from the gelf input for example. You should protect the output to process events from beats only. The filter looks like a noop for gelf based events.

Have you had a look at Logstash pipelines? A pipeline consists of inputs, filters and outputs. Using multiple pipelines, you can more easily separate data-flows, without having to use conditionals.

all events from gelf input has container_id key, which I check for in output, therefor events are ending up in right index.

events that coming in through beats input on another hand, looks like are ending up in right index as well, as I see them in %{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd} index, unless I'm wrong, that if statement in output works as expected. The issue here is @metadata is missing, now I can add yet another condition to check [@metadata][beat], however in best case scenario event would end up in same index or due to @metadata is missing after all would not even end up there anymore...

I'm reading and trying to understand Multiple Pipelines...

can you run a separate logstash config (don't use any of your current configurations) with this logstash config only?

input {
  beats {
    port => 5044
  }
}

output {
  stdout {
    codec => rubydebug {
      metadata => true
    }
  }
}

This should print you the raw events (without any filters), from beats only. Including the @metadata fields.

The docs are wrong. What you want to use is %{[beat][name]} and %{[beat][version]}. So it would be...

index => "%{[beat][name]}-%{[beat][version]}-%{+YYYY.MM.dd}"

@rcowart thanks again for your reply, however this configuration isn't exactly what I'm looking for.

in my case (per your configuration), indices that would get created: web0-6.2.3-2018.04.10, that would break visualization and/or dashboard in Kibana as beats pattern are: filebeat-* or metricbeat-*.

I re-configured my Logstash per Multiple Pipelines | Logstash Reference [6.2] | Elastic.

I adjusted output instead of elasticsearch to stdout, following is an example of one of the events:

logstash11  | {
logstash11  |     "@timestamp" => 2018-04-13T16:12:41.501Z,
logstash11  |         "offset" => 742656,
logstash11  |      "@metadata" => {
logstash11  |               "type" => "doc",
logstash11  |               "beat" => "filebeat",
logstash11  |           "pipeline" => "filebeat-6.2.3-apache2-access-default",
logstash11  |         "ip_address" => "10.142.0.8",
logstash11  |            "version" => "6.2.3"
logstash11  |     },
logstash11  |           "host" => "web0",
logstash11  |           "beat" => {
logstash11  |         "hostname" => "web0",
logstash11  |             "name" => "web0",
logstash11  |          "version" => "6.2.3"
logstash11  |     },
logstash11  |       "@version" => "1",
logstash11  |     "prospector" => {
logstash11  |         "type" => "log"
logstash11  |     },
logstash11  |         "source" => "/var/log/apache2/access.log",
logstash11  |        "message" => "127.0.0.1 - - [13/Apr/2018:16:12:40 +0000] \"GET /server-status?auto= HTTP/1.1\" 200 781 \"-\" \"Go-http-client/1.1\"",
logstash11  |        "fileset" => {
logstash11  |         "module" => "apache2",
logstash11  |           "name" => "access"
logstash11  |     },
logstash11  |           "tags" => [
logstash11  |         [0] "beats_input_codec_plain_applied"
logstash11  |     ]
logstash11  | }

You are right! I forgot that I set name to be the name of the beat in yml config file. Looking at your data, you have the option of using %{[beat][version]} or %{[@metadata"][version]}, and you would use %{[@metadata][beat]} for the beat name.

The event looks correct. You can use any field for creating the index name. You can even us a filter plugin to create a custom index name and add it to @metadata. The @metadata field is not index in Elasticsearch.

Does the problem still persist?

@steffens, even though I still not really sure what cause this not to work the first time, after configure logstash with multiple pipelines, that seems to address issue. I'm not seeing anything in Dashboard (yet), but it looks like I'm missing more Logstash configuration to parse logs correctly.

I was reading about multi-pipline and then you mentioned in one of your comment as well, so I marked that comment as "Solution".

Thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.