# %{\[@metadata\]\[beat\]}-%{\[@metadata\]\[version\]}-%{+YYYY.MM.dd}
**URL:** https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360
**Category:** Beats
**Created:** [April 9, 2018, 5:05pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360 "2018-04-09T17:05:40Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 9, 2018, 5:05pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/1 "2018-04-09T17:05:40Z")
I'm trying to follow [Configure the Logstash output | Metricbeat Reference (Accessing metadata fields)](https://www.elastic.co/guide/en/beats/metricbeat/current/logstash-output.html#_accessing_metadata_fields)
```
# curl --silent --request GET $ELASTICSEARCH_URI/_cat/indices | grep metadata
green open %{[@metadata][beat]}-%{[@metadata][version]}-2018.04.10 V3M9LavYTSa_7zcKEOh1yQ 5 1 6 0 272kb 139.3kb
#
```
elasticsearch's log:
> esm1 | [2018-04-10T00:00:20,446][INFO][o.e.c.m.MetaDataMappingService] [esm1] [%{[@metadata][beat]}-%{[@metadata][version]}-2018.04.10/1OIr7aShTB-9eBIhJYncKA] update\_mapping [doc]
logstash's (input/output):
```
# cat 10-input-beats.conf
input {
beats {
port => 5044
}
}
# grep -v password 30-output-elasticsearch.conf
output {
if [container_id] {
elasticsearch {
hosts => "elasticsearch:9200"
user => "elastic"
}
} else {
elasticsearch {
hosts => "elasticsearch:9200"
user => "elastic"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#document_type => "%{[@metadata][type]}"
}
}
}
#
```
[Beats input plugin | Logstash Reference](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#_description)
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 10, 2018, 11:14am UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/2 "2018-04-10T11:14:58Z")
Logstash will create this index name if the fields are not present in the event.
1. Which versions of Beats and Logstash have you installed?
2. There might be some filter in your configs removing/changing some of the events metadata.
---
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 10, 2018, 2:16pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/3 "2018-04-10T14:16:46Z")
1. 6.2.3
2. default configuration
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 10, 2018, 5:15pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/4 "2018-04-10T17:15:45Z")
Beats always send `@metadata` to Logstash. Grep all your logstash configs/directories for:
- `input`: another input might receive inputs without `@metadata`
- `metadata`: a filter might modify `@metadata`
---
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 10, 2018, 9:00pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/5 "2018-04-10T21:00:06Z")
following is my entire logstash configuration (minus my own comments and my password):
```
# grep -Ev '^#|password' *.conf
10-input-beats.conf:
10-input-beats.conf:input {
10-input-beats.conf: beats {
10-input-beats.conf: port => 5044
10-input-beats.conf: }
10-input-beats.conf:}
10-input-dead_letter_queue.conf:input {
10-input-dead_letter_queue.conf: dead_letter_queue {
10-input-dead_letter_queue.conf: path => "/usr/share/logstash/data/dead_letter_queue"
10-input-dead_letter_queue.conf: }
10-input-dead_letter_queue.conf:}
10-input-gelf.conf:
10-input-gelf.conf:input {
10-input-gelf.conf: gelf {
10-input-gelf.conf: codec => "json"
10-input-gelf.conf: }
10-input-gelf.conf:}
20-filter.conf:
20-filter.conf:filter {
20-filter.conf: json {
20-filter.conf: skip_on_invalid_json => true
20-filter.conf: source => "message"
20-filter.conf: }
20-filter.conf:}
20-filter.conf:
30-output-elasticsearch.conf:
30-output-elasticsearch.conf:output {
30-output-elasticsearch.conf: if [container_id] {
30-output-elasticsearch.conf: elasticsearch {
30-output-elasticsearch.conf: hosts => "elasticsearch:9200"
30-output-elasticsearch.conf: user => "elastic"
30-output-elasticsearch.conf: }
30-output-elasticsearch.conf: } else {
30-output-elasticsearch.conf: elasticsearch {
30-output-elasticsearch.conf: hosts => "elasticsearch:9200"
30-output-elasticsearch.conf: user => "elastic"
30-output-elasticsearch.conf: manage_template => false
30-output-elasticsearch.conf: index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
30-output-elasticsearch.conf: #document_type => "%{[@metadata][type]}"
30-output-elasticsearch.conf: }
30-output-elasticsearch.conf: }
30-output-elasticsearch.conf:}
#
```
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 11, 2018, 11:22am UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/6 "2018-04-11T11:22:16Z")
You have multiple inputs, but no filtering on conditions for actual source in your outputs. The weird index might have been created due to gelf or events comming from the dead letter queue.
How many outputs do you have?
You can protect the Elasticsearch output by adding a condition, checking for the presence of `[@metadata][beat]`.
---
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 11, 2018, 9:31pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/7 "2018-04-11T21:31:05Z")
sorry, I'm bit confused... you're saying to add a condition checking to `output` or to `filter`?
I mean, we can clearly see that events are ending up in that weirdly named index already..
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 12, 2018, 10:07am UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/8 "2018-04-12T10:07:44Z")
The output might process fields from the gelf input for example. You should protect the output to process events from beats only. The filter looks like a noop for gelf based events.
Have you had a look at [Logstash pipelines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html#multiple-pipelines)? A pipeline consists of inputs, filters and outputs. Using multiple pipelines, you can more easily separate data-flows, without having to use conditionals.
---
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 12, 2018, 3:00pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/9 "2018-04-12T15:00:11Z")
all events from `gelf` input has `container_id` key, which I check for in output, therefor events are ending up in right index.
events that coming in through beats input on another hand, looks like are ending up in right index as well, as I see them in `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` index, unless I'm wrong, that if statement in output works as expected. The issue here is `@metadata` is missing, now I can add yet another condition to check `[@metadata][beat]`, however in best case scenario event would end up in same index or due to @metadata _is_ missing after all would not even end up there anymore...
I'm reading and trying to understand Multiple Pipelines...
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 13, 2018, 11:51am UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/10 "2018-04-13T11:51:12Z")
can you run a separate logstash config (don't use any of your current configurations) with this logstash config only?
```auto
input {
beats {
port => 5044
}
}
output {
stdout {
codec => rubydebug {
metadata => true
}
}
}
```
This should print you the raw events (without any filters), from beats only. Including the `@metadata` fields.
---
### Author:  [@rcowart](https://discuss.elastic.co/u/rcowart)
#### Post date: [April 13, 2018, 1:58pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/11 "2018-04-13T13:58:15Z")
The docs are wrong. What you want to use is `%{[beat][name]}` and `%{[beat][version]}`. So it would be...
```auto
index => "%{[beat][name]}-%{[beat][version]}-%{+YYYY.MM.dd}"
```
---
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 13, 2018, 4:07pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/12 "2018-04-13T16:07:00Z")
@rcowart thanks _again_ for your reply, however this configuration isn't exactly what I'm looking for.
in my case (per your configuration), indices that would get created: `web0-6.2.3-2018.04.10`, that would break visualization and/or dashboard in Kibana as beats pattern are: `filebeat-*` or `metricbeat-*`.
---
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 13, 2018, 4:14pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/13 "2018-04-13T16:14:31Z")
I re-configured my Logstash per [Multiple Pipelines | Logstash Reference [6.2] | Elastic](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html#multiple-pipelines).
I adjusted output instead of `elasticsearch` to `stdout`, following is an example of one of the events:
```
logstash11 | {
logstash11 | "@timestamp" => 2018-04-13T16:12:41.501Z,
logstash11 | "offset" => 742656,
logstash11 | "@metadata" => {
logstash11 | "type" => "doc",
logstash11 | "beat" => "filebeat",
logstash11 | "pipeline" => "filebeat-6.2.3-apache2-access-default",
logstash11 | "ip_address" => "10.142.0.8",
logstash11 | "version" => "6.2.3"
logstash11 | },
logstash11 | "host" => "web0",
logstash11 | "beat" => {
logstash11 | "hostname" => "web0",
logstash11 | "name" => "web0",
logstash11 | "version" => "6.2.3"
logstash11 | },
logstash11 | "@version" => "1",
logstash11 | "prospector" => {
logstash11 | "type" => "log"
logstash11 | },
logstash11 | "source" => "/var/log/apache2/access.log",
logstash11 | "message" => "127.0.0.1 - - [13/Apr/2018:16:12:40 +0000] \"GET /server-status?auto= HTTP/1.1\" 200 781 \"-\" \"Go-http-client/1.1\"",
logstash11 | "fileset" => {
logstash11 | "module" => "apache2",
logstash11 | "name" => "access"
logstash11 | },
logstash11 | "tags" => [
logstash11 | [0] "beats_input_codec_plain_applied"
logstash11 | ]
logstash11 | }
```
---
### Author:  [@rcowart](https://discuss.elastic.co/u/rcowart)
#### Post date: [April 13, 2018, 4:50pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/14 "2018-04-13T16:50:43Z")
You are right! I forgot that I set name to be the name of the beat in yml config file. Looking at your data, you have the option of using `%{[beat][version]}` or `%{[@metadata"][version]}`, and you would use `%{[@metadata][beat]}` for the beat name.
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [April 16, 2018, 12:44pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/15 "2018-04-16T12:44:35Z")
The event looks correct. You can use any field for creating the index name. You can even us a filter plugin to create a custom index name and add it to `@metadata`. The `@metadata` field is not index in Elasticsearch.
Does the problem still persist?
---
### Author:  [@alexus](https://discuss.elastic.co/u/alexus)
#### Post date: [April 16, 2018, 3:40pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/16 "2018-04-16T15:40:24Z")
@steffens, even though I still not really sure what cause this not to work the first time, after configure logstash with multiple pipelines, that seems to address issue. I'm not seeing anything in Dashboard (yet), but it looks like I'm missing more Logstash configuration to parse logs correctly.
I was reading about multi-pipline and then you mentioned in one of your comment as well, so I marked that comment as "Solution".
Thanks!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 14, 2018, 3:40pm UTC](https://discuss.elastic.co/t/metadata-beat-metadata-version-yyyy-mm-dd/127360/17 "2018-05-14T15:40:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.