%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}

alexus · April 9, 2018, 5:05pm

I'm trying to follow Configure the Logstash output | Metricbeat Reference (Accessing metadata fields)

# curl --silent --request GET $ELASTICSEARCH_URI/_cat/indices | grep metadata
green open %{[@metadata][beat]}-%{[@metadata][version]}-2018.04.10 V3M9LavYTSa_7zcKEOh1yQ 5 1          6         0    272kb  139.3kb
#

elasticsearch's log:

esm1 | [2018-04-10T00:00:20,446][INFO ][o.e.c.m.MetaDataMappingService] [esm1] [%{[@metadata][beat]}-%{[@metadata][version]}-2018.04.10/1OIr7aShTB-9eBIhJYncKA] update_mapping [doc]

logstash's (input/output):

# cat 10-input-beats.conf 
input {
	beats {
		port => 5044
	}
}
# grep -v password 30-output-elasticsearch.conf
output {
	if [container_id] {
		elasticsearch {
			hosts => "elasticsearch:9200"
			user => "elastic"
		}
	} else {
		elasticsearch {
			hosts => "elasticsearch:9200"
			user => "elastic"
			manage_template => false
			index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
			#document_type => "%{[@metadata][type]}"
		}
	}
}
#

Beats input plugin | Logstash Reference

steffens · April 10, 2018, 11:14am

Logstash will create this index name if the fields are not present in the event.

Which versions of Beats and Logstash have you installed?
There might be some filter in your configs removing/changing some of the events metadata.

alexus · April 10, 2018, 2:16pm

6.2.3
default configuration

steffens · April 10, 2018, 5:15pm

Beats always send @metadata to Logstash. Grep all your logstash configs/directories for:

input: another input might receive inputs without @metadata
metadata: a filter might modify @metadata

alexus · April 10, 2018, 9:00pm

following is my entire logstash configuration (minus my own comments and my password):

# grep -Ev '^#|password' *.conf
10-input-beats.conf:
10-input-beats.conf:input {
10-input-beats.conf:	beats {
10-input-beats.conf:		port => 5044
10-input-beats.conf:	}
10-input-beats.conf:}
10-input-dead_letter_queue.conf:input {
10-input-dead_letter_queue.conf:	dead_letter_queue {
10-input-dead_letter_queue.conf:		path => "/usr/share/logstash/data/dead_letter_queue"
10-input-dead_letter_queue.conf:	}
10-input-dead_letter_queue.conf:}
10-input-gelf.conf:
10-input-gelf.conf:input {
10-input-gelf.conf:	gelf {
10-input-gelf.conf:		codec => "json"
10-input-gelf.conf:	}
10-input-gelf.conf:}
20-filter.conf:
20-filter.conf:filter {
20-filter.conf:	json {
20-filter.conf:		skip_on_invalid_json => true
20-filter.conf:		source => "message"
20-filter.conf:	}
20-filter.conf:}
20-filter.conf:
30-output-elasticsearch.conf:
30-output-elasticsearch.conf:output {
30-output-elasticsearch.conf:	if [container_id] {
30-output-elasticsearch.conf:		elasticsearch {
30-output-elasticsearch.conf:			hosts => "elasticsearch:9200"
30-output-elasticsearch.conf:			user => "elastic"
30-output-elasticsearch.conf:		}
30-output-elasticsearch.conf:	} else {
30-output-elasticsearch.conf:		elasticsearch {
30-output-elasticsearch.conf:			hosts => "elasticsearch:9200"
30-output-elasticsearch.conf:			user => "elastic"
30-output-elasticsearch.conf:			manage_template => false
30-output-elasticsearch.conf:			index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
30-output-elasticsearch.conf:			#document_type => "%{[@metadata][type]}"
30-output-elasticsearch.conf:		}
30-output-elasticsearch.conf:	}
30-output-elasticsearch.conf:}
#

steffens · April 11, 2018, 11:22am

You have multiple inputs, but no filtering on conditions for actual source in your outputs. The weird index might have been created due to gelf or events comming from the dead letter queue.

How many outputs do you have?

You can protect the Elasticsearch output by adding a condition, checking for the presence of [@metadata][beat].

alexus · April 11, 2018, 9:31pm

sorry, I'm bit confused... you're saying to add a condition checking to output or to filter?

I mean, we can clearly see that events are ending up in that weirdly named index already..

steffens · April 12, 2018, 10:07am

The output might process fields from the gelf input for example. You should protect the output to process events from beats only. The filter looks like a noop for gelf based events.

Have you had a look at Logstash pipelines? A pipeline consists of inputs, filters and outputs. Using multiple pipelines, you can more easily separate data-flows, without having to use conditionals.

alexus · April 12, 2018, 3:00pm

all events from gelf input has container_id key, which I check for in output, therefor events are ending up in right index.

events that coming in through beats input on another hand, looks like are ending up in right index as well, as I see them in %{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd} index, unless I'm wrong, that if statement in output works as expected. The issue here is @metadata is missing, now I can add yet another condition to check [@metadata][beat], however in best case scenario event would end up in same index or due to @metadata is missing after all would not even end up there anymore...

I'm reading and trying to understand Multiple Pipelines...

steffens · April 13, 2018, 11:51am

can you run a separate logstash config (don't use any of your current configurations) with this logstash config only?

input {
  beats {
    port => 5044
  }
}

output {
  stdout {
    codec => rubydebug {
      metadata => true
    }
  }
}

This should print you the raw events (without any filters), from beats only. Including the @metadata fields.

rcowart · April 13, 2018, 1:58pm

The docs are wrong. What you want to use is %{[beat][name]} and %{[beat][version]}. So it would be...

index => "%{[beat][name]}-%{[beat][version]}-%{+YYYY.MM.dd}"

alexus · April 13, 2018, 4:07pm

@rcowart thanks again for your reply, however this configuration isn't exactly what I'm looking for.

in my case (per your configuration), indices that would get created: web0-6.2.3-2018.04.10, that would break visualization and/or dashboard in Kibana as beats pattern are: filebeat-* or metricbeat-*.

alexus · April 13, 2018, 4:14pm

I re-configured my Logstash per Multiple Pipelines | Logstash Reference [6.2] | Elastic.

I adjusted output instead of elasticsearch to stdout, following is an example of one of the events:

logstash11  | {
logstash11  |     "@timestamp" => 2018-04-13T16:12:41.501Z,
logstash11  |         "offset" => 742656,
logstash11  |      "@metadata" => {
logstash11  |               "type" => "doc",
logstash11  |               "beat" => "filebeat",
logstash11  |           "pipeline" => "filebeat-6.2.3-apache2-access-default",
logstash11  |         "ip_address" => "10.142.0.8",
logstash11  |            "version" => "6.2.3"
logstash11  |     },
logstash11  |           "host" => "web0",
logstash11  |           "beat" => {
logstash11  |         "hostname" => "web0",
logstash11  |             "name" => "web0",
logstash11  |          "version" => "6.2.3"
logstash11  |     },
logstash11  |       "@version" => "1",
logstash11  |     "prospector" => {
logstash11  |         "type" => "log"
logstash11  |     },
logstash11  |         "source" => "/var/log/apache2/access.log",
logstash11  |        "message" => "127.0.0.1 - - [13/Apr/2018:16:12:40 +0000] \"GET /server-status?auto= HTTP/1.1\" 200 781 \"-\" \"Go-http-client/1.1\"",
logstash11  |        "fileset" => {
logstash11  |         "module" => "apache2",
logstash11  |           "name" => "access"
logstash11  |     },
logstash11  |           "tags" => [
logstash11  |         [0] "beats_input_codec_plain_applied"
logstash11  |     ]
logstash11  | }

rcowart · April 13, 2018, 4:50pm

You are right! I forgot that I set name to be the name of the beat in yml config file. Looking at your data, you have the option of using %{[beat][version]} or %{[@metadata"][version]}, and you would use %{[@metadata][beat]} for the beat name.

steffens · April 16, 2018, 12:44pm

The event looks correct. You can use any field for creating the index name. You can even us a filter plugin to create a custom index name and add it to @metadata. The @metadata field is not index in Elasticsearch.

Does the problem still persist?

alexus · April 16, 2018, 3:40pm

@steffens, even though I still not really sure what cause this not to work the first time, after configure logstash with multiple pipelines, that seems to address issue. I'm not seeing anything in Dashboard (yet), but it looks like I'm missing more Logstash configuration to parse logs correctly.

I was reading about multi-pipline and then you mentioned in one of your comment as well, so I marked that comment as "Solution".

Thanks!

system · May 14, 2018, 3:40pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.