Logstash's - Beat to Elasticsearch Configuration - metadata

Elastic Stack Beats
1

I'm trying to follow Beats input plugin | Logstash Reference [6.2] | Elastic

logstash's (input/output):

# cat pipeline/10-input-beats.conf 
input {
	beats {
		port => 5044
	}
}
# 
# cat pipeline/30-output-elasticsearch.conf  | grep -v password
output {
	if [container_id] {
		elasticsearch {
			hosts => "elasticsearch:9200"
			user => "elastic"
		}
	} else {
		elasticsearch {
			hosts => "elasticsearch:9200"
			user => "elastic"
			manage_template => false
			index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
			document_type => "%{[@metadata][type]}"
		}
	}
}
#

elasticsearch's log:

esm1             | [2018-02-25T02:40:04,197][INFO ][o.e.c.m.MetaDataCreateIndexService] [esm1] [%{[@metadata][beat]}-%{[@metadata][version]}-2018.02.25] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
esm1             | [2018-02-25T02:40:04,895][INFO ][o.e.c.m.MetaDataMappingService] [esm1] [%{[@metadata][beat]}-%{[@metadata][version]}-2018.02.25/AlutbrA8TOyG-rsfgzy5uw] create_mapping [%{[@metadata][type]}]
esm1             | [2018-02-25T02:40:05,236][INFO ][o.e.c.m.MetaDataMappingService] [esm1] [%{[@metadata][beat]}-%{[@metadata][version]}-2018.02.25/AlutbrA8TOyG-rsfgzy5uw] update_mapping [%{[@metadata][type]}]

elasticsearch's index with bogus name:

# curl --silent --request GET $ELASTICSEARCH_URI/_cat/indices | grep metadata
green open %{[@metadata][beat]}-%{[@metadata][version]}-2018.02.25 AlutbrA8TOyG-rsfgzy5uw 5 1          4         0  117.2kb   58.6kb
#

Why isn't %{[@metadata][beat]} and/or %{[@metadata][version]} isn't being translated to actual values?

Please advise.

2

In the beat's configuration I set name to the name of the beat (metricbeat, filebeat, etc). You would then want to use beat.version insead of metadata fields. So you would have...

index => "%{[beat][name]}-%{[beat][version]}-%{+YYYY.MM.dd}"

Remove the document _type bit. That will cause you problems in 6.x.

3

if logstash supposedly can do it on the fly, why change beat configuration?

4

@alexus I am just telling you how I do it, which works. I can't speak to any inaccuracies in the docs.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.