How to forward index from filebeat to elasticsearch via logstash

Elastic Stack Beats
1

Hello,

I have this script for filebeat which sends all logs to logstash:

filebeat:
  prospectors:
      paths:
       - D:\Logs\*
      input_type: log
  registry_file: "D:/ElasticSearch/filebeat-1.0.0-windows/registry"

  ### Elasticsearch as output
  #=elasticsearch:
    #=hosts: ["localhost:9200"]
    #=username: "admin2"
    #=password: "admin2"
    #=index: "dev"
	
  logstash:
    hosts: ["localhost:9202]
	#also I had tryed with dev with double quote
    index: dev
  console:
    pretty: true
	
shipper:  
logging:
  files:
    path: D:/ElasticSearch/filebeat-1.0.0-windows/Log
    rotateeverybytes: 10485760 # = 10MB

This is my script for logstash which forward message received from filebeat to elasticsearch:

input {
		
		beats {
		codec => "json"
		port => 9202
	}
}

output {
	stdout { codec => rubydebug }
	elasticsearch { 
		hosts => ["localhost:9200"] 
		user => "admin2"
		password => "admin2"
	}
}

I had followed this explanations: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html but durring this flow I had lost the original index which was setup into filebeat "dev" and instead, my new index is logstash-yy.mm.dd. And I have no idea if the issue is from filebeat or logstash.

When I had tested filebeat directly with elasticsearch the index was correct.

Do you have any idea what I have done wrong?

Thank you!
Ovidiu

2

I think in the elasticsearch output in the Logstash config, you also need to add this:

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

This is needed because when you set the value of index in the Filebeat configuration, that value is passed to Logstash from Filebeat via the @metadata.beat JSON field. See https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-details.html#logstash-output for more details.

1 Like
3

Yes, you're right! It's working.

Thank you!
Ovidiu