Changing winlogbeat from elasticsearch to logstash

[MColeman (Mark Coleman)]

Hi,
I started off a cluster with winlogbeat going directly to elasticsearch and using the pre-built dashboards. All that worked well out of the box. Now I'd like to send my winlogbeat data through logstash so I can do some email alerting out of logstash on the winlogbeat data and continue to use the pre-built dashboards.
I configured logstash to listen for the beats output with no issue, and I have a conditional elasticsearch output block to send the winlogbeat data into its own index like this:

input {
     beats {
           port => 5044
		   type => winlogbeat
        }
...
if [type] == "winlogbeat" {
	            elasticsearch {
	              hosts => ["https://localhost:9200"]	
				  index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

and that works fine. The issue seems to be when I go to the pre-built dashboards I get a failed shards message for the new winlogbeat index.

I guess my question is how can I make the logstash winlogbeat input match the winlogbeat input that goes directly to elasticsearch so that these dashboards continue to work?

Thanks!
Mark

[leandrojmp (Leandro Pereira)]

You need to configure the output as the example in the documentation so it will also use the ingest pipeline in Elasticsearch.

[MColeman (Mark Coleman)]

Hi Leandro! That pointed me in the right direction, although it differed from the documentation. I did have to load the ingest pipelines using this command

winlogbeat setup --pipelines
not 
winlogbeat setup --pipelines --modules sysmon,security

but then the inserts from logstash to elasticsearch would fail so I had to add this filter in logstash to choose the correct pipeline

#filter for winlogbeat
 if [type] == "winlogbeat"{
    mutate { add_field => { "[@metadata][pipeline]" => "winlogbeat-%{[agent][version]}-routing" } }
  } else if !([@metadata][pipeline]) {
    mutate { add_field => { "[@metadata][pipeline]" => "" } }
  }  
#end filter for winlogbeat

Now the dashboards are working and logstash is receiving the data like it should.

Thanks!
Mark