Migrating from Filebeat 5.5.2 to 6.0.0

Elastic Stack Beats
1

Hello World,
I am trying to upgrade my filebeat agent and doing the following steps (on Windows):

  1. Stopping the filebeat service
  2. uninstalling the filebeat with the uninstall-service-winlogbeat.ps1
  3. replacing the winlogbeat.yml in the 6.0.0 folder with the one that works from 5.5.2
  4. installing the service with install-service-winlogbeat.ps1
  5. Starting the service

No events coming in to ElasticSearch.... :frowning:
Stopping and Rolling back to the old 5.5.2 agent.. all documents/events coming in since last event.

Why?

2

After upgrading you should test your config and output. Perhaps you are affected by a breaking change in 6.0.

PS > .\winlogbeat.exe test config -e

PS > .\winlogbeat.exe test output -e

3

Seems like it is something in Logstash, because now i can get the events sent successfully in the winlogbeat logs, but they dont show up in elasticsearch.

I get this error in my logstash server:
[2017-11-16T20:35:35,062][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-2017.11.16", :_type=>"wineventlog", :_routing=>nil}, #LogStash::Event:0x1d52d6b3], :response=>{"index"=>{"_index"=>"winlogbeat-2017.11.16", "_type"=>"wineventlog", "_id"=>"gL5Uxl8BA5Maqd7NT5eO", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [winlogbeat-2017.11.16] as the final mapping would have more than 1 type: [doc, wineventlog]"}}}}

4

Did you install the latest index template from winlogbeat 6.0.0 to Elasticsearch? Since you are using Logstash you have to do in manually because the Beat isn't directly connection to ES.

# Write the template to disk.
PS> .\winlogbeat.exe export template | Out-File -Encoding UTF8 winlogbeat.template.json

# Install it to ES.
PS > Invoke-RestMethod -Method Put -ContentType "application/json" -InFile winlogbeat.template.json -Uri http://elasticsearch:9200/_template/winlogbeat-6.0.0

And after you have installed the index template to ES check your Logstash config to make sure that it is using a version in the index name like shown in these docs.

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
}
5

Another method of installing the template is like this:

PS C:\Program Files\Winlogbeat> .\winlogbeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["elasticsearch:9200"]'

where elasticsearch is the hostname/IP of your ES server.

6

You are correct, i did not update the template. But i have now. Is it enough to do this once? and then all winlogbeat agents will use that template? (I dont need to do that once per client?).

I updated and now they are coming in as they should.

Found this info, dont know if its because of that:

The Beats shipper automatically sets the type field on the event. You cannot override this setting in the Logstash config. If you specify a setting for the type config option in Logstash, it is ignored.

7

Correct you only need to do this once because it will apply to all indices created in elasticsearch that match winlogbeat-6.0.0-*. When you upgrade to 6.0.1 you should repeat this before starting the beat (the template should be there before any data shows up to ensure it gets index properly).

That statement about types in Logstash is unrelated to the problem. The type => log setting does nothing and that is all this statement is saying.

8

This topic was automatically closed after 21 days. New replies are no longer allowed.