Help please.. breaking changes?

Maekee · November 16, 2017, 7:54pm

Upgrading to 6.0.0 of the ELK stack
Can anyone please help me why i get this entrys over and over in the Logstash log

[2017-11-16T20:49:36,973][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-2017.11.16", :_type=>"wineventlog", :_routing=>nil}, #LogStash::Event:0x59ffd445], :response=>{"index"=>{"_index"=>"winlogbeat-2017.11.16", "_type"=>"wineventlog", "_id"=>"LcRhxl8BA5Maqd7NKABD", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [winlogbeat-2017.11.16] as the final mapping would have more than 1 type: [doc, wineventlog]"}}}}

This is my logstash.json

input {
beats {
port => 5044
type => "log"
}
}

filter {
if [task] == "Network Policy Server" {
mutate { add_tag => ["nps","radius"] }
}
}

output {
if "radius" in [tags] {
elasticsearch {
hosts => "elasticsearchhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-radius-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else if "dc" in [tags] {
elasticsearch {
hosts => "elasticsearchhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-dc-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else {
elasticsearch {
hosts => "elasticsearchhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}

theuntergeek · November 16, 2017, 8:16pm

You are apparently trying to send multiple types to a single index name. This is disallowed in 6.0.

See:

Breaking Changes in 6.0
Removal of Mapping Types

Christian_Dahlqvist · November 16, 2017, 8:20pm

Have you upgraded winlogbeat?

Maekee · November 16, 2017, 8:22pm

I have tried, but the events dont come into elasticsearch with the 6.0.0 version of the winlogbeat agent.

Dont know which part of the Beat->Logstash->ElasticSearch chain i have incorrect right now.
There is a breaking change somewhere.

theuntergeek · November 16, 2017, 8:23pm

My guess is that this is because you're trying to do:

document_type => "%{[@metadata][type]}"

The new Winlogbeat template is probably using doc by default (as that's what the defaults is for a single type per index in 6.0), which would cause the collision if you tried to add wineventlog as another type.

Maekee · November 16, 2017, 8:24pm

What does that mean?

It it because if this output logstash filter maybe?

output {
if "radius" in [tags] {
elasticsearch {
hosts => "elasticsearchserver:9200"
manage_template => false
index => "%{[@metadata][beat]}-radius-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else if "dc" in [tags] {
elasticsearch {
hosts => "elasticsearchserver:9200"
manage_template => false
index => "%{[@metadata][beat]}-dc-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else {
elasticsearch {
hosts => "elasticsearchserver:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}

theuntergeek · November 16, 2017, 8:32pm

Yes, it's because you're manually specifying the document_type with the value in @metadata. It might be as simple as just changing it to:

document_type => "doc"

Maekee · November 17, 2017, 7:42am

I removed the document_type from my logstash.json and updated the index name to a more static name, so it looks like this:

output {
if "radius" in [tags] {
elasticsearch {
hosts => "vascmdbtest01:9200"
manage_template => false
index => "winlogbeat-radius-%{+YYYY.MM.dd}"
}
} else if "dc" in [tags] {
elasticsearch {
hosts => "vascmdbtest01:9200"
manage_template => false
index => "winlogbeat-dc-%{+YYYY.MM.dd}"
}
} else {
elasticsearch {
hosts => "vascmdbtest01:9200"
manage_template => false
index => "winlogbeat-%{+YYYY.MM.dd}"
}
}
}

these bastards are still coming in:

[2017-11-17T08:39:20,548][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-2017.11.17", :_type=>"wineventlog", :_routing=>nil}, #LogStash::Event:0x42a0754], :response=>{"index"=>{"_index"=>"winlogbeat-2017.11.17", "_type"=>"wineventlog", "_id"=>"w9bqyF8BA5Maqd7N7qJR", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [winlogbeat-2017.11.17] as the final mapping would have more than 1 type: [doc, wineventlog]"}}}}

Where does the doc and wineventlog coming from??

mbsj · November 17, 2017, 10:09am

I have the same issue for a couple of my inputs. My configuration looks like this:

input {
tcp {
	port => 5560 
	codec => "json"
	type => ["performancecounter"]
  }
}

filter {
	if [type] == "performancecounter" {
		grok {
			match => { "CounterPath" => [ "\\\\%{DATA:hostname}\\%{DATA:countergroup}\(%{DATA:hostname}\\private\$\\%{DATA:service}\)\\%{GREEDYDATA:Counter}", 
										  "\\\\%{DATA:hostname}\\%{DATA:countergroup}\\%{GREEDYDATA:Counter}" ] }
	}

	date { 
		"match" => [ "MeasureTime", "yyyy-MM-dd HH:mm:ss,SSS" ]
		target => "@timestamp"
	}

  	mutate {
		replace => { "type" => "performancecounter" }
	}
  }
}

output {
	if [type] == "performancecounter" {
		elasticsearch { 
			hosts => ["172.22.22.195:9200"] 
			index => "performance-%{+YYYY.MM.dd}"
		}
	} 
}

With this i get the following error:

[2017-11-17T10:50:46,492][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"performance-2017.11.17", :_type=>"performancecounter", :_routing=>nil}, #<LogStash::Event:0x126f6108>], :response=>{"index"=>{"_index"=>"performance-2017.11.17", "_type"=>"performancecounter", "_id"=>"9KhjyV8BDxseqbvrQakY", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [performance-2017.11.17] as the final mapping would have more than 1 type: [performancecounter, json]"}}}}

I have tried to work around it by removing the type and using add_field to create a custom field, then check for that instead. But then i end up with this error:

[2017-11-17T10:58:27,378][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"performance-2017.11.17", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x345fcc96>], :response=>{"index"=>{"_index"=>"performance-2017.11.17", "_type"=>"doc", "_id"=>"dqhqyV8BDxseqbvrRboa", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Rejecting mapping update to [performance-2017.11.17] as the final mapping would have more than 1 type: [json, doc]"}}}}

Is there a way to strip this in the filter or output, removing one value or forcing a specific value? I have tried using mutate in the filter to replace the type field with a new (the same) value but it doesn't change anything.

Maekee · November 17, 2017, 12:12pm

I got help in the beat-forum here: Migrating from Filebeat 5.5.2 to 6.0.0

I had not uploaded a new beat template, after doing that, i updated the logstash.json and restarted the beat agents the errors disappeared.

Changed the logstash output to this:
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
Removed this from the input block: type => "log"

rsaeks · November 22, 2017, 6:01pm

I'm seeing a similar issue to this, but without even using beats. I've really tried to simplify our config for testing this out and have no references to type anywhere in my config:

input {
 udp {
    port => 5514
    tags => [ "cisco-switch" ]
  }
}
filter {
  if "cisco-switch" in [tags] {
    grok {
      match => [ "message" , "<%{NUMBER:message_type_id}>%{BASE10NUM:event_no}: %{CISCOTIMESTAMP:cisco_time} %{WORD}: %%{NOTSPACE:cisco_type}: %{GREEDYDATA:description} %{NOTSPACE:interface}, %{GREEDYDATA:status}"]
      match => [ "message" , "<%{NUMBER:message_type_id}>%{BASE10NUM:event_no}: %{CISCOTIMESTAMP:cisco_time} %{WORD}: %%{NOTSPACE:cisco_type}: %{GREEDYDATA:description} %{NOTSPACE:interface}: %{GREEDYDATA:status}"]
      add_tag => [ "cisco-switch-processed" ]
      add_tag => [ "30-filter-ciscoSwitch"]
      tag_on_failure => [ "cisco-switch-todo" ]
    }
    date {
      match => ["cisco_time", "MMM dd HH:mm:ss.SSS","MMM  d HH:mm:ss.SSS" ]
      timezone => "America/Chicago"
      target => [ "eventTime" ]
    }    
  }
}
output {
  if "cisco-switch-processed" in [tags] {
    elasticsearch { index => "ciscoswitch-%{+YYYY.MM.dd}"}
   }
  else {
    elasticsearch { index => "cleanup-%{+YYYY.MM.dd}"}
  }
}

But still see an error with "cisco-switch" listed as s type:

[2017-11-22T11:45:02,127][DEBUG][o.e.a.b.TransportShardBulkAction] [ciscoswitch-2017.11.22][0] failed to execute bulk item (index) BulkShardRequest [[ciscoswitch-2017.11.22][0]] containing [index {[ciscoswitch-2017.11.22][doc][8YrV5F8BPUaikWKCQbOo], source[{"cisco_type":"LINK-3-UPDOWN","message_type_id":"187","description":"Interface","message":"<187>705992: Nov 22 11:45:01.101 CST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/15, changed state to up","interface":"GigabitEthernet1/0/15","tags":["cisco-switch","cisco-switch-processed","30-filter-ciscoSwitch"],"cisco_time":"Nov 22 11:45:01.101","@timestamp":"2017-11-22T17:45:02.110Z","event_no":"705992","@version":"1","host":"192.168.52.4","eventTime":"2017-11-22T17:45:01.101Z","status":"changed state to up"}]}]
java.lang.IllegalArgumentException: Rejecting mapping update to [ciscoswitch-2017.11.22] as the final mapping would have more than 1 type: [cisco-switch, doc]

Any ideas on what else in the config file may need to be changed so "cisco-switch" no longer appears in the type?

Maekee · November 22, 2017, 7:32pm

As i understand, one change in 6.0.0 is that only one document type is supported, and in 7.0 types/mapping will be removed. But where do my types come from? beats?

Bharathkumarraju · November 23, 2017, 8:19am

/tmp/elastic_dev/filebeat/current/filebeat -c /tmp/elastic_dev/filebeat/config/filebeat.yml -e
2017/11/23 08:05:06.633737 beat.go:426: INFO Home path: [/tmp/elastic_dev/filebeat/current] Config path: [/tmp/elastic_dev/filebeat/current] Data path: [/tmp/elastic_dev/filebeat/current/data] Logs path: [/tmp/elastic_dev/filebeat/current/logs]
2017/11/23 08:05:06.633916 beat.go:433: INFO Beat UUID: ca5704f8-9b1a-4c94-8766-1dc76b119230
2017/11/23 08:05:06.633952 beat.go:192: INFO Setup Beat: filebeat; Version: 6.0.0
2017/11/23 08:05:06.634604 metrics.go:23: INFO Metrics logging every 30s
2017/11/23 08:05:06.635838 client.go:123: INFO Elasticsearch url: https://sample.test.raju.com:9200
2017/11/23 08:05:06.636048 client.go:123: INFO Elasticsearch url: https://sample.test.raju.com:9220
2017/11/23 08:05:06.636161 client.go:123: INFO Elasticsearch url: https://sample.test.raju.com:9230
2017/11/23 08:05:06.636812 module.go:80: INFO Beat name: 10.20.175.66
2017/11/23 08:05:06.641468 beat.go:260: INFO filebeat start running.
2017/11/23 08:05:06.642313 registrar.go:88: INFO Registry file set to: /tmp/elastic_dev/filebeat/current/data/registry
2017/11/23 08:05:06.642475 registrar.go:108: INFO Loading registrar data from /tmp/elastic_dev/filebeat/current/data/registry
2017/11/23 08:05:06.643372 registrar.go:119: INFO States Loaded from registrar: 4
2017/11/23 08:05:06.643439 crawler.go:44: INFO Loading Prospectors: 2
2017/11/23 08:05:06.643746 registrar.go:150: INFO Starting Registrar
2017/11/23 08:05:06.644503 prospector.go:103: INFO Starting prospector of type: log; id: 9119168733948319376
2017/11/23 08:05:06.645260 harvester.go:207: INFO Harvester started for file: /opt/hello1/test_ServiceAudit.log
2017/11/23 08:05:06.645842 prospector.go:103: INFO Starting prospector of type: log; id: 17106901312407876564
2017/11/23 08:05:06.645874 crawler.go:78: INFO Loading and starting Prospectors completed. Enabled prospectors: 2
2017/11/23 08:05:06.648357 harvester.go:207: INFO Harvester started for file: /opt/hello2/test_ProtocolAudit.log
2017/11/23 08:05:07.697281 client.go:651: INFO Connected to Elasticsearch version 6.0.0
2017/11/23 08:05:07.700284 client.go:651: INFO Connected to Elasticsearch version 6.0.0
2017/11/23 08:05:07.704069 client.go:651: INFO Connected to Elasticsearch version 6.0.0
2017/11/23 08:05:08.722058 client.go:465: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [service-audit-2017.11.23] as the final mapping would have more than 1 type: [log, doc]"}
2017/11/23 08:05:08.722107 client.go:465: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [protocol-audit-2017.11.23] as the final mapping would have more than 1 type: [log, doc]"}

Bharathkumarraju · November 23, 2017, 8:20am

{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [service-audit-2017.11.23] as the final mapping would have more than 1 type: [log, doc]"}
2017/11/23 08:05:08.722107 client.go:465: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [protocol-audit-2017.11.23] as the final mapping would have more than 1 type: [log, doc]"}

uschellh · November 27, 2017, 1:16pm

Hello,

we also suffer from this change, and i have no idea where to check now, that i got rid of types and changed the input filters and parsers to tags:

(as i tried putting in 1 type)
as the final mapping would have more than 1 type: [cisconx, WindowsEventLog]"}}}}
now change to tags:

tcp {
port => 5025
tags => "WindowsEventLog"

now other error details:

as the final mapping would have more than 1 type: [doc, cisconx

i really wonder where that cisconx comes from, its one of alot of types (we HAD)

see our conf.d folder

root@bngfvnjvhbns:/etc/logstash/conf.d# find . -type f -exec grep -H cisconx {} ;
./01_input.conf: tags => "cisconx"
./01_input.conf: tags => "cisconx"
./13_filter_cisconx.conf: if "cisconx" in [tags] {

our logstash index template: looks pretty stock i believe

(sorry for readability)

{"logstash":
{"order":0,"version":60001,"index_patterns":["logstash-"],"settings":
{"index":
{"refresh_interval":"5s"}
}
,"mappings":
{"default":
{"dynamic_templates":
[
{"message_field":
{"path_match":"message","match_mapping_type":"string","mapping":
{"type":"text","norms":false}
}
}
,
{"string_fields":
{"match":"","match_mapping_type":"string","mapping":
{"type":"text","norms":false,"fields":
{"keyword":
{"type":"keyword","ignore_above":256}
}
}
}
}
],"properties":
{"@timestamp":
{"type":"date"},"@version":
{"type":"keyword"},"geoip":
{"dynamic":true,"properties":
{"ip":{"type":"ip"},"location":
{"type":"geo_point"},"latitude":
{"type":"half_float"},"longitude":
{"type":"half_float"}}}}}},"aliases":{}}}

thanks for insights !! we dont want type cisconx

uschellh · December 8, 2017, 8:52am

that behaviour was gone the next day, idk what it has todo with the unchanged template for the daily indexes, but whatever

system · January 5, 2018, 8:53am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash 6.0 breaks winlogbeat agent indexing Logstash	3	1294	December 25, 2017
Events are not indexing after upgradtion 5.6 to 6.0 Elasticsearch	18	3904	December 20, 2017
Could Not Index Event to Elasticsearch Error? Elasticsearch	15	1120	October 11, 2022
WInlogbeats Error after upgrade Beats winlogbeat	2	338	May 20, 2020
Could not index event to Elasticsearch.Rejecting mapping update to [] as the final mapping would have more than 1 type: [_doc,syslog] Logstash	3	1696	July 22, 2020

Help please.. breaking changes?

Related topics