Metricbeat 6.3.2 upgrade breaks?

Elastic Stack Beats
1

CentOS 7.5.1804 (from /etc/centos-release)
ES 6.3.2 -- checked via curl
Metricbeat 6.3.2 (from debug log)
System module. (from JSON in debug log)

Debug logs don't show a lot that seems helpful. But there's a constant error in it. An example (slightly anonymized):

2018-07-30T14:31:09.705Z WARN elasticsearch/client.go:502 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbecfe62b2a128110, ext:10139927047, loc:(*time.Location)(0x3b01420)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"module":"system", "rtt":434659, "name":"process"}, "system":common.MapStr{"process":common.MapStr{"pid":7, "username":"xxxx", "memory":common.MapStr{"rss":common.MapStr{"bytes":0x0, "pct":0}, "share":0x0, "size":0x0}, "cpu":common.MapStr{"total":common.MapStr{"value":0, "pct":0, "norm":common.MapStr{"pct":0}}, "start_time":common.Time{wall:0x0, ext:63662610177, loc:(*time.Location)(nil)}}, "ppid":2, "pgid":0, "fd":common.MapStr{"limit":common.MapStr{"soft":0x400, "hard":0x1000}, "open":0x0}, "name":"migration/0", "state":"sleeping"}}, "host":common.MapStr{"name":"babe"}, "beat":common.MapStr{"name":"yyyy", "hostname":"yyyy", "version":"6.3.2"}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: Mapping definition for [system] has unsupported parameters: [properties : {ticks={type=long}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [system] has unsupported parameters: [properties : {ticks={type=long}}]"}}

I got a similar error after upgrading packet beat to 6.3.2, but that went away after forcing a template overwrite. Tried that here, no joy. Nothing seems to work.

2

Hi,

From which version are you upgrading?

Are you using custom indices or the default?

3

Using the default (did use custom a while back, but switched before going from 6.2 to 6.3.0)

Upgrade that broke was going from 6.3.0 to 6.3.2 for Metricbeat. Elasticsearch was still back at 6.2.x (I forget which point number it was. It had upgraded to 6.3.0, but the version actually running was still 6.2.x until after the upgrade to 6.3.2

I'm wondering if the fact the metricbeat fields.yml defines more fields than the ES default field limit has anything to do with it? -- Sorry, cranial flatulence...when I checked it last time I dropped a zero. Sigh.

4

Can you share the template Elasticsearch has for metricbeat-*?

5

I put the output from /_template/metricbeat-6.3.2 at https://gist.github.com/Paladin/d0ff4402c40364270130a12ee278719a because it was about 4x larger than the max for the message body here.

I should probably note that's the 6.3.2 template. There are more templates, but that should be the one metricbeat 6.3.2 uses by default, since I'm not specifying a pattern in metricbeat.yml

6

So. No Ideas?

My temptation is to nuke the entire metric data set and start over from scratch with no data, no indexes, no templates, nothing at all. see if it works.

7

And that's what I did. No idea what caused the issue, or how to fix it, so I just wiped everything and started over.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

9

For anyone that runs into this, the issue is caused by a mapping conflict between the current metricbeat-* index and new Metricbeat data. The easiest solution is delete the old template and current index to let Metricbeat create a fresh index with the upgraded template.

DELETE metricbeat-<today>
DELETE _template/metricbeat-<old>

Be sure the new template is written to the cluster.

metricbeat setup --template
10

I should be more specific about what happens here. The old Metricbeat template causes new data to have the wrong mapping. If preserving data is required, then these steps should work:

  1. place a write block on the "bad" metricbeat-<version>.* index with index.blocks.write: true
  2. delete the old template
  3. reindex the bad index it to an index with a different name
  4. delete the bad index

The new Metricbeat will then be free to index to metricbeat-<version> without a mapping conflict.