Problem with APM after upgrade to 7.10

Elastic Observability APM
,
1

Guys, today my APM is not working on the turn to create a new index.

The error is:

"log": "2021-05-28T00:00:02.816Z\tWARN\t[elasticsearch]\telasticsearch/client.go:408\tCannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x4d3f640, ext:63757756799, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"agent\":{\"ephemeral_id\":\"304360b5-5573-4c97-842a-ea5ac7229f21\",\"name\":\"java\",\"version\":\"1.12.0\"},\"ecs\":{\"version\":\"1.6.0\"},\"host\":{\"architecture\":\"amd64\",\"hostname\":\xxxxxxx\",\"ip\":\"10.23.78.129\",\"name\":\"xxxxxxxxxl\",\"os\":{\"platform\":\"Linux\"}},\"jvm\":{\"gc\":{\"count\":16,\"time\":7987}},\"labels\":{\"name\":\"PS MarkSweep\"},\"observer\":{\"ephemeral_id\":\"4ed24683-847f-40c3-b879-a52c91b0396a\",\"hostname\":\"apm-server-667cfd5674-rvpd9\",\"id\":\"xxxxxxxx\",\"type\":\"apm-server\",\"version\":\"7.10.2\",\"version_major\":7},\"process\":{\"pid\":8566,\"title\":\"/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.37.amzn1.x86_64/jre/bin/java\"},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"environment\":\"production\",\"language\":{\"name\":\"Java\",\"version\":\"1.8.0_171\"},\"name\":\"xxxxx\",\"node\":{\"name\":\"xxxxx\"},\"runtime\":{\"name\":\"Java\",\"version\":\"1.8.0_171\"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [labels.name] of type [scaled_float] in document with id 'ZDFFsHkBBnKI80FPTrz4'. Preview of field's value: 'PS MarkSweep'\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"PS MarkSweep\\\"\"}}\n",

Pls help meee

2

The error now is:

"2021-05-27T01:19:50.825Z\tWARN\t[elasticsearch]\telasticsearch/client.go:408\tCannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x33058600, ext:63757675182, loc:(*time.Location)(nil)}, Meta:{\"pipeline\":\"apm\"}, Fields:{\"agent\":{\"name\":\"nodejs\",\"version\":\"3.8.0\"},\"container\":{\"id\":\"183a65910409c969b99fbb8f15c0a910757c32f4543a9f859b63117a2b84c1a7\"},\"ecs\":{\"version\":\"1.6.0\"},\"host\":{\"architecture\":\"x64\",\"hostname\":\"183a65910409\",\"ip\":\"33333333\",\"name\":\"183a65910409\",\"os\":{\"platform\":\"linux\"}},\"labels\":{\"env\":\"production\",\"hostname\":\"183a65910409\"},\"observer\":{\"ephemeral_id\":\"4ed24683-847f-40c3-b879-a52c91b0396a\",\"hostname\":\"apm-server-667cfd5674-rvpd9\",\"id\":\"040df035-2207-4e1a-830d-4c33a4413337\",\"type\":\"apm-server\",\"version\":\"7.10.2\",\"version_major\":7},\"process\":{\"args\":[\"/usr/local/bin/node\",\"/app/bin/www\"],\"pid\":7,\"ppid\":1,\"title\":\"node\"},\"processor\":{\"event\":\"metric\",\"name\":\"metric\"},\"service\":{\"environment\":\"production\",\"framework\":{\"name\":\"express\",\"version\":\"4.17.1\"},\"language\":{\"name\":\"javascript\"},\"name\":\"xxxxxxx\",\"node\":{\"name\":\"183a65910409c969b99fbb8f15c0a910757c32f4543a9f859b63117a2b84c1a7\"},\"runtime\":{\"name\":\"node\",\"version\":\"14.15.1\"},\"version\":\"0.0.1\"},\"span\":{\"self_time\":{\"count\":0,\"sum\":{\"us\":0}},\"type\":\"app\"},\"transaction\":{\"name\":\"\",\"type\":\"request\"}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [labels.hostname] of type [scaled_float] in document with id 'O-Noq3kBBnKI80FPAuAe'. Preview of field's value: '183a65910409'\",\"caused_by\":{\"type\":\"number_format_exception\",\"reason\":\"For input string: \\\"183a65910409\\\"\"}}\n",
3

These two errors appear frequently

4

The error indicates that a numeric value has been previously sent and indexed for labels.hostname, which creates a numeric mapping for this field. Following documents with a different type for this field (string in your example) can not be indexed.
Are you instrumenting multiple different services where you set the hostname as label and if so, which APM language agents are you using?

5

Hello Silvia,

I'm using nodejs, java and javascript.

How do I solve this?

6

When instrumenting any service, ensure that the setLabel APIs (rumjs - apm.AddLabels, nodejs - apm.setLabel and java - transaction setLabel) use the same value type for the same keys accross all services. For instance, if you set the hostname as label key, ensure that the passed in values are always of type string or always of a numeric type, but do not mix them. The nodejs agent by default stringifies all values, but this can be disabled.

Label fields are indexed dynamically in Elasticsearch, the type is defined by the first value that gets indexed for a new label key. The above error indicates that the labels name and hostname are indexed as scaled_float, but the apm-server tries to index documents with a string value for these labels. If the type should be changed to string, then first fix the instrumentation in all your services as described above and then (assuming you are using default ILM) trigger a rollover to a new index by using the indices-rollover api. The new index will create a new mapping for dynamically indexed fields, again by deriving the type from the first indexed value.

7

Thanks for the answer Silvia.
I haven't changed anything, the only change made was the upgrade from version 7.5 to 7.10 is this type of upgrade recommended? Or would I be better off doing 7.5 to 7.6 and so on? After the upgrade it came to work for 3 days yet. Very strange

8

Upgrading from 7.5 to 7.10 should generally be fine. The errors rather indicate that something changed in the instrumented services, that the field values had been reported as numeric values at some point but (some) are now recorded as string values.

9

All right, it's so very strange because I not changed this in my services, the fields and the metrics so created by apm-server.

How to solve this? Any idea?

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.