Tried to parse field [body] as object, but found a concrete value

Elastic Observability APM
,
1

Kibana version:
Kibana 6.7.2

Elasticsearch version:
6.7.2

APM Server version:
6.7.2

APM Agent language and version:
Java 1.6.1

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
apt, ubuntu, template from kibana setup script

Fresh install or upgraded from other version?
fresh

Is there anything special in your setup?
APM-Agent connects directly to the APM-Server and the APM-Server directly to ES.

Template:

https://0bin.net/paste/xfQkjsWAI36+HjE-#R3XMRwZNoXKKxm1ubDpSqqJa1d2cxjdw7cQDImdLMzK

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Sometimes when I login to my application with springframework i get this error, and the transaction isn't indexed. But sometimes its working as expected.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
APM-Server Logs:

2019-05-15T09:25:28.633+0200    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x10366400, ext:63693501928, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"trace":common.MapStr{"id":"50b1d249ff2167f32e100668120cd808"}, "timestamp":common.MapStr{"us":1557905128272000}, "beat":common.MapStr{"name":"ESAMS01", "hostname":"ESAMS01", "version":"6.7.2"}, "host":common.MapStr{"name":"ESAMS01"}, "processor":common.MapStr{"name":"transaction", "event":"transaction"}, "transaction":common.MapStr{"duration":common.MapStr{"us":18626}, "type":"request", "result":"HTTP 2xx", "sampled":true, "span_count":common.MapStr{"dropped":common.MapStr{"total":0}, "started":4279}, "id":"ca835bc231e919e8", "name":"ThemeServlet#doGet"}, "context":common.MapStr{"user":map[string]interface {}{"username":"admin"}, "request":map[string]interface {}{"body":"[REDACTED]", "url":map[string]interface {}{"port":"9080", "pathname":"/analysis-ci/js/themes.js", "search":"v=2817", "protocol":"http", "full":"http://localhost:9080/analysis-ci/js/themes.js?v=2817", "hostname":"localhost"}, "socket":map[string]interface {}{"encrypted":false, "remote_address":"0:0:0:0:0:0:0:1"}, "http_version":"1.1", "method":"GET", "headers":map[string]interface {}{"accept-language":"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7", "host":"localhost:9080", "connection":"keep-alive", "user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) leWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36", "accept":"*/*", "referer":"http://localhost:9080/analysis-ci/index.jsp", "accept-encoding":"gzip, deflate, br"}, "cookies":map[string]interface {}{"JSESSIONID":[]interface {}{"5325DFBAC6BDE1A9D6C1A3A87B904216", "E1E6D76B0D74DA45401664DFDE5928ED"}, "session-expiry":"1557912328231", "server-time":"1557905128231", "session-flushed":"true", "client-time-offset":"17"}}, "response":map[string]interface {}{"headers":map[string]interface {}{"Cache-Control":"no-cache", "Set-Cookie":[]interface {}{"session-expiry=1557912328274; Path=/", "server-time=1557905128274; Path=/", "session-expiry=1557912328274; Path=/", "server-time=1557905128274; Path=/"}}, "finished":true, "headers_sent":false, "status_code":200}, "system":common.MapStr{"hostname":"PCWrRe", "architecture":"amd64", "platform":"Windows 10", "ip":"172.21.3.96"}, "process":common.MapStr{"pid":14624, "title":"C:\\PROGRA~1\\AMS\\ANALYS~1\\Pentaho\\java\\bin\\java.exe"}, "service":common.MapStr{"runtime":common.MapStr{"name":"Java", "version":"1.8.0_162"}, "name":"Analysis-CI_pcwrre", "agent":common.MapStr{"name":"java", "version":"1.6.1"}, "version":"1.04 beta 1 (2743)", "language":common.MapStr{"name":"Java", "version":"1.8.0_162"}}}}, Private:interface {}(nil)}, Flags:0x1} (status=400): {"type":"mer_parsing_exception","reason":"object ming for [context.request.body] tried to parse field [body] as object, but found a concrete value"}

2019-05-15T09:25:29.166+0200    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x1556a580, ext:63693501928, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"timestamp":common.MapStr{"us":1557905128358000}, "beat":common.MapStr{"name":"ESAMS01", "hostname":"ESAMS01", "version":"6.7.2"}, "host":common.MapStr{"name":"ESAMS01"}, "processor":common.MapStr{"name":"transaction", "event":"transaction"}, "transaction":common.MapStr{"result":"HTTP 2xx", "sampled":true, "span_count":common.MapStr{"dropped":common.MapStr{"total":0}, "started":1460}, "id":"d0582b39e646f5c2", "name":"Mantle_jsp#doGet", "duration":common.MapStr{"us":15820}, "type":"request"}, "context":common.MapStr{"user":map[string]interface {}{"username":"admin"}, "request":map[string]interface {}{"method":"GET", "headers":map[string]interface {}{"upgrade-insecure-requests":"1", "user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) leWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36", "accept":"text/html,lication/xhtml+xml,lication/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,lication/signed-exchange;v=b3", "referer":"http://localhost:9080/analysis-ci/index.jsp", "accept-encoding":"gzip, deflate, br", "accept-language":"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7", "host":"localhost:9080", "connection":"keep-alive"}, "cookies":map[string]interface {}{"session-flushed":"true", "client-time-offset":"17", "JSESSIONID":[]interface {}{"5325DFBAC6BDE1A9D6C1A3A87B904216", "E1E6D76B0D74DA45401664DFDE5928ED"}, "session-expiry":"1557912328308", "server-time":"1557905128308"}, "body":"[REDACTED]", "url":map[string]interface {}{"full":"http://localhost:9080/analysis-ci/Home", "hostname":"localhost", "port":"9080", "pathname":"/analysis-ci/Home", "protocol":"http"}, "socket":map[string]interface {}{"encrypted":false, "remote_address":"0:0:0:0:0:0:0:1"}, "http_version":"1.1"}, "response":map[string]interface {}{"finished":true, "headers_sent":false, "status_code":200, "headers":map[string]interface {}{"Set-Cookie":[]interface {}{"session-expiry=1557912328359; Path=/", "server-time=1557905128359; Path=/", "session-expiry=1557912328359; Path=/", "server-time=1557905128359; Path=/"}}}, "system":common.MapStr{"architecture":"amd64", "platform":"Windows 10", "ip":"172.21.3.96", "hostname":"PCWrRe"}, "process":common.MapStr{"pid":14624, "title":"C:\\PROGRA~1\\AMS\\ANALYS~1\\Pentaho\\java\\bin\\java.exe"}, "service":common.MapStr{"runtime":common.MapStr{"version":"1.8.0_162", "name":"Java"}, "name":"Analysis-CI_pcwrre", "agent":common.MapStr{"name":"java", "version":"1.6.1"}, "version":"1.04 beta 1 (2743)", "language":common.MapStr{"name":"Java", "version":"1.8.0_162"}}}, "trace":common.MapStr{"id":"837905dc41ec7a7fd2bec94c1b953678"}}, Private:interface {}(nil)}, Flags:0x1} (status=400): {"type":"mer_parsing_exception","reason":"object ming for [context.request.body] tried to parse field [body] as object, but found a concrete value"}
2

Hi,

Unfortunately, that looks like a bug in the APM Server. The best way forward for you is to upgrade to 7.0, which should have it fixed.

If you do so, pay special attention to breaking changes and compatibility requirements:

https://www.elastic.co/guide/en/apm/get-started/current/breaking-7.0.0.html
https://www.elastic.co/guide/en/apm/server/7.1/upgrading-to-70.html

I hope that helps,

Juan

3

Thx, i will try it.

Some Question aside, is there a fast way to bulk reindex the apm-indices then?

4

I upgraded to 7.0.1 but the problem still exists. Now i get this message and cant get the mapping to work again.

https://0bin.net/paste/hiETVba+C3KAQ38M#rahDs5DWv+UIVaQHpw5D8Q5ocJc4mrl6e8fQh+AooNB

5

Hi, I missed your previous message, sorry for that (there is no way to do bulk reindexing, you have to do it one by one).

So, you did reindex your 6.x indices to 7.0.1, and still getting that error in the new indices?
I assuming that apm-server is also 7.0.1, right?
Can you run ./apm-server -e setup template and show me the apm template again?

Juan

6

The APM-Server is also 7.0.1. I tried it with the reindexed indices and with the newly created from the apm-server.

The template is this (i changed "number_of_replicas" : "0" and dynamic to true "http" : { "dynamic" : true, )

https://0bin.net/paste/sraG4Hqb7e6p0A+H#Ng15mfurpphbfnoZqyLrZ3d1lGSRfWdNuqQ9bEWZoPu

7

Hi,

"http: dynamic" should be set to false, otherwise Elasticsearch will try to guess the data type from the first ingested event, which we don't want because it can be either a string or an object.

8

Hi,

ok, but how do I get new fields from http in ES automaticaly then?

And why is it that even when i refresh the index pattern in Kibana, the fields show up as "no cached mapping..."?

thx

9

Agreed on your points @gruselglatz. I think dynamic should be used if we want to add extra/custom fields on the predefined ECS fields, otherwise we get the "concrete value" exception. That is what I experience myself (Elastic Stack Docker 7.0.1 and 7.1.0).

As to refreshing the index pattern, it can get a bit delayed I guess. I've experienced on my PoC instance (Docker on a Mac Book Pro with 32 GB RAM) as much as 30 second delay to effect the index refresh on a 5-node ES cluster with under 100K records, 3 shards and 2 replicas.

10

Thx for your Answer @digitalron but my fields arent cached in any time. Are all your http. fields cached? Or any http.request. field? See this topic here Cant get Mapping to work

11

Hello,

What new fields are you trying to index? In your bin I don't see any we don't define already. This is the default mapping for 7.0:

./apm-server export template | jq [.mappings.properties.http]

[
  {
    "dynamic": false,
    "properties": {
      "request": {
        "properties": {
          "body": {
            "properties": {
              "bytes": {
                "type": "long"
              },
              "content": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "bytes": {
            "type": "long"
          },
          "headers": {
            "enabled": false,
            "type": "object"
          },
          "method": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "referrer": {
            "ignore_above": 1024,
            "type": "keyword"
          }
        }
      },
      "response": {
        "properties": {
          "body": {
            "properties": {
              "bytes": {
                "type": "long"
              },
              "content": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          },
          "bytes": {
            "type": "long"
          },
          "finished": {
            "type": "boolean"
          },
          "headers": {
            "enabled": false,
            "type": "object"
          },
          "status_code": {
            "type": "long"
          }
        }
      },
      "version": {
        "ignore_above": 1024,
        "type": "keyword"
      }
    }
  }
]

You can try to ingest 2 events, 1 with an object body and 1 with a string body:
curl -v -H "Content-Type: application/x-ndjson" -X POST http://localhost:8200/intake/v2/events --data-binary @<(curl -s https://raw.githubusercontent.com/elastic/apm-server/v7.0.1/testdata/intake-v2/transactions.ndjson)

curl -v -H "Content-Type: application/x-ndjson" -X POST http://localhost:8200/intake/v2/events --data-binary @<(curl -s https://raw.githubusercontent.com/elastic/apm-server/v7.0.1/testdata/intake-v2/errors.ndjson)

And then check Elasticsearch:
curl -H "Content-Type: application/json" -X POST "http://localhost:9200/apm*/_search?pretty&filter_path=hits.hits._source.http.request.body"

{
  "hits" : {
    "hits" : [
      {
        "_source" : {
          "http" : {
            "request" : {
              "body" : {
                "original" : "Hello World"
              }
            }
          }
        }
      },
      {
        "_source" : {
          "http" : {
            "request" : {
              "body" : {
                "original" : {
                  "additional" : {
                    "req" : "additional information",
                    "bar" : 123
                  },
                  "str" : "hello world"
                }
              }
            }
          }
        }
      }
    ]
  }
}

So the default template should work for you unless you have other different requirement.

Does that help?

Juan

12

Here is a sample discovery result:

34
Screen Shot 2019-05-29 at 14.22.34.png1110×501 64.6 KB

In the template file, I added references to the http fields, explicit on the numeric and using wildcards for anything that can be safely guessed as a string. I also enabled the headers in the sections for both response and request:

 "headers": {
    "enabled": true,
    "type": "object"
  }

The initial results were not indexed but I just refreshed on the index management page.

13

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.