Tried to parse field [body] as object, but found a concrete value

server
java
#1

Kibana version:
Kibana 6.7.2

Elasticsearch version:
6.7.2

APM Server version:
6.7.2

APM Agent language and version:
Java 1.6.1

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
apt, ubuntu, template from kibana setup script

Fresh install or upgraded from other version?
fresh

Is there anything special in your setup?
APM-Agent connects directly to the APM-Server and the APM-Server directly to ES.

Template:

https://0bin.net/paste/xfQkjsWAI36+HjE-#R3XMRwZNoXKKxm1ubDpSqqJa1d2cxjdw7cQDImdLMzK

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Sometimes when I login to my application with springframework i get this error, and the transaction isn't indexed. But sometimes its working as expected.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
APM-Server Logs:

2019-05-15T09:25:28.633+0200    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x10366400, ext:63693501928, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"trace":common.MapStr{"id":"50b1d249ff2167f32e100668120cd808"}, "timestamp":common.MapStr{"us":1557905128272000}, "beat":common.MapStr{"name":"ESAMS01", "hostname":"ESAMS01", "version":"6.7.2"}, "host":common.MapStr{"name":"ESAMS01"}, "processor":common.MapStr{"name":"transaction", "event":"transaction"}, "transaction":common.MapStr{"duration":common.MapStr{"us":18626}, "type":"request", "result":"HTTP 2xx", "sampled":true, "span_count":common.MapStr{"dropped":common.MapStr{"total":0}, "started":4279}, "id":"ca835bc231e919e8", "name":"ThemeServlet#doGet"}, "context":common.MapStr{"user":map[string]interface {}{"username":"admin"}, "request":map[string]interface {}{"body":"[REDACTED]", "url":map[string]interface {}{"port":"9080", "pathname":"/analysis-ci/js/themes.js", "search":"v=2817", "protocol":"http", "full":"http://localhost:9080/analysis-ci/js/themes.js?v=2817", "hostname":"localhost"}, "socket":map[string]interface {}{"encrypted":false, "remote_address":"0:0:0:0:0:0:0:1"}, "http_version":"1.1", "method":"GET", "headers":map[string]interface {}{"accept-language":"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7", "host":"localhost:9080", "connection":"keep-alive", "user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) leWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36", "accept":"*/*", "referer":"http://localhost:9080/analysis-ci/index.jsp", "accept-encoding":"gzip, deflate, br"}, "cookies":map[string]interface {}{"JSESSIONID":[]interface {}{"5325DFBAC6BDE1A9D6C1A3A87B904216", "E1E6D76B0D74DA45401664DFDE5928ED"}, "session-expiry":"1557912328231", "server-time":"1557905128231", "session-flushed":"true", "client-time-offset":"17"}}, "response":map[string]interface {}{"headers":map[string]interface {}{"Cache-Control":"no-cache", "Set-Cookie":[]interface {}{"session-expiry=1557912328274; Path=/", "server-time=1557905128274; Path=/", "session-expiry=1557912328274; Path=/", "server-time=1557905128274; Path=/"}}, "finished":true, "headers_sent":false, "status_code":200}, "system":common.MapStr{"hostname":"PCWrRe", "architecture":"amd64", "platform":"Windows 10", "ip":"172.21.3.96"}, "process":common.MapStr{"pid":14624, "title":"C:\\PROGRA~1\\AMS\\ANALYS~1\\Pentaho\\java\\bin\\java.exe"}, "service":common.MapStr{"runtime":common.MapStr{"name":"Java", "version":"1.8.0_162"}, "name":"Analysis-CI_pcwrre", "agent":common.MapStr{"name":"java", "version":"1.6.1"}, "version":"1.04 beta 1 (2743)", "language":common.MapStr{"name":"Java", "version":"1.8.0_162"}}}}, Private:interface {}(nil)}, Flags:0x1} (status=400): {"type":"mer_parsing_exception","reason":"object ming for [context.request.body] tried to parse field [body] as object, but found a concrete value"}

2019-05-15T09:25:29.166+0200    WARN    elasticsearch/client.go:539     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x1556a580, ext:63693501928, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"timestamp":common.MapStr{"us":1557905128358000}, "beat":common.MapStr{"name":"ESAMS01", "hostname":"ESAMS01", "version":"6.7.2"}, "host":common.MapStr{"name":"ESAMS01"}, "processor":common.MapStr{"name":"transaction", "event":"transaction"}, "transaction":common.MapStr{"result":"HTTP 2xx", "sampled":true, "span_count":common.MapStr{"dropped":common.MapStr{"total":0}, "started":1460}, "id":"d0582b39e646f5c2", "name":"Mantle_jsp#doGet", "duration":common.MapStr{"us":15820}, "type":"request"}, "context":common.MapStr{"user":map[string]interface {}{"username":"admin"}, "request":map[string]interface {}{"method":"GET", "headers":map[string]interface {}{"upgrade-insecure-requests":"1", "user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) leWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36", "accept":"text/html,lication/xhtml+xml,lication/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,lication/signed-exchange;v=b3", "referer":"http://localhost:9080/analysis-ci/index.jsp", "accept-encoding":"gzip, deflate, br", "accept-language":"de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7", "host":"localhost:9080", "connection":"keep-alive"}, "cookies":map[string]interface {}{"session-flushed":"true", "client-time-offset":"17", "JSESSIONID":[]interface {}{"5325DFBAC6BDE1A9D6C1A3A87B904216", "E1E6D76B0D74DA45401664DFDE5928ED"}, "session-expiry":"1557912328308", "server-time":"1557905128308"}, "body":"[REDACTED]", "url":map[string]interface {}{"full":"http://localhost:9080/analysis-ci/Home", "hostname":"localhost", "port":"9080", "pathname":"/analysis-ci/Home", "protocol":"http"}, "socket":map[string]interface {}{"encrypted":false, "remote_address":"0:0:0:0:0:0:0:1"}, "http_version":"1.1"}, "response":map[string]interface {}{"finished":true, "headers_sent":false, "status_code":200, "headers":map[string]interface {}{"Set-Cookie":[]interface {}{"session-expiry=1557912328359; Path=/", "server-time=1557905128359; Path=/", "session-expiry=1557912328359; Path=/", "server-time=1557905128359; Path=/"}}}, "system":common.MapStr{"architecture":"amd64", "platform":"Windows 10", "ip":"172.21.3.96", "hostname":"PCWrRe"}, "process":common.MapStr{"pid":14624, "title":"C:\\PROGRA~1\\AMS\\ANALYS~1\\Pentaho\\java\\bin\\java.exe"}, "service":common.MapStr{"runtime":common.MapStr{"version":"1.8.0_162", "name":"Java"}, "name":"Analysis-CI_pcwrre", "agent":common.MapStr{"name":"java", "version":"1.6.1"}, "version":"1.04 beta 1 (2743)", "language":common.MapStr{"name":"Java", "version":"1.8.0_162"}}}, "trace":common.MapStr{"id":"837905dc41ec7a7fd2bec94c1b953678"}}, Private:interface {}(nil)}, Flags:0x1} (status=400): {"type":"mer_parsing_exception","reason":"object ming for [context.request.body] tried to parse field [body] as object, but found a concrete value"}
(Juan Alvarez) #2

Hi,

Unfortunately, that looks like a bug in the APM Server. The best way forward for you is to upgrade to 7.0, which should have it fixed.

If you do so, pay special attention to breaking changes and compatibility requirements:

https://www.elastic.co/guide/en/apm/get-started/current/breaking-7.0.0.html
https://www.elastic.co/guide/en/apm/server/7.1/upgrading-to-70.html

I hope that helps,

Juan

#3

Thx, i will try it.

Some Question aside, is there a fast way to bulk reindex the apm-indices then?