I'm receiving the following error, whether I'm monitoring this application with a filebeat daemonset deployed to Kubernetes using autodiscovery or with Elastic Agent and the Kubernetes integration:
Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.November, 3, 13, 10, 44, 213072493, time.Location(\"\")), Meta: {...event data trimmed ...}, Private:(*input_logfile.updateOp)(0xc0052895f0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"mapper [scope.server] cannot be changed from type [keyword] to [long]\"}, dropping event!
The following is the event, cleaned up and prettified in JSON format. While there is no actual scope.server
object, there is a scope.headers.server
object, but that's an array.
These logs are being generated by APM, and there is no additional processing happening to them, other than being converted from the NDJSON object. I assume that a scope.server
field is being injected into the event somewhere, but I have no idea where.
{
"agent":
{
"ephemeral_id": "7a886739-dd57-4022-9b19-c31c09dcc652",
"id": "cb047285-c25c-404d-afc8-d7eea65b906d",
"name": "my-elastic-agent-hvphf",
"type": "filebeat",
"version": "8.5.0"
},
"cloud":
{
...
},
"container":
{
...
},
"data_stream":
{
"dataset": "kubernetes.container_logs",
"namespace": "default",
"type": "logs"
},
"ecs":
{
"version": "1.6.0"
},
"elastic_agent":
{
"id": "cb047285-c25c-404d-afc8-d7eea65b906d",
"snapshot": false,
"version": "8.5.0"
},
"event":
{
"dataset": "kubernetes.container_logs"
},
"host":
{
...
},
"input":
{
"type": "filestream"
},
"kubernetes":
{
...
},
"log":
{
"file":
{
"path": "/var/log/containers/my-app-54765bd557-lsghd_my-app_my-app-cb8c91bbe89a0f5b2a20d6401ae97f38ede923ddc41dc64e7468b348f3ed330e.log"
},
"flags":
[
"multiline"
],
"logger": "elasticapm.transport.http",
"offset": 9497849
},
"log.level": "debug",
"message": "Sent request, url=https://apm.example.com:8200/intake/v2/events size=1.23kb status=202",
"scope":
{
"app": "\\u003cfastapi.applications.FastAPI object at 0x7feafd9b1e80\\u003e",
"asgi":
{
"spec_version": "2.1",
"version": "3.0"
},
"client":
[
"127.0.0.6",
60693
],
"endpoint": "\\u003cfunction health_check at 0x7feafc372a60\\u003e",
"fastapi_astack": "\\u003ccontextlib.AsyncExitStack object at 0x7feafc0682b0\\u003e",
"headers":
[
[
"b'host'",
"b'10.1.2.3:80'"
],
[
"b'user-agent'",
"b'kube-probe/1.21+'"
],
[
"b'accept'",
"b'*/*'"
],
[
"b'connection'",
"b'close'"
],
[
"b'accept-encoding'",
"b'gzip'"
]
],
"http_version": "1.1",
"method": "GET",
"path": "/status",
"query_string": "b''",
"raw_path": "b'/status'",
"root_path": "",
"router": "\\u003cfastapi.routing.APIRouter object at 0x7feafda2b340\\u003e",
"scheme": "http",
"server":
[
"10.1.2.3",
80
],
"type": "http"
},
"service":
{
"name": "my-app"
},
"span":
{
"id": "6a78464c27819a72"
},
"status_code": 200,
"stream": "stderr",
"trace":
{
"id": "b84a2ece988145a4e1c0991b900da761"
},
"transaction":
{
"id": "77f19eccbeaedb68"
}
}
Update
I'm temporarily deploying an Elasticsearch pipeline to remove the scope.server
field. Now I'm getting the same error on scope.client
. Temporarily removing that as well.