Error Indexing Logs generated by APM

[DougR]

I'm receiving the following error, whether I'm monitoring this application with a filebeat daemonset deployed to Kubernetes using autodiscovery or with Elastic Agent and the Kubernetes integration:

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.November, 3, 13, 10, 44, 213072493, time.Location(\"\")), Meta: {...event data trimmed ...}, Private:(*input_logfile.updateOp)(0xc0052895f0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"mapper [scope.server] cannot be changed from type [keyword] to [long]\"}, dropping event!

The following is the event, cleaned up and prettified in JSON format. While there is no actual scope.server object, there is a scope.headers.server object, but that's an array.

These logs are being generated by APM, and there is no additional processing happening to them, other than being converted from the NDJSON object. I assume that a scope.server field is being injected into the event somewhere, but I have no idea where.

{
    "agent":
    {
        "ephemeral_id": "7a886739-dd57-4022-9b19-c31c09dcc652",
        "id": "cb047285-c25c-404d-afc8-d7eea65b906d",
        "name": "my-elastic-agent-hvphf",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "cloud":
    {
        ...
    },
    "container":
    {
        ...
    },
    "data_stream":
    {
        "dataset": "kubernetes.container_logs",
        "namespace": "default",
        "type": "logs"
    },
    "ecs":
    {
        "version": "1.6.0"
    },
    "elastic_agent":
    {
        "id": "cb047285-c25c-404d-afc8-d7eea65b906d",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event":
    {
        "dataset": "kubernetes.container_logs"
    },
    "host":
    {
        ...
    },
    "input":
    {
        "type": "filestream"
    },
    "kubernetes":
    {
        ...
    },
    "log":
    {
        "file":
        {
            "path": "/var/log/containers/my-app-54765bd557-lsghd_my-app_my-app-cb8c91bbe89a0f5b2a20d6401ae97f38ede923ddc41dc64e7468b348f3ed330e.log"
        },
        "flags":
        [
            "multiline"
        ],
        "logger": "elasticapm.transport.http",
        "offset": 9497849
    },
    "log.level": "debug",
    "message": "Sent request, url=https://apm.example.com:8200/intake/v2/events size=1.23kb status=202",
    "scope":
    {
        "app": "\\u003cfastapi.applications.FastAPI object at 0x7feafd9b1e80\\u003e",
        "asgi":
        {
            "spec_version": "2.1",
            "version": "3.0"
        },
        "client":
        [
            "127.0.0.6",
            60693
        ],
        "endpoint": "\\u003cfunction health_check at 0x7feafc372a60\\u003e",
        "fastapi_astack": "\\u003ccontextlib.AsyncExitStack object at 0x7feafc0682b0\\u003e",
        "headers":
        [
            [
                "b'host'",
                "b'10.1.2.3:80'"
            ],
            [
                "b'user-agent'",
                "b'kube-probe/1.21+'"
            ],
            [
                "b'accept'",
                "b'*/*'"
            ],
            [
                "b'connection'",
                "b'close'"
            ],
            [
                "b'accept-encoding'",
                "b'gzip'"
            ]
        ],
        "http_version": "1.1",
        "method": "GET",
        "path": "/status",
        "query_string": "b''",
        "raw_path": "b'/status'",
        "root_path": "",
        "router": "\\u003cfastapi.routing.APIRouter object at 0x7feafda2b340\\u003e",
        "scheme": "http",
        "server":
        [
            "10.1.2.3",
            80
        ],
        "type": "http"
    },
    "service":
    {
        "name": "my-app"
    },
    "span":
    {
        "id": "6a78464c27819a72"
    },
    "status_code": 200,
    "stream": "stderr",
    "trace":
    {
        "id": "b84a2ece988145a4e1c0991b900da761"
    },
    "transaction":
    {
        "id": "77f19eccbeaedb68"
    }
}

Update

I'm temporarily deploying an Elasticsearch pipeline to remove the scope.server field. Now I'm getting the same error on scope.client. Temporarily removing that as well.