Error Indexing Logs generated by APM

I'm receiving the following error, whether I'm monitoring this application with a filebeat daemonset deployed to Kubernetes using autodiscovery or with Elastic Agent and the Kubernetes integration:

Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.November, 3, 13, 10, 44, 213072493, time.Location(\"\")), Meta: {...event data trimmed ...}, Private:(*input_logfile.updateOp)(0xc0052895f0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"mapper [scope.server] cannot be changed from type [keyword] to [long]\"}, dropping event!

The following is the event, cleaned up and prettified in JSON format. While there is no actual scope.server object, there is a scope.headers.server object, but that's an array.

These logs are being generated by APM, and there is no additional processing happening to them, other than being converted from the NDJSON object. I assume that a scope.server field is being injected into the event somewhere, but I have no idea where.

{
    "agent":
    {
        "ephemeral_id": "7a886739-dd57-4022-9b19-c31c09dcc652",
        "id": "cb047285-c25c-404d-afc8-d7eea65b906d",
        "name": "my-elastic-agent-hvphf",
        "type": "filebeat",
        "version": "8.5.0"
    },
    "cloud":
    {
        ...
    },
    "container":
    {
        ...
    },
    "data_stream":
    {
        "dataset": "kubernetes.container_logs",
        "namespace": "default",
        "type": "logs"
    },
    "ecs":
    {
        "version": "1.6.0"
    },
    "elastic_agent":
    {
        "id": "cb047285-c25c-404d-afc8-d7eea65b906d",
        "snapshot": false,
        "version": "8.5.0"
    },
    "event":
    {
        "dataset": "kubernetes.container_logs"
    },
    "host":
    {
        ...
    },
    "input":
    {
        "type": "filestream"
    },
    "kubernetes":
    {
        ...
    },
    "log":
    {
        "file":
        {
            "path": "/var/log/containers/my-app-54765bd557-lsghd_my-app_my-app-cb8c91bbe89a0f5b2a20d6401ae97f38ede923ddc41dc64e7468b348f3ed330e.log"
        },
        "flags":
        [
            "multiline"
        ],
        "logger": "elasticapm.transport.http",
        "offset": 9497849
    },
    "log.level": "debug",
    "message": "Sent request, url=https://apm.example.com:8200/intake/v2/events size=1.23kb status=202",
    "scope":
    {
        "app": "\\u003cfastapi.applications.FastAPI object at 0x7feafd9b1e80\\u003e",
        "asgi":
        {
            "spec_version": "2.1",
            "version": "3.0"
        },
        "client":
        [
            "127.0.0.6",
            60693
        ],
        "endpoint": "\\u003cfunction health_check at 0x7feafc372a60\\u003e",
        "fastapi_astack": "\\u003ccontextlib.AsyncExitStack object at 0x7feafc0682b0\\u003e",
        "headers":
        [
            [
                "b'host'",
                "b'10.1.2.3:80'"
            ],
            [
                "b'user-agent'",
                "b'kube-probe/1.21+'"
            ],
            [
                "b'accept'",
                "b'*/*'"
            ],
            [
                "b'connection'",
                "b'close'"
            ],
            [
                "b'accept-encoding'",
                "b'gzip'"
            ]
        ],
        "http_version": "1.1",
        "method": "GET",
        "path": "/status",
        "query_string": "b''",
        "raw_path": "b'/status'",
        "root_path": "",
        "router": "\\u003cfastapi.routing.APIRouter object at 0x7feafda2b340\\u003e",
        "scheme": "http",
        "server":
        [
            "10.1.2.3",
            80
        ],
        "type": "http"
    },
    "service":
    {
        "name": "my-app"
    },
    "span":
    {
        "id": "6a78464c27819a72"
    },
    "status_code": 200,
    "stream": "stderr",
    "trace":
    {
        "id": "b84a2ece988145a4e1c0991b900da761"
    },
    "transaction":
    {
        "id": "77f19eccbeaedb68"
    }
}

Update

I'm temporarily deploying an Elasticsearch pipeline to remove the scope.server field. Now I'm getting the same error on scope.client. Temporarily removing that as well.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.