Failed to parse field [msg.RequestPort] of type [long]

Elastic Observability Logs
1

Filebeat cannot parse and drop the logs when receives the different type of input.
"RequestPort":"-" (it usually contains the long type number)

Error:

{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [msg.RequestPort] of type [long] in document with id 'DOCUMENT_ID'. Preview of field's value: '-'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"-\\\"\"}}, dropping event!"
2

Hi @ayarosh,

this looks like the target index in Elasticsearch doesn't have a well-defined mapping for the msg.RequestPort field. This caused Elasticsearch to guess it as long based on the first document it received, which contained that field. If you want to be able to ingest the string "-" as a valid value, you'd have to define the field to be of the keyword type in the index template that is applied to the index.

3

I'm using the ingest pipeline for parsing json logs.

{
  "processors": [
    {
      "json": {
        "field": "message",
        "target_field": "msg"
      }
    },
    {
      "date_index_name": {
        "field": "@timestamp",
        "index_name_prefix": "index",
        "date_rounding": "d"
      }
    }
  ]
}

Is there any way to replace the dash with zero value?

4

Yes, the set processor can set a value conditionally if the appropriate if property is given. In your case it could be something like (careful, untested :innocent:):

{
  "processors": [
    {
      "json": {
        "field": "message",
        "target_field": "msg"
      }
    },
    {
      "date_index_name": {
        "field": "@timestamp",
        "index_name_prefix": "index",
        "date_rounding": "d"
      }
    },
    {
      "set": {
        "field": "msg.RequestPort",
        "value": 0,
        "if": "ctx?.msg?.RequestPort == '-'"
      }
    }
  ]
}

Hope this helps?

1 Like
5

Thanks Felix!
I solved my problem; the new logs is coming with 0 port.
I'm not losing them anymore.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.