Failed to parse field [msg.RequestPort] of type [long]

[ayarosh (Artur Yarosh)]

Filebeat cannot parse and drop the logs when receives the different type of input.
"RequestPort":"-" (it usually contains the long type number)

Error:

{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [msg.RequestPort] of type [long] in document with id 'DOCUMENT_ID'. Preview of field's value: '-'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"For input string: \\\"-\\\"\"}}, dropping event!"

[weltenwort (Felix Stürmer)]

Hi @ayarosh,

this looks like the target index in Elasticsearch doesn't have a well-defined mapping for the msg.RequestPort field. This caused Elasticsearch to guess it as long based on the first document it received, which contained that field. If you want to be able to ingest the string "-" as a valid value, you'd have to define the field to be of the keyword type in the index template that is applied to the index.

[ayarosh (Artur Yarosh)]

I'm using the ingest pipeline for parsing json logs.

{
  "processors": [
    {
      "json": {
        "field": "message",
        "target_field": "msg"
      }
    },
    {
      "date_index_name": {
        "field": "@timestamp",
        "index_name_prefix": "index",
        "date_rounding": "d"
      }
    }
  ]
}

Is there any way to replace the dash with zero value?

[weltenwort (Felix Stürmer)]

Yes, the set processor can set a value conditionally if the appropriate if property is given. In your case it could be something like (careful, untested ):

{
  "processors": [
    {
      "json": {
        "field": "message",
        "target_field": "msg"
      }
    },
    {
      "date_index_name": {
        "field": "@timestamp",
        "index_name_prefix": "index",
        "date_rounding": "d"
      }
    },
    {
      "set": {
        "field": "msg.RequestPort",
        "value": 0,
        "if": "ctx?.msg?.RequestPort == '-'"
      }
    }
  ]
}

Hope this helps?

[ayarosh (Artur Yarosh)]

Thanks Felix!
I solved my problem; the new logs is coming with 0 port.
I'm not losing them anymore.