Netflow.flow_id data type incorrectly set to long instead of unsigned-long

Elastic Stack Beats
1

In /etc/filebeats/fields.yaml file, netflow.flow_id is set to type long.

        - name: flow_id
          type: long

As per NetFlow standard, it should be unsigned long.

Because of this, events are getting dropped with error:

(status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:1477] failed to parse field [netflow.flow_id] of type [long] in document with id 'Lba4XZYB_Zr7jpsbS6pn'. 
Preview of field's value: '11133470846251699209'\",\"caused_by\":{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:1497] Numeric value (11133470846251699209) out of range of long (-9223372036854775808 - 9223372036854775807)\\n at [Source

Is it possible to fix it? Do we have any workaround for this?

2

Hi @vjineo Welcome to the community.

What version are you on?

Looks like a duplicate of this... Please be patient new users it take a few minutes for topics to show up. I will close that one.

Ohhh I think you will need to set that to keyword, I see on the other modules it is set to keyword

        - name: flow_id
          type: keyword

It might be worth opening an Issue....

3

Thanks @stephenb for the quick reply. I'm on 8.18 version

ubuntu@ubuntu-ELK:~$ filebeat version
filebeat version 8.18.0 (amd64), libbeat 8.18.0 [b907b0fa1e1b7400d4e222d6b6ca9d1b77c08d97 built 2025-04-09 17:09:00 +0000 UTC]
ubuntu@ubuntu-ELK:~$

I'll raise an issue in the Github page that you shared, thanks for confirming.

In the meantime, do we have any workaround for this? I was going through this page:

Thought of taking backup like this:

(venv) ubuntu@ubuntu-ELK:~$ sudo filebeat export template > filebeat.template.json
(venv) ubuntu@ubuntu-ELK:~$

Modify the type to keyword in the template.json and figure out a way to force filebeat to use this.

P.S: Apologies for the double post. I thought I posted in wrong category and then tried to fix it.

4

The how to load is just a few lines lower

filebeat export template > filebeat.template.json
<fix>
curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_index_template/filebeat-8.18.0 -d@filebeat.template.json

Try that... you can also just go into Kibana and manually fix the template...

5

That worked, thanks a lot @stephenb.

Initially I faced below issue and I couldn't figure out how to solve that:

ubuntu@ubuntu-ELK:~$ curl -XPUT http://localhost:9200/_data_stream/filebeat-8.18.0
{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"data_stream [filebeat-8.18.0] already exists"}],"type":"resource_already_exists_exception","reason":"data_stream [filebeat-8.18.0] already exists"},"status":400}ubuntu@ubuntu-ELK:~$

Ended up bringing up fresh instance and it worked. Used below sequence (filebeat started last):

ubuntu@ubuntu-ELK:~$ sudo sudo filebeat export template > filebeat.template.json
ubuntu@ubuntu-ELK:~$ vim filebeat.template.json
ubuntu@ubuntu-ELK:~$ curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_index_template/filebeat-8.18.0 -d@filebeat.template.json
{"acknowledged":true}ubuntu@ubuntu-ELK:~$
ubuntu@ubuntu-ELK:~$ sudo filebeat modules enable netflow
Enabled netflow
ubuntu@ubuntu-ELK:~$ sudo sed -i 's/preset: balanced/preset: throughput/g' /etc/filebeat/filebeat.yml
ubuntu@ubuntu-ELK:~$ sudo sed -i 's/enabled: false/enabled: true/g' /etc/filebeat/modules.d/netflow.yml
ubuntu@ubuntu-ELK:~$ sudo sed -i 's/netflow_host: localhost/netflow_host: 0.0.0.0/g' /etc/filebeat/modules.d/netflow.yml
ubuntu@ubuntu-ELK:~$ sudo sed -i 's/netflow_port: 2055/&\n      max_message_size: 40KiB/' /etc/filebeat/modules.d/netflow.yml
ubuntu@ubuntu-ELK:~$
ubuntu@ubuntu-ELK:~$ sudo systemctl start filebeat
ubuntu@ubuntu-ELK:~$
ubuntu@ubuntu-ELK:~$ curl -XPUT http://localhost:9200/_data_stream/filebeat-8.18.0
{"acknowledged":true}ubuntu@ubuntu-ELK:~$
ubuntu@ubuntu-ELK:~$
1 Like
6

I've ILM policy configured and when rollover happened, it changed back to "long".

Verified using below command:

GET filebeat-*/_mapping/field/netflow.flow_id

 {
  ".ds-filebeat-8.18.0-2025.04.23-000004": {
    "mappings": {
      "netflow.flow_id": {
        "full_name": "netflow.flow_id",
        "mapping": {
          "flow_id": {
            "type": "long"
          }
        }
      }
    }
  }
}

Could someone guide me in handling this?

7

Made the following changes and it seems to work after rollover now:

-Edit '/etc/filebeat/filebeat.yml' and add setting 'setup.template.enabled: false'
-Executed this

curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_index_template/filebeat-8.18.0 -d@filebeat.template.json

And this:

curl -X DELETE "http://localhost:9200/_data_stream/filebeat-8.18.0"

And finally this:

curl -XPUT http://localhost:9200/_data_stream/filebeat-8.18.0

Now it is good:

{
  ".ds-filebeat-8.18.0-2025.04.23-000002": {
    "mappings": {
      "netflow.flow_id": {
        "full_name": "netflow.flow_id",
        "mapping": {
          "flow_id": {
            "type": "keyword"
          }
        }
      }
    }
  },
  ".ds-filebeat-8.18.0-2025.04.23-000001": {
    "mappings": {
      "netflow.flow_id": {
        "full_name": "netflow.flow_id",
        "mapping": {
          "flow_id": {
            "type": "keyword"
          }
        }
      }
    }
  }
}