# Mapping Netflow Data

**URL:** <https://discuss.elastic.co/t/mapping-netflow-data/1411>\
**Category:** Elasticsearch\
**Created:** [May 27, 2015, 5:16pm UTC](https://discuss.elastic.co/t/mapping-netflow-data/1411 "2015-05-27T17:16:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ginja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ginja/32/418_2.png) [@Ginja](https://discuss.elastic.co/u/Ginja)\
**Post date:** [May 27, 2015, 5:16pm UTC](https://discuss.elastic.co/t/mapping-netflow-data/1411/1 "2015-05-27T17:16:27Z")

</div>

Hi All,

Just some context, I originally raised [this issue](https://github.com/elastic/kibana/issues/3994) in Kibana's issue tracker, but was told that it's an Elasticsearch issue and that it would be best to raise it here.

When attempting to view my Netflow data using Kibana 4.0.2 (Build 6004), I get the following warning in a yellow banner:

```auto
Courier Fetch: 5 of 5 shards failed.

```

Using Chrome's Developer tools I pulled out the query, payload, and response. All of which are below.

I can view this data in Kibana 4 if I let Elasticsearch create a mapping dynamically for this index, but I would like to use a custom one so that it's optimized. Also, the generated template produces a lot of parsing errors in Elasticsearch's logs (Numeric value out of range of long, etc...). I've included the mapping I'm trying to use below, and I don't see anything wrong with it. Netflow data includes bigger numbers than `"type": "long"` can handle, so I needed to use `"type": "string"` for some fields.

Request

```bash
curl -XPOST http://fqdn.omitted.com:5601/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1432705287095

```

Request Payload

```json
{
  "index": "customindex-*",
  "ignore_unavailable": true
}\n
{
  "size": 500,
  "sort": {
    "@timestamp": "desc"
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    }
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "30s",
        "pre_zone": "-07:00",
        "pre_zone_adjust_large_interval": true,
        "min_doc_count": 0,
        "extended_bounds": {
          "min": 1432704390127,
          "max": 1432705290128
        }
      }
    }
  },
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gte": 1432704390134,
                  "lte": 1432705290134
                }
              }
            }
          ],
          "must_not": []
        }
      }
    }
  },
  "fields": [
    "*",
    "_source"
  ],
  "script_fields": {},
  "fielddata_fields": [
    "@timestamp"
  ]
}

```

Response

```json
{
  "responses": [
    {
      "took": 44,
      "timed_out": false,
      "_shards": {
        "total": 5,
        "successful": 0,
        "failed": 5,
        "failures": [
          {
            "index": "customindex-2015.05.27",
            "shard": 0,
            "status": 500,
            "reason": "RemoteTransportException[[fqdn.omitted.com][inet[/192.168.1.110:9300]][indices:data/read/search[phase/fetch/id]]]; nested: ElasticsearchIllegalStateException[No matching token for number_type [BIG_INTEGER]]; "
          },
          {
            "index": "customindex-2015.05.27",
            "shard": 1,
            "status": 500,
            "reason": "RemoteTransportException[[fqdn.omitted.com][inet[/192.168.1.110:9300]][indices:data/read/search[phase/fetch/id]]]; nested: ElasticsearchIllegalStateException[No matching token for number_type [BIG_INTEGER]]; "
          },
          {
            "index": "customindex-2015.05.27",
            "shard": 2,
            "status": 500,
            "reason": "RemoteTransportException[[fqdn.omitted.com][inet[/192.168.1.110:9300]][indices:data/read/search[phase/fetch/id]]]; nested: ElasticsearchIllegalStateException[No matching token for number_type [BIG_INTEGER]]; "
          },
          {
            "index": "customindex-2015.05.27",
            "shard": 3,
            "status": 500,
            "reason": "RemoteTransportException[[fqdn.omitted.com][inet[/192.168.112.177:9300]][indices:data/read/search[phase/fetch/id]]]; nested: ElasticsearchIllegalStateException[No matching token for number_type [BIG_INTEGER]]; "
          },
          {
            "index": "customindex-2015.05.27",
            "shard": 4,
            "status": 500,
            "reason": "RemoteTransportException[[fqdn.omitted.com][inet[/192.168.1.110:9300]][indices:data/read/search[phase/fetch/id]]]; nested: ElasticsearchIllegalStateException[No matching token for number_type [BIG_INTEGER]]; "
          }
        ]
      },
      "hits": {
        "total": 86519,
        "max_score": null,
        "hits": []
      },
      "aggregations": {
        "2": {
          "buckets": [
            {
              "key_as_string": "2015-05-27T05:16:30.000Z",
              "key": 1432703790000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:17:00.000Z",
              "key": 1432703820000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:17:30.000Z",
              "key": 1432703850000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:18:00.000Z",
              "key": 1432703880000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:18:30.000Z",
              "key": 1432703910000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:19:00.000Z",
              "key": 1432703940000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:19:30.000Z",
              "key": 1432703970000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:20:00.000Z",
              "key": 1432704000000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:20:30.000Z",
              "key": 1432704030000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:21:00.000Z",
              "key": 1432704060000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:21:30.000Z",
              "key": 1432704090000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:22:00.000Z",
              "key": 1432704120000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:22:30.000Z",
              "key": 1432704150000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:23:00.000Z",
              "key": 1432704180000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:23:30.000Z",
              "key": 1432704210000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:24:00.000Z",
              "key": 1432704240000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:24:30.000Z",
              "key": 1432704270000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:25:00.000Z",
              "key": 1432704300000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:25:30.000Z",
              "key": 1432704330000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:26:00.000Z",
              "key": 1432704360000,
              "doc_count": 0
            },
            {
              "key_as_string": "2015-05-27T05:26:30.000Z",
              "key": 1432704390000,
              "doc_count": 4209
            },
            {
              "key_as_string": "2015-05-27T05:27:00.000Z",
              "key": 1432704420000,
              "doc_count": 7270
            },
            {
              "key_as_string": "2015-05-27T05:27:30.000Z",
              "key": 1432704450000,
              "doc_count": 6646
            },
            {
              "key_as_string": "2015-05-27T05:28:00.000Z",
              "key": 1432704480000,
              "doc_count": 7181
            },
            {
              "key_as_string": "2015-05-27T05:28:30.000Z",
              "key": 1432704510000,
              "doc_count": 6612
            },
            {
              "key_as_string": "2015-05-27T05:29:00.000Z",
              "key": 1432704540000,
              "doc_count": 6753
            },
            {
              "key_as_string": "2015-05-27T05:29:30.000Z",
              "key": 1432704570000,
              "doc_count": 6509
            },
            {
              "key_as_string": "2015-05-27T05:30:00.000Z",
              "key": 1432704600000,
              "doc_count": 10295
            },
            {
              "key_as_string": "2015-05-27T05:30:30.000Z",
              "key": 1432704630000,
              "doc_count": 13073
            },
            {
              "key_as_string": "2015-05-27T05:31:00.000Z",
              "key": 1432704660000,
              "doc_count": 14627
            },
            {
              "key_as_string": "2015-05-27T05:31:30.000Z",
              "key": 1432704690000,
              "doc_count": 3344
            }
          ]
        }
      }
    }
  ]
}

```

Template & Mapping

```json
{
  "template": "customindex-*",
  "settings": {
    "index.refresh_integererval": "5s",
    "index.number_of_shards": "5"
  },
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": false
      }
    },
    "netflow": {
      "properties": {
        "@timestamp": {
          "type": "date",
          "format": "dateOptionalTime"
        },
        "@version": {
          "type": "string"
        },
        "host": {
          "type": "string"
        },
        "netflow": {
          "properties": {
            "flow_seq_num": {
              "type": "long"
            },
            "flowset_id": {
              "type": "long"
            },
            "nf_f_conn_id": {
              "type": "long"
            },
            "nf_f_dst_addr_ipv4": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_dst_intf_id": {
              "type": "long"
            },
            "nf_f_dst_port": {
              "type": "long"
            },
            "nf_f_egress_acl_id": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_event_time_msec": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_flow_create_time_msec": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_fwd_flow_delta_bytes": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_rev_flow_delta_bytes": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_flow_bytes": {
              "type": "long"
            },
            "nf_f_fw_event": {
              "type": "long"
            },
            "nf_f_fw_ext_event": {
              "type": "long"
            },
            "nf_f_icmp_code": {
              "type": "long"
            },
            "nf_f_icmp_type": {
              "type": "long"
            },
            "nf_f_icmp_type_ipv6": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_icmp_code_ipv6": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_ingress_acl_id": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_protocol": {
              "type": "long"
            },
            "nf_f_src_addr_ipv4": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_src_intf_id": {
              "type": "long"
            },
            "nf_f_src_port": {
              "type": "long"
            },
            "nf_f_username": {
              "type": "string"
            },
            "nf_f_xlate_dst_addr_ipv4": {
              "type": "string"
            },
            "nf_f_xlate_dst_port": {
              "type": "long"
            },
            "nf_f_xlate_src_addr_ipv4": {
              "type": "string",
              "index": "not_analyzed"
            },
            "nf_f_xlate_src_port": {
              "type": "long"
            },
            "version": {
              "type": "long"
            }
          }
        },
        "type": {
          "type": "string"
        }
      }
    }
  }
}

```

Sample Document

```auto
{
    "@timestamp" => "2015-05-27T06:51:08.000Z",
       "netflow" => {
                         "version" => 9,
                    "flow_seq_num" => 2196973,
                      "flowset_id" => 263,
                    "nf_f_conn_id" => 21064372,
              "nf_f_src_addr_ipv4" => 2836759729,
                   "nf_f_src_port" => 51349,
                "nf_f_src_intf_id" => 15,
              "nf_f_dst_addr_ipv4" => 2866430306,
                   "nf_f_dst_port" => 80,
                "nf_f_dst_intf_id" => 14,
                   "nf_f_protocol" => 6,
                  "nf_f_icmp_type" => 0,
                  "nf_f_icmp_code" => 0,
        "nf_f_xlate_src_addr_ipv4" => 2856329729,
        "nf_f_xlate_dst_addr_ipv4" => 2856430306,
             "nf_f_xlate_src_port" => 51349,
             "nf_f_xlate_dst_port" => 80,
                   "nf_f_fw_event" => 2,
               "nf_f_fw_ext_event" => 2015,
            "nf_f_event_time_msec" => 1452309468866,
                 "nf_f_flow_bytes" => 45
    },
      "@version" => "1",
          "type" => "netflow",
          "host" => "192.168.1.100"
}

```

---

<div class="post-metadata">

**Author:** ![Ginja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ginja/32/418_2.png) [@Ginja](https://discuss.elastic.co/u/Ginja)\
**Post date:** [May 28, 2015, 1:57am UTC](https://discuss.elastic.co/t/mapping-netflow-data/1411/2 "2015-05-28T01:57:26Z")

</div>

Figured it out, or a workaround at least. I just needed to convert all/most fields to string with Logstash before I shipped off the logs to elasticsearch.

```auto
filter {
  mutate { convert => { "[netflow][version]" => "string" } }
  mutate { convert => { "[netflow][flow_seq_num]" => "string" } }
  mutate { convert => { "[netflow][flowset_id]" => "string" } }
  mutate { convert => { "[netflow][nf_f_conn_id]" => "string" } }
  mutate { convert => { "[netflow][nf_f_src_addr_ipv4]" => "string" } }
  mutate { convert => { "[netflow][nf_f_src_port]" => "string" } }
  mutate { convert => { "[netflow][nf_f_src_intf_id]" => "string" } }
  mutate { convert => { "[netflow][nf_f_dst_addr_ipv4]" => "string" } }
  mutate { convert => { "[netflow][nf_f_dst_port]" => "string" } }
  mutate { convert => { "[netflow][nf_f_dst_intf_id]" => "string" } }
  mutate { convert => { "[netflow][nf_f_protocol]" => "string" } }
  mutate { convert => { "[netflow][nf_f_icmp_type]" => "string" } }
  mutate { convert => { "[netflow][nf_f_icmp_code]" => "string" } }
  mutate { convert => { "[netflow][nf_f_xlate_src_addr_ipv4]" => "string" } }
  mutate { convert => { "[netflow][nf_f_xlate_dst_addr_ipv4]" => "string" } }
  mutate { convert => { "[netflow][nf_f_xlate_src_port]" => "string" } }
  mutate { convert => { "[netflow][nf_f_xlate_dst_port]" => "string" } }
  mutate { convert => { "[netflow][nf_f_fw_event]" => "string" } }
  mutate { convert => { "[netflow][nf_f_fw_ext_event]" => "string" } }
  mutate { convert => { "[netflow][nf_f_event_time_msec]" => "string" } }
  mutate { convert => { "[netflow][nf_f_flow_bytes]" => "string" } }
  mutate { convert => { "[netflow][nf_f_fwd_flow_delta_bytes]" => "string" } }
  mutate { convert => { "[netflow][nf_f_rev_flow_delta_bytes]" => "string" } }
  mutate { convert => { "[netflow][nf_f_flow_create_time_msec]" => "string" } }
  mutate { convert => { "[netflow][nf_f_ingress_acl_id]" => "string" } }
  mutate { convert => { "[netflow][nf_f_egress_acl_id]" => "string" } }
  mutate { convert => { "[netflow][nf_f_username]" => "string" } }
}

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 28, 2015, 9:25am UTC](https://discuss.elastic.co/t/mapping-netflow-data/1411/3 "2015-05-28T09:25:25Z")

</div>

You can do those mutates in the one call, eg;

```auto
mutate {
    convert => ["[netflow][version],string,"[netflow][flow_seq_num]",string, etc etc]
  }

```

Or you should be able to, not sure how it handle nested fields to be honest!

---

<div class="post-metadata">

**Author:** ![Ginja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ginja/32/418_2.png) [@Ginja](https://discuss.elastic.co/u/Ginja)\
**Post date:** [May 28, 2015, 7:29pm UTC](https://discuss.elastic.co/t/mapping-netflow-data/1411/4 "2015-05-28T19:29:43Z")

</div>

Thanks for the tip. This was the only way I could make it work all in one mutate block:

```auto
  mutate { 
    convert => ["[netflow][version]", "string" ]
    convert => ["[netflow][flow_seq_num]", "string" ]
    convert => ["[netflow][flowset_id]", "string" ]
    convert => ["[netflow][nf_f_conn_id]", "string" ]
    convert => ["[netflow][nf_f_src_addr_ipv4]", "string" ]
    convert => ["[netflow][nf_f_src_port]", "string" ]
    convert => ["[netflow][nf_f_src_intf_id]", "string" ]
    convert => ["[netflow][nf_f_dst_addr_ipv4]", "string" ]
    convert => ["[netflow][nf_f_dst_port]", "string" ]
    convert => ["[netflow][nf_f_dst_intf_id]", "string" ]
    convert => ["[netflow][nf_f_protocol]", "string" ]
    convert => ["[netflow][nf_f_icmp_type]", "string" ]
    convert => ["[netflow][nf_f_icmp_code]", "string" ]
    convert => ["[netflow][nf_f_xlate_src_addr_ipv4]", "string" ]
    convert => ["[netflow][nf_f_xlate_dst_addr_ipv4]", "string" ]
    convert => ["[netflow][nf_f_xlate_src_port]", "string" ]
    convert => ["[netflow][nf_f_xlate_dst_port]", "string" ]
    convert => ["[netflow][nf_f_fw_event]", "string" ]
    convert => ["[netflow][nf_f_fw_ext_event]", "string" ]
    convert => ["[netflow][nf_f_event_time_msec]", "string" ]
    convert => ["[netflow][nf_f_flow_bytes]", "string" ]
    convert => ["[netflow][nf_f_fwd_flow_delta_bytes]", "string" ]
    convert => ["[netflow][nf_f_rev_flow_delta_bytes]", "string" ]
    convert => ["[netflow][nf_f_flow_create_time_msec]", "string" ]
    convert => ["[netflow][nf_f_ingress_acl_id]", "string" ]
    convert => ["[netflow][nf_f_egress_acl_id]", "string" ]
    convert => ["[netflow][nf_f_username]", "string" ]
  }

```

---

<div class="post-metadata">

**Author:** ![Gabriel\_Rosca](https://avatars.discourse-cdn.com/v4/letter/g/e19adc/32.png) [@Gabriel\_Rosca](https://discuss.elastic.co/u/Gabriel_Rosca)\
**Post date:** [July 6, 2015, 4:54pm UTC](https://discuss.elastic.co/t/mapping-netflow-data/1411/5 "2015-07-06T16:54:19Z")

</div>

Hi Ginja,

Not sure why you want everything as sting you are losing a lot of functionality like IP range and SUM per destination port for example.

Here is my Elasticsearch template for netflow. See if that helps you.

Works fine for me ...

curl -XPUT localhost:9200/\_template/netflow -d '  
{  
"order" : 2,  
"template" : "logstash-ptc-netflow\*",  
"mappings" : {  
"netflow" : {  
"dynamic\_templates" : [ {  
"message\_field" : {  
"mapping" : {  
"index" : "analyzed",  
"omit\_norms" : true,  
"type" : "string"  
},  
"match" : "message",  
"match\_mapping\_type" : "string"  
}  
}, {  
"string\_fields" : {  
"mapping" : {  
"index" : "analyzed",  
"omit\_norms" : true,  
"type" : "string",  
"fields" : {  
"raw" : {  
"index" : "not\_analyzed",  
"ignore\_above" : 256,  
"type" : "string"  
}  
}  
},  
"match" : "\*",  
"match\_mapping\_type" : "string"  
}  
} ],  
"\_all" : {  
"enabled" : true  
},  
"properties" : {  
"@version" : {  
"type" : "string",  
"index" : "not\_analyzed",  
"doc\_values": true  
},  
"@timestamp" : {  
"type" : "date",  
"format" : "dateOptionalTime",  
"doc\_values": true  
},  
"type" : {  
"type" : "string",  
"index" : "not\_analyzed",  
"doc\_values": true  
},  
"flow\_seq\_num": {  
"index": "not\_analyzed",  
"type": "long",  
"doc\_values": true  
},  
"engine\_type": {  
"index": "not\_analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"engine\_id": {  
"index": "not\_analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"sampling\_algorithm": {  
"index": "not\_analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"sampling\_interval": {  
"index": "not\_analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"flow\_records": {  
"index": "not\_analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"ipv4\_src\_addr": {  
"index": "analyzed",  
"type": "ip",  
"doc\_values": true  
},  
"ipv4\_dst\_addr": {  
"index": "analyzed",  
"type": "ip",  
"doc\_values": true  
},  
"ipv4\_next\_hop": {  
"index": "analyzed",  
"type": "ip",  
"doc\_values": true  
},  
"input\_snmp": {  
"index": "not\_analyzed",  
"type": "long",  
"doc\_values": true  
},  
"output\_snmp": {  
"index": "not\_analyzed",  
"type": "long",  
"doc\_values": true  
},  
"in\_pkts": {  
"index": "analyzed",  
"type": "long",  
"doc\_values": true  
},  
"in\_bytes": {  
"index": "analyzed",  
"type": "long",  
"doc\_values": true  
},  
"first\_switched": {  
"index": "not\_analyzed",  
"type": "date",  
"doc\_values": true  
},  
"last\_switched": {  
"index": "not\_analyzed",  
"type": "date",  
"doc\_values": true  
},  
"l4\_src\_port": {  
"index": "analyzed",  
"type": "long",  
"doc\_values": true  
},  
"l4\_dst\_port": {  
"index": "analyzed",  
"type": "long",  
"doc\_values": true  
},  
"tcp\_flags": {  
"index": "analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"protocol": {  
"index": "analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"src\_tos": {  
"index": "analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"src\_as": {  
"index": "analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"dst\_as": {  
"index": "analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"src\_mask": {  
"index": "analyzed",  
"type": "integer",  
"doc\_values": true  
},  
"dst\_mask": {  
"index": "analyzed",  
"type": "integer",  
"doc\_values": true  
}  
}  
}  
},  
"settings" : {  
"index": {  
"refresh\_interval" : "5s",  
"store.throttle.max\_bytes\_per\_sec" : "200mb",  
"translog.flush\_threshold\_size": "200mb"  
}  
}  
}'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:03am UTC](https://discuss.elastic.co/t/mapping-netflow-data/1411/6 "2017-07-06T00:03:24Z")

</div>