### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 3, 2018, 5:25am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/1 "2018-04-03T05:25:02Z")
I've been having this issue fora while now, I've posted on github but the problem is not the netflow codec
>
>
> \- Version: logstash-codec-netflow-3.5.2
> logstash 5.6.3
> \- Operating System: Ubu…ntu 16.04
> \- Config File:
> \` input {
> udp {
> tags =\> \["netflow"\]
> type =\> "netflow"
> port =\> 5046
> codec =\> netflow {
> versions =\> \[9\]
> }
> }
>
>
> output {
> if \[type\] == "netflow" {
> elasticsearch { hosts =\> \["http://localhost:9200"\] }
> stdout { codec =\> rubydebug }
> #file { path =\> "/log/netflow.log" }
> }\`
>
> \- Sample Data:
> \`\[2017-10-19T07:52:24,549\]\[WARN \]\[logstash.codecs.netflow \] Template length doesn't fit cleanly into flowset {:template\_id=\>257, :template\_length=\>143, :record\_length=\>1392}\`
>
> \- Steps to Reproduce:
> Originally I was only collecting netflow for CISCO2901/K9
> I'm not sure if this has started happening since I started ingesting the netflow from a cisco-wlc (5520) as well
> I see it's a known issue because they might use the same template id?
> People are working on logstash-input-netflow which might resolve the issue.
> Since then I've removed the cisco-wlc netflow but the messages in the sample data still appear few times every minute
jorritfolmer has explained why and given me some insight but I thought I ask here for a bit more help
> Blockquote
> 2018-04-03T10:12:48,021][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash-2018.04.03", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x467c29d1], :response=\>{"index"=\>{"\_index"=\>"logstash-2018.04.03", "\_type"=\>"doc", "\_id"=\>"emzZiGIBVDzVCvaTWDoP", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse [netflow.event\_time\_msec]", "caused\_by"=\>{"type"=\>"json\_parse\_exception", "reason"=\>"Numeric value (16787034570129189063) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@5783d39f; line: 1, column: 224]"}}}}}
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [April 3, 2018, 6:22am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/2 "2018-04-03T06:22:15Z")
It looks like that number is outside if the bounds of what Elasticsearch can store as a precise value, and Elasticsearch is rejecting the request to store the document.
~~While a type of a field cannot be changed on an existing index, you may be better off using a `double`, which can store larger numbers at the cost of precision.~~ _EDIT: further discovery showed that this number is wrong, not that the type should be changed._
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [April 3, 2018, 6:30am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/3 "2018-04-03T06:30:37Z")
> [@VamPikmin](#):
>
> ```auto
> [netflow.event_time_msec]
>
> ```
A milisecond-level granularity timestamp of 16787034570129189063 is roughly [500 million years in the future](https://www.google.com/search?q=16787034570129189063+milliseconds+in+years&oq=16787034570129189063+milliseconds+in+years&aqs=chrome..69i57j6.12765j0j7&client=ms-android-google&sourceid=chrome-mobile&ie=UTF-8). Something is fishy here.
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 3, 2018, 6:39am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/4 "2018-04-03T06:39:26Z")
Thank you for your reply.
I need to narrow it down to a device, but I have a feeling it's the Cisco ASA. Do you have anything to suggest?
Should I check the time and zone on the ASA?
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [April 3, 2018, 6:45am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/5 "2018-04-03T06:45:56Z")
When I removed seven decimal places, it looks like a relatively appropriate milliseconds-since-epoch value (March of 2023, just five years in the future?).
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 3, 2018, 7:05am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/6 "2018-04-03T07:05:58Z")
That's helpful, thank you.
I am just using @timestamp, nothing fancy that I know of. Don't have access to the firewall so I can't check the date there
> Blockquote [2018-04-03T16:42:46,107][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash-2018.04.03", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x28d91fe6], :response=\>{"index"=\>{"\_index"=\>"logstash-2018.04.03", "\_type"=\>"doc", "\_id"=\>"WRQ-imIBuAdp7i0vXvbJ", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse [netflow.event\_time\_msec]", "caused\_by"=\>{"type"=\>"json\_parse\_exception", "reason"=\>"Numeric value (16520087337464545448) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@6b38de4d; line: 1, column: 383]"}}}}}
> $ date (on the server)
> Tuesday 3 April 16:57:21 AEST 2018
I have tried deleting the template and adding one with netflow.event\_time \_msec and a type of long, but didn't really work. The type still says number, this is just the default logstash template. This is before you suggested to use a double, would I add it in the same spot - the logstash template?
I might set up ip tables to only accept netflow from the ASA just to narrow down to see if that's where the problem lies, do you think?
The netflow data is coming in from the ASA, and other devices, but this warning must mean some data might be missing?
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [April 3, 2018, 6:29pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/7 "2018-04-03T18:29:19Z")
I have revised my comment about changing the field's type -- it's the value in this particular document that is wrong, not the underlying type in Elasticsearch.
- How is `[netflow.event_time_msec]` populated? (e.g., is there anything in your logstash configuration that explicitly touches the field? Is there anything in your netflow configuration that explicitly sets the event's timestamp?)
- Is March of 2023 (a milisecond-granularity timestamp roughly five years in the future) an appropriate value for an event?
When encountering a wildly-off timestamp, what is your desired behaviour? dropping the event?
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 3, 2018, 10:35pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/8 "2018-04-03T22:35:00Z")
I've tried searching for the value when I get status 400 and the exception and here's what it looks like, I've only removed the geoip output and the real outside ip
Looks like it's not the ASA but the 2901 Router causing it, maybe an ACL causing a packet to be malformed, don't know 😊
> Blockquote [2018-04-04T08:13:25,292][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"netflow-2018.04.03", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x54630c5d], :response=\>{"index"=\>{"\_index"=\>"netflow-2018.04.03", "\_type"=\>"doc", "\_id"=\>"JSWSjWIBuAdp7i0vaC-j", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse [netflow.flow\_start\_msec]", "caused\_by"=\>{"type"=\>"json\_parse\_exception", "reason"=\>"Numeric value (14340042958798061568) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@66e2e84; line: 1, column: 309]"}}}}}
root# cat netflow.log | grep 14340042958798061568
> Blockquote
> {"syslog\_severity":"notice","netflow":{"icmp\_code":0,"flowset\_id":257,"ipv4\_dst\_addr":"some\_ip\_address","ingress\_acl\_id":"00007d3b-59d07d3b-59b40000","egress\_acl\_id":"08b90000-00090002-0005c0a8","xlate\_dst\_addr\_ipv4":"2.192.168.101","xlate\_src\_port":14784,"l4\_dst\_port":0,"flow\_start\_msec":14340042958798061568,"ipv4\_src\_addr":"0.0.0.0","flow\_seq\_num":3324889,"fw\_ext\_event":1536,"username":"Aˬe9\u0006\u0000\u0017e
> +\u0000\u0000À¨Ç\u0006\u0018\u0011\u001B\u0001\u0000\u0000\u0000\u0000};%}:þÈ\u0000\u0000\u0003m\u0000\u0000\u0000\u0006\u0000\u0002\u0000\u0005À¨\u0006À¨e\u0006\u0006\u0002ñ¡#è\u0000\u0000À¨Ç","input\_snmp":22992,"conn\_id":403773953,"version":9,"protocol":0,"icmp\_type":0,"xlate\_src\_addr\_ipv4":"13.0.5.0","xlate\_dst\_port":43139,"event\_time\_msec":9595789153602158760,"fw\_event":65,"l4\_src\_port":32059,"output\_snmp":2387},"syslog\_severity\_code":5,"syslog\_facility":"user-level","type":"netflow","tags":["netflow","Cisco 2901 Router","GeoIP-DST","\_geoip\_lookup\_failure","netflow-message"],"host":"192.168.199.1","@timestamp":"2018-04-03T22:13:36.000Z","syslog\_facility\_code":1}
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 3, 2018, 10:42pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/9 "2018-04-03T22:42:26Z")
> [@yaauie](#):
>
> Is there anything in your netflow configuration that explicitly sets the event's timestamp?)
In the logstash config this is what I have
filter {
if [type] == "netflow" {
grok {
match =\> {
"message" =\> "%{GREEDYDATA}" }
```
add_tag => ["netflow-message"]
remove_field => ["@version"]
} #grok
syslog_pri {
syslog_pri_field_name => "syslog5424_pri"
}
} #syslog
} #filter
```
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 4, 2018, 10:26pm UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/10 "2018-04-04T22:26:14Z")
> [@yaauie](#):
>
> I have revised my comment about changing the field's type -- it's the value in this particular document that is wrong, not the underlying type in Elasticsearch.
Thanks for that
> [@yaauie](#):
>
> When encountering a wildly-off timestamp, what is your desired behaviour? dropping the event?
I'd like to know what's causing it. If I'm only ingesting logs from the router I don't see the warning. When I add the firewall in the mix, ingesting on the same udp port the warnings start popping up in the log file.
---
### Author:  [@yaauie](https://discuss.elastic.co/u/yaauie)
#### Post date: [April 5, 2018, 12:56am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/11 "2018-04-05T00:56:21Z")
You may be interested in enabling the dead-letter-queue. It's a feature that gives Logstash somewhere to put events that can't be delivered so that you can dig through them later without holding up your pipeline.
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 6, 2018, 5:17am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/12 "2018-04-06T05:17:37Z")
Thanks yaauie, will try that. I have enabled the dead-letter-queue however since I still had the output going to a file instead of elasticsearch I didn't see any exceptions overnight.
I've changed that now, will have to wait and see what happens
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [April 9, 2018, 12:02am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/13 "2018-04-09T00:02:45Z")
This is weird, when I enable the dead letter queue and set it to a path, the exceptions stop showing up however the dead letter queue logs don't show anything
Soon as I disable dead\_letter\_queue I get them again.
I added the following to my logstash config
dead\_letter\_queue {
path =\> "/loggy/dead\_letter\_queue"
commit\_offsets =\> true
pipeline\_id =\> "main"
}
I've enabled the dead letter queue in logstash.yml
dead\_letter\_queue.enable: true
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 7, 2018, 12:02am UTC](https://discuss.elastic.co/t/logstash-output-elasticsearch-for-netflow/126505/14 "2018-05-07T00:02:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.