Logstash receives the attached packets (template and flow packets) but it seems to drop the flow packet with the error
[WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 2048 from observation domain id 1010, because no template to decode it with has been received. This message will usually go away after 1 minute.
Subsequent flow packets seem to parsed and it is showing up in kibana UI.
I need help in figuring out why this first flow packet alone is dropped even though template packet is coming before the flow packet.
I'm using these versions logstash 6.4.3 logstash-codec-netflow (3.14.1)
I have a netflow codec configuration for UDP messages similar to this codec {
** versions => [5,9,10]**
** target => "ipfix"**
** cache_save_path => "/usr/..."**
** ipfix_definitions => "...path to ipfix.yml file"** } Always, the first flow packet for a new domain ID is dropped by logstash.
This is likely due to the fact that Logstash is multithreaded, and will start multiple input and pipeline.workers. While the template record is being processed by one worker, the flow record is grabbed by another. Since the template isn't processed yet, and available to decode the flow, you get an error.
You could try setting both UDP input and pipeline workers to 1. However I wouldn't suggest it for production. If you have even a moderate volume of flow data, you will need all of the workers (and corresponding cores) as possible.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.