# Can not decode Citrix Netscaler IPFIX messages **URL:** \ **Category:** Logstash\ **Created:** [August 12, 2020, 4:53am UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654 "2020-08-12T04:53:09Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![tkuronen](https://avatars.discourse-cdn.com/v4/letter/t/3ec8ea/32.png) [@tkuronen](https://discuss.elastic.co/u/tkuronen)\ **Post date:** [August 12, 2020, 4:53am UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/1 "2020-08-12T04:53:09Z")
Is anyone here successfully decoded a IPFIX data send by Citrix Netscaler ? - Version: Logstash (7.7.1) / logstash-codec-netflow (4.2.1) - Operating System: RHEL 7.8 Logstash Input config: ```auto input { udp { id => "logstash_netscaler_input" port => 12208 codec => netflow { versions => [10] target => ipfix cache_save_path => "/tmp" ipfix_definitions => "/etc/logstash/conf.d/ipfix.yaml" include_flowset_id => true } type => ipfix } } ``` File ipfix.yaml is exactly this one: > Traffic is received from Netscaler to Logstash but it seems that the IPFIX template can not be read. We get the following Warnings to logstash-plain.log from id's 256 - 332: `Can't (yet) decode flowset id 256 from observation domain id 0, because no template to decode it with has been received. This message will usually go away after 1 minute.` ... and this Warning from enterprise field: `Unsupported enterprise field {:type=>638, :enterprise=>5951, :length=>2}` However it can be confirmed by taking a tcpdump capture from Logstash server that templates are received. When starting Logstash, it warns that: `Template Cache does not exist {:file_path=>"/tmp/ipfix_templates.cache"}` However the cache file is not generated even if pipeline is running for hours, although templates are sent by Netscaler every 60 seconds. ---
**Author:** ![cknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cknz/32/9640_2.png) [@cknz](https://discuss.elastic.co/u/cknz)\ **Post date:** [August 12, 2020, 11:28am UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/2 "2020-08-12T11:28:09Z")
I have done something similar in the past for the AppFlow stuff that comes out of the Citrix Netscaler platform; before there was a plugin for IPFIX. Those flowset IDs look very familiar. The conclusion I came away with was that the Netscalers don't actually send all of the templates (that may not be true in later versions; I've found AppFlow logging to be unsatisfactory, so haven't persued it since for much). When I approached the problem, I basically captured / reverse-engineered some of templates myself. I wrote a problem that used a forked version of libipfix (see this HISTORICAL archive at [https://github.com/cameronkerrnz/libipfix](https://github.com/cameronkerrnz/libipfix) -- it does have some parsing issues currently) that emitted JSON to a file, which was then tailed by logstash. Hope that helps. PS. If /tmp/ipfix\_templates.cache is not being written to, things to consider: - permissions, including SELinux - SystemD private temporary directory may be in enabled - what does strace say about it? ---
**Author:** ![tkuronen](https://avatars.discourse-cdn.com/v4/letter/t/3ec8ea/32.png) [@tkuronen](https://discuss.elastic.co/u/tkuronen)\ **Post date:** [August 13, 2020, 7:57am UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/3 "2020-08-13T07:57:35Z")
Thanks for the tips. I tried to also change the cache\_save\_path to /var/lib/logstash/cache/ and added an empty json {} to file ipfix\_templates.cache When Logstash is starting, if I understand correctly it reads the json and checks that file is writable: ```auto 8532 open("/var/lib/logstash/cache/ipfix_templates.cache", O_RDONLY|O_CLOEXEC) = 70 8532 fcntl(70, F_GETFD) = 0x1 (flags FD_CLOEXEC) 8532 fcntl(70, F_SETFD, FD_CLOEXEC) = 0 8532 fstat(70, {st_mode=S_IFREG|0644, st_size=3, ...}) = 0 8532 fcntl(70, F_GETFD) = 0x1 (flags FD_CLOEXEC) 8532 ioctl(70, TCGETS, 0x7effeabf8640) = -1 ENOTTY (Inappropriate ioctl for device) 8532 fstat(70, {st_mode=S_IFREG|0644, st_size=3, ...}) = 0 8532 lseek(70, 0, SEEK_CUR) = 0 8532 read(70, "{}\n", 3) = 3 8532 read(70, "", 1024) = 0 8532 close(70) = 0 8532 access("/var/lib/logstash/cache/ipfix_templates.cache", W_OK) = 0 ``` However after that, it does nothing to the cache files. However taking tcpdump and inspecting it with Wireshark, it can be confirmed that template packages are received. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 13, 2020, 2:47pm UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/4 "2020-08-13T14:47:42Z")
> [@tkuronen](#): > > However the cache file is not generated even if pipeline is running for hours As far as I can see [persist()](https://github.com/logstash-plugins/logstash-codec-netflow/blob/19b70af9fcab715baba7601d3620eea096f562bc/lib/logstash/codecs/netflow.rb#L589) is never called, so the cache will never be written to disk. persist() was defined, presumably to replace the [old call](https://github.com/logstash-plugins/logstash-codec-netflow/pull/158/files#diff-6b0a745358ff99061b0234545f80d7c4L348) to save\_templates\_cache. But the old call was just removed and not replaced with a call to persist. Perhaps @yaauie will comment. (That might explain how often folks complain about the 'no template to decode it with has been received' message.) ---
**Author:** ![cknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cknz/32/9640_2.png) [@cknz](https://discuss.elastic.co/u/cknz)\ **Post date:** [August 15, 2020, 8:23am UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/5 "2020-08-15T08:23:24Z")
I think Badger is correct. But there is also the memory-based cache to consider, and if there is a warning that there is no-such template known, that would indicate that the template never got entered into the memory cache, so there would appear to be two problems. Thinking the persistance code-path through a bit: - the templates get written to disk by do\_persist - do\_persist is called from both persist and do\_register - do\_register is **not** called from LogStash::Codecs::Netflow#register (L61) but **is** called from TemplateRegistry#register (L567) - register docstring indicates that that if invalid\_template is thrown it won't be cached - register is called from the the template processing parts of decode\_ipfix (L306) and decode\_netflow9 (L203) - persist itself seems never to be called, but that shouldn't matter; persistence should still occur when a new template is received. In `decode_ipfix`, it would seem (I'm not a Ruby developer yet), that `ipfix_field_for` (L299) must succeed in order to reach the call to `@ipfix_template.register`. It is in ipfix\_field\_for that we see the warning mentioned: `Unsupported enterprise field {:type=>638, :enterprise=>5951, :length=>2}` Enterprise number 5951 is indeed Citrix. The code would suggest that it will only take enterprises-type pairs that it knows about in `@ipfix_fields` (note: now we're talking about fields, not templates). The fact that the warning is about an unsupported field, rather than an unsupported enterprise helps to narrow the focus..... I think 638 is a type that came later in the development of the Citrix Netscaler (Application Delivery Controller) appliance. So where are these fields defined? L73 tells me in netflow/ipfix.yaml, which has been provided. But you'll see that the section of that file that pertains to Citrix (5951) is [L1260-L2157](https://github.com/logstash-plugins/logstash-codec-netflow/blob/master/lib/logstash/codecs/netflow/ipfix.yaml#L1260-L2157), and the newer types are not included. So if you're not seeing any persistance, it would suggest that the only templates you have configured to be sent are for those that come after 541. What needs to happen is for a complete set of those types to be dug up. That's the problem with AppFlow, as this mapping is not published (or at least, was not published in an up-to-date form when I last looked for it some years ago). You might wonder why we need this mappings at all, isn't IPFIX meant to be flexible? Well, yes and no. The template records that get sent out are a bit like structure definitions, but they only tell you that a particular message is formulated from a particular set of fields (Information Elements, or IEs for short, or those specific to each Enterprise, or EIEs), not what those fields are (how big they are, what sort of datatype it contains or what you might call them). Thus, the receiver needs to know the following: - how to interpret each IE (all of them within each fieldset it receives) - what to do with each IE (eg. what name to give it, or what it means semantically) You can gain some partial appreciation for what it was done this way when you look at the history of NetFlow (in particular the various subversions within NetFlow version 8).... but it still leaves a bit of a sour taste in the mouth of us that don't necessarily want to point an IPFIX exporter at something that isn't a vendor element manager (that is coded to know what to do semantically with the IEs it receives). If you can't find the information you're looking for (I once tried asking our technical account manager, but nothing came of that), then you'll just have to reverse engineer it from the template messages that come out and then make successive guesses..... good luck; you'll appreciate why I don't much like AppFlow for logging. These days, I would only use it for Load Balanced Virtual Servers that aren't HTTP (and thus can't include the client IP in a request header). If memory recalls, I think Splunk had some table of these: [https://github.com/splunk/ipfix](https://github.com/splunk/ipfix), but that's [out of date](https://github.com/splunk/ipfix/issues/1) too and hasn't been updated in 7 years or more, and Splunk has since [dropped support for IPFIX](https://splunkbase.splunk.com/app/1801/). [Reading between the lines a bit](https://docs.citrix.com/en-us/citrix-adc/13/ns-ag-appflow-intro-wrapper-con/ns-ag-appflow-config-tsk.html), I see that that Citrix have introduced a 'logstream' transport for AppFlow, as a alternative to IPFIX.... I can imagine why that would be attractive for various reasons, so its not unreasonably to consider AppFlow as being less-and-less useful in future. PS. I'm really not sure what kind of template you're interested in, but if you're wanting performance-management, you may get better ROI by focussing on RUM instead. Continue down the AppFlow path only if you are suitably desperate and if no other mechanism will suffice (submit a support ticket to see how to do what you might want using syslog logging or such). OR, consider how you might instead get the insight you need from Netscaler MAS... though I have no experience trying that. Hope that helps 🙂 Cameron ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 15, 2020, 1:30pm UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/6 "2020-08-15T13:30:57Z")
> [@cknz](#): > > do\_register is **not** called from LogStash::Codecs::Netflow#register (L61) but **is** called from TemplateRegistry#register do\_persist is only called from do\_register inside the catch block for :invalid\_template. So if you only receive valid templates there will be no persistence. ---
**Author:** ![tkuronen](https://avatars.discourse-cdn.com/v4/letter/t/3ec8ea/32.png) [@tkuronen](https://discuss.elastic.co/u/tkuronen)\ **Post date:** [August 17, 2020, 7:29am UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/7 "2020-08-17T07:29:01Z")
Thanks for the answers. > [@cknz](#): > > So if you're not seeing any persistance, it would suggest that the only templates you have configured to be sent are for those that come after 541. Actually templates are sent also for fields below that 541. I have created also an issue [IPFIX templates sent by Citrix Netscaler are not cached · Issue #192 · logstash-plugins/logstash-codec-netflow · GitHub](https://github.com/logstash-plugins/logstash-codec-netflow/issues/192) There is a screenshot from Wireshark where can be seen that eg. template for field 256. ...and pipeline also outputs nothing if configuring an output to a file for a test. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 14, 2020, 7:29am UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/8 "2020-09-14T07:29:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [October 30, 2020, 7:17pm UTC](https://discuss.elastic.co/t/can-not-decode-citrix-netscaler-ipfix-messages/244654/9 "2020-10-30T19:17:17Z")
> [@Badger](#): > > do\_persist is only called from do\_register inside the catch block for :invalid\_template. So if you only receive valid templates there will be no persistence. Catch blocks in ruby are weird, and I'm not 100% sure why I used one here because the implementing code never actually throws the related symbol. Catch blocks will execute the provided code in the block, allowing a specific thrown symbol to blow through the stack to the catcher. So persistence happens _unless_ the `:invalid_template` is thrown inside the block (or an exception is thrown in the block, which would blow even further through the stack). There are tests covering persisting of the cache to disk, so I'm unsure what is failing in this situation and why.