# Problem searching Filebeat netflow module logs **URL:** \ **Category:** Beats\ **Tags:** beats-module, filebeat\ **Created:** [September 18, 2020, 7:35pm UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166 "2020-09-18T19:35:31Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 18, 2020, 7:35pm UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/1 "2020-09-18T19:35:31Z")
I'm really sorry if this isn't the right tag for this, I'm just not sure where to start. I recently installed Filebeat, enabled the netflow module and pointed my Edgerouter X to my machine running the Elastic stack. I'm getting logs but I'm having trouble searching. Some searches work while others (IP addresses) don't. For example, here's an event that contains Nuremberg: ![Elastic1](https://us1.discourse-cdn.com/elastic/original/3X/7/d/7d3ec5a2d31121e0f23137391529af4590eda53d.png) If I search for Nuremberg I get results like one would expect: ![Elastic2](https://us1.discourse-cdn.com/elastic/original/3X/e/c/ec2873707492c9f78ab886474df9b3d246f7ddcb.png) Now if I search for an IP address in that same event (207.180.192.206) I don't get anything returned: ![Elastic3](https://us1.discourse-cdn.com/elastic/original/3X/e/5/e56f51dff8175b899b02ba7264b894297617ef2a.png) destination.ip is searchable: ![Elastic4](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66afe46d898b0e5e70d40ec747acc0024b716f1b.png) I'm not sure what I'm doing wrong. Can anyone help me out? Thanks ---
**Author:** ![francescouk](https://avatars.discourse-cdn.com/v4/letter/f/7feea3/32.png) [@francescouk](https://discuss.elastic.co/u/francescouk)\ **Post date:** [September 19, 2020, 1:39am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/2 "2020-09-19T01:39:21Z")
Im not an expert but have you tried ```auto destination.ip: "207.180.192.206" ``` ---
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 19, 2020, 2:39am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/3 "2020-09-19T02:39:28Z")
Absolutely that works. Using that logic though, to get the destination.geo.city\_name as I did in the screenshot I'd have to specify that field and the city name: `destination.geo.city_name:"Nuremberg"` but I don't. I can simply search Nuremberg. It's something to do with integers, I'm sure. Anything with a number fails to return results, unless specifically specifying field and exact terms as you mentioned. This isn't the case in my Winlogbeat index. I can put in any IP and return results. Screenshot taken moments ago. ![Elastic5](https://us1.discourse-cdn.com/elastic/original/3X/7/5/75f3196eddf8b6038de778ec01f8ccba9bff03f4.png) This is why I'm thinking there's some config issue with Filebeat but I'm really struggling to find out what the problem is. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 19, 2020, 3:12am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/4 "2020-09-19T03:12:52Z")
Hi @Sylancer 1st I don't think you have a config issue. 2nd the proper KQL Syntax for both your searches are `destination.geo.city_name: "Nuremberg"` `destination.ip: "207.180.192.206"` Those should always work as expected (you can skip the rest if that is all you need) Why just putting in Nuremberg works is a little more nuanced. `destination.geo.city_name` is of type `keyword` When you just enter a Text like `Nuremberg` with out the field name Discover will search for that text in `keyword` and / or `text` fields it is a short hand search and it can be a bit confusing but still useful for a quick search. In this case it matches `destination.geo.city_name` exactly which is the only way to match a keyword. If you have` "San Jose"` as a city name in `destination.geo.city_name` you would need to search on `"San Jose"` in quotes otherwise the tokenizer will split it into `San` and `Jose` (without quotes) and it will not find either because the `destination.geo.city_name` is a `keyword` and the value is`"San Jose"` so it must be an _exact_ match. When you just enter `"207.180.192.206"` that is only found in the `ip` types fields so search does not find it (not to mention it tokenizes it into 200 180 192 206 if you put it in without quotes) So in short when you just put text in the search bar without the field names it will search the `text` fields for a token that matches and the `keyword` fields for an _ **exact match** _ if you enter EN-US without quotes it will break it up into EN and US.... Yes Elastic is a Search Engine ... and it works like one 🙂 So long story use the KQL syntaxt described [here](https://www.elastic.co/guide/en/kibana/7.9/kuery-query.html) ... its pretty easy, there is type ahead built into the search bar and you will get expected results. Hope this helps a bit... ---
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 19, 2020, 11:36pm UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/5 "2020-09-19T23:36:51Z")
Hi @stephenb Thanks so much for the detailed answer. I really appreciate it. So if I understand correctly, I can `shorthand` search a `keyword` or `text` field, but the `destination.ip` is not of type `keyword` or `text`. So could I change it to `keyword` or `text`? Would that give the results I'm looking for? I ask because I work with the Elastic Stack every day in my regular job and I'm certainly able to `shorthand` search IP addresses. I'm just trying to learn how Elastic works on my own by building/experimenting on my own network. Thanks again for your help. ---
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 20, 2020, 12:23am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/6 "2020-09-20T00:23:38Z")
So I tried the exact match on another field `event.duration` like you mentioned and it didn't return any records. I tried without the quotes and even increasing the time range. Any idea why it's not finding the exact match? We see `event.duration` is `110001.0` ![Elastic1](https://us1.discourse-cdn.com/elastic/original/3X/0/1/0171982b43aa89fe8d8f179669e69f80d448d198.png) And my exact match query: `event.duration : "110001.0"` ![Elastic2](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9c35cd76b344683d2b885a54ab452c7f324ca9bd.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 20, 2020, 12:25am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/7 "2020-09-20T00:25:10Z")
> [@Sylancer](#): > > So if I understand correctly, I can `shorthand` search a `keyword` or `text` field, but the `destination.ip` is not of type `keyword` or `text` . So could I change it to `keyword` or `text` ? Would that give the results I'm looking for? Me personally I would not change the type because the `ip` type allows a lot of other searching / query benefits like searching by a CIDR rule etc. > **[IP field type | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ip.html)** in addition the built in Security / SIEM app expects the IPs to be the correct `ip` type. > [@Sylancer](#): > > I ask because I work with the Elastic Stack every day in my regular job and I'm certainly able to `shorthand` search IP addresses. If IPs are in also in log messages or other text fields etc yes you can find them this / shorthand way, but you also may be finding the IP for a different reason. See My Example below in this case it found the `"198.98.61.24"` because `source.address` is a `text` field but it did not find it via the `source.ip` which is an `ip` field. The highlight yellow indicates which field was found during the search... often there is a `text` and `ip` field, that may be what you are seeing. ![Screen Shot 2020-09-19 at 5.09.29 PM](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4c006e679f4f59d5c411db57120086cca12f5799.png) ![Screen Shot 2020-09-19 at 5.13.33 PM](https://us1.discourse-cdn.com/elastic/original/3X/1/f/1faf10d0f3aaf3a84bdadfc6f4545157aa91a8af.png) Soooo...... If you can search is there a problem? I would not change the `ip` types in the Netflow modules or any of the other prebuilt modules _ **but** _ that is the great thing about Elasticsearch it is Open you can change what you like .... but you may affect / invalidate other parts of the Security App, Dashboards, Alerts, Detection, Anomaly Detection jobs etc. if you do. If you feel strongly you could add fields, but I would not change the existing mappings for the Pre Built Modules. The formal search is better if you know what field you want to search. `destination.ip: "207.180.192.206"` the shorthand is good for a quick search but if you know you want to check to specifically search `source.ip` or `destination.ip` etc... I would use the KQL syntax ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 20, 2020, 12:50am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/8 "2020-09-20T00:50:43Z")
@Sylancer > [@Sylancer](#): > > So I tried the exact match on another field `event.duration` like you mentioned and it didn't return any records. I tried without the quotes and even increasing the time range. Any idea why it's not finding the exact match? > > We see `event.duration` is `110001.0` Interesting... What version are you on... `event.duration` should be a `long` not a `float` ... if it is a `long` you should be able to search for it ..... floats are a whole different matter (in short you do search on exact floats ... for all the float point / math reasons) you search on a range, gt, lt, lte etc.... BUT that said `event.duration` should be a `long` ... can you try `event.duration : "110001"` Also if there are commas etc you need to take them out... example `110,001` you would search on `event.duration : "110001"` Perhaps there is a bug in the display or more likely did you change any display settings in advanced settings? and just in case This is what the number format should be... this controls the formatting in Kibana _ **not** _ how the data stored or the field type ![Screen Shot 2020-09-19 at 5.47.44 PM](https://us1.discourse-cdn.com/elastic/original/3X/5/b/5b846d26b809b88dde860aa4c415fe84f3744b0d.png) If I set it to the below I see the problem you are having... perhaps that is it... ![Screen Shot 2020-09-19 at 5.55.18 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/a/dab3e579e2fdebd6cc67084a672f82e7ef26f5f3.png) ---
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 20, 2020, 6:40am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/9 "2020-09-20T06:40:32Z")
Hey @stephenb Re: your earlier msg, you're right. The IP is probably in another log. That makes sense. Re: the event duration issue. I haven't changed anything in Advanced settings. Mine look identical to your screenshot: ![Elastic5](https://us1.discourse-cdn.com/elastic/original/3X/1/0/103063528593b50bd6317bdc8e191ea24febfb0e.png) I _think_ I found the location that tells me if `event.duration` is a `long` and it looks like it is: ![Elastic3](https://us1.discourse-cdn.com/elastic/original/3X/9/c/9c1176f3d710e8b3bb303bf1402ebcfe34016afe.png) And this image shows it's a number field ![Elastic4](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b9af30e532b2ab7d7617533a53a3a6910d2ee161.png) And I tried your suggestion of `event.duration: "3141"` as per the screenshot above but still nothing. Same results with or without the .0 ![Elastic6](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fcb84fe0bc398482d462650b5e3e4a6ee59df9dc.png) What a weird problem ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 20, 2020, 3:10pm UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/10 "2020-09-20T15:10:34Z")
@Sylancer Yeah... that is weird ... if you are interested ...which now I am 😉 ..we can check a few things. One thought typically we are not searching for an exact duration (or number for that fact) we often search ranges, so I am not sure if this is a real use case or we just stumbled on it but lets take a look either way if you are interested. Lets check a couple things... First in discover try this and lets see if it pulls back the record (perhaps more) `event.duration > "3140" AND event.duration < "3142"` Next lets get the exact `_index` and doc `_id` and look at the Example ![Screen Shot 2020-09-20 at 7.59.33 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/b/1b5d88c4763b1088b362fd8b94d2f1671864953f.png) I run `GET filebeat-24h-2020.09.19/_doc/zSquo3QBLVR631guRBpU` and I see ``` { "_index" : "filebeat-24h-2020.09.19", "_type" : "_doc", "_id" : "zSquo3QBLVR631guRBpU", "_version" : 1, "_seq_no" : 500401, "_primary_term" : 1, "found" : true, "_source" : { "input" : { "type" : "log" }, "event" : { "duration" : 24806000000, "created" : "2020-01-11T20:15:39.417Z", "category" : "network_traffic", "start" : "2020-01-11T20:14:14.093Z", "dataset" : "suricata.eve", ``` Screen shot note the blue indicates its is a number... ![Screen Shot 2020-09-20 at 8.05.13 AM](https://us1.discourse-cdn.com/elastic/original/3X/f/d/fd10bd99a76b834125fa76dfa33a3d10f46cef7f.png) Do this let me know what you see.... ---
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 21, 2020, 2:43am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/11 "2020-09-21T02:43:48Z")
Hey @stephenb Thanks to your screenshots I think I have a really big clue. So I tried a range like you suggested and didn't get any results but I decided to filter on one of the durations. Then I checked out the filter: ![Elastic1](https://us1.discourse-cdn.com/elastic/original/3X/5/8/58682462b856dc47067d74a49b9b639fd1f1fc41.png) I decided to search on the value in the filter and bingo!! ![Elastic2](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f93f91d299b4da3b72d0879c115eb934d46ec1a.png) I checked out the index and it looks like the following: ![Elastic3](https://us1.discourse-cdn.com/elastic/original/3X/8/4/84cc12da8ad7a8a0f547eef24656de0ccec2d2d2.png) So it looks like as long as I search by nanoseconds I get results. I'm not exactly sure how to fix/change this but I think this is the reason I'm not able to search for what's displayed in Kibana. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 21, 2020, 4:54am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/12 "2020-09-21T04:54:11Z")
Interesting and good find!!! Remind Me Stack Version and Filebeat Version and this is Netflow modules correct / confirm? Here is the docs on this.... > **[Numeric field formatters | Kibana Guide \[7.9\] | Elastic](https://www.elastic.co/guide/en/kibana/current/field-formatters-numeric.html)** It looks like (but you could break it 🙂 ) ...but I don't think you will if you only change the Output Format display to Nanoseconds and 0 Decimal Places. I would NOT change the Input Format I will have to poke into that I do not have anything with handy with nano seconds / same mapping type But I think you found it... I learn something everyday Thanks! ---
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 21, 2020, 8:24pm UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/13 "2020-09-21T20:24:02Z")
Stack version is 7.9.1, or at least I think that's the info you're looking for. ![Elastic4](https://us1.discourse-cdn.com/elastic/original/3X/0/4/04491e8cab15bc9324abdb8e55e7416f751aafd4.png) Filebeat is: `filebeat version 7.9.1 (amd64), libbeat 7.9.1 [ad823eca4cc74439d1a44351c596c12ab51054f5 built 2020-09-01 19:58:51 +0000 UTC]` Confirmed Netflow module for Filebeat. I tried changing the Input Format and the Output Format to seconds and removing the decimal. I don't think it's actually displaying in seconds though: ![Elastic5](https://us1.discourse-cdn.com/elastic/original/3X/6/6/668f28b7718915c38b0228892684a257a019a17d.png) I wonder if it's my router setting the Input Format? I'm going to have to look into that. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [September 21, 2020, 8:44pm UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/14 "2020-09-21T20:44:17Z")
@Sylancer Yup 7.9.1 I would _ **not** _ have changed the input format... only the output Yes you would need to match the input format and the source format. I think the input was fine unless you are saying the values are fundamentally wrong by an order of magnitude I would have probably left input nano and output milli with 0 decimal places And again I think we are doing academics at this point you will usually search durations by ranges... But it been good to see this, let me know what you find out ---
**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\ **Post date:** [September 22, 2020, 4:19am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/15 "2020-09-22T04:19:07Z")
> I would _ **not** _ have changed the input format... only the output ...oh. Well it looks like Kibana knows what's best for me and went and changed it back to nanoseconds anyway. This brings me back to my original problem. I have to search on nanoseconds and not milliseconds like I set to be displayed in the Index Pattern. I tried outputting to Logstash so I might be able to mutate the field but then I lose all the nice Dashboards from the Filebeat setup. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 20, 2020, 6:19am UTC](https://discuss.elastic.co/t/problem-searching-filebeat-netflow-module-logs/249166/16 "2020-10-20T06:19:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.