PAcketbeat Flow wih CIDR search on Kibana


(Tzathenah Peaneah Halatuituia) #1

Hi
i have a packetbeat flow index with dest.ip :x.x.x.x which is working fine. What i need to do now is if it is possible to search based on CIDR notation instead of individual ips . Like if i want to find how much byte to and fro a particular CIDR address for a given time .
Really appreciate your kind help in advacne


(Robert Cowart) #2

Specifically for CIDR notation you would use a term query. This requires that the field is indexed as the type ip. Other options to query for ranges of IP addresses are the query_string and range queries. Below are examples of each. The field used in the examples is from ElastiFlow but you should get the idea of how they work:

term:

{
  "query": {
    "term": {
      "flow.dst_addr": "192.168.12.0/24"
    }
  }
}

query_string:

{
  "query": {
    "query_string": {
      "query": "flow.dst_addr: \"192.168.0.0/24\""
    }
  }
}
{
  "query": {
    "query_string": {
      "query": "flow.dst_addr:[192.168.12.1 TO 192.168.12.255]"
    }
  }
}

range:

{
  "query": {
    "range": {
      "flow.dst_addr": {
        "gte": "192.168.12.1",
        "lte": "192.168.12.255"
      }
    }
  }
}

Rob

Robert Cowart (rob@koiossian.com)
www.koiossian.com
True Turnkey SOLUTIONS for the Elastic Stack


(Tzathenah Peaneah Halatuituia) #3

Hi Rob sorry for the silly question i am aware that this code need to b execute somewhere . Please note i have been use elk stack for quite sometime but i an yet familiar with it throughly. So really i don't understand where this code should be and also is this allow me to do search on the Kibana search field . i hope you understand


(Robert Cowart) #4

The search bar uses Lucene syntax by default, which is the syntax for a query_string query. So put this in the search bar:

dest.ip: "192.168.1.0/24"

NOTE: The quotes around the CIDR subnet are important. It won't work without them.


(Tzathenah Peaneah Halatuituia) #5

thanks Rob ... now i can see a result ... but only when i used individual ips not the CIDR notaiotn like dest.ip"8.8.0.0/16"


(Robert Cowart) #6

You are missing a colon: dest.ip"8.8.0.0/16"should be dest.ip: "8.8.0.0/16"


(Tzathenah Peaneah Halatuituia) #7

pack

Hi Rob
This is what is shown


(Tzathenah Peaneah Halatuituia) #8

this is the success search
pack

thanks Rob


(Tzathenah Peaneah Halatuituia) #9

However i can do this ... i think the maximum is a /24

dest.ip:[192.168.2.1 TO 192.168.2.255]


(Robert Cowart) #10

You should look at the index pattern and ensure that dest.ip was indexed as an IP and not a string. Here is an example from my system that shows this working:

All data:

10.0.0.0/8

192.168.3.0/24

Rob

Robert Cowart (rob@koiossian.com)
www.koiossian.com
True Turnkey SOLUTIONS for the Elastic Stack


(Tzathenah Peaneah Halatuituia) #11


Hi Rob
I believe it is index as required as this is what i have similar to your but the only difference is i cannot use the CIDR


(Robert Cowart) #12

Notice how your you have a t next to dest-ip in your screenshot. This indicates that the field is indexed as a string (or text).

Look at mine...
46
The little PC icon indicates it is indexed as an IP address.

Go to Management --> Index Patterns and find the field. It must be an IP, like this...
02

I expect you will see a string. As I mentioned earlier the field MUST be indexed as the type ip for CIDR queries to work. The range query only looked like it was working because it was basically treating it with alphabetical sorting.

I just took a look at the packetbeat 6.2.4 index template and it does in fact type dest.ip as a keyword.

"dest": {
  "properties": {
    "ip": {
      "ignore_above": 1024,
      "type": "keyword"
    },

I didn't think about this initially. We don't use any beats data in its out-of-the-box form (data model is just too whacky), and when we normalize it to our own schema we handle the data types properly.

I recommend that you open as issue for this in the Beats repository on GitHub.


(Tzathenah Peaneah Halatuituia) #13

any suggestion what i should do with this .....actually this is a result from packetbeat


(Robert Cowart) #14

Like I said...

They need to fix this in Beats.


(Tzathenah Peaneah Halatuituia) #15

thanks

what are you using ??? syslog + logstash ???


(Tzathenah Peaneah Halatuituia) #16

i try to change this but not work either


(Robert Cowart) #17

Our solutions take in a number of log and network flow sources, normalize this data to common schemas, and provide a suite of visualizations, dashboards, alerts and anomaly detection methods that work seamless across all data sources.

We are primarily focused on security and network flow analytics. However we have also helped our customers in the other areas, such as performance management, and even IoT/Smart Cities projects.

We currently send Packetbeat data to Logstash, where it is transformed to meet the requirements of our customers. In the future this function will likely be moved into Kafka Streams jobs (faster and more scalable).

Most of our customers understand the potential of what can be achieved with the Elastic Stack. However they lack the time required to learn all of the little (sometimes undocumented) details needed to build it themselves. We typically deliver in just a couple days, what it would take them months to do on their own.

We also contribute to the community with projects for basic network flow and syslog collection.


(Robert Cowart) #18

NOTE: if you change the index template, making dest.ip (and other IP addresses) an ip type, you will have to wait until a new index is created. Index templates only have effect at index creation.

You also need to prevent Packetbeat from overwriting your template, as it will by default.
https://www.elastic.co/guide/en/beats/packetbeat/master/packetbeat-template.html


(Tzathenah Peaneah Halatuituia) #19

so you suggest i delete the index and recreate


(system) #20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.