How to monitor network traffic of specific URL by using packebeat

Elastic Stack Beats
1

Hi All,

I am using packetbeat to monitor application server network traffic and building the dashboard based on the data captured.

I would like to know is there any way in packetbeat to specify the URL (Ex:- https://www.elastic.co/) to find the network traffic IN & OUT for specific user.

If it has any option please kindly point me out the right documentation and i will understand it first.

Please kindly share any thoughts it would be very helpful.

Thanks,
Ganeshbabu R

2

I'm not sure I understand your use-case. Can you give us an example of what exactly do you want to achieve?

3

Sorry for the delay response @adrisr

Consider this is my sample scenario,
Using packetbeat we want to monitor the application URL and try to find out the network traffic of IN & OUT..

Ex:- We are having some list of application URL and if one user is accessing that URL and we would like to find out network traffic in his machine & vice versa

I was trying to find something similar to this. This is for finding out specific process running in machine.

Please let me know if this can be achievable using packetbeat
Regards,
Ganeshbabu R

4

@adrisr

Any of your thoughts on this ?

Regards,
Ganeshbabu R

5

Sorry for the delay.

I understand you want something like this?

19
Screen Shot 2018-02-01 at 11.07.19.png3196×814 227 KB

If not please explain clearly your setup (involved devices) and what do you mean by "user" (host, process?)

6

@adrisr

Yes adrian..
This is what I want needed..

Let me be more specific,

When "X" user login to this host (Host is nothing but the application url (EX:- http://www.servicenow.com) and I want find the network In/out for the particular site.

Please Kindly share the reference documentation and let me know your thoughts how to configured it packetbeat.yml

Thanks,
Ganeshbabu R

7

more questions:

  1. Do you care about traffic to a single domain (servicenow.com), or there is more than one domain?

  2. Do you need to differentiate traffic to different paths inside the same domain, or you don't care at all about paths?

  3. Is traffic using plain HTTP or HTTPS?

  4. Are you hosting a single domain in this HTTP server?

8

Yes there will be lot of domains and we want to take care of each one of those.

Currently my intention was to find the traffic for the domains and I hope the packetbeat will b able to the capture the path & differentiate inside the each domain and if we find different path will use it and show it in the dashboard.

Yes we are using plain HTTP

We are having multiple domain and we hosted some domain in http and others in https

Let me know your thoughts.

Thanks,
Ganeshbabu R

9

What you want is to use the http protocol decoder in packetbeat. It is enabled by default. You should extended it with two extra options, send_headers and send_all_headers:

- type: http
  # Configure the ports where to listen for HTTP traffic. You can disable
  # the HTTP protocol by commenting out the list of ports.
  ports: [80, 8080, 8000, 5000, 8002]
  send_headers: true
  send_all_headers: true

This will provide you with events like this:

{
  "@timestamp": "2018-02-02T09:36:19.757Z",
  "responsetime": 95,
  "type": "http",
  "bytes_in": 708,
  "bytes_out": 33861,
  "client_port": 54386,
  "path": "/",
  "direction": "out",
  "port": 80,
  "method": "GET",
  "query": "GET /",
  "http": {
    "request": {
      "params": "",
      "headers": {
        "host": "example.com",
        "accept-encoding": "gzip, deflate",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6)",
        "content-length": 0
      }     
    },    
    "response": {
      "headers": {
        "connection": "Keep-Alive",
        "server": "Apache/2.2.17",
        "content-length": 33376,
        "content-type": "text/html",
        "expires": "Thu, 01 Jan 1980  0:00:00 GMT"
      },    
      "code": 200,
      "phrase": "OK"
    }
  },
  "proc": "",
  "status": "OK", 
  "client_ip": "192.168.1.39",
  "ip": "62.99.163.135"
}

Now in Kibana you can build a nice visualisation using the fields in the http events. The fields you want are:

  • http.request.header.host: The server name part of the url ("www.hostnamenow.com")
  • path: The path component of the URL.
  • client_ip: The IP address of the client
  • bytes_in: Amount of data sent from client to server.
  • bytes_out: Amount of data sent from server to client.

If you're not familiar with visualisations, what you can do first is a table visualisation, then you can experiment with other types.

  • first add a filter type:"http" so that only http events get used.
  • change the default Metric type from count to sum, to sand use the field bytes_out, as what you ultimately want is to sum all bytes sent.
  • add a new Metric of type sum for the bytes_in field.
  • Now you need to start adding buckets and sub-buckets to create aggregations. Use the Terms aggregation, first on the http.request.headers.host field. Then add a sub-bucket using the client_ip field.

This will give you a table similar to the one I showed you in a previous message.

Kibana visualisations are very powerful but they require practice to understand how the work. Have a look at the Kibana user guide.

3 Likes
10

@adrisr

Thanks for the clear explanation.

I am trying it and I will get back to you with the update.

Regards,
Ganeshbabu R

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.