HTTPS Traffic visualization on url

[sholzhauer]

Hello, hope everyone is in good health.

We want to monitor the amount of data (gb's) which are being send to a https endpoint from our logstash servers. To do so we installed packetbeat with the attached configuration.

Usually this would be a case of searching for all traffic on TLS to the ipaddresses of the endpoint, however these are relatively dynamic as it's a AWS loadbalancer. The challenge which we face is some of the events logged contain a destination.domain with the endpoint, where as the ones with byte fields do not but do contain the IP addresses.

Does anyone have any experience/solutions/pointers how we could monitor this traffic properly?

packetbeat.yaml

#============================== Network device ================================
​
packetbeat:
  interfaces:
    device: any
​
#================================== Flows =====================================
​
packetbeat.flows:
  # Enable Network flows. Default: true
  #enabled: true
​
  # Set network flow timeout. Flow is killed if no packet is received before being
  # timed out.
  timeout: 30s
​
  # Configure reporting period. If set to -1, only killed flows will be reported
  period: 10s
​
  # Set to true to publish fields with null values in events.
  #keep_null: false
​
#========================== Transaction protocols =============================
​
packetbeat.protocols:
  - type: http
    enabled: true
    ports: [80]
    tags: ["elk", "logstash"]
  - type: tls
    enabled: true
    ports: [443]
    tags: ["elk", "logstash"]
​
#================================ Processors ===================================

[cheiligers]

Hi @sholzhauer,
I'm not sure if your question is about how to visualize the HTTPS traffic based on url or if it's related to how to set up the packetbeat.yaml configuration to monitor the traffic. If it's the latter, the Beats channel is the appropriate place to post in.

If it's a visualization you're asking about, there's a handy reference in the docs for that. You'll want to select the stack version you're working with in the docs.

[sholzhauer]

Hello @cheiligers,

The challenge is in finding/combining the events.

We cant build a visualization based on IP because these are dynamic and the events containing the "domain" where the data is being send to, do not contain byte fields.

So in short: how can we monitor amount of data based on a domain 'example.com'?

[cheiligers]

@sholzhauer, can you provide an example of a doc you're working with? I need to see what it contains (maybe 2 or 3 of them, since the IP is dynamic) and I can try to help you figure out the combination of events for such a visualization.

[sholzhauer]

Hi @cheiligers

Sure, the first event contains the domain example.com with the ip 127.0.0.7.

{
	"_index": "packetbeat-2020.04.06",
	"_type": "_doc",
	"_id": "aaaaaaaaaaaaaaaa",
	"_version": 1,
	"_score": null,
	"_source": {
		"cloud": {
			"key": "value"
		},
		"agent": {
			"id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa",
			"ephemeral_id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa",
			"version": "7.6.1",
			"type": "packetbeat",
			"hostname": "hostname"
		},
		"event": {
			"dataset": "tls",
			"category": "network_traffic",
			"duration": 15892000,
			"end": "2020-04-06T15:26:00.247Z",
			"kind": "event",
			"start": "2020-04-06T15:26:00.231Z"
		},
		"@version": "1",
		"type": "tls",
		"client": {
			"ip": "172.25.101.24",
			"port": 40308
		},
		"server": {
			"domain": "example.com",
			"port": 443,
			"ip": "127.0.0.7"
		},
		"tags": [
			"elk",
			"logstash",
			"beats_input_raw_event"
		],
		"destination": {
			"domain": "example.com",
			"ip": "127.0.0.7",
			"port": 443
		},
    "network": {
      "type": "ipv4",
      "direction": "outbound",
      "transport": "tcp",
      "protocol": "tls",
      "community_id": "1:5LSg8UfDqhHoSi6ZWoZ+c08QHps="
    },
    "ecs": {
			"version": "1.4.0"
		},
		"host": {
			"mac": [
				"00:00:00:00:00:00"
			],
			"ip": [
				"172.25.101.24",
				"000::000:000:000:000"
			],
			"architecture": "x86_64",
			"hostname": "hostname",
			"id": "ec25772e440441406d2a2f89789bb5ae",
			"name": "hostname",
			"os": {
				"key": "value"
			},
			"containerized": false
		},
		"status": "OK",
		"@timestamp": "2020-04-06T15:26:00.231Z",
		"tls": "tls_fields",
		"source": {
			"port": 40308,
			"ip": "172.25.101.24"
		}
	},
	"sort": [
		1586186760231
	]
}

The second contains the byte fields which i want to visualize

{
  "_index": "packetbeat-2020.04.06",
	"_type": "_doc",
	"_id": "aaaaaaaaaaaaaaaa",
	"_version": 1,
	"_score": null,
  "_source": {
    "host": {
			"mac": [
				"00:00:00:00:00:00"
			],
			"ip": [
				"172.25.101.24",
				"000::000:000:000:000"
			],
			"architecture": "x86_64",
			"hostname": "hostname",
			"id": "ec25772e440441406d2a2f89789bb5ae",
			"name": "hostname",
			"os": {
				"key": "value"
			},
			"containerized": false
		},
		"network": {
      "community_id": "1:5LSg8UfDqhHoSi6ZWoZ+c08QHps=",
      "packets": 19,
      "bytes": 8006,
      "transport": "tcp",
      "type": "ipv4"
    },
    "event": {
      "start": "2020-04-06T15:26:00.649Z",
      "end": "2020-04-06T15:26:00.650Z",
      "duration": 866345,
      "category": "network_traffic",
      "action": "network_flow",
      "dataset": "flow",
      "kind": "event"
    },
    "flow": {
      "final": false,
      "id": "EAz/////AP//////CAwAAAESy/IXrBllGLsBdJ17YgEAAAAAAA"
    },
    "cloud": {
			"key": "value"
		},
		"source": {
      "packets": 11,
      "bytes": 2257,
      "ip": "172.25.101.24",
      "port": 40308
    },
    "@version": "1",
    "type": "flow",
    "@timestamp": "2020-04-06T15:26:10.000Z",
    "destination": {
      "packets": 8,
      "bytes": 5749,
      "ip": "127.0.07",
      "port": 443
    },
    "ecs": {
      "version": "1.4.0"
    },
    "tags": [
      "tag"
    ],
    "agent": {
			"id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa",
			"ephemeral_id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa",
			"version": "7.6.1",
			"type": "packetbeat",
			"hostname": "hostname"
		
  },
  "sort": [
    1586186770000
  ]
}

So ultimately how do i combine these events so i can visualize the amount of data going to example.com?

[cheiligers]

I see what the difficulty is here: The two documents from your example have different mappings, so the fields you want to visualize don't exist in the same doc. That makes visualizing the data a little tricky.

I was able to create a table with the data from these 2 docs (you should be able to do the same for a metric or gauge visualization)

You'll be able to visualize the data if the fields you want to visualize (the domain fields and the bytes fields) are in the same doc. You could try reindexing using a pipeline with a script processor that compares the docs and searches for a common, unique, field (or work on an event timestamp range to cross correlate the docs). Another option is to try the experimental
ES transform API to pivot the data into the shape you need.
I hope that helps!