# HTTPS Traffic visualization on url **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-monitoring\ **Created:** [March 24, 2020, 7:56am UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783 "2020-03-24T07:56:38Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer)\ **Post date:** [March 24, 2020, 7:56am UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783/1 "2020-03-24T07:56:39Z")
Hello, hope everyone is in good health. We want to monitor the amount of data (gb's) which are being send to a https endpoint from our logstash servers. To do so we installed packetbeat with the attached configuration. Usually this would be a case of searching for all traffic on TLS to the ipaddresses of the endpoint, however these are relatively dynamic as it's a AWS loadbalancer. The challenge which we face is some of the events logged contain a destination.domain with the endpoint, where as the ones with byte fields do not but do contain the IP addresses. Does anyone have any experience/solutions/pointers how we could monitor this traffic properly? packetbeat.yaml ```auto #============================== Network device ================================ ​ packetbeat: interfaces: device: any ​ #================================== Flows ===================================== ​ packetbeat.flows: # Enable Network flows. Default: true #enabled: true ​ # Set network flow timeout. Flow is killed if no packet is received before being # timed out. timeout: 30s ​ # Configure reporting period. If set to -1, only killed flows will be reported period: 10s ​ # Set to true to publish fields with null values in events. #keep_null: false ​ #========================== Transaction protocols ============================= ​ packetbeat.protocols: - type: http enabled: true ports: [80] tags: ["elk", "logstash"] - type: tls enabled: true ports: [443] tags: ["elk", "logstash"] ​ #================================ Processors =================================== ``` ---
**Author:** ![cheiligers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheiligers/32/73114_2.png) [@cheiligers](https://discuss.elastic.co/u/cheiligers)\ **Post date:** [April 3, 2020, 4:34pm UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783/2 "2020-04-03T16:34:08Z")
Hi @sholzhauer, I'm not sure if your question is about how to visualize the HTTPS traffic based on url or if it's related to how to set up the `packetbeat.yaml` configuration to monitor the traffic. If it's the latter, the [Beats](https://discuss.elastic.co/c/beats/28) channel is the appropriate place to post in. If it's a visualization you're asking about, there's a handy reference in the [docs](https://www.elastic.co/guide/en/kibana/current/visualize.html) for that. You'll want to select the stack version you're working with in the docs. ---
**Author:** ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer)\ **Post date:** [April 6, 2020, 9:35am UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783/3 "2020-04-06T09:35:19Z")
Hello @cheiligers, The challenge is in finding/combining the events. We cant build a visualization based on IP because these are dynamic and the events containing the "domain" where the data is being send to, do not contain byte fields. So in short: how can we monitor amount of data based on a domain '[example.com](http://example.com)'? ---
**Author:** ![cheiligers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheiligers/32/73114_2.png) [@cheiligers](https://discuss.elastic.co/u/cheiligers)\ **Post date:** [April 6, 2020, 2:40pm UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783/4 "2020-04-06T14:40:49Z")
@sholzhauer, can you provide an example of a doc you're working with? I need to see what it contains (maybe 2 or 3 of them, since the IP is dynamic) and I can try to help you figure out the combination of events for such a visualization. ---
**Author:** ![sholzhauer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sholzhauer/32/110282_2.png) [@sholzhauer](https://discuss.elastic.co/u/sholzhauer)\ **Post date:** [April 6, 2020, 3:48pm UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783/5 "2020-04-06T15:48:17Z")
Hi @cheiligers Sure, the first event contains the domain `example.com` with the ip `127.0.0.7`. ```json { "_index": "packetbeat-2020.04.06", "_type": "_doc", "_id": "aaaaaaaaaaaaaaaa", "_version": 1, "_score": null, "_source": { "cloud": { "key": "value" }, "agent": { "id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa", "ephemeral_id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa", "version": "7.6.1", "type": "packetbeat", "hostname": "hostname" }, "event": { "dataset": "tls", "category": "network_traffic", "duration": 15892000, "end": "2020-04-06T15:26:00.247Z", "kind": "event", "start": "2020-04-06T15:26:00.231Z" }, "@version": "1", "type": "tls", "client": { "ip": "172.25.101.24", "port": 40308 }, "server": { "domain": "example.com", "port": 443, "ip": "127.0.0.7" }, "tags": [ "elk", "logstash", "beats_input_raw_event" ], "destination": { "domain": "example.com", "ip": "127.0.0.7", "port": 443 }, "network": { "type": "ipv4", "direction": "outbound", "transport": "tcp", "protocol": "tls", "community_id": "1:5LSg8UfDqhHoSi6ZWoZ+c08QHps=" }, "ecs": { "version": "1.4.0" }, "host": { "mac": [ "00:00:00:00:00:00" ], "ip": [ "172.25.101.24", "000::000:000:000:000" ], "architecture": "x86_64", "hostname": "hostname", "id": "ec25772e440441406d2a2f89789bb5ae", "name": "hostname", "os": { "key": "value" }, "containerized": false }, "status": "OK", "@timestamp": "2020-04-06T15:26:00.231Z", "tls": "tls_fields", "source": { "port": 40308, "ip": "172.25.101.24" } }, "sort": [ 1586186760231 ] } ``` The second contains the byte fields which i want to visualize ```json { "_index": "packetbeat-2020.04.06", "_type": "_doc", "_id": "aaaaaaaaaaaaaaaa", "_version": 1, "_score": null, "_source": { "host": { "mac": [ "00:00:00:00:00:00" ], "ip": [ "172.25.101.24", "000::000:000:000:000" ], "architecture": "x86_64", "hostname": "hostname", "id": "ec25772e440441406d2a2f89789bb5ae", "name": "hostname", "os": { "key": "value" }, "containerized": false }, "network": { "community_id": "1:5LSg8UfDqhHoSi6ZWoZ+c08QHps=", "packets": 19, "bytes": 8006, "transport": "tcp", "type": "ipv4" }, "event": { "start": "2020-04-06T15:26:00.649Z", "end": "2020-04-06T15:26:00.650Z", "duration": 866345, "category": "network_traffic", "action": "network_flow", "dataset": "flow", "kind": "event" }, "flow": { "final": false, "id": "EAz/////AP//////CAwAAAESy/IXrBllGLsBdJ17YgEAAAAAAA" }, "cloud": { "key": "value" }, "source": { "packets": 11, "bytes": 2257, "ip": "172.25.101.24", "port": 40308 }, "@version": "1", "type": "flow", "@timestamp": "2020-04-06T15:26:10.000Z", "destination": { "packets": 8, "bytes": 5749, "ip": "127.0.07", "port": 443 }, "ecs": { "version": "1.4.0" }, "tags": [ "tag" ], "agent": { "id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa", "ephemeral_id": "aaaaaa-aaaa-aaaa-aaaa-aaaaaaaa", "version": "7.6.1", "type": "packetbeat", "hostname": "hostname" }, "sort": [ 1586186770000 ] } ``` So ultimately how do i combine these events so i can visualize the amount of data going to [example.com](http://example.com)? ---
**Author:** ![cheiligers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cheiligers/32/73114_2.png) [@cheiligers](https://discuss.elastic.co/u/cheiligers)\ **Post date:** [April 6, 2020, 5:33pm UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783/6 "2020-04-06T17:33:24Z")
I see what the difficulty is here: The two documents from your example have different mappings, so the fields you want to visualize don't exist in the same doc. That makes visualizing the data a little tricky. I was able to create a table with the data from these 2 docs (you should be able to do the same for a metric or gauge visualization) ![Screen Shot 2020-04-06 at 10.23.21](https://us1.discourse-cdn.com/elastic/original/3X/9/f/9f4470c58a8209b1a46bc12debcce4a144e6e10f.png) You'll be able to visualize the data if the fields you want to visualize (the domain fields and the bytes fields) are in the same doc. You could try [reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/docs-reindex.html) using a [pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/pipeline-processor.html) with a [script processor](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/script-processor.html) that compares the docs and searches for a common, unique, field (or work on an event timestamp range to cross correlate the docs). Another option is to try the experimental [ES transform API](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) to pivot the data into the shape you need. I hope that helps! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 4, 2020, 5:33pm UTC](https://discuss.elastic.co/t/https-traffic-visualization-on-url/224783/7 "2020-05-04T17:33:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.