# \[ES5.1.2\] Help with Detecting DNS Tunnels with PacketBeat+Watcher **URL:** \ **Category:** Beats\ **Tags:** packetbeat\ **Created:** [January 23, 2017, 6:17pm UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522 "2017-01-23T18:17:23Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Snowman](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Snowman](https://discuss.elastic.co/u/Snowman)\ **Post date:** [January 23, 2017, 6:17pm UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/1 "2017-01-23T18:17:23Z")
Hi, thanks a lot for your reply. I am using Packetbeat - 5.1.2 (My Kibana and also Elastic are 5.1.2) from elasticsearch repo. curl -XGET "[http://localhost:9200/packetbeat-\*/dns/\_search?pretty&q=dns.question.etld\_plus\_one:2](http://localhost:9200/packetbeat-*/dns/_search?pretty&q=dns.question.etld_plus_one:2)" outputs : [PASTEBIN LINK](http://pastebin.com/di6QeGZy) curl -XGET "[http://localhost:9200/packetbeat-2017.01.23/\_mapping/field/dns.question.](http://localhost:9200/packetbeat-2017.01.23/_mapping/field/dns.question.)\*" outputs : `{"packetbeat-2017.01.23":{"mappings":{"dns":{"dns.question.class":{"full_name":"dns.question.class","mapping":{"class":{"type":"text","fielddata":true}}},"dns.question.type":{"full_name":"dns.question.type","mapping":{"type":{"type":"text","fielddata":true}}},"dns.question.etld_plus_one":{"full_name":"dns.question.etld_plus_one","mapping":{"etld_plus_one":{"type":"text","fielddata":true}}},"dns.question.name":{"full_name":"dns.question.name","mapping":{"name":{"type":"keyword","fields":{"analyzed":{"type":"text","analyzer":"domain_name_analyzer","search_analyzer":"keyword"}}}}},"dns.question.name.analyzed":{"full_name":"dns.question.name.analyzed","mapping":{"analyzed":{"type":"text","analyzer":"domain_name_analyzer","search_analyzer":"keyword"}}}}}}}` Regards, Martin * * * > Hi Martin, could you please post your question to [Beats - Discuss the Elastic Stack](https://discuss.elastic.co/c/beats/packetbeat) and then let's move the discussion there. > Please include what version of Packetbeat you are using. > I noticed that the by\_domain bucket keys are strange in your results. The keys should always end in dots (like [google.com](http://google.com). or [amazon.co.uk](http://amazon.co.uk).). If you run a query like curl -XGET "[http://localhost:9200/packetbeat-\*/dns/\_search?pretty&q=dns.question.etld\_plus\_one:2](http://localhost:9200/packetbeat-*/dns/_search?pretty&q=dns.question.etld_plus_one:2)" this should show some documents that correspond to the "2" domain. Please include this info in your post. > What is the output of the following query to show the field mappings being used the dns.question dictionary? > curl -XGET "[http://localhost:9200/packetbeat-2017.01.23/\_mapping/field/dns.question.](http://localhost:9200/packetbeat-2017.01.23/_mapping/field/dns.question.)\*" > Regards, > Andrew * * * > Hi Andrew, > I wanted to test DNS tunneling detection with ES 5.1.2 ( [https://github.com/elastic/examples/tree/master/packetbeat\_dns\_tunnel\_detection1](https://github.com/elastic/examples/tree/master/packetbeat_dns_tunnel_detection1) ) based on [Detecting DNS Tunnels with Packetbeat and Watcher | Elastic Blog](https://www.elastic.co/blog/detecting_dns_tunnels_with_packetbeat_and_watcher) that you wrote some time ago, however i came to problem with output of painless script / unique\_hostnames\_watch.json script. > It outputs really weird domains, ie. domain “com” or “1”,”2” or "support" and so on. I attached output from : > curl -XPUT [http://localhost:9200/\_watcher/watch/\_execute?pretty](http://localhost:9200/_watcher/watch/_execute?pretty) -d@unique\_hostnames\_watch.json > I am relatively new to elasticsearch/kibana/packetbeam and i was unable to reverse whats going on and where parsing of dns.question.etld\_plus\_one field fails. This leads me to three conclusions : > 1/ i made some ridiculous mistake and my parsing is not working as it should 🙂 > 2/ guide you provided produce in our infrastructure huge amount of false-positive requests (I was unable to find them with Discovery tool in Kibana) > 3/ dunno whats going on because i am new to elasticstack > Moreover i was unable to create visualisation Kibana like you described in [elastic.co](http://elastic.co) blog - like [https://www.elastic.co/assets/blt8aad375573dc7783/unique-fqdn-per-domain.png](https://www.elastic.co/assets/blt8aad375573dc7783/unique-fqdn-per-domain.png) . I am able to create graph of requests per unique domains, but not for unique FQDNs per Second level domain. I would like to setup our infrastructure in next 30 days and then request for X-PACK pricing offer from elastic because of Alerting feature is payed. > I would be really happy to hear some advices from you. > Thanks. > Regards, > Martin ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [January 23, 2017, 6:44pm UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/2 "2017-01-23T18:44:45Z")
It looks like you do not have the index template for Packetbeat installed. The `dns.question.etld_plus_one` field is mapped incorrectly as a `text` type field with fielddata enabled. Because of this the field is being analyzed and tokenized resulting in the strange terms aggregation bucketing. By default Packetbeat should install the index template if you are using the Elasticsearch output, but if you have configured it differently or disabled this option then you need to manually install the index template. See _[Loading the Index Template in Elasticsearch](https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-template.html)_ for more details. Is the template installed? `curl -XGET 'http://localhost:9200/_template/packetbeat?pretty'` Install the template: `curl -XPUT 'http://localhost:9200/_template/packetbeat?pretty' -d@/etc/packetbeat/packetbeat.template.json` You'll need to delete any data that you have indexed in order to correct the field mappings after installing the index template. `curl -XDELETE 'http://localhost:9200/packetbeat-*?pretty'` Then after Packetbeat writes at least one DNS request to ES, if you re-run this query to check the field mappings you should see that `dns.question.etld_plus_one` is a `keyword` type instead of `text`. `curl -XGET 'http://localhost:9200/packetbeat-*/_mapping/field/dns.question.*?pretty'` ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [January 23, 2017, 7:21pm UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/3 "2017-01-23T19:21:00Z")
[Here](https://gist.github.com/andrewkroh/d2f65319f22f8923cae06f65a6461e20#file-dns-tunnel-visualizations-json) are the visualizations I use. You can save the json to file then import them into Kibana (Management -\> Saved Objects -\> Import). Then you can add them to a dashboard of your own. ![](https://us1.discourse-cdn.com/elastic/original/2X/7/733de1821acd8be94e5dede605ed9e692056dd90.png) ---
**Author:** ![Snowman](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Snowman](https://discuss.elastic.co/u/Snowman)\ **Post date:** [February 3, 2017, 9:21am UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/4 "2017-02-03T09:21:18Z")
It worked like charm! Thanks, I didn't realize that template is not installed. Now everything works as it should. However I encountered problem with reverse domains : `83.in-addr.arpa.={unique_hostnames=302, total_bytes_in=13623.0, total_bytes=31943.0, total_requests=382, total_bytes_out=18320.0}, 217.in-addr.arpa.={unique_hostnames=367, total_bytes_in=60966.0, total_bytes=245222.0, total_requests=3595, total_bytes_out=184256.0}` Is it possible to put regexp to dns.question.etld\_plus\_one "must not" terms ? So i will be able to whitelist all in-addr.arpa domains. Meanwhile I tried to import you graph but it throws error : `Saved "field" parameter is now invalid. Please select a new field.` Visualisation objects will import but when I try to view them there is some problem with "field input". It shows error like : `Visualize: "field" is a required parameter`. `Saved "field" parameter is now invalid. Please select a new field.` ![](https://us1.discourse-cdn.com/elastic/original/2X/d/d6bcc0076b249a8217ec94601780753246e4c20d.png) Dunno why, do you have ES 5.1.x like me ? Thanks in advance. Regards, Martin ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [February 3, 2017, 9:30pm UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/5 "2017-02-03T21:30:05Z")
You can filter reverse lookups from the query by excluding records where `dns.question.type` is `PTR`. Not sure what's going on with the dashboards, I did test importing them on a clean instance before sharing them. But in any case it should be easy to fix them. You might need to refresh your index mappings in Kibana (Management -\> Index Patterns -\> packetbeat-\* -\> then click on the button to "refresh field list"). The red field in your image is supposed to be set to `dns.question.etld_plus_one`. ---
**Author:** ![Snowman](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Snowman](https://discuss.elastic.co/u/Snowman)\ **Post date:** [February 13, 2017, 5:43pm UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/6 "2017-02-13T17:43:23Z")
Graphs now working fine after field refresh. > [@andrewkroh](#): > > You can filter reverse lookups from the query by excluding records where dns.question.type is PTR. I moddified json like : "must\_not": { "terms": { "dns.question.type": "PTR" }, "terms": { "dns.question.etld\_plus\_one": [ "[yahoo.com](http://yahoo.com)." ] } It throws : `failed to execute [search] input for watch [_inlined_], reason [[terms] query does not support [dns.question.type]]` , any suggestions ? Thanks for patience. ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [February 15, 2017, 8:50pm UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/7 "2017-02-15T20:50:47Z")
It looks like the query is malformed, try using something like this as part of the query. ```auto { "query": { "bool": { "filter": { "range": { "@timestamp": { "from": "now-4h" } } }, "must_not": [ { "terms": { "dns.question.etld_plus_one": [ "yahoo.com." ] } }, { "term": { "dns.question.type": "PTR" } } ] } } } ``` ---
**Author:** ![Snowman](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@Snowman](https://discuss.elastic.co/u/Snowman)\ **Post date:** [March 9, 2017, 10:05am UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/8 "2017-03-09T10:05:22Z")
Hi Andrew, everything is working as expected. Thanks for your help. I would like to proceed because my trial license has expired. I found [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions) link I want to ensure if i have to buy Gold subscription because its the lowest package with Alerting (Watcher). Is it was mentioned upper I focus my infrastructure to Detect DNS Tunnels. Thanks for reply. BR ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [March 11, 2017, 6:03am UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/9 "2017-03-11T06:03:47Z")
Hi Martin, I asked one of the sales people to reach out to you. They will be able to answer all of your subscription questions. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 8, 2017, 6:03am UTC](https://discuss.elastic.co/t/es5-1-2-help-with-detecting-dns-tunnels-with-packetbeat-watcher/72522/10 "2017-04-08T06:03:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.