SIEM can't detect DNS activity to Internet

robertitox · June 10, 2020, 4:53pm

Dear, I have running an ELK server 7.7 with Elastic SIEM. I have enabled the related pre-built nmap and tcpdump signals detection rules and I could detect these type of "attacks".

But now I have enabled the "DNS Activity to Internet" signal detection rule, it looks for events in the filebeat-* index pattern by default, but after try to connect from some hosts to Internet using 8.8.8.8 Google DNS, the SIEM can't detect this dangerous traffic.

After that I duplicated the signal detection rule and I add the packetbeat-* index pattern in order to search the DNS events, but I still can't detect nothing at all.

Please can you tell me how to config filebeat or packetbeat in order to detect DNS traffic to Internet? I tell you again that the pre-built "DNS Activity to Internet" signal detection rule look into filebeat-* index pattern.

Thanks a lot in advance.

Craig_Chamberlain · June 10, 2020, 6:06pm

Hi Robert - so the "DNS Activity to the Internet Rule" should work with most any network data using the Elastic Common Schema (ECS.) This schema defines common fields like source.ip, destination.port and destination.ip which are used by this rule. The search used by the rule is this:

destination.port:53 and source.ip:(10.0.0.0/8 or 172.16.0.0/12 or 192.168.0.0/16) and not destination.ip:(10.0.0.0/8 or 127.0.0.0/8 or 169.254.169.254/32 or 172.16.0.0/12 or 192.168.0.0/16 or 224.0.0.251 or 224.0.0.252 or 255.255.255.255 or "::1" or "ff02::fb")

Let's troubleshoot this. If you manually run this search against your Packetbeat index, you should see events returned for the DNS traffic you described. If no events are returned, I can think of a few possible reasons.

The Packetbeat instance monitoring this traffic is older and not using ECS fields? I tested this search just now in Packetbeat 7.6.1. Most recent versions of Packetbeat should generate data compatible with this rule.
The Packetbeat instance has a vantage point that sees post-NAT traffic and sees the packets sourcing from a routable / public IP address instead of a private IP. Try searching for "destination.ip: 8.8.8.8 and destination.port: 53" and see if that finds the events coming from a different source IP.
The Packetbeat instance did not record the packets because they did not traverse a network interface, mirror, span or tap monitored by Packetbeat?
The Packetbeat instance did not record the packets because they were dropped by a network interface, mirror, span or tap monitored by Packetbeat?
Could the DNS packets have been dropped by an intermediate firewall or router ACL before they traversed a network segment monitored by Packetbeat? Were the DNS queries successful?

robertitox · June 10, 2020, 7:33pm

Dear Craig, thanks for your response.

I have an ELK server 7.7.0 and I have two Linux boxes with packetbeat and filebeat 7.7.0.

Server and beat boxes are in the same network segment, no NAT and no any other thing in the middle. In the beat box I try to query 4.4.4.4 Google DNS, I run tcpdump and I can see packets going to 4.4.4.4:53. I have not response from 4.4.4.4 because it is blocket by firewall, but the Linux box with packetbeat tries to get it so I suppose the DNS Internet activity detection rule is fired up.

When I run the below query against the packetbeat index:

GET /packetbeat-7.7.0/_search
{
"query": { "match_all": {} },
"sort": [
{ "destination.ip": "asc" }
]
}

I obtain and see that ip: "4.4.4.4" (Google DNS) and not destination.ip, so I'm confused....can you help again please? Regards !!!

"hits" : [
{
"_index" : "packetbeat-7.7.0-2020.05.14-000001",
"_type" : "_doc",
"_id" : "QizsnnIBuVNl7NxaIPft",
"_score" : null,
"_source" : {
"@timestamp" : "2020-06-10T15:49:48.959Z",
"server" : {
"ip" : "4.4.4.4",
"port" : 53
},
"query" : "class IN, type A, www.sintys.gob.ar",
"source" : {
"ip" : "172.31.28.7",
"port" : 46490,
"bytes" : 35
},
"status" : "Error",
"event" : {
"start" : "2020-06-10T15:49:48.959Z",
"kind" : "event",
"category" : "network_traffic",
"dataset" : "dns"
},
"type" : "dns",
"client" : {
"ip" : "172.31.28.7",
"port" : 46490,
"bytes" : 35
},
"resource" : "www.sintys.gob.ar",
"dns" : {
"op_code" : "QUERY",
"header_flags" : [
"RD"
],
"type" : "query",
"flags" : {
"recursion_desired" : true,
"recursion_available" : false,
"authentic_data" : false,
"checking_disabled" : false,
"authoritative" : false,
"truncated_response" : false
},
"response_code" : "NOERROR",
"id" : 53781,
"additionals_count" : 0,
"question" : {
"class" : "IN",
"etld_plus_one" : "sintys.gob.ar",
"registered_domain" : "sintys.gob.ar",
"top_level_domain" : "gob.ar",
"subdomain" : "www",
"name" : "www.sintys.gob.ar",
"type" : "A"
},
"authorities_count" : 0,
"answers_count" : 0
},
"host" : {
"containerized" : false,
"ip" : [
"172.31.28.7"
],
"mac" : [
"00:50:56:b2:97:b1"
],
"hostname" : "MITLTOCS-Agent02",
"architecture" : "x86_64",
"name" : "MITLTOCS-Agent02",
"os" : {
"name" : "Debian GNU/Linux",
"kernel" : "4.9.0-3-amd64",
"codename" : "stretch",
"platform" : "debian",
"version" : "9 (stretch)",
"family" : "debian"
},
"id" : "1436e041f70b4887887077ee84a0bae1"
},
"network" : {
"direction" : "outbound",
"community_id" : "1:PpSYmtSvRXwFbj45QauJiIpR63E=",
"bytes" : 35,
"type" : "ipv4",
"transport" : "udp",
"protocol" : "dns"
},
"agent" : {
"version" : "7.7.0",
"type" : "packetbeat",
"ephemeral_id" : "bfa97b80-78d2-4810-9b30-9ab964af614f",
"hostname" : "MITLTOCS-Agent02",
"id" : "649569ab-b23d-487d-af4b-50654252e501"
},
"ecs" : {
"version" : "1.5.0"
},
"error" : {
"message" : "No response to this query was received"
},
"method" : "QUERY",
"destination" : {
"port" : 53,
"ip" : "4.4.4.4"
}
},
"sort" : [
"4.4.4.4"
]
},

Craig_Chamberlain · June 10, 2020, 7:42pm

If you search the packetbeat-* index in Discover, do you see results like this?

Time	source.ip		destination.ip	destination.port
		10.10.10.10		4.4.4.4			53

robertitox · June 11, 2020, 12:07am

Dear Craig, effectively in Discover --> Packetbeat I can see an output like you told me:

Jun 10, 2020 @ 13:25:00.000 172.31.28.6 8.8.8.8 53

Jun 10, 2020 @ 13:24:50.000 172.31.28.6 8.8.8.8 53

Jun 10, 2020 @ 13:24:40.000 172.31.28.6 8.8.8.8 53

I tell you again the server 172.31.28.6 is trying to resolve public hostnames against 8.8.8.8, but 8.8.8.8 is blocked by the corporate firewall, so there is no incoming DNS traffic.

And the DNS Internet Activity signal detection rule searches into filebeat-* by default, and I've added packetbeat-*.

I'd appreciate your help please.

Regards !!!

Craig_Chamberlain · June 11, 2020, 12:34am

Can you share a screenshot of the modified rule?

robertitox · June 11, 2020, 1:42am

Yes, of course Craig, here it is.

Thanks a lot again!!!

Craig_Chamberlain · June 11, 2020, 1:57am

OK that looks good of course. Is there anything in "Failure History"?

I see no "last signal" status so I assume the Signal Count and Signals sections further down in the rule are empty. Can you confirm there are no signals for this rule by searching for *DNS* in the Detections page?

Back in the custom rules list, if you click "Monitoring" can you share the values for this rule? Are there any errors or gaps in the monitoring data or stats for this rule?

Craig_Chamberlain · June 11, 2020, 2:02am

One other thing occurred to me - in your Packetbeat search results, can you confirm that those fields are named like this and not something else -

source.ip                   destination.ip           destination.port
172.31.28.6               8.8.8.8                        53

robertitox · June 11, 2020, 1:45pm

Hi Craig, I tell you:

In the Packetbeat results, the fields are named as I've shown to you:

source.ip / destination.ip / destination.port

In "Failure History", there is nothing at all, it's empty.
I see no "last signal" status and the Signal Count and Signals sections further down in the rule are empty. I confirm there are no signals for this rule by searching for *DNS* in the Detections page.
The rule monitoring:

Please, If there is something else I'd can check, tell me what.

Greetings !!!

Craig_Chamberlain · June 11, 2020, 2:14pm

OK let’s rule out timestamps. Which field are you using for the timestamp in your packetbeat index?

Please try these steps;

Modify your cloned DNS rule to lookback five hours instead of five minutes.
Generate the DNS traffic and verify the events are found in Discover.
Search for DNS signals to see if the rule fired on the events.

If no results, please repeat with a longer lookback if possible?

robertitox · June 11, 2020, 4:07pm

Craig,

In my custom DNS Activity rule I've modified the look-back time to 5 hours:

After that I've generated the DNS traffic to 8.8.8.8 Google DNS and I can found the events in Discover looking at Packetbeat index:

Finally I went to Detection --> Signals but no DNS rule was fired at all.

After that I've repeated with a look-back time of 10 hours, but no DNS rule was fired again.

The packetbeats from my test servers send the output to Elasticsearch, not to Logstash.

Is there any chance to view a related log file for a deeper analysis ???

Thanks a lot again !!!

Craig_Chamberlain · June 11, 2020, 4:51pm

OK looking at your screenshots I think you may be in the EST time zone? Let's check further for timestamp issues. Can you do the following:

check the system time and time zone on your endpoint
check the same on the Packetbeat host
Generate the DNS traffic again and note the system time on the host where you ran this.
Search for the Packetbeat events again for this DNS traffic and gather the values of the following fields from one or more Packetbeat events:
```
 event.start
 event.end
 @timestamp
```
Do the timestamps match the system time you noted in step 3?

robertitox · June 11, 2020, 8:57pm

Dear Craig, I tell you:

ELK server --> Attached to NTP server / GMT-3
Packetbeat host --> Attached to the same NTP server / GMT-3

I executed the "date" command on both hosts and the date and time are the same.

I generated DNS traffic at Packetbeat host:

host www.google.com 8.8.8.8

These are Packetbeats events on Discovery:

I reviewed the Timezone for Date formatting:

And the DNS Internet Activity rule is not fired at all...

I could put to work the Nmap, Tcpdump and Whoami rules....but couldn't this DNS and the SMTP to Internet rules. In both DNS and SMTP to Internet rules, the default index is filebeat-*, but with just this index they don't work because some fields like destination.port don't appear in filebeat index.

Any idea?

Thanks a lot again !!!

Craig_Chamberlain · June 11, 2020, 9:29pm

I'm guessing you are running the other rules (nmap, tcpdump, whoami) on auditbeat data? I see the auditbeat index in the rule details - does the search also return events from the auditbeat index for the dns connections to 8.8.8.8, if you connect from an auditbeat instance? Does the rule create a signal on this activity from an auditbeat instance?

Craig_Chamberlain · June 11, 2020, 9:31pm

one more thing - it's not possible the dates in the Packetbeat events were shifted into the future is it? Did those timestamps agree with the local system time when you generated the DNS activity?

There is one issue where timestamps can be handled incorrectly which is covered here: https://github.com/elastic/kibana/issues/57649

As a workaround, if you check and update your field mapping/beats config/ingest pipeline to ensure @timestamp is being ingested as ISO 8601 with timezone designators (e.g. 2020-02-11T23:59:55Z ) that may be the fix.

robertitox · June 12, 2020, 1:38am

Hi Craig, I tell you now:

Auditbeat index does not have the DNS events, it doesn't work.
In the duplicate DNS rule, I deleted filebeat-* and auditbeat-* patterns and I just defined packetbeat-*, and in this case it works OK !!!
In the signal detection window the @timestamp for a given signal is the rule running time, and it doesn't match the time.event_start and time.event_end from the packetbeat-* pattern....I think this is OK, isn't it ???
Now I think I can't use a rule with more than one index to search into, this is my problem and I don't know the reason.
I couldn't understand the time definition problem you pointed to me. Maybe I have to read in depth, but my question is this: is there any easy way to check if my Date/Time format is OK ???

Special thanks again !!!

robertitox · June 16, 2020, 4:39pm

Dear Craig, how are you?

I want to tell you that after ELK upgrade from 7.7.0 to 7.7.1, the SNS Activity to Internet works OK inspecting two indexes. Maybe it was a bug, resolved in this new stable version.

Thanks a lot, cheers !

Craig_Chamberlain · June 16, 2020, 6:27pm

Hi Robert - glad to hear that. I tested this myself in 7.7.1 and it works ok here too. If you should receive hits from this rule, you can also use Auditbeat or Winlogbeat data to identify and investigate the process generating such DNS traffic, if you wish. Regards, Craig

robertitox · June 17, 2020, 12:54pm

Dear Craig, just a last question please:

When you say:

"If you should receive hits from this rule, you can also use Auditbeat or Winlogbeat data to identify and investigate the process generating such DNS traffic"

Does this imply that I have to investigate the process generating the DNS traffic in Discover option from Kibana, selecting and searching into auditbeat-* and filebeat-*? Or is there any specific way to do this analysis? Because now I need to know if the DNS query to the Internet was made by host command, dig command, etc.

Thanks a lot again !!!