Parse subdomain from fqdn in logstash

Elastic Stack Logstash
1

Hi,

I have several events coming in logstash with info of web link accessed.

example:
DNS query is called for the name crl.sectigo.com, type 28
DNS query is called for the name repo.maven.apache.org, type 2
DNS query is called for the name crl.maven.apache.random.org, type 1

I parsed them using grok in a separate field.

example:
crl.sectigo.com
repo.maven.apache.org
crl.maven.apache.random.org

I want to have one more field with just last 2 subdomains.

example:
sectigo.com
apache.org
random.org

I tried many logics with grok etc. but of no use. Thing is I can get a random domain name, no clue with how many subdomains in it. I need to have only last 2 in a field.

2

Hi,

This grok pattern give you the result you want :
%{WORD:name}.%{WORD:TLD}$
With this pattern, you sai you want two words separated by a dot and they are placed at the end of your field (with the character $)

Cad.

3

You have to be careful with this because country tlds have an extra characters on the front of a domain. So qbc.co becomes qbc.co.uk and this logic would only return co.uk. I don't think logstash has a registered domain processor but filebeat and elasticsearch do. See Registered Domain | Filebeat Reference [7.12] | Elastic.

4

The 'tld' filter exists precisely for this purpose. To do this correctly is very non-trivial and cannot be done algorithmically; you need to use the 'public suffix' data, which is what the 'tld' plugin (and others like it) use.

Cheers,
Cameron

(PS. In case you come across a plugin I wrote called logstash-filter-dnssummary, I would suggest you stick with the 'tld' plugin, just because its more maintained; unless perhaps you care about Unicode and IDNA)

5

Thanks all for tld suggestion. Thing is, my logstash server does not have access to internet. And tld filter needs to be manually installed. It does not come with the bundle. Any suggestions?

6

Use the plugin manager.

7

Or if you only lack direct access to the internet, but can still go through a HTTP proxy, you could use it via a proxy (port 3128 is the default port for a Squid proxy; other proxies vary)

https_proxy=myproxy.example.com:3128 http_proxy=myproxy.example.com:3128 logstash_plugin install logstash-filter-tld
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.