Migrating packetbeat ingestion from Elasticsearch to Logstash

[parthmaniar (Parth D. Maniar)]

Hello,

I currently push packetbeat logs from hosts directly to elasticsearch with a geo-pipeline. I want to migrate this compute load to a dedicated logstash pipeline. Stack is running at 7.7.1.

I need help with:

1. Geo-Pipeline configuration using GeoMind DB.
2. DNS configuration assistance as I am getting PTR hits on my recurssive DNS servers and there are no resolutions. Example:
   
   

input {
  beats {
    port => 7044
  }
}

filter {
  dns {
    reverse => [ "src_host" ]
    nameserver => [ "DNS server IP" ]
    action => "replace"
    hit_cache_size => 4096
    hit_cache_ttl => 900
    failed_cache_size => 512
    failed_cache_ttl => 900
    }

  geoip {
    source => "src_ip"
    target => "geoip"
    database => "/opt/logstash/vendor/geoip/GeoLite2-City.mmdb"
  }

}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    ssl => true
    user => redacted
    password => redacted
    cacert => '/etc/logstash/elasticsearch-ca.pem'
    ssl_certificate_verification => false
    ilm_enabled => false
  }
}

Does this seem correct?

What changes do I need to make for the DNS queries not to be PTR ones and fail on the recursive DNS server?

[ptamba]

@parthmaniar

nameserver => [ "DNS server IP" ]

based on this i think you need to change this to :

nameserver => { 
  address = [“dns server ip”] 
}

and if you want dns resolution rather than reverse DNS resolution, use resolve instead of reverse

[parthmaniar (Parth D. Maniar)]

Hi Thank you for the reply. I am trying to get hostnames (if published) of IP addresses that the endpoint is trying to connect to. So a reverse DNS lookup. The parameter for the same would be "resolve", & hence I reckon my configuration is correct?: Instead of using my own DNS server (unbound), I am changing the IP to 9.9.9.9. Lets see if that fixes the PTR issue.

filter {
  dns {
    reverse => [ "src_host" ]
    nameserver => [ "DNS server IP" ]
    action => "replace"
    hit_cache_size => 4096
    hit_cache_ttl => 900
    failed_cache_size => 512
    failed_cache_ttl => 900
    }

[ptamba]

@parthmaniar

nameserver => [ "DNS server IP" ]

this still needs to be adjusted to follow the correct syntax as required by the documentation

[parthmaniar (Parth D. Maniar)]

Please pardon me but where am I going wrong. Just unable to grasp. Sorry

[ptamba]

@ptamba

nameserver => { 
  address => [“dns server ip”] 
}

instead of

@parthmaniar

nameserver => [ "DNS server IP" ]

[parthmaniar (Parth D. Maniar)]

Thank you very much and I am sorry you had to spoon feed me that.

[parthmaniar (Parth D. Maniar)]

I changed the configuration but I am still seeing PTR record request such as 45.247.250.180.in-addr.arpa.

It should be doing a reverse lookup and for the IP, there is no hostname defined. I am not suspecting my DNS server and the way it handles PTR requests. Also is the appending of .in.addr.arpa correct?

[ptamba]

you could try asking your dns server for PTR records of the said IP (using nslookup or dig) and compare the DNS server answer vs the result you get logstash though I doubt it will be different provided the config is correct.

[parthmaniar (Parth D. Maniar)]

Thank you very much. Marking one of your posts as the answer for future reference. Thank you.