I'm not sure what the best way to tackle an issue I'm having in regards to
DNS resolution of IP addresses in my ES documents.
For the background, I'm using logstash as a netflow collector --> ES. I was
previously using the dns filter of logstash to reverse lookup IP fields in
realtime but that caused performance issues and it seems like records were
being lost. So my question is - is it more efficient for me to continue
trying to tackle this in logstash (before records are placed into ES) or
would it make more sense for me to do something after the record is in ES?
I don't have an issue with the delay of having the DNS resolution, so I
imagine going through the previous hour, every hour to batch update records.
If someone could point me in the right direction I'd greatly appreciate it.
I'm very new to the whole ELK stack so apologies if I'm missing something
obvious.
I'm not sure what the best way to tackle an issue I'm having in regards to
DNS resolution of IP addresses in my ES documents.
For the background, I'm using logstash as a netflow collector --> ES. I
was previously using the dns filter of logstash to reverse lookup IP fields
in realtime but that caused performance issues and it seems like records
were being lost. So my question is - is it more efficient for me to
continue trying to tackle this in logstash (before records are placed into
ES) or would it make more sense for me to do something after the record is
in ES? I don't have an issue with the delay of having the DNS resolution,
so I imagine going through the previous hour, every hour to batch update
records.
If someone could point me in the right direction I'd greatly appreciate
it. I'm very new to the whole ELK stack so apologies if I'm missing
something obvious.
For the background, I'm using logstash as a netflow collector --> ES. I was
previously using the dns filter of logstash to reverse lookup IP fields in
realtime but that caused performance issues and it seems like records were
being lost. So my question is - is it more efficient for me to continue
trying to tackle this in logstash (before records are placed into ES) or
would it make more sense for me to do something after the record is in ES?
I don't have an issue with the delay of having the DNS resolution, so I
imagine going through the previous hour, every hour to batch update records.
I've found that running a caching nameserver on the logstash server and
setting /etc/resolv.conf to use the local name server massively improves
the performance of the dns filter in logstash. Otherwise, you lots of
off-server dns lookups which take time.
For the background, I'm using logstash as a netflow collector --> ES. I
was
previously using the dns filter of logstash to reverse lookup IP fields
in
realtime but that caused performance issues and it seems like records
were
being lost. So my question is - is it more efficient for me to continue
trying to tackle this in logstash (before records are placed into ES) or
would it make more sense for me to do something after the record is in
ES?
I don't have an issue with the delay of having the DNS resolution, so I
imagine going through the previous hour, every hour to batch update
records.
I've found that running a caching nameserver on the logstash server and
setting /etc/resolv.conf to use the local name server massively improves
the performance of the dns filter in logstash. Otherwise, you lots of
off-server dns lookups which take time.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
