Mass update of records - dns resolution

Howdy,

I'm not sure what the best way to tackle an issue I'm having in regards to
DNS resolution of IP addresses in my ES documents.

For the background, I'm using logstash as a netflow collector --> ES. I was
previously using the dns filter of logstash to reverse lookup IP fields in
realtime but that caused performance issues and it seems like records were
being lost. So my question is - is it more efficient for me to continue
trying to tackle this in logstash (before records are placed into ES) or
would it make more sense for me to do something after the record is in ES?
I don't have an issue with the delay of having the DNS resolution, so I
imagine going through the previous hour, every hour to batch update records.

If someone could point me in the right direction I'd greatly appreciate it.
I'm very new to the whole ELK stack so apologies if I'm missing something
obvious.

Thanks in advance.

--Mike

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ac8302b7-2975-4d55-9554-33368decb08c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You could do this with a script, you'd have to develop that yourself though.

But ultimately you may better off doing this in Logstash, just head over to
Redirecting to Google Groups and ask for
help on sorting it out :slight_smile:

On 2 January 2015 at 15:00, Mike Sheinberg m.sheiny@gmail.com wrote:

Howdy,

I'm not sure what the best way to tackle an issue I'm having in regards to
DNS resolution of IP addresses in my ES documents.

For the background, I'm using logstash as a netflow collector --> ES. I
was previously using the dns filter of logstash to reverse lookup IP fields
in realtime but that caused performance issues and it seems like records
were being lost. So my question is - is it more efficient for me to
continue trying to tackle this in logstash (before records are placed into
ES) or would it make more sense for me to do something after the record is
in ES? I don't have an issue with the delay of having the DNS resolution,
so I imagine going through the previous hour, every hour to batch update
records.

If someone could point me in the right direction I'd greatly appreciate
it. I'm very new to the whole ELK stack so apologies if I'm missing
something obvious.

Thanks in advance.

--Mike

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ac8302b7-2975-4d55-9554-33368decb08c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ac8302b7-2975-4d55-9554-33368decb08c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9k4RGJ2DydRyiBzkru8T7dSNAq04MqLMRz9buCcoDgVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Mike Sheinberg wrote on 1/1/2015 11:00 PM:

For the background, I'm using logstash as a netflow collector --> ES. I was
previously using the dns filter of logstash to reverse lookup IP fields in
realtime but that caused performance issues and it seems like records were
being lost. So my question is - is it more efficient for me to continue
trying to tackle this in logstash (before records are placed into ES) or
would it make more sense for me to do something after the record is in ES?
I don't have an issue with the delay of having the DNS resolution, so I
imagine going through the previous hour, every hour to batch update records.

I've found that running a caching nameserver on the logstash server and
setting /etc/resolv.conf to use the local name server massively improves
the performance of the dns filter in logstash. Otherwise, you lots of
off-server dns lookups which take time.

--[Lance]

--
GPG Fingerprint: 409B A409 A38D 92BF 15D9 6EEE 9A82 F2AC 69AC 07B9
CACert.org Assurer

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54A61D01.7000700%40bearcircle.net.
For more options, visit https://groups.google.com/d/optout.

1 Like

Oooooooooooo - That's a good idea Lance!

On Thu, Jan 1, 2015 at 11:22 PM, Lance A. Brown lance@bearcircle.net
wrote:

Mike Sheinberg wrote on 1/1/2015 11:00 PM:

For the background, I'm using logstash as a netflow collector --> ES. I
was
previously using the dns filter of logstash to reverse lookup IP fields
in
realtime but that caused performance issues and it seems like records
were
being lost. So my question is - is it more efficient for me to continue
trying to tackle this in logstash (before records are placed into ES) or
would it make more sense for me to do something after the record is in
ES?
I don't have an issue with the delay of having the DNS resolution, so I
imagine going through the previous hour, every hour to batch update
records.

I've found that running a caching nameserver on the logstash server and
setting /etc/resolv.conf to use the local name server massively improves
the performance of the dns filter in logstash. Otherwise, you lots of
off-server dns lookups which take time.

--[Lance]

--
GPG Fingerprint: 409B A409 A38D 92BF 15D9 6EEE 9A82 F2AC 69AC 07B9
CACert.org Assurer

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/0XQdFiqD22w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/54A61D01.7000700%40bearcircle.net
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMnwvFyBU1tT_XXEwRPM__Y9zRyGwst2hCx4RXAPR-GqPehYUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.