How to filter logs on remote server before shipping to logstash server

Elastic Stack Elasticsearch
1

Hi all,

I am new to ELK and my organization is interested in implementing this
framework.
I setup ELK on my machine and trying to collect logs from remote server.
But the logs on the remote server are huge in size (in GigaBytes).
I see we can use logstash shipper or logstash forwarder. But I don't want
to forward all the data to central ELK server for indexing.

  1. So I want to filter data on the remote servers locally before being sent
    to ELK server. How can we do this?

Also we have many such remote servers from which I want to collect filtered
logs on ELK server.

  1. Can someone suggest the recommended architecture to collect 'filtered'
    logs from multiple remote server and forward them to central ELK server?

  2. Also what could be the performance impact and CPU utilization to have
    logstash filtering on each remote server,

    when compared to one central logstash server with logstash forwarder on
    each remote servers?

Thanks,
Vilas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f5cad331-bf0a-445c-bdfd-6ba72df7c2b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

On Thursday, November 06, 2014 at 20:27 CET,
Vilas Reddy pvilasreddy@gmail.com wrote:

I am new to ELK and my organization is interested in implementing this
framework.
I setup ELK on my machine and trying to collect logs from remote
server.
But the logs on the remote server are huge in size (in GigaBytes).
I see we can use logstash shipper or logstash forwarder. But I don't
want to forward all the data to central ELK server for indexing.

May I suggest you take this question to the logstash-users mailing list
as it's unrelated to Elasticsearch itself. While some Logstash folks
follow this list as well the message is more on-topic there.

https://groups.google.com/forum/#!forum/logstash-users

[...]

--
Magnus Bäck | Software Engineer, Development Tools
magnus.back@sonymobile.com | Sony Mobile Communications

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141110075106.GC6370%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

3

Thanks Magnus.
Submitted to the suggested logstash group.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb83a0c1-6086-4af3-bd4e-6a8ff33cc1d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.