Duplicate and forward logs to a new instance of Elasticsearch

Elastic Stack Logstash
1

Hi,
I am fairly new to ELK and I am seeking some advice. I have inherited an ELK stack which is processing logs from multiple sources and I am looking into forwarding some of those logs to another instance of Elasticsearch.

Our current set up is (Cent-OS 7):
Logstash -> Elasticsearch (5.6) -> Kibana

What I want to achieve is to forward some of the logs to a new instance of Elasticsearch and also keep the logs in my current stack. What would be the best way to achieve this?

I was told in the elasticsearch forum that I would have to "duplicate the event to a second output". Any advice is greatly appreciated. Thanks.

2

Add another elasticsearch{} entry to the output{} section of the logstash configuration. Just change the IP/Hostname :slight_smile:

3

Thanks AquaX,
I was able to forward all the output to my 2nd cluster but I believe what I need is a bit different. I'll try to explain:

I want to keep processing all my logs from different sources with logstash and then send them to the localhost instance of elasticsearch. (This is my current setting which I want to keep untouched).

I want to now processs the sames logs with logstash but "extract" and "modify" the same logs, and then send them to my 2nd instance of elasticsearch.

I see this like having another pipeline processing the same logs but using a different output. Is this possible/reasonable?

Thanks in advance!
Camilo

4

Yes, you could use pipeline to pipeline communication to unconditionally send events to two pipelines that process them in different ways.

5

I am running logstash 5.6.6 so I guess pipeline to pipeline won't work for me.

I have different config files for every input and one output file pointing to localhost.

output {
elasticsearch {
hosts => ["localhost:9200"]
}

In my output file I have tried to filter by type and then send those types to my 2nd instance of elasticsearch.
I also configured a completely new input with an output to my 2nd instance of elasticsearch.
In both scenarios my filter did not work and all the information (from all my inputs) was sent to my 2nd instance of elasticsearch as well.

I am a bit lost on how to move forward with this.

edit: typo

6

You could try using lumberjack outputs and a beats inputs to the same effect.

7

Will give that a try and let u know how it goes

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.