How to run multiple logstash instances for s3 input

Elastic Stack Logstash
1

Hello.

I am trying to install multiple logstash instances for s3 input but it seems to be impossible because each logstash saves a sincedb file locally and even if the sincedb file is shared between the logstash instances, the same object of s3 may be processed simultaneously by multiple logstash instances.

Do you have any idea?

2 Likes
2

What does your input look like?

3

Here is my input sample.

input {
s3 {
access_key_id => "XXXXXXXXXXXXXXXXXXXXX"
secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXX"
region => "XXXXXXXXXX"
bucket => "XXXXXXXX"
prefix => "XXXXX/XXXXX/"
tags => [ "XXXXXXXX","XXXXXXX" ]
type => "elb"
sincedb_path => "/var/lib/logstash/.sincedb_XXXXXXXXXX_elb"
temporary_directory => "/tmp/logstash/input_XXXXXXXXXX_elb"
}
}

4

So just do;

input {
s3 {
access_key_id => "XXXXXXXXXXXXXXXXXXXXX"
secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXX"
region => "XXXXXXXXXX"
bucket => "XXXXXXXX"
prefix => "XXXXX/XXXXX/"
tags => [ "XXXXXXXX","XXXXXXX" ]
type => "elb"
sincedb_path => "/var/lib/logstash/.sincedb_XXXXXXXXXX_elb"
temporary_directory => "/tmp/logstash/input_XXXXXXXXXX_elb"
}
s3 {
access_key_id => "YYYY"
secret_access_key => "YYYY"
region => "YYYY"
bucket => "YYYY"
prefix => "YYYY/YYYY/"
tags => [ "YYYY","YYYY" ]
type => "elb"
sincedb_path => "/var/lib/logstash/.sincedb_YYYY_elb"
temporary_directory => "/tmp/logstash/input_YYYY_elb"
}
}

Where YYYY and the XXXXXXXXXXXXXXXXXXXXX values are different.

5

The multiple instances are autoscaled backend instances of ELB and the instances have a same configuration as below.

input {
beats {
...
}

s3 {
...
}

I want them to process same bucket and prefix of s3.

6

Hi Mark - I think Hugo's problem is that he has a single s3 bucket that he wants to process with multiple logstash instances (i.e. for scalability, as there is a lot of data), and he is finding that each instance will process all s3 bucket files, rather than half each, as they do not share a sincedb.

I am also interested in solving this problem.

Thanks,
Andrew

2 Likes
7

Yes, astickler.

This problem is still not solved. :frowning: