Logstash w/ s3 output plugin - slow/delay

Elastic Stack Logstash
1

Hello,

I'm using Logstash 7.17.10 with S3 output plugin, and getting poor performance (possibly not related to performance)

my pipeline:

input {
    elasticsearch {
        docinfo => true
        docinfo_fields => [ "_id" ]
        hosts => ["${ELASTICSEARCH_HOSTS}"]
        index => 'index*'
        password => "${ELASTICSEARCH_PASSWORD}"
        query => ' ... '
        schedule => '0 * * * *'
        user => "${ELASTICSEARCH_USER}"
    }
}

filter {
    ruby { code => 'event.set("[@metadata][suffix]", Time.now.strftime("%Y-%m-%dT%H"));' }
}

output {
    s3 {
        access_key_id => "${MINIO_ACCESS_KEY}"
        additional_settings => {
            force_path_style => true
        }
        bucket => "${MINIO_BUCKET}"
        codec => json_lines {
            target => ""
        }
        endpoint => "${MINIO_ENDPOINT}"
        prefix => "a-b-c-d/e-%{[@metadata][_index]}-%{[@metadata][suffix]}.json"
        secret_access_key => "${MINIO_SECRET_KEY}"
        validate_credentials_on_root_bucket => false
    }
}

it can easily 30+ minutes before i see objects on my s3 storage.

also both elasticsearch and logstash are residing on same network and so as minio (storage) as well and if i configure logstash to output file locally, it doesn't takes long at all to complete, and i'm able to push file manually to s3 quick as well (so it's not network).

please advise.
thank you in advance!

2

does it have anything to do with time_file ?

3

Could be, the default rotation_strategy is size_and_time, so a file will only be pushed to s3 after it reaches the specified time (15 minutes per default) or the specified size ( ~ 5 MB per default).

But also keep in mind the the this output plugin only officially supports AWS S3, any S3 compatible it is not supported, it may work, but it also may have issues.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.