Logstash S3 output plugin with AWS emulation with Localstack

Elastic Stack Logstash
1

Hi All, after I spent a lot of time configuring and connecting Logstash to Localstack, I had many errors and no guide to follow .

In my case I have 2 dockers , one is logstash docker with S3 output pluging , second docker is Localstack docker that emulates the AWS services.

I will paste docker-compose.yml and logstash config files to make it easy for you to do the same and save your time.

This is the config file for logstash , some simple input line generator

VERY IMPORTANT
Keys - must be = "test"
add "endpoint" to S3 output plugin with port 4566 , this is the only port after Localstack update on 09.2020
add next lines to S3 output otherwise, you will have a lot of errors

           validate_credentials_on_root_bucket => "false"
           additional_settings => { "force_path_style" => true }

input {
  generator {
    lines => [
      "line 1",
      "line 2",
      "line 3"
    ]
    # Emit all lines 3 times.
    count => 3
  }
}

output {

   stdout { codec  => rubydebug {metadata => true} }

  
    s3{
           access_key_id => "test"
        secret_access_key => "test"
           endpoint => "http://yourIPaddress:4566/"
           region => "us-east-1"
           bucket => "yourBucketName"   
           size_file => 2048
           time_file => 1
           codec => "json_lines"
           prefix => "addSomePrefix"
           validate_credentials_on_root_bucket => "false"
           additional_settings => { "force_path_style" => true }
        }

}

Second docker-compose file for Localstack

 version: '3.5'
services:    
     localstack:
        container_name: localstack_main
        image: localstack/localstack
        network_mode: bridge
        ports:
          - "4566:4566"
          - "4571:4571"
          - "${PORT_WEB_UI-8080}:${PORT_WEB_UI-8080}"       
        environment:
          - SERVICES=s3
          - DEBUG=1
          - DATA_DIR=${DATA_DIR- }
          - PORT_WEB_UI=${PORT_WEB_UI- }
          - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR- }
          - KINESIS_ERROR_PROBABILITY=${KINESIS_ERROR_PROBABILITY- }
          - DOCKER_HOST=unix:///var/run/docker.sock
          - HOST_TMP_FOLDER=${TMPDIR}
        volumes:
          - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
          - "/var/run/docker.sock:/var/run/docker.sock"

Do not forget to create a manually bucket you want to use in S3
by running from cmd

"aws --endpoint-url=http://localhost:4566 s3 mb s3://yourBucketName"

I hope it will save you a lot of time.

Goodluck.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.