# Reading existing files with S3 Logstash Input Plugin

**URL:** <https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702>\
**Category:** Logstash\
**Created:** [March 4, 2019, 10:05am UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702 "2019-03-04T10:05:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![febrezo](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@febrezo](https://discuss.elastic.co/u/febrezo)\
**Post date:** [March 4, 2019, 10:05am UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702/1 "2019-03-04T10:05:07Z")

</div>

Hi!

I'm trying to reach some files we have in a S3 instance with Logstash. My .conf file looks like this:

```
input {
  s3 {
    "access_key_id" => "my_key"
    "secret_access_key" => my_secret_key"
    "bucket" => "my_bucket"
  }
}

output {
  stdout{}
}

```

The output shown in the terminal is as follows:

```
Sending Logstash logs to /home/my_user/.../tmp/logstash-6.6.1/logs which is now configured via log4j2.properties
[2019-03-04T10:59:41,177][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-04T10:59:41,190][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-03-04T10:59:59,101][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-04T10:59:59,335][INFO][logstash.outputs.rabbitmq] Connected to RabbitMQ at 
[2019-03-04T10:59:59,427][INFO][logstash.inputs.s3] Registering s3 input {:bucket=>"my_bucket", :region=>"us-east-1"}
[2019-03-04T10:59:59,808][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x7d681967 run>"}
[2019-03-04T10:59:59,852][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-04T11:00:00,133][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
S3 client configured for "us-east-1" but the bucket "my_bucket" is in "eu-west-1"; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts
[2019-03-04T11:00:02,701][ERROR][logstash.inputs.s3] S3 input: Unable to list objects in bucket {:prefix=>nil, :message=>"The request signature we calculated does not match the signature you provided. Check your key and signing method."}

```

It is messing me up a little bit taking into account that the prefix seems not to be a compulsory argument... I have tested with a tiny Python script using the credentials and bucket I provide in the Logstash config file that the files are already there, but I cannot make Losgstash start indexing the previously stored files. **Is Logstash's S3 plugin only monitoring new files by default?** In this case, how can I make Logstash to start processing all the previously stored files?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 4, 2019, 1:20pm UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702/2 "2019-03-04T13:20:32Z")

</div>

You need to set the region option on the input.

---

<div class="post-metadata">

**Author:** ![febrezo](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@febrezo](https://discuss.elastic.co/u/febrezo)\
**Post date:** [March 4, 2019, 4:14pm UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702/3 "2019-03-04T16:14:14Z")

</div>

Oops. Is it compulsory? AFAIK, it seems that it is not required. Even, if not properly provded, wouldn't it only be a bit slower?

```
Sending Logstash logs to /home/my_user/.../tmp/logstash-6.6.1/logs which is now configured via log4j2.properties
[2019-03-04T17:10:48,896][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-04T17:10:48,911][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.6.1"}
[2019-03-04T17:11:08,558][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-04T17:11:08,779][INFO][logstash.outputs.rabbitmq] Connected to RabbitMQ at 
[2019-03-04T17:11:08,891][INFO][logstash.inputs.s3] Registering s3 input {:bucket=>"my_bucket", :region=>"eu-west-1"}
[2019-03-04T17:11:09,231][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x73d24490 run>"}
[2019-03-04T17:11:09,275][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-04T17:11:09,538][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2019-03-04T17:11:11,370][ERROR][logstash.inputs.s3] S3 input: Unable to list objects in bucket {:prefix=>nil, :message=>"The request signature we calculated does not match the signature you provided. Check your key and signing method."}

```

The error seems to be the same and it might be linked to the **prefix =\> NIL** message, which keeps being shown online after 60 seconds… Note that I am still not pushing new files to the S3 instance. I only want to get "old" files...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 4, 2019, 4:26pm UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702/4 "2019-03-04T16:26:31Z")

</div>

You are right, it's not compulsory. I misread your log.

---

<div class="post-metadata">

**Author:** ![febrezo](https://avatars.discourse-cdn.com/v4/letter/f/96bed5/32.png) [@febrezo](https://discuss.elastic.co/u/febrezo)\
**Post date:** [March 5, 2019, 10:08am UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702/5 "2019-03-05T10:08:29Z")

</div>

I'm starting to think that it may be an issue with the encoding of the objects because of the answers in this link. [https://stackoverflow.com/questions/30518899/how-to-fix-the-request-signature-we-calculated-does-not-match-the-signature-er#30519762](https://stackoverflow.com/questions/30518899/how-to-fix-the-request-signature-we-calculated-does-not-match-the-signature-er#30519762)

Are bucket names using a "-" character error prone? And paths having a "/" like "LOGS/ES/BOT-2019\_02\_28.log"?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 2, 2019, 10:08am UTC](https://discuss.elastic.co/t/reading-existing-files-with-s3-logstash-input-plugin/170702/6 "2019-04-02T10:08:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.