# Not able to create an indexes for files stored in S3
**URL:** https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759
**Category:** Logstash
**Created:** [June 6, 2018, 8:32am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759 "2018-06-06T08:32:08Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)
#### Post date: [June 6, 2018, 8:32am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/1 "2018-06-06T08:32:08Z")
Hi Team,
Able to execute and create an index in kibana if I keep the \*.csv in local machine but, I don't if I keep them in S3 bucket. Below are the config details.
input {
s3 {
access\_key\_id =\> "Access KEY"
secret\_access\_key =\> "Secret Access Key"
bucket =\> "gtologs"
prefix =\> "test/" # (No folder has created in bucket. File name is test.csv.)
region =\> "us-east-1"
}
}
filter {
csv {
columns =\> ["id","name","age","money"]
}
}
output {
elasticsearch {
hosts =\> ["localhost:9200"]
index =\> "gto"
document\_id =\> "%{id}"
# stdout { codec =\> rubydebug }
```
}
```
}
Can someone please help me to fix this issue.
Regards,
Panneer S
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 10, 2018, 8:31pm UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/2 "2018-06-10T20:31:13Z")
> prefix =\> "test/" # (No folder has created in bucket. File name is test.csv.)
But test/ is not a prefix of test.csv.
---
### Author:  [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)
#### Post date: [June 11, 2018, 6:08am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/3 "2018-06-11T06:08:21Z")
Hi Magnus,
Thanks for the reply. I have done the necessary changes but still I am having the same issues.
PFB, the input conf and output details, for your reference.
**Input conf:**
input {
s3 {
access\_key\_id =\> "accesskey"
secret\_access\_key =\> "secretkey"
bucket =\> "gtologs"
prefix =\> "dummy/test.csv"
region =\> "us-east-1"
}
}
filter {
csv {
columns =\> ["id","name","age","money"]
# separator =\> ","
```
}
```
}
output {
elasticsearch {
hosts =\> ["[ec2-10-10-10-10.compute-1.amazonaws.com:9200](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200)"]
index =\> "ctsgto"
document\_id =\> "%{id}"
# stdout { codec =\> rubydebug }
```
}
```
}
**Output details:**
[root@elkserver conf.d]# /usr/share/logstash/bin/logstash -f s3.conf
WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO] 2018-06-11 06:23:21.912 [main] scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}
[INFO] 2018-06-11 06:23:21.919 [main] scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}
[WARN] 2018-06-11 06:23:22.453 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO] 2018-06-11 06:23:22.643 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=\>"6.2.4"}
[INFO] 2018-06-11 06:23:22.834 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=\>9600}
[INFO] 2018-06-11 06:23:35.910 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}
[INFO] 2018-06-11 06:23:36.284 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/)]}}
[INFO] 2018-06-11 06:23:36.286 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/), :path=\>"/"}
[WARN] 2018-06-11 06:23:36.424 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=\>"[http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/)"}
[INFO] 2018-06-11 06:23:36.649 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es\_version=\>6}
[WARN] 2018-06-11 06:23:36.650 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}
[INFO] 2018-06-11 06:23:36.653 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=\>nil}
[INFO] 2018-06-11 06:23:36.656 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}
[INFO] 2018-06-11 06:23:36.663 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//ec2-10-10-10-10.compute-1.amazonaws.com:9200](https://ec2-10-10-10-10.compute-1.amazonaws.com:9200)"]}
[INFO] 2018-06-11 06:23:36.672 [[main]-pipeline-manager] s3 - Registering s3 input {:bucket=\>"gtologs", :region=\>"us-east-1"}
[INFO] 2018-06-11 06:23:36.742 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"}
[INFO] 2018-06-11 06:23:36.774 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=\>1, :pipelines=\>["main"]}
Please confirm, is index will take time to create since the logs are in S3 bucket?
I am facing this issue for the past 2 weeks. So, I request you to kindly provide your guidance in order to fix this issue
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 11, 2018, 7:07am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/4 "2018-06-11T07:07:16Z")
Things to try:
- Increase Logstash's log level.
- Remove the `prefix` option altogheter.
---
### Author:  [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)
#### Post date: [June 11, 2018, 8:53am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/5 "2018-06-11T08:53:37Z")
Hi Magnus,
Have removed the prefix and executed but it was throwing an error.
Then, I have changed the prefix as "prefix =\> "dummy/" and executed. It was working and the index has created as expected but unable to view any information as mentioned in \*.csv file.
In index page, I couldn't see any information as mentioned in excel sheet. Kindly help.
PFA, the logstash.log link in below, for your reference.
[https://s3.amazonaws.com/gtologs/logs/logstash-plain.log](https://s3.amazonaws.com/gtologs/logs/logstash-plain.log)
Regards,
Panneer S
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 11, 2018, 9:59am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/6 "2018-06-11T09:59:02Z")
> In index page,
Which index page?
> I couldn't see any information as mentioned in excel sheet.
So what **do** you see?
---
### Author:  [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)
#### Post date: [June 11, 2018, 10:22am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/7 "2018-06-11T10:22:40Z")
Hi Magnus,
1. Under Discover page, I couldn't see any information for the created index.
2. I have seen some info with errors. So, I deleted that index and tried to create a new one but this time, I am not sure I couldn't even create an index
**Here is my conf file:**
input {
s3 {
access\_key\_id =\> "accesskey"
secret\_access\_key =\> "secretkey"
bucket =\> "gtologs"
prefix =\> "dummy/"
region =\> "us-east-1"
}
}
filter {
csv {
columns =\> ["id","name","age","money"]
}
}
output {
elasticsearch {
hosts =\> ["[ec2-10-10-10-10.compute-1.amazonaws.com:9200](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200)"]
index =\> "gto"
document\_id =\> "%{id}"
}
}
\ ***.CSV file:**
[root@elkserver conf.d]# cat test.csv
id,name,age,money
1234,jyoti,38,200000
5678,rannjan,58,4000000
7890,panda,68,8000000
8904,jyoti ranjan panda,88,980000000
**Output:**
[root@elkserver conf.d]# /usr/share/logstash/bin/logstash -f s3.conf
WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO] 2018-06-11 10:16:32.812 [main] scaffold - Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}
[INFO] 2018-06-11 10:16:32.821 [main] scaffold - Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}
[WARN] 2018-06-11 10:16:33.384 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO] 2018-06-11 10:16:33.587 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=\>"6.2.4"}
[INFO] 2018-06-11 10:16:33.750 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=\>9600}
[INFO] 2018-06-11 10:16:46.621 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}
[INFO] 2018-06-11 10:16:46.993 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/)]}}
[INFO] 2018-06-11 10:16:46.996 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/), :path=\>"/"}
[WARN] 2018-06-11 10:16:47.123 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=\>"[http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/](http://ec2-10-10-10-10.compute-1.amazonaws.com:9200/)"}
[INFO] 2018-06-11 10:16:47.351 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es\_version=\>6}
[WARN] 2018-06-11 10:16:47.351 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}
[INFO] 2018-06-11 10:16:47.355 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=\>nil}
[INFO] 2018-06-11 10:16:47.358 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}
[INFO] 2018-06-11 10:16:47.365 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//ec2-10-10-10-10.compute-1.amazonaws.com:9200](https://ec2-10-10-10-10.compute-1.amazonaws.com:9200)"]}
[INFO] 2018-06-11 10:16:47.374 [[main]-pipeline-manager] s3 - Registering s3 input {:bucket=\>"gtologs", :region=\>"us-east-1"}
[INFO] 2018-06-11 10:16:47.540 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\"}
[INFO] 2018-06-11 10:16:47.561 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=\>1, :pipelines=\>["main"]}
[INFO] 2018-06-11 10:16:48.693 [[main]\"/usr/share/logstash/data/plugins/inputs/s3/sincedb\_0eb55c321eb3e531487f435a77836115"}
Please specify, what went wrong here? Thanks for your ongoing support
Regards,
Panneer S
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 11, 2018, 10:54am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/8 "2018-06-11T10:54:16Z")
> So, I deleted that index and tried to create a new one but this time, I am not sure I couldn't even create an index
Did you also delete the sincedb file where Logstash's current position in the S3 files are stored?
---
### Author:  [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)
#### Post date: [June 11, 2018, 11:14am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/9 "2018-06-11T11:14:45Z")
Hi Magnus,
Yes. I deleted all the sincedb files and created a new index and I able to see them under discover page with the below error
12 Visualize: Zero or negative time interval not supported
Error: Request to Elasticsearch failed: {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"Zero or negative time interval not supported"}],"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":[{"shard":0,"index":"filebeat-6.2.4-2018.06.08","node":"KbysAteuSbaOVVsL-IktGA","reason":{"type":"illegal\_argument\_exception","reason":"Zero or negative time interval not supported"}}]},"status":400}
at [http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:712440](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:712440)
at Function.Promise.try ([http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:503538](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:503538))
at [http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:502926](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:502926)
at Array.map ()
at Function.Promise.map ([http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:502884](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:502884))
at callResponseHandlers ([http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:712018](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:712018))
at [http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:701368](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/commons.bundle.js?v=16627:1:701368)
at processQueue ([http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:132456](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:132456))
at [http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:133349](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:133349)
at Scope.$digest ([http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:144239](http://ec2-10-10-10-10.compute-1.amazonaws.com:5601/bundles/vendors.bundle.js?v=16627:58:144239))
Regards,
Panneer S
---
### Author:  [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)
#### Post date: [June 11, 2018, 11:17am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/10 "2018-06-11T11:17:38Z")
Please find the link of the error screen shot
[ ](https://s3.amazonaws.com/gtocloudlogs/Logs/discover+page.JPG)
Regards,
Panneer S
---
### Author:  [@PanneerSelvam](https://discuss.elastic.co/u/PanneerSelvam)
#### Post date: [June 12, 2018, 4:44am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/11 "2018-06-12T04:44:58Z")
Hi Magnus,
Now, I was able to read s3 logs from EL and view the indexes in Kibana without issues.
Regards,
Panneer S
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 10, 2018, 4:45am UTC](https://discuss.elastic.co/t/not-able-to-create-an-indexes-for-files-stored-in-s3/134759/12 "2018-07-10T04:45:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.