# S3 input and Elasticsearch output in Logstash **URL:** \ **Category:** Logstash\ **Created:** [September 29, 2015, 1:07pm UTC](https://discuss.elastic.co/t/s3-input-and-elasticsearch-output-in-logstash/31340 "2015-09-29T13:07:27Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![AymenChetoui](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aymenchetoui/32/4156_2.png) [@AymenChetoui](https://discuss.elastic.co/u/AymenChetoui)\ **Post date:** [September 29, 2015, 1:07pm UTC](https://discuss.elastic.co/t/s3-input-and-elasticsearch-output-in-logstash/31340/1 "2015-09-29T13:07:27Z")
Hi all, I am trying to get familiar with S3 plugin in Logstash in two steps : 1 - Pushing logs to S3 as output 2 - Getting logs from S3 as input 1 - The Logstash conf file looks like this : ``` output { s3{ access_key_id => "***" secret_access_key => " ******" endpoint_region => " **-*** -*" bucket => "bucket-name" size_file => 2048 codec => "json" canned_acl => "private" } } ``` The logs I am sending look like this : ``` {"_id":"55fee3c726045db681acd139","index":0,"guid":"199836ff-bbfd-4a2f-9e66-452e72342e24","isActive":false,"balance":"$3,565.23","picture":"http://placehold.it/32x32","age":38,"eyeColor":"green","name":{"first":"Brandie","last":"Blackwell"}} ... ``` Which seems to look pretty fine as I have logs stored in S3 in different files with 2K size : ``` {"_id":"55fee3c726045db681acd139","index":0,"guid":"199836ff-bbfd-4a2f-9e66-452e72342e24","isActive":false,"balance":"$3,565.23","picture":"http://placehold.it/32x32","age":38,"eyeColor":"green","name":{"first":"Brandie","last":"Blackwell"},"@version":"1","@timestamp":"2015-09-29T12:20:12.679Z","host":"*.*.*.*"} ... ``` 2 - However when getting logs as input an try to send them to Elasticsearch issues came across : The config file : ``` input { s3{ access_key_id => " ****" secret_access_key => " *******" region => " **-*** -*" bucket => "bucket-name" codec => "json" } } output { elasticsearch { host => "*.*.*.*" protocol => "http" } } ``` Losgtash is not able to retrieve logs from S3 and send them correctly to Elasticsearch : > failed action with response of 400, dropping action: ["index", {:\_id=\>nil, :\_index=\>"logstash-2015.09.29", :\_type=\>"logs", :\_routing=\>nil}, #\0}, @lut={}\>, @cancelled=false, @data={"\_id"=\>"55fee3c7643281f83e914515", "index"=\>70, "guid"=\>"891dfae1-c48d-483a-a065-f61f37ed41a3", "isActive"=\>true, "balance"=\>"$3,263.89", "picture"=\>"[http://placehold.it/32x32](http://placehold.it/32x32)", "age"=\>26, "eyeColor"=\>"green", "name"=\>{"first"=\>"Parrish", "last"=\>"Baxter"}, "@version"=\>"1", "@timestamp"=\>"2015-09-29T12:59:31.543Z", "host"=\>"_._._._"}, @metadata={"retry\_count"=\>0}, @accessors=#\"55fee3c7643281f83e914515", "index"=\>70, "guid"=\>"891dfae1-c48d-483a-a065-f61f37ed41a3", "isActive"=\>true, "balance"=\>"$3,263.89", "picture"=\>"[http://placehold.it/32x32](http://placehold.it/32x32)", "age"=\>26, "eyeColor"=\>"green", "name"=\>{"first"=\>"Parrish", "last"=\>"Baxter"}, "@version"=\>"1", "@timestamp"=\>"2015-09-29T12:59:31.543Z", "host"=\>"_._._._"}, @lut={"message"=\>[{"\_id"=\>"55fee3c7643281f83e914515", "index"=\>70, "guid"=\>"891dfae1-c48d-483a-a065-f61f37ed41a3", "isActive"=\>true, "balance"=\>"$3,263.89", "picture"=\>"[http://placehold.it/32x32](http://placehold.it/32x32)", "age"=\>26, "eyeColor"=\>"green", "name"=\>{"first"=\>"Parrish", "last"=\>"Baxter"}, "@version"=\>"1", "@timestamp"=\>"2015-09-29T12:59:31.543Z", "host"=\>"_._._._"}, "message"], "type"=\>[{"\_id"=\>"55fee3c7643281f83e914515", "index"=\>70, "guid"=\>"891dfae1-c48d-483a-a065-f61f37ed41a3", "isActive"=\>true, "balance"=\>"$3,263.89", "picture"=\>"[http://placehold.it/32x32](http://placehold.it/32x32)", "age"=\>26, "eyeColor"=\>"green", "name"=\>{"first"=\>"Parrish", "last"=\>"Baxter"}, "@version"=\>"1", "@timestamp"=\>"2015-09-29T12:59:31.543Z", "host"=\>"_._._._"}, "type"]}\>\>] {:level=\>:warn} Could you please help me find out what I am missing here? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 29, 2015, 2:00pm UTC](https://discuss.elastic.co/t/s3-input-and-elasticsearch-output-in-logstash/31340/2 "2015-09-29T14:00:33Z")
Check your ES logs, a 400 response should report something in those. ---
**Author:** ![AymenChetoui](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aymenchetoui/32/4156_2.png) [@AymenChetoui](https://discuss.elastic.co/u/AymenChetoui)\ **Post date:** [September 29, 2015, 2:37pm UTC](https://discuss.elastic.co/t/s3-input-and-elasticsearch-output-in-logstash/31340/3 "2015-09-29T14:37:49Z")
The Elasticsearch logs say that it **failed to execute bulk item (index) index** and **failed to parse [\_id] :** > [2015-09-29 14:11:55,883][INFO][node] [Commander Kraken] version[1.4.4], pid[30103], build[c88f77f/2015-02-19T13:05:36Z] > [2015-09-29 14:11:55,884][INFO][node] [Commander Kraken] initializing ... > [2015-09-29 14:11:55,944][INFO][plugins] [Commander Kraken] loaded [cloud-aws], sites [bigdesk, head] > [2015-09-29 14:12:00,568][INFO][node] [Commander Kraken] initialized > [2015-09-29 14:12:00,568][INFO][node] [Commander Kraken] starting ... > [2015-09-29 14:12:00,659][INFO][transport] [Commander Kraken] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[localhost/127.0.0.1:930$[2015-09-29 14:12:00,668][INFO][discovery] [Commander Kraken] S3-IO-Cluster/vrI8A3B5TNutb6ZrFQT1eQ > [2015-09-29 14:12:05,027][INFO][cluster.service] [Commander Kraken] new\_master [Commander Kraken][vrI8A3B5TNutb6ZrFQT1eQ][ip-_-_-_-_][inet[localhost/127.$[2015-09-29 14:12:05,054][INFO][http] [Commander Kraken] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[localhost/127.0.0.1:920$[2015-09-29 14:12:05,054][INFO][node] [Commander Kraken] started > [2015-09-29 14:12:05,061][INFO][gateway] [Commander Kraken] recovered [0] indices into cluster\_state > [2015-09-29 14:24:19,788][INFO][cluster.metadata] [Commander Kraken] [logstash-2015.09.29] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [\_d$[2015-09-29 14:24:20,191][DEBUG][action.bulk] [Commander Kraken] [logstash-2015.09.29][4] failed to execute bulk item (index) index {[logstash-2015.09.29][$org.elasticsearch.index.mapper.MapperParsingException: failed to parse [\_id] > at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:416) > at org.elasticsearch.index.mapper.internal.IdFieldMapper.parse(IdFieldMapper.java:295) > at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:709) > at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:500) > at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:542) > at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:491) > at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:392) > at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:444) > at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:150) > at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOper$ > at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction$ > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.elasticsearch.index.mapper.MapperParsingException: Provided id [AVAZfWaivQTTvmzMp1dt] does not match the content one [55fee3c726045db681acd139] > at org.elasticsearch.index.mapper.internal.IdFieldMapper.parseCreateField(IdFieldMapper.java:310) > at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:406) > ... 13 more Obviously Elasticsearch thinks the second JSON log as the value to the index field or something. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:27am UTC](https://discuss.elastic.co/t/s3-input-and-elasticsearch-output-in-logstash/31340/4 "2017-07-06T05:27:47Z")