# The s3 input for creating the index does not work with preffix and csv files **URL:** \ **Category:** Logstash\ **Created:** [January 5, 2024, 11:29pm UTC](https://discuss.elastic.co/t/the-s3-input-for-creating-the-index-does-not-work-with-preffix-and-csv-files/350496 "2024-01-05T23:29:48Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Marcos\_Daniel\_Santos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcos_daniel_santos/32/124171_2.png) [@Marcos\_Daniel\_Santos](https://discuss.elastic.co/u/Marcos_Daniel_Santos)\ **Post date:** [January 5, 2024, 11:29pm UTC](https://discuss.elastic.co/t/the-s3-input-for-creating-the-index-does-not-work-with-preffix-and-csv-files/350496/1 "2024-01-05T23:29:49Z")
Hi everyone, I have a bucekt s3 with 2 csv files, one that is a securityhub repot and the other a guardduty report, both AWS services and security. I'm using the preffix to get my object, using the logstash -f file.conf --debug command, I see that it finds the securityhub.csv file but it skips the file and doesn't create the index. Follow the command output: > Blockquote DEBUG] 2024-01-05 23:15:22.286 [pool-3-thread-1] jvm - collector name {:name=\>"ConcurrentMarkSweep"} [DEBUG] 2024-01-05 23:15:22.564 **[[main]\"security\_hub\_results.csv"}** [DEBUG] 2024-01-05 23:15:22.571 **[[main]\"security\_hub\_results.csv"}** [DEBUG] 2024-01-05 23:15:23.641 **[[main]\"LogStash::Inputs::S3"}** [DEBUG] 2024-01-05 23:15:23.649 [[main]\"main", :thread=\>"#\"} [DEBUG] 2024-01-05 23:15:23.670 [[main]-pipeline-manager] javapipeline - Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"} [DEBUG] 2024-01-05 23:15:23.749 [[main]-pipeline-manager] javapipeline - Shutdown waiting for worker thread {:pipeline\_id=\>"main", :thread=\>"#\"} [DEBUG] 2024-01-05 23:15:23.755 [[main]-pipeline-manager] csv - Closing {:plugin=\>"LogStash::Filters::CSV"} [DEBUG] 2024-01-05 23:15:23.756 [[main]-pipeline-manager] pluginmetadata - Removing metadata for plugin eb8f5d7a915e774ee908322cb49b5311ddb4d0226fad4637788d9e1b34fe1466 [DEBUG] 2024-01-05 23:15:23.757 [[main]-pipeline-manager] stdout - Closing {:plugin=\>"LogStash::Outputs::Stdout"} [DEBUG] 2024-01-05 23:15:23.758 [[main]-pipeline-manager] pluginmetadata - Removing metadata for plugin e33fca5295dce67aa1bd189c267cd1c06e1766ee0d2faa12d1bbe10126075298 [DEBUG] 2024-01-05 23:15:23.762 [[main]-pipeline-manager] elasticsearch - Closing {:plugin=\>"LogStash::Outputs::Elasticsearch"} [DEBUG] 2024-01-05 23:15:23.782 [[main]-pipeline-manager] elasticsearch - Stopping sniffer [DEBUG] 2024-01-05 23:15:23.784 [[main]-pipeline-manager] elasticsearch - Stopping resurrectionist [DEBUG] 2024-01-05 23:15:24.456 [[main]-pipeline-manager] elasticsearch - Waiting for in use manticore connections [DEBUG] 2024-01-05 23:15:24.465 [[main]-pipeline-manager] elasticsearch - Closing adapter #LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x2151f761 [DEBUG] 2024-01-05 23:15:24.472 [[main]-pipeline-manager] PoolingHttpClientConnectionManager - Connection manager is shutting down [DEBUG] 2024-01-05 23:15:24.473 [[main]-pipeline-manager] DefaultManagedHttpClientConnection - http-outgoing-0: Close connection [DEBUG] 2024-01-05 23:15:24.473 [[main]-pipeline-manager] PoolingHttpClientConnectionManager - Connection manager shut down [DEBUG] 2024-01-05 23:15:24.473 [[main]-pipeline-manager] pluginmetadata - Removing metadata for plugin 44c25b32a23622ecc5a22d3366fb3d2abf9d77c296a652a04e3897b03080cf2b [DEBUG] 2024-01-05 23:15:24.476 [[main]-pipeline-manager] javapipeline - Pipeline has been shutdown {:pipeline\_id=\>"main", :thread=\>"#\"} [INFO] 2024-01-05 23:15:24.487 [[main]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=\>"main"} [DEBUG] 2024-01-05 23:15:24.525 [LogStash::Runner] agent - Shutting down all pipelines {:pipelines\_count=\>0} [DEBUG] 2024-01-05 23:15:24.553 [LogStash::Runner] agent - Converging pipelines state {:actions\_count=\>1} [DEBUG] 2024-01-05 23:15:24.571 [Converge PipelineAction::Delete] agent - Executing action {:action=\>LogStash::PipelineAction::Delete/pipeline\_id:main} [INFO] 2024-01-05 23:15:24.605 [Converge PipelineAction::Delete] pipelinesregistry - Removed pipeline from registry successfully {:pipeline\_id=\>:main} [DEBUG] 2024-01-05 23:15:24.620 [LogStash::Runner] os - Stopping [DEBUG] 2024-01-05 23:15:24.661 [LogStash::Runner] jvm - Stopping [DEBUG] 2024-01-05 23:15:24.667 [LogStash::Runner] persistentqueue - Stopping [DEBUG] 2024-01-05 23:15:24.667 [LogStash::Runner] deadletterqueue - Stopping [DEBUG] 2024-01-05 23:15:24.756 [Api Webserver] agent - API WebServer has stopped running [INFO] 2024-01-05 23:15:24.757 [LogStash::Runner] runner - Logstash shut down. > Blockquote follow my s3 input: ```auto input { s3 { bucket => "s3-bucket" region => "sa-east-1" id => "my-securityhub" prefix => "security_hub_results.csv" exclude_pattern => "/finding\.csv$" role_arn => "myrole" # type => "s3" #sincedb_path => "/etc/logstash/sincedb/s3-sincedb" codec => "plain" #interval => 15 watch_for_new_files => false additional_settings => { force_path_style => true follow_redirects => false } } } ``` When I set the prefix to "" and even using exclude\_pattern it creates the index but with the information from the 2 files in the root of the bucket. I've already done several tests and it's strange that the documentation talks about how to use the tool but it doesn't really work properly. I've also tried removing the exclude\_pattern and using only the prefix pointing to the file I want to use, which is security\_hub\_results.csv" I hope someone can clarify my doubts about this. ---
**Author:** ![Marcos\_Daniel\_Santos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcos_daniel_santos/32/124171_2.png) [@Marcos\_Daniel\_Santos](https://discuss.elastic.co/u/Marcos_Daniel_Santos)\ **Post date:** [January 5, 2024, 11:39pm UTC](https://discuss.elastic.co/t/the-s3-input-for-creating-the-index-does-not-work-with-preffix-and-csv-files/350496/2 "2024-01-05T23:39:11Z")
Hi guys, I removed the .csv at the end of the file name and it worked. prefix =\> "security\_hub\_results" I find it strange because although .csv is the file format, in the s3 bucket the ket is: security\_hub\_results.csv Taking advantage of the opportunity, I am creating this index in question from the command line. I had separated 2 pipelines, one for each filed and conf, however, this one from security hub gives me an error regarding the type of the CreatedAt field. When I run it manually, as was done just now, it creates the idnex without any problems, could anyone tell me the reason for this? This is the error it returns: "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [CreatedAt] of type [date] in document with id 'arn:aws:securityhub :sa-east-1:accountid:security-control/EC2.19/finding/6230424d-8c59-40cc-9b33-dc9093123f9a'. Preview of field's value: 'CRITICAL'", "caused\_by"=\>{"type"= \>"illegal\_argument\_exception", "reason"=\>"failed to parse date field [CRITICAL] with format [strict\_date\_optional\_time||epoch\_millis]", "caused\_by"=\>{"type"=\>"date\_time\_parse\_exception", "reason"=\>" Failed to parse with all enclosed parsers"}}}}}} I tried doing several things, using filter to change the data type but nothing worked. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 6, 2024, 3:15am UTC](https://discuss.elastic.co/t/the-s3-input-for-creating-the-index-does-not-work-with-preffix-and-csv-files/350496/3 "2024-01-06T03:15:54Z")
> [@Marcos\_Daniel\_Santos](#): > > [[main]\"security\_hub\_results.csv"} The prefix has to be a prefix, it cannot be the whole filename. See [this](https://github.com/logstash-plugins/logstash-input-s3/blob/b7f42d7cd7c09b5343e5e60fe2bd7971f039eecd/lib/logstash/inputs/s3.rb#L361) test in the code. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 6, 2024, 12:33pm UTC](https://discuss.elastic.co/t/the-s3-input-for-creating-the-index-does-not-work-with-preffix-and-csv-files/350496/4 "2024-01-06T12:33:04Z")
> [@Marcos\_Daniel\_Santos](#): > > "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [CreatedAt] of type [date] in document with id 'arn:aws:securityhub :sa-east-1:accountid:security-control/EC2.19/finding/6230424d-8c59-40cc-9b33-dc9093123f9a'. Preview of field's value: 'CRITICAL'", "caused\_by"=\>{"type"= \>"illegal\_argument\_exception", "reason"=\>"failed to parse date field [CRITICAL] with format [strict\_date\_optional\_time||epoch\_millis]", "caused\_by"=\>{"type"=\>"date\_time\_parse\_exception", "reason"=\>" Failed to parse with all enclosed parsers"}}}}}} This is a mapping error, it is returned by Elasticsearch. It means that you are trying to index a field that has a value that is not supported by the current mapping. In this case the field `CreatedAt` is mapped as `date` in Elasticsearch, but in this document it has the value _CRITICAL_ , which of course is not a date and will be rejected, you need to check your parsing to see why this is happening. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 3, 2024, 12:33pm UTC](https://discuss.elastic.co/t/the-s3-input-for-creating-the-index-does-not-work-with-preffix-and-csv-files/350496/5 "2024-02-03T12:33:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.