# GCS Input not working as expected **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [June 19, 2024, 8:49am UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713 "2024-06-19T08:49:08Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Random\_BB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/random_bb/32/126785_2.png) [@Random\_BB](https://discuss.elastic.co/u/Random_BB)\ **Post date:** [June 19, 2024, 8:49am UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713/1 "2024-06-19T08:49:08Z")
I have an AWS Domain to which I am trying to Ingest Json codec files from GCS bucket using Logstash. I can see a log stating that my logstash has connected to domain and a few other logs stating it is trying to fetch logs from GCS bucket. This is my logstash configuration. ```auto input { google_cloud_storage { bucket_id => "test-bucket" json_key_file => "/etc/logstash/credentials.json" codec => "json_lines" } } filter { } output { opensearch { hosts => "https://.us-east-1.es.amazonaws.com:443" user => "admin" password => "admin" index => "logstash-test-1" ssl_certificate_verification => true } } ``` The same file works expected when the input is a file via ConfigMap. But when done via GCS, it doesn't push. I did see these logs. ```auto [2024-06-19T08:28:41,025][INFO][logstash.inputs.googlecloudstorage][main] ProcessedDb created in: /usr/share/logstash/data/plugins/inputs/google_cloud_storage/db [2024-06-19T08:28:41,027][INFO][logstash.inputs.googlecloudstorage][main] Turn on debugging to explain why blobs are filtered. [2024-06-19T08:28:41,028][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"} [2024-06-19T08:28:41,033][INFO][logstash.inputs.googlecloudstorage][main][6278fa388e5b5004f390348cab6962e1c49ff5ef2e012a1436a636cecb12a3c8] Fetching blobs from test-bucket [2024-06-19T08:28:41,045][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} ``` I can see a log stating blobs are fetched, but it is not getting uploaded. Any reason why they are being filtered out? Using the official logstash docker image - [docker.elastic.co/logstash/logstash:8.8.2](http://docker.elastic.co/logstash/logstash:8.8.2) on which I have installed gcs input plugin. Posting it here as I am using Elasticsearch logstash image. Any help would be appreciated, I have been stuck here for days now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 19, 2024, 8:49am UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713/2 "2024-06-19T08:49:08Z")
OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See [What is OpenSearch and the OpenSearch Dashboard? | Elastic](https://www.elastic.co/elasticsearch/opensearch) for more details. (This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: ) ---
**Author:** ![Random\_BB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/random_bb/32/126785_2.png) [@Random\_BB](https://discuss.elastic.co/u/Random_BB)\ **Post date:** [June 19, 2024, 8:50am UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713/3 "2024-06-19T08:50:16Z")
Removed #awses, #opensearch ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 19, 2024, 11:14am UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713/4 "2024-06-19T11:14:58Z")
> [@Random\_BB](#): > > `[INFO][logstash.inputs.googlecloudstorage][main] Turn on debugging to explain why blobs are filtered.` This is very good advice. If you do it then the [input will log](https://github.com/logstash-plugins/logstash-input-google_cloud_storage/blob/079b5f182d0b18278a10a55360d82f19f1bbebb7/lib/logstash/inputs/cloud_storage/blob_filter.rb#L22) whether each blob matches each of the conditions that could prevent it being processed. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [June 19, 2024, 12:22pm UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713/5 "2024-06-19T12:22:17Z")
> [@Random\_BB](#): > > I am trying to Ingest Json codec files from GCS bucket using Logstash What are the extension of the files in your bucket? If you check the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-google_cloud_storage.html#plugins-inputs-google_cloud_storage-file_matches) you will see that the option `file_matches` per default will only match `*.log` and `*.log.gz` files, if your files have other extensions they will be filtered out. The default pattern is this: `.*\.log(\.gz)?` If your files are `json` files you may need to change this setting to something like: `file_matches => ".*\.json"` ---
**Author:** ![Random\_BB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/random_bb/32/126785_2.png) [@Random\_BB](https://discuss.elastic.co/u/Random_BB)\ **Post date:** [June 20, 2024, 6:30am UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713/6 "2024-06-20T06:30:21Z")
Files are just plain text files without any extension. Meaning, they are plain txt files with each line representing a single json object. But I missed the default part. Thanks for the help! Let me see how to use them from here. ---
**Author:** ![Random\_BB](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/random_bb/32/126785_2.png) [@Random\_BB](https://discuss.elastic.co/u/Random_BB)\ **Post date:** [June 20, 2024, 7:17am UTC](https://discuss.elastic.co/t/gcs-input-not-working-as-expected/361713/7 "2024-06-20T07:17:24Z")
> [@leandrojmp](#): > > file\_matches =\> ".\*.json" I added file\_matches =\> ".\*" so that it picks up all files and it worked. Thanks @leandrojmp