# How to configure GCS as filebeat input **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 10, 2021, 6:02pm UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453 "2021-12-10T18:02:55Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![arvin495](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arvin495/32/98571_2.png) [@arvin495](https://discuss.elastic.co/u/arvin495)\ **Post date:** [December 10, 2021, 6:02pm UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/1 "2021-12-10T18:02:55Z")
Hello Team, We are storing our audit logs in GCS bucket. we would like to ingest them to Elasticsearch when required - not regularly - using filebeat. I have checked S3 option where it let us use s3 like storages as input using providers. I'm using following configuration but it is not writing any data however when I test the filebeat configuration it is fine. I doubt my input configuration is not right in someway. Please check the following and help me understand what's wrong ```auto filebeat.inputs: - type: gcp project_id: gcp-project-xxx bucket_name: log-bucket credentials_file: /tmp/service-account-key.json output.elasticsearch: hosts: "https://es-test-xxx.aivencloud.com" username: "avnadmin" password: "xxxxx" indices: - index: 'restore-test' ``` ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [December 11, 2021, 12:16am UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/2 "2021-12-11T00:16:13Z")
There is no `gcp` input, it's `gcp-pubsub` and ur config isn't valid for that. See [GCP Pub/Sub input | Filebeat Reference [7.16] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-gcp-pubsub.html) on how to configure that. If all you want to do is read log files from an S3 compatible bucket, see the `aws-s3` input on how to poll a bucket, [AWS S3 input | Filebeat Reference [7.16] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html). ---
**Author:** ![arvin495](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arvin495/32/98571_2.png) [@arvin495](https://discuss.elastic.co/u/arvin495)\ **Post date:** [December 12, 2021, 5:07pm UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/3 "2021-12-12T17:07:56Z")
First I thought the same @legoguy1000 untill I found this [AWS S3 input | Filebeat Reference [master] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-aws-s3.html#_provider) ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [December 12, 2021, 5:26pm UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/4 "2021-12-12T17:26:11Z")
I'm confused, what on that page made u think u couldn't poll a Gcp bucket that is S3 compatible? See [amazon web services - How to access Google Cloud Storage bucket using aws-cli - Stack Overflow](https://stackoverflow.com/a/50162192) on how to do it via the AWS cli. ---
**Author:** ![arvin495](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arvin495/32/98571_2.png) [@arvin495](https://discuss.elastic.co/u/arvin495)\ **Post date:** [December 13, 2021, 3:08am UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/5 "2021-12-13T03:08:14Z")
apologies for miscommunication from my end. what I meant was we could use gcp provider in fileabeat input configuration the way we are using it for s3 as follows ```auto filebeat.inputs: - type: aws-s3 non_aws_bucket_name: test-s3-bucket number_of_workers: 5 bucket_list_interval: 300s access_key_id: xxxxxxx secret_access_key: xxxxxxx endpoint: https://s3.example.com:9000 expand_event_list_from_field: Records ``` I just want to know how we can do the same to fetch GCS objects to Elasticsearch. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [December 13, 2021, 4:03am UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/6 "2021-12-13T04:03:09Z")
You would configure it like so. Also u can test the AWS cli the same way using the link from the previous post to test the keys and endpoint. ```auto filebeat.inputs: - type: aws-s3 non_aws_bucket_name: test-s3-bucket number_of_workers: 5 bucket_list_interval: 300s access_key_id: xxxxxxx secret_access_key: xxxxxxx endpoint: https://storage.googleapis.com ``` ---
**Author:** ![arvin495](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arvin495/32/98571_2.png) [@arvin495](https://discuss.elastic.co/u/arvin495)\ **Post date:** [December 13, 2021, 4:25am UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/7 "2021-12-13T04:25:11Z")
Thanks @legoguy1000 I tried above configuration however I couldn't understand the part where it is still checking for bucket\_arn or queue\_url even though I provided non\_aws\_bucket\_name. error: ```auto WARN [aws-s3] awss3/config.go:54 neither queue_url nor bucket_arn were provided, input aws-s3 will stop INFO [crawler] beater/crawler.go:141 Starting input (ID: 17738867761700079737) INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1 INFO [input.aws-s3] compat/compat.go:111 Input aws-s3 starting {"id": "F62D1E3EA5C30879"} INFO [input.aws-s3] compat/compat.go:124 Input 'aws-s3' stopped {"id": "F62D1E3EA5C30879"} ``` should we do any changes on type? probably changing aws-s3 to gcp-gcs (I'm not sure) ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [December 13, 2021, 1:13pm UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/8 "2021-12-13T13:13:01Z")
My apologies, the ability to poll non AWS buckets was only added to 8.0.0 and want backpack to 7.x. You'll have to wait until 8.0 is released to be able to too what I explained. But to provide a bit more clarification, you can't just change the input names. There is a specific list of inputs that can be used, [Configure inputs | Filebeat Reference [7.16] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#filebeat-input-types). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 10, 2022, 3:13pm UTC](https://discuss.elastic.co/t/how-to-configure-gcs-as-filebeat-input/291453/9 "2022-01-10T15:13:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.