# How often does logstash send data to google cloud storage?

**URL:** <https://discuss.elastic.co/t/how-often-does-logstash-send-data-to-google-cloud-storage/257973>\
**Category:** Logstash\
**Created:** [December 8, 2020, 10:53am UTC](https://discuss.elastic.co/t/how-often-does-logstash-send-data-to-google-cloud-storage/257973 "2020-12-08T10:53:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![iooi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iooi/32/79231_2.png) [@iooi](https://discuss.elastic.co/u/iooi)\
**Post date:** [December 8, 2020, 10:53am UTC](https://discuss.elastic.co/t/how-often-does-logstash-send-data-to-google-cloud-storage/257973/1 "2020-12-08T10:53:28Z")

</div>

# Config

pipeline.yml

```auto
    output {
        stdout { codec => rubydebug }
        google_cloud_storage {
          bucket => "my_nginx"
          json_key_file => "/my/secret/path/credentials.json"
          temp_directory => "/tmp/logstash-gcs"
          log_file_prefix => "logstash_gcs"
          max_file_size_kbytes => 1024
          output_format => "plain"
          date_pattern => "%Y-%m-%dT%H:00"
          flush_interval_secs => 2
          gzip => false
          gzip_content_encoding => false
          uploader_interval_secs => 60
          include_uuid => true
          include_hostname => true
        }
    }

```

# Logs

```auto
    [INFO] 2020-12-08 10:15:36.955 [[main]-pipeline-manager] googlecloudstorage - Using temporary directory: /tmp/logstash-gcs
    [INFO] 2020-12-08 10:15:37.036 [[main]-pipeline-manager] googlecloudstorage - Initializing Google API client, key: /my/secret/path/credentials.json

```

# /tmp/logstash-gcs file

```auto
    $ cat /tmp/logstash-gcs/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log
    2020-12-08T10:16:21.284Z {name=filebeat-86ljd} 192.168.49.2 - - [08/Dec/2020:10:16:12 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "-"

```

* * *

When got data, it can find it in the stdout, but after that check on the gcs, didn't find any data there.

---

<div class="post-metadata">

**Author:** ![iooi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iooi/32/79231_2.png) [@iooi](https://discuss.elastic.co/u/iooi)\
**Post date:** [December 8, 2020, 11:06am UTC](https://discuss.elastic.co/t/how-often-does-logstash-send-data-to-google-cloud-storage/257973/2 "2020-12-08T11:06:03Z")

</div>

It uploaded after a long while. About 44 minutes. New log:

```auto
[INFO] 2020-12-08 11:00:03.069 [[main]>worker0] googlecloudstorage - Rotated out file: /tmp/logstash-gcs/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log

[INFO] 2020-12-08 11:00:03.084 [pool-9-thread-1] googlecloudstorage - Uploading file to my_nginx/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log

[INFO] 2020-12-08 11:00:05.511 [pool-9-thread-1] googlecloudstorage - Uploaded file to my_nginx/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 5, 2021, 11:06am UTC](https://discuss.elastic.co/t/how-often-does-logstash-send-data-to-google-cloud-storage/257973/3 "2021-01-05T11:06:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.