How often does logstash send data to google cloud storage?

Elastic Stack Logstash
1

Config

pipeline.yml

    output {
        stdout { codec => rubydebug }
        google_cloud_storage {
          bucket => "my_nginx"
          json_key_file => "/my/secret/path/credentials.json"
          temp_directory => "/tmp/logstash-gcs"
          log_file_prefix => "logstash_gcs"
          max_file_size_kbytes => 1024
          output_format => "plain"
          date_pattern => "%Y-%m-%dT%H:00"
          flush_interval_secs => 2
          gzip => false
          gzip_content_encoding => false
          uploader_interval_secs => 60
          include_uuid => true
          include_hostname => true
        }
    }

Logs

    [INFO ] 2020-12-08 10:15:36.955 [[main]-pipeline-manager] googlecloudstorage - Using temporary directory: /tmp/logstash-gcs
    [INFO ] 2020-12-08 10:15:37.036 [[main]-pipeline-manager] googlecloudstorage - Initializing Google API client, key: /my/secret/path/credentials.json

/tmp/logstash-gcs file

    $ cat /tmp/logstash-gcs/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log
    2020-12-08T10:16:21.284Z {name=filebeat-86ljd} 192.168.49.2 - - [08/Dec/2020:10:16:12 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" "-"

When got data, it can find it in the stdout, but after that check on the gcs, didn't find any data there.

2

It uploaded after a long while. About 44 minutes. New log:

[INFO ] 2020-12-08 11:00:03.069 [[main]>worker0] googlecloudstorage - Rotated out file: /tmp/logstash-gcs/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log

[INFO ] 2020-12-08 11:00:03.084 [pool-9-thread-1] googlecloudstorage - Uploading file to my_nginx/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log

[INFO ] 2020-12-08 11:00:05.511 [pool-9-thread-1] googlecloudstorage - Uploaded file to my_nginx/logstash_gcs_logstash-nginx-to-gcs-0_2020-12-08T10:00.part001.ccf37ea7-673a-4468-b35f-867b35d4aa77.log
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.