Using this config for logstash's output. It's using /tmp/logstash-gcs
as a local folder. Send to GCS when file become 1024 kbytes.
input {
beats {
port => 5044
}
}
filter {}
output {
google_cloud_storage {
bucket => "gcs-bucket-name"
json_key_file => "/secrets/service_account/credentials.json"
temp_directory => "/tmp/logstash-gcs"
log_file_prefix => "logstash_gcs"
max_file_size_kbytes => 1024
output_format => "json"
date_pattern => "%Y-%m-%dT%H:00"
flush_interval_secs => 2
gzip => false
gzip_content_encoding => false
uploader_interval_secs => 60
include_uuid => true
include_hostname => true
}
}
After restart machine, can it continuous do the job schedule without any data loss?
Does it has a queue feature to manage as pub/sub?