Output only s3 file create events to elastic search and not the contents of the files

Elastic Stack Logstash
1

Hello team,

Is there a config that enables Logstash s3 input plugin to output only s3 filenames under the bucket where Logstash config is polling on?

The current below config is outputting file contents with message fields excluded to elastic search. I have all the info required in log file names (as a status prefix) and the contents of the log files are not needed at this point. Kindly let me know if the same is not possible.

input {
  s3 {
    "region" => "us-east-2"
    "bucket" => "s3-bucket"
    "prefix" => "<pre-fix>"
    "interval" => "10"
"exclude_pattern" => ".txt\z|.sh\z"
"additional_settings" => {
      "force_path_style" => true
      "follow_redirects" => false
                }
  }
}

filter {
  mutate {
    remove_field => [ "message" ]
    add_field => {
      "file" => "%{[@metadata][s3][key]}"
    }
  }
}
output {
  elasticsearch {
    hosts => ["vcp_endpoint"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}
2

For example,

If pipline_1234/logs/done_1213_study.log is the S3 key, and below are the file contents of the file

[timestamp] started pipeline for subject 1234
[timestamp] download raw data
[timestamp] processing intermediate step
[timestamp] ....... 50 intermittent log lines
[timestamp] completed processing for subject 1234

I just need only the log file key to be logged in to elastic search and not the contents to elastic search.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.