Can i define S3 output file content?

[brandonbai (Brandon)]

Hi,

When I use the following code to remove the message field,

filter {
....

 if "_grokparsefailure" not in [tags] {
    mutate {
      remove_field => [ "message" ]
    }
  }
....
}

the log content output to s3 becomes the following content:

2019-07-26T02:38:56.000Z {name=f2b7eda7958a} %{message}

From this I guess the s3 plugin will only get and output the message field.

Is there any way to make:

1. The content output to elasticsearch does not contain the message field;
2. The content output to s3 contains only the message field

logstash version: 6.6.0

Thanks!

[Badger]

@brandonbai

2019-07-26T02:38:56.000Z {name=f2b7eda7958a} %{message}

That's the output from event.to_s, which is what the line codec uses if you do not supply a format. So supply a format...

output { s3 { codec => line { format => "..." ...

[brandonbai (Brandon)]

Hi Badger,

As you said, I also saw the relevant introduction in the official document.

Thanks!