Logstash 2.4 - S3 output strange ${message} in file

Hi -

The S3 output seems to be working in that I'm getting messages, but instead of the parsed output I'm getting a message variable ${message}. This happens in both the temp location and in S3.

Output -
root@ip-xx-xxx-xxx-xxx:/tmp/logstash# cat ls.s3.ip-xx-xxx-xxx-xxx.2016-11-04T20.30.part2.txt
2016-11-04T20:30:27.373Z ip-xx-xxx-xxx-xxx %{message}

Sample config section
s3 {
bucket => "zz-test-bucket"
prefix => "logstash/"
region => "us-east-1"
temporary_directory => "/tmp/logstash"
size_file => 2048
time_file => 5

Any thoughts on what I'm doing wrong?


Do your events have a message field?

No I remove that field after the grok filter.

Well, that certainly explains things. What do you want your S3 logs to look like?

Actually, for now I'm just passing (or would like to) _grokparsefailure logs to S3. The rest is going to kinesis and ES.

Json objects would be fine. Is this just a matter of changing the codec?

Yes, use the json_lines codec.

that worked...thx.