I am experiencing some strange behavior with logstash, using a combination of codecs on a input/file and an output/s3. There seems so be an issue with the output/s3 logstash plugin, as I cannot get any part files to upload to S3 unless I specify a codec in the output/s3 plugin.
I am tailing java application log files, so ideally I am using the input/file plugin to watch all log files in a directory and make sure any stack traces encountered (and their new lines) are wrapped up into the same logstash event. I do so like this:
input {
file {
path => "C:/some/directory/logs/*"
codec => multiline {
pattern => "^%{DATESTAMP}"
negate => true
what => "previous"
}
}
}
This will properly append my stack traces to their parent events. Then I want to perform two different output/s3 operations (essentially recreating the raw log line by line, and also uploading the event json):
output {
s3{
access_key_id => "mykey"
secret_access_key => "myseckey"
region => "us-east-1"
bucket => "somebucket"
size_file => 10000
upload_workers_count => 2
restore => true
prefix => "rawlogs/"
temporary_directory => "C:/Temp/LogStash/raw"
time_file => 5
}
s3{
access_key_id => "mykey"
secret_access_key => "myseckey"
region => "us-east-1"
bucket => "somebucket"
size_file => 10000
upload_workers_count => 2
restore => true
prefix => "jsoneventlogs/"
temporary_directory => "C:/Temp/LogStash/json"
time_file => 5
codec => "json_lines"
}
}
The S3 upload that uses the "json_lines" codec works fine, but the raw log upload that uses the default "plain" codec does not work at all. The files sit in my temp directory and never get pushed to S3. I have tried to use the "line" codec, but still the same behavior. If I remove the "multiline" codec from the input/file plugin and use it in my output/S3 raw plugin, then they will upload to S3 just fine, but each newline in a stacktrace is coming in as its own event, thus the codec does not seem to do its job.
Any idea why the output/S3 plugin only seems to work with json_lines and multiline?