Logstash-output-s3 not working with auditbeat

[computermaster0101]

I have setup auditbeat on a centos6 server. it ships to logstash which ships to s3. Below is my config

input {
beats {
port => 9210
}
}
output {
s3{
region => "us-east-1"
bucket => "NAME-REMOVED"
prefix => "auditLogs/%{+YYYY}/%{[fields][env]}/%{[fields][stack]}/%{[beat][hostname]}/%{+MM}/%{+dd}/"
time_file => 60
}
}

An object is created in the correct location in the s3 bucket but it contains the following

2018-07-16T05:48:58.036Z {name=SERVER-NAME-REMOVED} %{message}

I've setup WinLogBeat on Windows server and it ships to the same logstash and the messages are coming through. After doing in-depth troubleshooting, i believe the issue is that auditbeat is sending a messages object instead of a message object. Has anyone else experienced this? Should this be a new ticket in github?

ps. I've removed PII from the above

[computermaster0101]

Good morning community!

I wanted to check in on this as it has been a week. Is there any other information i can supply or any other validations I should complete before opening a ticket for a bug?

[computermaster0101]

Hey Everyone!

Here is another update. I was able to setup and configure auditd and filebeat. Going this route everything shows up in the s3 bucket as expected. Im not sure if this is a bug in the way auditbeat ships vs how all the other beats ship but i did not change anything in the logstash config.

[yaauie (Ry Biesemeyer)]

The default format string for the S3 output plugin includes a reference to the message field, but if the event being pushed does not have a message field, the format string is not expanded.

I'm not familiar with the exact format of AuditBeat, but if you add a Stdout Output Plugin, you may be able to determine the "shape" of the output and coerce it into place:

output {
  if [message]
    s3 {
      # ...
    }
  } else {
    stdout {
      codec => rubydebug
    }
  }

Once you know the "shape" of the events, you can add a filter that will compose your message using the sprintf syntax:

filter {
  if not [message] {
    mutate => {
      add_field => {
        "message" => ""
      }
    }
  }
}

OR: if auditbeat is sending messages, and you want to split these out to be individual message objects, you can use the Split Filter Plugin:

filter {
  if [messages] and not [message] {
    split {
      field => "messages"
      target => "message"
      remove_field => "messages"
    }
  }
}