Filebeats custom fields not sending with message

[jspeer (Josh Speer)]

I'm running filebeats 5.5.1, the 2 fields that I've added into the prospector aren't sending with the logs. Here is my prospector:

- input_type: log
    paths:
        - /etc/filebeat/log/*.log
    fields: 
        log_type: promolog
        type: pricing
    fields_under_root: true

Here is the message I'm sending to the log

echo "This is a test" >> test.log

Here is what logstash is seeing

2017-09-15T16:39:48.593Z 1843d9713bf0 This is a test

[andrewkroh]

How do you have Logstash configured? Where is that output coming from? Please share your Logstash config.

[jspeer (Josh Speer)]

Here is the combined logstash config, it's separated into separate files for input, filter, and output

input {
  beats {
    port => 5044
    ssl => false
    #ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    #ssl_key => "/etc/pki/tls/private/logstash-beats.key"
 }
}
filter {
  if [fields][log_type] == "promolog" {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:logtimestamp} %{WORD:msg} %
{SYSLOG5424SD:jclass} - %{WORD:rtype} %{SPACE}:%{SPACE}%{GREEDYDATA:payload}" }
  }
    date {
        match => ["logtimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
        target => "@timestamp"
        add_field => { "debug" => "timestampMatched"}
  }

   csv {
       source => "payload"
       separator => "|"
       remove_field => ["payload"]
  }

  mutate {
      add_field  => {
       "reqtype"  => "%{column2}"
       "ordernum" => "%{column3}"
       "brand"    => "%{column4}"
       "promo"    => "%{column5}"
     }
   } 
 }
 }
output {

   stdout {}
   elasticsearch {
   hosts => ["localhost"]
   manage_template => false
   index => "pricing-%{YYYY.MM.dd}"
   document_type => "pricing"
   user => "elastic"
   password => "changeme"

}
}

[andrewkroh]

Try using a different codec with the stdout output. This should cause all fields to be logged.

output {
  stdout { codec => rubydebug { metadata => true } }
}

[jspeer (Josh Speer)]

Perfect, thank you! I'm now seeing the fields in the logstash output.