Filebeats custom fields not sending with message

I'm running filebeats 5.5.1, the 2 fields that I've added into the prospector aren't sending with the logs. Here is my prospector:

- input_type: log
    paths:
        - /etc/filebeat/log/*.log
    fields: 
        log_type: promolog
        type: pricing
    fields_under_root: true

Here is the message I'm sending to the log

echo "This is a test" >> test.log

Here is what logstash is seeing

2017-09-15T16:39:48.593Z 1843d9713bf0 This is a test

How do you have Logstash configured? Where is that output coming from? Please share your Logstash config.

Here is the combined logstash config, it's separated into separate files for input, filter, and output

input {
  beats {
    port => 5044
    ssl => false
    #ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    #ssl_key => "/etc/pki/tls/private/logstash-beats.key"
 }
}
filter {
  if [fields][log_type] == "promolog" {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:logtimestamp} %{WORD:msg} %
{SYSLOG5424SD:jclass} - %{WORD:rtype} %{SPACE}:%{SPACE}%{GREEDYDATA:payload}" }
  }
    date {
        match => ["logtimestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
        target => "@timestamp"
        add_field => { "debug" => "timestampMatched"}
  }

   csv {
       source => "payload"
       separator => "|"
       remove_field => ["payload"]
  }

  mutate {
      add_field  => {
       "reqtype"  => "%{column2}"
       "ordernum" => "%{column3}"
       "brand"    => "%{column4}"
       "promo"    => "%{column5}"
     }
   } 
 }
 }
output {

   stdout {}
   elasticsearch {
   hosts => ["localhost"]
   manage_template => false
   index => "pricing-%{YYYY.MM.dd}"
   document_type => "pricing"
   user => "elastic"
   password => "changeme"

}
}

Try using a different codec with the stdout output. This should cause all fields to be logged.

output {
  stdout { codec => rubydebug { metadata => true } }
}

Perfect, thank you! I'm now seeing the fields in the logstash output.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.