New user - Simple problem with adding a field with mutate

Elastic Stack Logstash
1

I want to add a new field based on the content of the source field from filebeat. The filebeat-* index shows SOME logs get the new field, and other logs are empty. However, the logstash-* index has file defined for every log file. Don't understand.

Tnx

input {

beats {
port => 5044
}
}

filter {
mutate {
add_field => {
"file" => "%{source}"
}
}
}

2

If the source field doesn't always exist you shouldn't unconditionally include the mutate filter. Use a conditional:

if [source] {
  mutate {
    add_field => {
      "file" => "%{source}"
    }
  }
}
3

Thank you, magnus. Could I get you to look at my latest config? I made my jboss filebeat send the type as "jboss" so I could do some processing on it. I want to extract the debug level. Again, some logs I see the level field added and some I don't, yet the message exists and my pattern matches as confirmed in the grok debugger.

filter {
    mutate {
        add_field => {
            "file" => "%{source}"
        }
    }
    if [message] {
    if [type] == "jboss" {
        grok {
            match => { "message" => "%{TIME:time} %{WORD:level} .*" }
        }
        grok {
            add_field => { "level" => "%{level}" }
        }
    }
    }
}

Example line that isn't generating the level field.

10:23:09,766 INFO [stdout] (Curator-TreeCache-0) Listen: Update path: /realtime/subscriptions/network/160072/160072/TLOC, timestamp is: 1472836989577

4 
if [message] {

if [type] == "jboss" {

Equivalent:

if [message] and [type] == "jboss" {

grok {

        add_field => { "level" => "%{level}" }
    }

I've seen people make this exact mistake a few times before. What made you add this?

Example line that isn't generating the level field.

Works just fine:

$ cat test.config
input { stdin {} }
output { stdout { codec => rubydebug } }
filter {
  grok {
    match => ["message", "%{TIME:time} %{WORD:level} .*"]
  }
}
$ echo '10:23:09,766 INFO [stdout] (Curator-TreeCache-0) Listen: Update path: /realtime/subscriptions/network/160072/160072/TLOC, timestamp is: 1472836989577' | logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "10:23:09,766 INFO [stdout] (Curator-TreeCache-0) Listen: Update path: /realtime/subscriptions/network/160072/160072/TLOC, timestamp is: 1472836989577",
      "@version" => "1",
    "@timestamp" => "2016-09-03T15:35:47.428Z",
          "host" => "bertie",
          "time" => "10:23:09,766",
         "level" => "INFO"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}