Mutate Split unable to get variables

Elastic Stack Logstash
1

Logstash version 6.6.1

I am attempting to split out the [path] to get the file name. Not all of my logs have extensions, so I am attempting to split things by the / rather than the grok example that I found with the ".log" at the end. But I have been unable to get even basic split functionality working.

Here is my dmesg.conf:

input {
  pipeline {
    address => dmesg_log
  }
}
filter {
  mutate {
    split => ["path", "/"]
  }
  mutate {
    add_field => { "shortHostname" => "%{path[-1]}" }
  }
}

When I run this through logstash I receive a warning:

[2019-03-07T09:08:41,478][WARN ][org.logstash.FieldReference] Detected ambiguous Field Reference `path[-1]`, which we expanded to the path `[path, -1]`; in a future release of Logstash, ambiguous Field References will not be expanded.

When I check the output, I find that logstash creates the shortHostname and sets it to "%{path[-1]}"

 {"shortHostname":"%{path[-1]}","tags":["beats_input_codec_plain_applied"],"host":{"id":"e815db59563c4267b2ffb999d0cabc42","containerized":tr     ue,"os":{"family":"redhat","version":"7 (Core)","platform":"centos","codename":"Core","name":"CentOS Linux"},"architecture":"x86_64","name":     "localhost.localdomain"},"@version":"1","source":"/var/log/dmesg","prospector":{"type":"log"},"fields":{"dmesg":true},"message":"[    6.6199     73] type=1305 audit(1551742248.385:4): audit_pid=567 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1","input":     {"type":"log"},"log":{"file":{"path":"/var/log/dmesg"}},"beat":{"version":"6.6.1","hostname":"localhost.localdomain","name":"localhost.local     domain"},"offset":120579,"@timestamp":"2019-03-07T17:08:42.061Z"}

From there, I opted to copy and paste the code snippet directly from the documentation to verify that I didn't have any major issues with my syntax. But I ran into the exact same problem:

input {
  pipeline {
    address => dmesg_log
  }
}
filter {
  mutate {
    split => ["hostname", "."]
    add_field => { "shortHostname" => "{%hostname[0]}" }
  }
}

When I run this, I don't receive any errors or warning but I have the same output issue:

{"shortHostname":"{%hostname[0]}","tags":["beats_input_codec_plain_applied"],"host":{"id":"e815db59563c4267b2ffb999d0cabc42","containerized"     :true,"os":{"family":"redhat","version":"7 (Core)","platform":"centos","codename":"Core","name":"CentOS Linux"},"architecture":"x86_64","nam     e":"localhost.localdomain"},"@version":"1","source":"/var/log/dmesg","prospector":{"type":"log"},"fields":{"dmesg":true},"message":"[    6.6     19973] type=1305 audit(1551742248.385:4): audit_pid=567 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1","inpu     t":{"type":"log"},"log":{"file":{"path":"/var/log/dmesg"}},"beat":{"version":"6.6.1","hostname":"localhost.localdomain","name":"localhost.lo     caldomain"},"offset":120579,"@timestamp":"2019-03-07T17:29:14.679Z

I am unsure what it is that I am doing wrong, especially when I am copying and pasting directly from the documentation.
I'm not sure if the split is actually happening, or if I need to use variables a different way.

Do you have any suggestions?

2 
add_field => { "shortHostname" => "%{[path][-1]}" }

I did not know you could do an array reference like that, but it works! Which documentation did you copy and paste from?

3

This was the documentation that I pulled it from:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-split

And how did you get yours to work?
When I used your example I still have the same problem

{"shortHostname":"%{[path][-1]}","tags":["beats_input_codec_plain_applied"],"host":{"id":"e815db59563c4267b2ffb999d0cabc42","containerized":     true,"os":{"family":"redhat","version":"7 (Core)","platform":"centos","codename":"Core","name":"CentOS Linux"},"architecture":"x86_64","name     ":"localhost.localdomain"},"@version":"1","source":"/var/log/dmesg","prospector":{"type":"log"},"fields":{"dmesg":true},"message":"[    6.61     9973] type=1305 audit(1551742248.385:4): audit_pid=567 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1","input     ":{"type":"log"},"log":{"file":{"path":"/var/log/dmesg"}},"beat":{"version":"6.6.1","hostname":"localhost.localdomain","name":"localhost.loc     aldomain"},"offset":120579,"@timestamp":"2019-03-07T19:00:01.479Z"}

Here is my dmesg.conf:

filter {
  mutate {
    split => ["path", "/"]
    add_field => { "shortHostname" => "%{[path][-1]}" }
  }
}
4

That event does not have a field [path], it has [log][file][path]. Try

split => ["[log][file][path]", "/"]
add_field => { "shortHostname" => "%{[log][file][path][-1]}" }
5

Dangit, the farthest I went was [file][path] I didn't try [log][file][path].

That fixed my problem. Once again, thank you for your help with this.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.