Burga
March 15, 2020, 9:27am
1
Hi , i'm creating index name dynamically but when I'm passing the field to output
i'm getting the variable name ([@metadata ][index]):
mutate {
add_field => { "[@metadata][index_prefix]" => "system-secure" }
}
output :
output {@metadata ][index_prefix]-%{+YYYY.MM.dd}"@metadata ][index_prefix]"} }
and getting --> Rate: %[@metadata ][index_prefix]
please assist .
rcowart
March 15, 2020, 9:33am
2
Burga
March 15, 2020, 11:40am
3
after the change I'm getting the same result @metadata ][index_prefix]}"} }
result:
Mar 15 13:39:06 il-infra-logs2 logstash[17753]: %{[@metadata ][index_prefix]}
rcowart
March 15, 2020, 12:01pm
4
Usually that happens when the field isn't set. To better see what is included in the event, add an output with the rubydebug codec and enable inclusion of metadata.
output {
stdout {
codec => rubydebug {
metadata => "true"
}
}
}
BTW, please enclose your examples in a proper code block, as I did above, so it displays easier. You do this by putting three back ticks on the lines before and after your code.
Rob
How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations What is the best storage technology for Elasticsearch?
Burga
March 15, 2020, 12:35pm
5
yes the field wasn't set , somehow it didn't get into the
if [fileset][module] == "system" {
and I don't know why when I put the line right after the filter it sets the field but after the
[fileset][module] == "system" it doesn't
it doesn't
My code is
filter {
mutate {
add_field => { "[@metadata][index_prefix]" => "filter" }
}
if [fileset][module] == "system" {
mutate {
add_field => { "[@metadata][index_prefix]" => "system" }
}
if [fileset][name] == "auth" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$",
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
pattern_definitions => {
"GREEDYMULTILINE"=> "(.|\n)*"
}
remove_field => "message"
}
date {
match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
geoip {
source => "[system][auth][ssh][ip]"
target => "[system][auth][ssh][geoip]"
}
}
else if [fileset][name] == "syslog" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
remove_field => "message"
}
date {
match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => [ "il-infra-es1", "il-infra-es2", "il-infra-es3" ]
manage_template => false
index => "%[@metadata][index_prefix]-%{+YYYY.MM.dd}"
}
stdout { codec => line {format => "%{[@metadata][index_prefix]}"} }
}
In my original code I'm trying to create different indexes to different types (syslog,auth )
the second day I'm sitting on this issue , can you please direct me what I'm doing wrong
rcowart
March 15, 2020, 1:19pm
6
Did you output using rubydebug, including metadata => true, as I suggested? I think it would help to see what is going on here.
Let's reduce this to a minimum...
mutate {
add_field => { "[@metadata][index_prefix]" => "filter" }
}
if [fileset][module] == "system" {
mutate {
add_field => { "[@metadata][index_prefix]" => "system" }
}
...
}
That first mutate will set [@metadata][index_prefix] to filter.
If [fileset][module] equals system the second mutate will add another value to [@metadata][index_prefix]. The result being an array of two values, [ filter, system ]. You should be able to see this if you output the values as I suggested.
In this case when refer to %{[@metadata][index_prefix]} Logstash doesn't know which value to use, and you just get the variable name.
If you intend this second mutate to replace the original value, you need to use replace instead of add_field. In fact, if I am setting a field's value using mutate and I know I never want that field to hold more than a single value, I will always use replace. If the field doesn't already exist, it will be created. So there is no downside to using replace.
BTW, don't forget to change index => "%[@metadata][index_prefix]-%{+YYYY.MM.dd}" to include braces index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}".
Rob
How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations What is the best storage technology for Elasticsearch?
Burga
March 15, 2020, 1:44pm
7
Yes I ran it as you suggested and I get no value set for this field when
it is placed inside the if
if [fileset][module] == "system" {
mutate {
add_field => { "[@metadata][index_prefix]" => "system" }
}
when it outside it has the value of "filter" , I suppose it means that it simply doesn't get inside because the
[fileset][module] == "system"
is false ?
this is what i get :
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "log" => {Mar 15 15:36:24 il-infra-logs2 logstash[758]: "@metadata " => { Mar 15 15:36:24 il-infra-logs2 logstash[758]: "version" => "7.6.1", Mar 15 15:36:24 il-infra-logs2 logstash[758]: "ip_address" => "192.168.127.230", Mar 15 15:36:24 il-infra-logs2 logstash[758]: "beat" => "filebeat", Mar 15 15:36:24 il-infra-logs2 logstash[758]: "type" => "_doc"
how can I check what is the value of [fileset][module] ?
rcowart
March 15, 2020, 1:58pm
8
If you don't see it in the rubydebug output, then it doesn't exist.
rcowart
March 15, 2020, 2:02pm
9
I think what you are looking for in current versions of Beats is event.module.
system
April 12, 2020, 5:50pm
11
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
