Passing to output index field but getting [@metadata][index]

Hi , i'm creating index name dynamically but when I'm passing the field to output

i'm getting the variable name ([@metadata][index]):

 mutate {
  add_field => { "[@metadata][index_prefix]" => "system-secure" }
}

output :

output {
elasticsearch {
hosts => [ "il-infra-es1", "il-infra-es2", "il-infra-es3" ]
manage_template => false
index => "%[@metadata][index_prefix]-%{+YYYY.MM.dd}"
}
stdout { codec => line {format => "Rate: %[@metadata][index_prefix]"} }
}

and getting --> Rate: %[@metadata][index_prefix]

please assist .

You are missing braces. %[@metadata][index_prefix] should be %{[@metadata][index_prefix]}.

Rob

GitHub YouTube LinkedIn
How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations
What is the best storage technology for Elasticsearch?

after the change I'm getting the same result :slight_smile:
stdout { codec => line {format => "%{[@metadata][index_prefix]}"} }

result:

Mar 15 13:39:06 il-infra-logs2 logstash[17753]: %{[@metadata][index_prefix]}

Usually that happens when the field isn't set. To better see what is included in the event, add an output with the rubydebug codec and enable inclusion of metadata.

output {
  stdout {
    codec => rubydebug {
      metadata => "true"
    }
  }
}

BTW, please enclose your examples in a proper code block, as I did above, so it displays easier. You do this by putting three back ticks on the lines before and after your code.

Rob

GitHub YouTube LinkedIn
How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations
What is the best storage technology for Elasticsearch?

yes the field wasn't set , somehow it didn't get into the


if [fileset][module] == "system" {

and I don't know why when I put the line right after the filter it sets the field but after the

 [fileset][module] == "system" it doesn't 

it doesn't

My code is

filter {

    
	   mutate {
      add_field => { "[@metadata][index_prefix]" => "filter" }
    }

  if [fileset][module] == "system" {
  
  
	   mutate {
      add_field => { "[@metadata][index_prefix]" => "system" }
    }
  
    if [fileset][name] == "auth" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} %{DATA:[system][auth][ssh][method]} for (invalid user )?%{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]} port %{NUMBER:[system][auth][ssh][port]} ssh2(: %{GREEDYDATA:[system][auth][ssh][signature]})?",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: %{DATA:[system][auth][ssh][event]} user %{DATA:[system][auth][user]} from %{IPORHOST:[system][auth][ssh][ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:\[%{POSINT:[system][auth][pid]}\])?: Did not receive identification string from %{IPORHOST:[system][auth][ssh][dropped_ip]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:\[%{POSINT:[system][auth][pid]}\])?: \s*%{DATA:[system][auth][user]} :( %{DATA:[system][auth][sudo][error]} ;)? TTY=%{DATA:[system][auth][sudo][tty]} ; PWD=%{DATA:[system][auth][sudo][pwd]} ; USER=%{DATA:[system][auth][sudo][user]} ; COMMAND=%{GREEDYDATA:[system][auth][sudo][command]}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:\[%{POSINT:[system][auth][pid]}\])?: new group: name=%{DATA:system.auth.groupadd.name}, GID=%{NUMBER:system.auth.groupadd.gid}",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{DATA:[system][auth][useradd][name]}, UID=%{NUMBER:[system][auth][useradd][uid]}, GID=%{NUMBER:[system][auth][useradd][gid]}, home=%{DATA:[system][auth][useradd][home]}, shell=%{DATA:[system][auth][useradd][shell]}$",
                  "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:\[%{POSINT:[system][auth][pid]}\])?: %{GREEDYMULTILINE:[system][auth][message]}"] }
        pattern_definitions => {
          "GREEDYMULTILINE"=> "(.|\n)*"
        }
        remove_field => "message"
      }
      date {
        match => [ "[system][auth][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
      geoip {
        source => "[system][auth][ssh][ip]"
        target => "[system][auth][ssh][geoip]"
      }
	 
	  
	  
    }
    else if [fileset][name] == "syslog" {
      grok {
        match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSINT:[system][syslog][pid]}\])?: %{GREEDYMULTILINE:[system][syslog][message]}"] }
        pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
        remove_field => "message"
      }
      date {
        match => [ "[system][syslog][timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
	  
	  
    }
  }

output {
  elasticsearch {
    hosts => [ "il-infra-es1", "il-infra-es2", "il-infra-es3" ]
    manage_template => false
    index => "%[@metadata][index_prefix]-%{+YYYY.MM.dd}"
  }
stdout { codec => line {format => "%{[@metadata][index_prefix]}"} }
}

In my original code I'm trying to create different indexes to different types (syslog,auth )

the second day I'm sitting on this issue , can you please direct me what I'm doing wrong

Did you output using rubydebug, including metadata => true, as I suggested? I think it would help to see what is going on here.

Let's reduce this to a minimum...

mutate {
  add_field => { "[@metadata][index_prefix]" => "filter" }
}

if [fileset][module] == "system" {
  mutate {
    add_field => { "[@metadata][index_prefix]" => "system" }
  }

  ...

}

That first mutate will set [@metadata][index_prefix] to filter.

If [fileset][module] equals system the second mutate will add another value to [@metadata][index_prefix]. The result being an array of two values, [ filter, system ]. You should be able to see this if you output the values as I suggested.

In this case when refer to %{[@metadata][index_prefix]} Logstash doesn't know which value to use, and you just get the variable name.

If you intend this second mutate to replace the original value, you need to use replace instead of add_field. In fact, if I am setting a field's value using mutate and I know I never want that field to hold more than a single value, I will always use replace. If the field doesn't already exist, it will be created. So there is no downside to using replace.

BTW, don't forget to change index => "%[@metadata][index_prefix]-%{+YYYY.MM.dd}" to include braces index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}".

Rob

GitHub YouTube LinkedIn
How to install Elasticsearch & Kibana on Ubuntu - incl. hardware recommendations
What is the best storage technology for Elasticsearch?

Yes I ran it as you suggested and I get no value set for this field when

it is placed inside the if

if [fileset][module] == "system" {
  
  
	   mutate {
      add_field => { "[@metadata][index_prefix]" => "system" }
    }

when it outside it has the value of "filter" , I suppose it means that it simply doesn't get inside because the

[fileset][module] == "system"

is false ?

this is what i get :

Mar 15 15:36:24 il-infra-logs2 logstash[758]: "log" => {
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "file" => {
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "path" => "/var/log/messages"
Mar 15 15:36:24 il-infra-logs2 logstash[758]: },
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "offset" => 18645146
Mar 15 15:36:24 il-infra-logs2 logstash[758]: },
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "@metadata" => {
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "version" => "7.6.1",
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "ip_address" => "192.168.127.230",
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "beat" => "filebeat",
Mar 15 15:36:24 il-infra-logs2 logstash[758]: "type" => "_doc"
Mar 15 15:36:24 il-infra-logs2 logstash[758]: },

how can I check what is the value of [fileset][module] ?

If you don't see it in the rubydebug output, then it doesn't exist.

I think what you are looking for in current versions of Beats is event.module.

Thank you