Filebeat fields value unable to use it in logstash configuration file

Elastic Stack Logstash
#1

I have set custom name in filebeat input section. However, in logstash configuration file it was unable to be use

Filebeat config file

- type: log

  enabled: true

  paths:
  ...
  fields:
    - name_of_index: group-1
    - name_of_log: apache

Logstash.conf

input {
  beats {
    port => 5044
    ...
    type => "%{[field][name_of_log]}"
  }
}

filter {
  if "%{[field][name_of_log]}" in ["apache"] {
    ...
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}" 
  }
}

output in elasticsearch

%[fields][name_of_index]-2020-01-05

I not sure which parr it is wrong able to help in this?

#2

looks like you have typos...

In filebeat no - See Here

  fields:
    name_of_index: group-1
    name_of_log: apache

The fields are under fields several places you only have the term field no s.
Also you could probably simplify perhaps look here.

input {
  beats {
    port => 5044
    ...
    type => "%{[fields][name_of_log]}"
  }
}

filter {
  if [type] == "apache" {
    ...
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}" 
  }
}
#3

hi can i know if i use type, it means table name right?

another thing would like to know whyc below not combining the idnex rather it giving 2 seperate sets of index?

- type: log

  enabled: true

  paths: /var/log/apache
  ...
  fields:
    - name_of_index: group-1
    - name_of_log: apache


- type: log

  enabled: true

  paths: /var/log/message
  ...
  fields:
    - name_of_index: group-1
    - name_of_log: message
#4

No type means in the input type see here It is not the index name or type in this case it is log which means reading a log file

I would suggest perhaps referring to the documentation

#5

thanks

#6

Hi i check in logstash input

type => "%{[fields][name_of_log]}"

it is printing the string rather than the value in the output section. so in output it is showing as

 "type" : "%{[fields][name_of_log]}".

why there are such behaviour

#7

Apologies I was cut and pasting...

At the input you does not appear have access to the fields yet so you can only set static values. (My bad I will need to look closer at that)

So to accomplish what I think you want you would do this..

input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][name_of_log] == "apache" {
    ...
  } else if [fields][name_of_log]  == "nginx" {
   ...
  }  else {
    ...
  }

}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}" 
  }
}
#8

You could also do it like this which might be a little cleaner

input {
  beats {
    port => 5044
  }
}

filter {

   mutate { 
        add_field => { "type" => "%{[fields][name_of_log]]}" }
    }

  if [type] == "apache" {
    ...
  } else if [type]  == "nginx" {
   ...
  }  else {
    ...
  }

}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[fields][name_of_index]}-%{+YYYY-MM-dd}" 
  }
}
#9

I see you opened another thread ...

Hi @navin1093 I did not realize you were trying to set _type as the poster stated that is not longer supported if you just want to set a normal field named type you can still do that, but it is just like any other field.

In reality, with the removal of types ( "_type" : "_doc" always ) , You should think of an index as a table not a database...