How to use filebeat fields name value in logstash config

Elastic Stack Beats
1

Hi All

Below are my config files for 2 filebeats & logstash.. I set the fields index=my_data_1 in filebeat config. & send to logstash.. in the logstash i want use the value passed in fields.index from filebeat to use it for indexing when sent to elastic search.. so that both filebeat agents using the same logstash can send data to different index names.

But, [fields][index] is not working.. im unable to get the value set in filebeat..

With the below configuration the index name in created in ES cluster is [fields][index]-2017.03.23

"[@metadata][index]" => "[fields][index]" ==> not working

Logstash configs:

```
input {
  beats {
     port => "9997"
  }
}

filter {
  mutate {
    replace => {
      "[@metadata][index]" => "[fields][index]"   ===> This is not working..
    }
  }
}


output {
  elasticsearch {
     hosts => ["10.205.233.191:8089","10.205.236.248:8089","10.205.235.211:8089"]
     index => "%{[@metadata][index]}-%{+YYYY.MM.dd}"
     #document_type => "%{[@metadata][type]}"
  }
}
```

File beat -1 config:

filebeat.prospectors:
- input_type: log
  paths:
    -  <path to file>

  fields:
     index:  my_data

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:9997"]

File beat -2 config:

```
filebeat.prospectors:
- input_type: log
  paths:
    -  <path to file>

  fields:
     index:  my_data_2

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:9997"]
```
1 Like
2
  • Could you use file output to verify that the field is set on the filebeat side? (should be the case)
  • Could you to use fields index directly for the fields?
index => "%{[fields][index]}-%{+YYYY.MM.dd}"
1 Like
3

Thanks for the reply.. But I need to put in filter as highlighted above.. When I put in the filter section, it didn't work.

But if work for if condition in the filter section..it works fine.. but not when I used it assign
e.g:
if [fields][index] == "abc" { #### ==> Works
"[@metadata][index]" => "[fields][index]" ===> This is not working..
}

4

check the syntax. I think the right hand side must be a format-string, that is "%{[field][index]}".

1 Like
6

Worked. Thanks.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.