How to set the index name to the value of tags set by filebeat

Elastic Stack Logstash
1

I am using filebeat to collect some logs.
I have set the value of tags for each log, and I want to use that value as the name of the index.
Is that possible?

vi /etc/filebeat/filebeat.yml
... snip ...
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/A.log
  tags: ["A", "foo", "bar"]
- type: log
  enabled: true
  paths:
    - /path/to/B.log
  tags: ["B", "hoge"]
... snip ...

With the following method, it will be "[tags][0]-2021.07.28" instead of the value of tags.
(Actually, we want it to be "A-2021.07.28", "B-2021.07.28", etc.)

vi /etc/logstash/logstash-sample.config
... snip ...
output {
  elasticsearch {
    hosts => ["localhost"]
    index => "%{[tags][0]}-%{+YYYY.MM.dd}"
}
... snip ...

As a workaround for now, I'm using if-else statements, but I'm having trouble with the increasing number of log types.

vi /etc/logstash/logstash-sample.config
... snip ...
if ( "A" in [tags][0] ) [
  output {
    elasticsearch {
      hosts => ["localhost"]
      index => "A-%{+YYYY.MM.dd}"
  }
}
else if ( "B" in [tags][0] ) [
  output {
    elasticsearch {
      hosts => ["localhost"]
      index => "B-%{+YYYY.MM.dd}"
  }
}
... snip ...

Do you have any good ideas?

Also, I'd really like to concatenate the values I set for tags (different lengths for different logs) into the name of the index.
(For example, I want to use "A-foo-bar-2021.07.28" or "B-hoge-2021.07.28.")

Do you have any good ideas about this as well?

2

Don't use tags, have filebeat create a new field instead.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/A.log
  tags: ["foo", "bar"]
  fields:
       index_name: A

- type: log
  enabled: true
  paths:
    - /path/to/B.log
  tags: ["hoge"]
  fields:
       index_name: B

Then with the output you don't have to use all those if statements:

    output {
      elasticsearch {
        index => "%{[fields][index_name]}-%{+YYYY.MM.dd}"
      }
    }
1 Like
3

Very good idea!!!
That's exactly what I needed to know!!!
Thanks for letting me know!

4

Basically, the log is being output with the name I specified, but
For some reason, it also creates an index with the following name.

%{[fields][index_name]}-2021.07.29

The size of the document is not large, but I can't look inside it with elasticsearch-head.
(The search is in progress and no results are returned)

Can you tell me why?

5

I can't seem to get the ElasticSearchAPI to display properly.
Is there any way to fix this?

curl -XGET 'localhost:9200/%{[fields][index_name]}-2021.07.29/_settings?pretty'
curl: (3) [globbing] nested braces not supported at pos 18
6

If that happens then the field [fields][index_name] does not exist. Double check your input by outputting to stdout and make sure that you define a field name. If you want you can also include an if statement before the output to set a default index name
Check out this example:

    output {
    if [fields][index_name] {
       elasticsearch {
         index => "%{[fields][index_name]}-%{+YYYY.MM.dd}"
       }
     } else {
       elasticsearch {
         index => "defaultindex-%{+YYYY.MM.dd}"
       }
     }
       
    }
7

Thank you for your answer.

That's a very good idea to set the default index name in the if-else!
I had already done that support as well.

However, I had written the following to determine if the field name exists

if([fields][index_name] ! = "") {
... snip ...

Following your idea, I would write something like this

if([fields][index_name]) {
... snip ...

Since the index is not created immediately, we don't know the result yet.
I will report the results when they are available.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.