More indexes in Filebeat

Elastic Stack Kibana
1

I want to have some search statistics. I have only one log file, where i can separate the lines with a attribute.
I want to use one filebeat, one elasticsearch and more kibana.
I try to have in filebeat more indexes:
index1-*
index2-*
etc.

Is this possible?

2

Yes it's possible.

Directly at the filbeat level, not to my knowledge. But you can separate the elements according to the tags in your output.

**EXAMPLE **

    output {
    	if "XXXX" not in [tags] {
    		elasticsearch {
    			index => "idx-%{+YYYY.MM.dd}"
    			host => ""
    			port => ""
    		}
    	} else {
    		elasticsearch {
    			index => "idx-2-%{+YYYY.MM.dd}"
    			host => ""
    			port => ""
    		}
    	}
    }
3

Unfortunately, i don't know where should i enter this configuration.

I have the following output to elasticsearch in the filebeat.yml.

output.elasticsearch:
hosts: "localhost:9200"
index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
pipeline: "shop-search-log"
indices:

  • index: "aposalis-%{[beat.version]}-%{+yyyy.MM.dd}"
    when.contains:
    include_lines: "aposalis"
  • index: "marienbrunn-%{[beat.version]}-%{+yyyy.MM.dd}"
    when.contains:
    include_lines: "marienbrunn"
4

You have to install Logstash and specify all the beats agents that you have to send the data collected to logstash.

Then, in logstash, it has a configuration file of which there are three parts:

  • Input
  • filter
  • output

In the input part, it will be a beat. must be specified in this conf file.

In filter, if you want to parse your collected logs.

And finally in the part, output, you have to define the output. it will be elasticsearch.

Let me know if you want an example of a typical configuration file.

5

Ok, thank you very much.

Without Logstash is this not possible?

6

yes, you can do it without logstash. but I did not know before.
I searched, I found against I have not tested but it must work

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "logs-%{[beat.version]}-%{+yyyy.MM.dd}"
  indices:
    - index: "critical-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "CRITICAL"
    - index: "error-%{[beat.version]}-%{+yyyy.MM.dd}"
      when.contains:
        message: "ERR"

Source : https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

let me know, if it works well

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.