Problem creating index

Elastic Stack Beats
1

Good morning,

I have a filebeat on one server, and I would need you to create a new index, different from what I have configured on other servers, so that when the index is created the Kibana can absorb it under another name.

What I have configured in the filebeat.yml is the following:

output.logstash:
# The Logstash hosts
hosts: ["logstash.com:5044"]
index: "filebeat-MAXIMO_UP-%{+yyyy.MM.dd}"

And the logstash in the following way in the ELK server, so that it recognizes me the common indexes, and the one of the specific server that I have commented above, that later would be more servers. My output-elasticsearch.conf is the following:

output {
  elasticsearch { hosts => ["logstash.com:9200"]
    hosts => "logstash.com:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-MAXIMO_UP-%{+yyyy.MM.dd}"
  }
}

But the logstash does not recognize it, when creating an index with Kibana it does not appear, where could it have the error?

Greetings and thanks

2

Hi @vdelburgo, welcome to the Elastic community forums!

It looks like you are specifying the index setting twice in your Logstash Elasticsearch output plugin.

    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-MAXIMO_UP-%{+yyyy.MM.dd}"

I don't think this is supported. I'm not sure but I assume the plugin is only honoring one of those settings.

Instead, I think what you want is something like this:

        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

When Beats send events to Logstash, each event contains a @metadata field. That field contains a beat field. By default the value of that field is the name of the Beat, e.g. filebeat. In your case, this field will contain the correct index name, depending on which Beat is sending the event.

Hope that makes sense.

Shaunak

3

Hello Shaunak,

OK, I can prove it. But the question is if, for example, I have three servers that I want to group in the 'index1' and another 3 servers that I want to group in the 'index2' how do I make those groupings? I don't know if I can explain...

Thanks and regards

4

When you say "servers", you mean Filebeat instances, right? In that case, what you are doing by setting the desired index prefix under output.logstash.index is correct.

So for the first three Filebeat instances you would set output.logstash.index: index1 and for the other 3 you would set output.logstash.index: index2.

Then in your Logstash pipeline's Elasticsearch output settings, you would set index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}". This will cause the data coming from your first three Filebeat instances to be indexed into index1-2020.04.23 and data coming from your other three Filebeat instances to be indexed into index2-2020.04.23 (assuming today's date just as an example).

5

Perfect Shaunak, it works for me...

Is there a way to create the indexes automatically in Kibana? I mean, so I don't have to click on 'create index' every day

Greetings and thank you very much for your help

6

Sorry, but I'm not quite sure what you mean by this. Kibana has the notion of index patterns to match multiple indices with wildcards. Perhaps that's what you're looking for?

7

Sorry, that was a silly question... :roll_eyes:

Thanks and regards for your help

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.