Multiple index in elasticsearch from filebeat

Elastic Stack Beats
1

Hi , I want to create a two outputs filters in logstash , that depends on the "type" field sent from filebeat:

     filebeat:
       prospectors:
        -
          paths:
                  - /var/log/app.log
               
          input_type: log
          document_type: app
        
        paths:
                  - /var/log/stuff.log
               
          input_type: log
          document_type: stuff

I have in logstash two filterss:

filterapp.conf

  filter {
    if[type] == "app" {
    
    .......         
}

filterfoo.conf

  filter {
    if[type] == "stuff " {
    
    .......         
}

output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I would like to create 2 outputs with 2 indexes

app-*
stuff-*

How Can I tell to output {} to send the logs to app index or stuff index?

2

I dont know if this is the correct way , but it seems to be working:

output {

if [type] == "git" {

elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "stash-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

else
{

elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

3

That is the best way :slight_smile:

4

yeah, seems to work.

You can use the mutate filter to overwrite @metadata.type with another value in the filter section. Like:

filter {
  if [type] == "git" {
    mutate ...
  }
}

elasticsearch {
  hosts => ["localhost:9200"]
  sniffing => true
  manage_template => false
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
  }
}

Using @metadata in logstash is for 'private' use in your script. You can set/remove any fields at will. Logstash will remove @metadata when serializing the event to JSON.

Instead of overwriting '@metadata.beat', you can also create @metadata.index in the filter and use index => "{[@metadata][index]}".

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.