Add tag to filebeat modules

[rodri.gz (RGZ-ELK)]

hello!

I am trying to send the logs with the filebeat mysql module to my logstash and it already obtains the pipeline and does the parsing correctly.

How can I differentiate both inputs ( Slowlog and errror ) to apply different filters to them?

for example:

  if [log][file][path] == "/var/lib/mysql/g99dnap797-slow.log" {
    elasticsearch {
      hosts => ["https://x:9200","https://y:9200","https://z:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-tecnico-mysql-%{[agent][hostname]}-%{+YYYY.MM.dd}"
      ssl => true
      manage_template => false
      user => "elastic"
      password => "xxxxxxx"
      pipeline => "filebeat-7.13.2-mysql-slowlog-pipeline"
    }
  }

the module:

- module: mysql
  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  # Slow logs
  slowlog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

[legoguy1000 (Alex)]

There are other fields in @metadata that u may be able to use our should be able to use event.dataset which in this case would be mysql.error and mysql.slowlog

[rodri.gz (RGZ-ELK)]

first of all , thanks for answering!

I'm trying in this way but it doesnt work.

After this I have an else condition and all the data is skipping the dataset and passing through the else, why?

  if [event][dataset] == "apache.error" {
    elasticsearch {
      hosts => ["https://xxx:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-tecnico-apache-%{[agent][hostname]}-%{+YYYY.MM.dd}"
      ssl => true
      manage_template => false
      user => "elastic"
      password => "xxxxxxxt"
      pipeline => "filebeat-7.13.2-apache-error-pipeline"
    }
  }


  if [event][dataset] == "mysql.error"  {
    elasticsearch {
      hosts => ["https://xxxxx:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-tecnico-mysql-%{[agent][hostname]}-%{+YYYY.MM.dd}"
      ssl => true
      manage_template => false
      user => "elastic"
      password => "xxxxxxxxxx"
      pipeline => "filebeat-7.13.2-mysql-error-pipeline"
    }
  }

thanks you!

[stephenb (Stephen Brown)]

So I think you can do this simpler something like

In the filter block do the if / else on the event.data type and set a field or tag with part of the index name like index-subtype

then your output could look something like

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{index-subtype}"
      pipeline => "%{[@metadata][pipeline]}" 
      user => "elastic"
      password => "secret"
    }
  } else {
    elasticsearch {
      hosts => "http://localhost:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{index-subtype}"
      user => "elastic"
      password => "secret"
    }
  }
}

Unclear why you want to separate by hostname... that often leads to many many indices with small shard which leads to over sharding... The host.name and agent.hostname is in every event and can be easily filtered ... just a thought

[rodri.gz (RGZ-ELK)]

ok, I'll try it that way.

Regarding the name of the indexes, we put the hostname because it is a cluster with quite specific data. Anyway, could you perform a reindex with a date variable so that it just removes the hostname?

for example:

POST _reindex
{
  "source": {
    "index": "filebeat-7.13.2-apache*"
  },
  "dest": {
    "index": "filebeat-7.13.2-apache-%{[@timestamp]}"
  }
}

[stephenb (Stephen Brown)]

That is not going to work that destination index will be the literal of what you defined... and of course timestamp go down to the ms so you would probably not want to do that...

You would probably want something like ....

POST _reindex
{
  "source": {
    "index": "filebeat-7.13.2-apache-*-2021.11.*"
  },
  "dest": {
    "index": "filebeat-7.13.2-apache-2021.11"
  }
}