Issue with index

[bob-bza]

filebeat 1:

• input_type: log
  paths:
  • /var/log/ambari.log
    document_type: ambari
• input_type: log
  paths:
  • /var/log/zookeeper.log
    document_type: zookeeper

tags: ["masternode"]

filebeat 2:

• input_type: log
  paths:
  • /var/log/hive.log
    document_type: hive
• input_type: log
  paths:
  • /var/log/zookeeper.log
    document_type: zookeeper

tags: ["datanode"]

Logstash
input {
beats {
port => 5044
}
}

output{
if "masternode" in [tags] {
if [type] == "ambari" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "ambarinodelog"
}
}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}
if "masternode" in [tags] {
if [type] == "hive" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "hivelog"
}
}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}
}

what i was trying to was
i have multiple filebeats with different tags.
want to send all zookeeper logs to single index irrespective of tags does it works?
If yes please guide me proper required changes in my configuration

[magnusbaeck (Magnus Bäck)]

Just don't put the elasticsearch output that writes to the zookeeperlog index inside the conditionals.

[bob-bza]

@magnusbaeck

Just don't put the elasticsearch output that writes to the zookeeperlog index inside the conditionals.

output{
if "masternode" in [tags] {
if [type] == "ambari" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "ambarinodelog"
}
}
}
if "masternode" in [tags] {
if [type] == "hive" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "hivelog"
}
}
}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}

are you suggesting for this approch?

[jkuang (Jimmy Kuang)]

FYI. The if condition for your hive and zookeeper is incorrect.

The second if statement should be

if "datanode" in [tags]

and not

if "masternode" in [tags]

since your filebeat2 is tags: ["datanode"]

[bob-bza]

@jkuang

FYI. The if condition for your hive and zookeeper is incorrect.

The second if statement should be

if "datanode" in [tags]

and not

if "masternode" in [tags]

since your filebeat2 is tags: ["datanode"]

can you please show in detail, i dont have much idea what you are telling about

[jkuang (Jimmy Kuang)]

Look for datanode below:

output{
if "masternode" in [tags] {
if [type] == "ambari" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "ambarinodelog"
}
}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}
if "datanode" in [tags] {
if [type] == "hive" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "hivelog"
}
}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}
}

[bob-bza]

yes got you. thats an example i posted . sorry my mistake . i forget to add datanode in tags.

can i know exact syntax for same index(zookeeper) for multiple tags(datanode, slavenodes,masters,edge etc etc)

i want all zookeeper logs logs to same index

@jkuang

output{
if "masternode" in [tags] {
if [type] == "ambari" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "ambarinodelog"
}
}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}
if "datanode" in [tags] {
if [type] == "hive" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "hivelog"
}
}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}
}

[magnusbaeck (Magnus Bäck)]

If you want all Zookeeper logs sent to the same place regardless of tag, don't put your zookeeperlogs outputs inside any if "whatever" in [tags] conditionals.

[jkuang (Jimmy Kuang)]

Please ignore my previous post. Use the syntax below:

output
{
if "masternode" in [tags] {
if [type] == "ambari" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "ambarinodelog"
}
}
}
if "datanode" in [tags] {
if [type] == "hive" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "hivelog"
}
}

}
if [type] == "zookeeper" {
elasticsearch {
hosts => ["endpoint"]
action => "index"
index => "zookeeperlog"
}
}
}