About repeating parameters when using if/else conditions

[espala (Salih)]

Hello there,

I'm using the config I've added as "original" below on my logstash server. My configuration works fine.

my purpose;
I have a lot of servers and I collect the general system logs of all of them on elk. I installed filebeat on each client server, I set the log settings and add their hostnames as tags.

I create an index specifically for each server hostname. For each of my servers, I use if/else patterns over and over. Unfortunately I have to.

my problem;
In every if/else pattern, I am constantly writing other parameters along with the index value. In fact, I aim to do this with less effort and without repeating the same settings.

so I don't want to type parameters like "host", "user", "password" over and over. I want to put the "index" parameter, which can change constantly, into the if/else pattern. I put the if/else sections two columns forward, two columns back, tried all combinations.

---

For example, when I run the following configuration, I get an error (below). I may be blinded by spending so much time on the case. I may be missing something very simple. Does anyone have any advice on this?

test config;

...

output {
  elasticsearch {
    hosts => ["https://elk.domain.com:9200"]
    manage_template => false
    user => "loguser"
    password => "password"

  if "server1" in [tags] {
    index => "server1-logs-%{+yyyy.MM.dd}"
  }

  if "server2" in [tags] {
    index => "server2-logs-%{+yyyy.MM.dd}"
  }

...
...
...

  else {
    index => "noindex-%{+yyyy.MM.dd}"
  }

  }  
}

Error messages;
Nov 17 10:33:58 elk-server logstash[19301]: [2021-11-17T10:33:58,092][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 24, column 6 (byte 760) after output {\n elasticsearch {\n\n if ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in block in converge_state'"]}

original config;

input {
  beats {
    port => 5044
  }
}

filter {
  if [input][type] == "log" {
    mutate {
      "remove_field" => [ '[meta][cloud][availability_zone]','[meta][cloud][instance_id]','[meta][cloud][instance_name]','[meta][cloud][machine_type]','[meta][cloud][project_id]','[meta][cloud][provider]','source','[host][os][codename]','[host][os][family]','[host][os][name]','[host][os][platform]','[host][os][version]','[host][architecture]','[host][containerized]','[beat][hostname]','[beat][timezone]','[beat][version]','[beat][name]','[prospector][type]','[offset]','[host][id]','[error][message]','[event][dataset]','[fileset][name]', '[@version]' ]
      "remove_tag" => ["beats_input_codec_plain_applied"]
    }
  }
}


output {

  if "server1" in [tags] {
    elasticsearch {
      hosts => ["https://elk.domain.com:9200"]
      index => "server1-logs-%{+yyyy.MM.dd}"
      manage_template => false
      user => "loguser"
      password => "password"
    }
  }
  elseif "server2" in [tags] {
    elasticsearch {
      hosts => ["https://elk.domain.com:9200"]
      index => "server2-logs-%{+yyyy.MM.dd}"
      manage_template => false
      user => "loguser"
      password => "password"
    }
  }

...
...
...

  else {
    elasticsearch {
      hosts => ["https://elk.domain.com:9200"]
      index => "noindex-%{+yyyy.MM.dd}"
      manage_template => false
      user => "loguser"
      password => "password"
    }
  }

}

[AquaX (Andreas Helmer)]

Since filebeat includes the name of the server in the host.name field (iirc) you can put the server name field in the index output.

This will save you a lot of if statements

elasticsearch {
      hosts => ["https://elk.domain.com:9200"]
      index => "%{[host][name]}-logs-%{+yyyy.MM.dd}"
      manage_template => false
      user => "loguser"
      password => "password"
    }

EDIT: FIXED %{host.name} to %{[host][name]}

[espala (Salih)]

First of, thank you for your answer and interest.

Actually I'm tried this but failed. Unfortunately I must to create a noindex with an if/else in all cases. But now I tried without if/else, it seems to fail.

Actually, I don't delete this field with "remove_tag", it should find it. I could not find information about same standard environment variables in the articles. I also define node name in filebeat config.

I tried the config you sent. It creates an index directly with the name "%{host.name}-logs-..."

The config I tried to do with if/else before;

output {
# I tried "host.name" and "tags" fields. and I also used "==" statements, replacing "in" with strings. I also changed the strings.

  if %{host.name} in [host][name] {  
    elasticsearch {
      hosts => ["https://elk.domain.com:9200"]
      index => "%{host.name}-logs-%{+yyyy.MM.dd}"
      manage_template => false
      user => "loguser"
      password => "password"
    }
  }

  elseif {
    elasticsearch {
      hosts => ["https://elk.domain.com:9200"]
      index => "noindexes-%{+yyyy.MM.dd}"
      manage_template => false
      user => "loguser"
      password => "password"
    }
  }
}

[Badger]

@espala

index => "%{host.name}-logs-%{+yyyy.MM.dd}"

That refers to a field that has a stop in its name. If [host] is an object that contains a [name] field then this should be index => "%{[host][name]}-logs-%{+yyyy.MM.dd}"

[AquaX (Andreas Helmer)]

Yep! My bad. @espala Badgers correction should work.