How to push same document into multiple indexes?

Elastic Stack Logstash
1

Hello,
I collecting logs via TCP input and I want specific log to be in logstash index and also in monitoring index. But logs with "time" in [tags] are only in monitoring index, not logstash. Why?

My Logstash output config look like:

output {
    if "time" in [tags] {
      elasticsearch {
    hosts => ["192.168.1.1:9200"]
    index => "monitoring-%{+YYYY.ww}"
      }
    }
    else if [appname] == "java" {
     elasticsearch {
    hosts => ["192.168.1.1:9200"]
    index => "java-%{+YYYY.MM.dd}"
      }
    }
    else if [appname] != "java" {
      elasticsearch {
    hosts => ["192.168.1.1:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
      }
     }
    }
2

But logs with "time" in [tags] are only in monitoring index, not logstash. Why?

Because that's how if ... else conditionals work. If the first condition is true the event will be sent to the monitoring index and the other conditions won't be considered at all.

3

Thank you for your answer. Is there any approach how I can achieve what I need?

4

Yes, but it's not clear exactly what you need. I think you're looking for this:

  • If the event has the tag "time" send it to the monitoring index.
  • If the event's appname field is "java" send to the java index, else send to the logstash index.
5
  • If the event has the tag "time" send it to the monitoring index AND to the logstash index.
  • If the event's appname field is "java" send to the java index
  • Everything else send to the logstash index.
6

That's basically what I said, and it's obvious that describing such logical conditions with prose easily turns ambiguous.

I'm not sure what to do here. I could suggest something that satisfies what I think you want, but if you're not at all familiar with conditionals I don't know if that helps.

7

Thank you for your time and for your advice. I was able to set what I need with this setting:

output {
    if "time" in [tags] {
      elasticsearch {
    hosts => ["192.168.1.1:9200"]
    index => "monitoring-%{+YYYY.ww}"
      }
      elasticsearch {
    hosts => ["192.168.1.1:9200"]
    index => "logstash-%{+YYYY.ww}"
      }
    }
    else if [appname] == "java" {
     elasticsearch {
    hosts => ["192.168.1.1:9200"]
    index => "java-%{+YYYY.MM.dd}"
      }
    }
    else if [appname] != "java" {
      elasticsearch {
    hosts => ["192.168.1.1:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
      }
     }
    }
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.