Single input and multiple output

[kalla]

Hi,

It is possible to create multiple index with different column values (In elastic search) with 1 input (beats) ?

My logstash.conf is:

input {
  beats {
    port => 5044
  }
}

filter {
  csv {
    separator => ","
    columns => ["A", "B", "C", "D"]
  }

  mutate {
    remove_field => ["message", "prospector", "tags", "beat", "offset", "@version", "host"]
  }
}

output {
    elasticsearch {
      hosts => "http://localhost:9200"
      index => "index-%{[A]}"
      manage_template => false
    }

    elasticsearch {
      hosts => "http://localhost:9200"
      index => "index-%{[B]}"
      manage_template => false
    }
}

ex.
index-%{[A]} will have all the csv columns
and index-%{[B]} will have only column B

How to configure logstash output part?

[Shaoranlaos (Christian Stockhaus)]

i don't now if it is possible on logstash side but you could use the mapping configuration on elasticsearch side to disable the fields(columns) that should not be in an index

https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html

[magnusbaeck (Magnus Bäck)]

You can use a clone filter to splice each event in two. You can then e.g. use a prune filter to delete all fields in the cloned event except the one field you want to keep (and make sure you keep the @timestamp field; see https://github.com/logstash-plugins/logstash-filter-prune/issues/22). The cloned event will be identical to the original event except for a tag that you can use to distinguish it and apply extra filters and pick the other output.