Can I filter the Output data separately?!

Elastic Stack Logstash
1

We have the following env: LS 2.0.x; ES 2.0.x and K 4.2.x

The input comes from the trace logs that an application produces, we filter the data and then wish to populate two separate (but related) indices: orders and orders_alt. Each one of these has a separate mapping structure of data.

We are having trouble controlling what fields go with what Index. We end up with the same fields in both Indices because ES creates them dynamically and we have to way to filter between the Outputs.

How can we better control what goes into which index? Can we somehow remove fields in the Output section?! The MUTATE is part of filtering not output...

2

Currently Logstash has a single event pipeline so for a given message the same set of fields will be sent to all outputs. It sounds like you'd have to run multiple Logstash instances.

3

You should be able to use the clone filter to generate one event per output and format these differently.

4

Reading the documentation on that filter, sounds like it would create a COPY of the event to which we could then add fields as needed - that part sounds good, but how do I tell Logstash to send the original event to 'orders' index while sending a copy of the event + custom fields to the 'orders_alt' index?

5

You should be able to use the clone filter3 to generate one event per output and format these differently.

Ah, forgot about clone. Yes, that's the way to go.

[...] but how do I tell Logstash to send the original event to 'orders' index while sending a copy of the event + custom fields to the 'orders_alt' index?

The filter's clones option sets the type field of the clone(s), so just add conditionals to selectively apply filters and send them to different outputs.

filter {
  clone {
    clones => ["cloned"]
  }
  if [type] == "cloned" {
    # this filter
  } else {
    # that filter
  }
}
6

I'm sorry, I still have trouble understanding how the CLONE filter affects the processing pipeline. Suppose we read in a line from the log, and we do a match and then find the stuff we want, then we call the CLONE filter on that event.... does that mean that the pipeline will then have 2 events - one original and then the Clone event (with a different type)!?

Is my thinking correct on that, plz?

Thx -

7

Yes, that's right. Why don't you just try it out?

$ cat test.config
input { stdin { type => "original" } }
output { stdout { codec => "rubydebug" } }
filter {
  clone {
    clones => ["clone"]
  }
}
$ echo hello | /opt/logstash/bin/logstash -f test.config
Settings: Default filter workers: 1
Logstash startup completed
{
       "message" => "hello",
      "@version" => "1",
    "@timestamp" => "2015-12-01T20:27:56.049Z",
          "type" => "original",
          "host" => "hallonet"
}
{
       "message" => "hello",
      "@version" => "1",
    "@timestamp" => "2015-12-01T20:27:56.049Z",
          "type" => "clone",
          "host" => "hallonet"
}
Logstash shutdown completed
8

I tried it with my config locally and it sure works! Thank you so much -

10

I have written a blog post that gives detailed steps on how to use Logstash to filter data in different ways, and then to drive that data to different outputs depending on which filters have been applied. See https://alexmarquardt.com/2018/08/31/using-logstash-to-drive-filtered-data-from-a-single-source-into-multiple-output-destinations/