Select fields for multiple outputs

Elastic Stack Logstash
1

Hi all!
I have a config:

input {
  rabbitmq {
    host => "x.x.x.x"
    port => 5672
    vhost => "/"
    user => "guest"
    password => "guest"
    queue => "yyyy"
    durable => true
    codec => "es_bulk"
  }
}
filter {
  mutate {
    remove_field => [ "@version", "@timestamp" ]
  }
}
output {
  elasticsearch {
    protocol => "http"
    host => "x.x.x.x"
    port => 9200
    cluster => "test_cluster"
    action => "%{[@metadata][action]}"
    document_id => "%{[@metadata][_id]}"
    document_type => "%{[@metadata][_type]}"
    index => "%{[@metadata][_index]}"
  }
  stdout {
    codec => rubydebug { metadata => true }
  }
}

The input data is something like:

{"index":{"_index":"data","_type":"en","_id":1}}
{"Text":"text1","Language":"en","Date":"2015-09-02T00:00:00Z"}

On parse we get:

{
     "Language".[0;37m => .[0m.[0;33m"en".[0m,
         "Date".[0;37m => .[0m.[0;33m"2015-09-02T00:00:00Z".[0m,
         "Text".[0;37m => .[0m.[0;33m"text1".[0m,
    "@metadata".[0;37m => .[0m{
             "_index".[0;37m => .[0m.[0;33m"data".[0m,
              "_type".[0;37m => .[0m.[0;33m"en".[0m,
                "_id".[0;37m => .[0m.[1;34m1.[0m,
             "action".[0;37m => .[0m.[0;33m"index".[0m,
        "retry_count".[0;37m => .[0m.[1;34m0.[0m
    }
}

And now I'm stuck. I need send to "stdout" only field "Text". Can you help me to change my config to select to "stdout" output only "Text" field?

2

Don't use the rubydebug codec. Use the plain codec or just omit that option. I believe you'll still get the timestamp (not sure if you can do anything about that) but you can use a simple sed filter or something to remove that.

3

"stdout" output with "rubydebug" codec is just for example. I'll use "rabbitmq" as second output with "json" codec. But I need to filter fields before sending data to that output.

I'd found the same question, but answer seems wrong:
[http://stackoverflow.com/questions/25156476/logstash-how-to-output-a-selection-of-fields][1]
[1]: http://stackoverflow.com/questions/25156476/logstash-how-to-output-a-selection-of-fields
Please advice.

4

Oh. There's no generic way of filtering what fields to send to a particular output.

5

Hm... So, the best way is try to modify output plugin to own need, right?

6

Yes, or modify the base class for outputs to add a generic feature to filter the fields to include.

7

Understood, thanks.

8

Relevant GitHub issue: