Select fields for multiple outputs


Hi all!
I have a config:

input {
  rabbitmq {
    host => "x.x.x.x"
    port => 5672
    vhost => "/"
    user => "guest"
    password => "guest"
    queue => "yyyy"
    durable => true
    codec => "es_bulk"
filter {
  mutate {
    remove_field => [ "@version", "@timestamp" ]
output {
  elasticsearch {
    protocol => "http"
    host => "x.x.x.x"
    port => 9200
    cluster => "test_cluster"
    action => "%{[@metadata][action]}"
    document_id => "%{[@metadata][_id]}"
    document_type => "%{[@metadata][_type]}"
    index => "%{[@metadata][_index]}"
  stdout {
    codec => rubydebug { metadata => true }

The input data is something like:


On parse we get:

     "Language".[0;37m => .[0m.[0;33m"en".[0m,
         "Date".[0;37m => .[0m.[0;33m"2015-09-02T00:00:00Z".[0m,
         "Text".[0;37m => .[0m.[0;33m"text1".[0m,
    "@metadata".[0;37m => .[0m{
             "_index".[0;37m => .[0m.[0;33m"data".[0m,
              "_type".[0;37m => .[0m.[0;33m"en".[0m,
                "_id".[0;37m => .[0m.[1;34m1.[0m,
             "action".[0;37m => .[0m.[0;33m"index".[0m,
        "retry_count".[0;37m => .[0m.[1;34m0.[0m

And now I'm stuck. I need send to "stdout" only field "Text". Can you help me to change my config to select to "stdout" output only "Text" field?

(Magnus Bäck) #2

Don't use the rubydebug codec. Use the plain codec or just omit that option. I believe you'll still get the timestamp (not sure if you can do anything about that) but you can use a simple sed filter or something to remove that.


"stdout" output with "rubydebug" codec is just for example. I'll use "rabbitmq" as second output with "json" codec. But I need to filter fields before sending data to that output.

I'd found the same question, but answer seems wrong:
Please advice.

(Magnus Bäck) #4

Oh. There's no generic way of filtering what fields to send to a particular output.


Hm... So, the best way is try to modify output plugin to own need, right?

(Magnus Bäck) #6

Yes, or modify the base class for outputs to add a generic feature to filter the fields to include.


Understood, thanks.

(Magnus Bäck) #8

Relevant GitHub issue:

(system) #9