How to pass temporary data from filter to output in logstash

Elastic Stack Logstash
1

How can I pass temporary data from a filter to output in logstash?

I did like this:

filter {
  mutate { add_field => { "[@metadata][TEMP_DATA]" => "%{custom_index}" } }
  mutate { remove_field => ["custom_index"] }
}

output {
  elasticsearch {
    index => "%{[@metadata][TEMP_DATA]}"
  }
}

But it did not work.
Does anyone have an idea how I could do that?

2

In what way did it not work?

3

It is weird because I can see at the Monitoring page in Kibana that the Logstash Node is receiving data. But when I go into the Discover page in Kibana I do not see any data.

I have created an index in Elasticsearch and when I check the index in Kibana I don't see any data.

I am using Logstash 5.5.2.

4

The output plugin looks as follows:

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    codec => json
    manage_template => false
    hosts => [ "localhost:9200" ]
    user => "elastic"
    password => "changeme"
    index => "%{[@metadata][TEMP_DATA]}"
    document_type => "ObjectEvent"
    document_id => "%{epc}"
  }
}
5

Use Elasticsearch's cat indices API to check what indices you have. Do you by any chance have an index named, literally, %{[@metadata][TEMP_DATA]}? Or %{custom_index}?

6

I already have done that and nothing shows up. I have used the stdout output plugin to display what I actually want to send to elasticsearch and it displays everything perfectly.

7

If I have something like this:

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    codec => json
    manage_template => false
    hosts => [ "localhost:9200" ]
    user => "elastic"
    password => "changeme"
    index => "%{custom_index}"
    document_type => "ObjectEvent"
    document_id => "%{epc}"
  }
}

And I don't remove the custom_index field from the filter, it works.
But I don't want to send the custom_index field to elasticsearch.
That's why I am using @metadata.

8

Hi @magnusbaeck,

I just tried something else. Instead of TEMP_DATA, I put another used %{[@metadata][endpoint]}.

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    codec => json
    manage_template => false
    hosts => [ "localhost:9200" ]
    user => "elastic"
    password => "changeme"
    index => "%{[@metadata][endpoint]}"
    document_type => "ObjectEvent"
    document_id => "%{epc}"
  }
}

and an index %{[@metadata][endpoint]} has been created. But with TEMP_DATA nothing changes.
Do you know an way to give a value to an index and not send this value as a field to elasticsearch?

Is this actually possible?

9

Comment out your elasticsearch output and change your stdout output to stdout { codec => rubydebug { metadata => true } }. What does an example event look like?

10

I also have done that. But this is not appearing.

grafik
grafik.png1464×346 59.3 KB

I have actually two @metadata fields. One is [@metadata][EPC_t] and the another one is [@metadata][endpoint]. Just [@metadata][EPC_t] is been displayed as you can see.

11

It works! BUT... I don't know why.

I changed the position where I write the following code:

filter {
  mutate { add_field => { "[@metadata][endpoint]" => "%{endpoint}" } }
  mutate { remove_field => ["endpoint"] }
}

and placed this after the following code:

  if [EPC_t] {
    mutate { add_field => { "[@metadata][EPC_t]" => "%{EPC_t}" } }
    mutate { remove_field => ["EPC_t"] }
  }

If I try to put the addition of [@metadata][endpoint] field somewhere else before the EPC_t metadata field, it does not work.

Maybe I am missing some logstash basics? Or metadata basics? I don't get it. Both fields have nothing to do with each other....

it works...