Exclude Graphite fields from Elasticsearch

[Exocomp]

In the filter stage of logstash I'm creating some fields based on metricbeat data that are meant to be for Graphite specifically.

Then in the output stage of logstash I'm sending the data to Graphite and Elasticsearch, for ex:

		graphite {
			host => [ "192.168.1.128" ]
			include_metrics => [ "hosts.*" ]
			fields_are_metrics => true
		}
		elasticsearch {
		    hosts => [ "192.168.1.121:9200" ]
			index => "hosts-%{+YYYY.MM.dd}"
			document_id => "%{fingerprint}"
			action => "create"
		}

Unfortunately this results with the fields I wanted in Graphite only to be in Elasticsearch also.

What are some options where I can avoid this and get the Graphite fields to be excluded from Elasticsearch ?

[magnusbaeck (Magnus Bäck)]

You'll have to clone the original event with the clone filter and add the extra fields only to the clone that you send to graphite. Use conditionals in the output block to send events with extra fields to graphite and all other events to elasticsearch.

[Exocomp]

@magnusbaeck

For the graphite fields I use the ruby filter to generate them, if I use the clone filter would I still be able to use the ruby filter as well?

I had a look at the clone documentation but it is not clear so I will have to try it out but just checking if you might know. So what I'm thinking is that I use the clone filter and add a custom tag and fields I need. And then later in the filter pipeline I use a conditional check against that custom tag and then use a ruby filter to do the work of adding the Graphite fields. Would this be possible ?

[magnusbaeck (Magnus Bäck)]

For the graphite fields I use the ruby filter to generate them, if I use the clone filter would I still be able to use the ruby filter as well?

Yes, of course.

So what I'm thinking is that I use the clone filter and add a custom tag and fields I need. And then later in the filter pipeline I use a conditional check against that custom tag and then use a ruby filter to do the work of adding the Graphite fields. Would this be possible ?

That's exactly the idea.

[Exocomp]

@magnusbaeck

The clone filter worked exactly like I wanted it to. Thanks for the suggestion!