Logstash Elasticsearch template for doc_values


(Elvar) #1

How do I go about modifying the default template for Logstash for it to use doc_values for as many fields as possible?

With Elasticsearch 2.0 doc_values will be default for all fields instead of optional, so by then this problem should solve itself. Does this mean that I could instead of changing the Logstash template I could change the default in Elasticsearch?

I have found these discussions on the subject, but I'm not sure what is the right method.



(Mark Walkom) #2

I'd use the first one, if you have specific questions please ask :slight_smile:


(Elvar) #3

Will that then apply to all .raw fields only?

With Elasticsearch 2.0 new default, will that then only apply to .raw fields as well but will integer and timestamp fields also use raw?

(gaddamn I will have to modify lots of dashboards that should use the raw field anyway :slight_smile: )


(Mark Walkom) #4

That's correct, you will need to add doc values to any other fields manually till 2.0 (which isn't far off anyway).


(Elvar) #5

I'm still using Kibana3 and when using the table filter there it doesnt seem to be aware of the .raw field when ordering events in the table. Is there any way to use the .raw field there?

Is it right of me to assume that because of the table filter, just by modifying the @timestamp field for doc_values would speed things up and save memory?


(Mark Walkom) #6

It should, yeah.

Might be easier if you raise another thread for the other one to keep things clear :slight_smile:


(Elvar) #7

I will take one step at a time and modify the raw fields first

Thanks for the clarification! :smile:


(Elvar) #8

One more question directly related to this.

Should I modify the template already stored in Elasticsearch or should I modify the template specified in Logstash?


(Mark Walkom) #9

I'd copy the existing LS one and then store it somewhere custom, then manage it in the ES output in the LS conf.


(system) #10