Logstash Elasticsearch template for doc_values

Elastic Stack Logstash
1

How do I go about modifying the default template for Logstash for it to use doc_values for as many fields as possible?

With Elasticsearch 2.0 doc_values will be default for all fields instead of optional, so by then this problem should solve itself. Does this mean that I could instead of changing the Logstash template I could change the default in Elasticsearch?

I have found these discussions on the subject, but I'm not sure what is the right method.

2

I'd use the first one, if you have specific questions please ask :slight_smile:

3

Will that then apply to all .raw fields only?

With Elasticsearch 2.0 new default, will that then only apply to .raw fields as well but will integer and timestamp fields also use raw?

(gaddamn I will have to modify lots of dashboards that should use the raw field anyway :slight_smile: )

4

That's correct, you will need to add doc values to any other fields manually till 2.0 (which isn't far off anyway).

5

I'm still using Kibana3 and when using the table filter there it doesnt seem to be aware of the .raw field when ordering events in the table. Is there any way to use the .raw field there?

Is it right of me to assume that because of the table filter, just by modifying the @timestamp field for doc_values would speed things up and save memory?

6

It should, yeah.

Might be easier if you raise another thread for the other one to keep things clear :slight_smile:

7

I will take one step at a time and modify the raw fields first

Thanks for the clarification! :smile:

8

One more question directly related to this.

Should I modify the template already stored in Elasticsearch or should I modify the template specified in Logstash?

9

I'd copy the existing LS one and then store it somewhere custom, then manage it in the ES output in the LS conf.