Mapping optimizations for Aggregations


(Shahar Mor) #1

Hi,

We are using ES + Kibana for aggregations only.

Is there anything we can do to optimize the mapping and remove stuff we don't need?

Our main concern is memory usage. Maybe specific filteres/terms/norms or things like that that are not really documented anywhere...

Our ES version is 1.5.2


#2

Hi! Just disable indexing fields you never needed index: no and enable doc_values: true for fields you are using for aggregations. Of course you can also set not_analized for strings and other fields that should not be analyzed at all. Hope that help.

PS. try to upgrade to latest 1.7.x because there were some optimizations and bugfixes.


(Mark Walkom) #3

Upgrade to 2.X instead, you'll get better performance :slightly_smiling:


(Shahar Mor) #4

I would really like to do that, but i can't take my whole cluster down.

I need a way to reindex all the data to a new cluster. @warkolm know any?


(Mark Walkom) #5

Do you need to reindex though? Have you run the migration plugin to check?


(Shahar Mor) #6

I ran the migration plugin and there were no errors.

However, when i tried to reindex some data to a new cluster i faced some issues with a geoip field.
I don't see any other way without reindexing as i can't take the cluster down.


(Mark Walkom) #7

But why are you reindexing?
If the migration plugin found nothing then you can do an in-situ upgrade! (Of course I'd still take a backup)


(Shahar Mor) #8

Following the table here: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html i must do a complete cluster restart no?


(Mark Walkom) #9

Ah right, sorry I missed that bit. You do yes.

I use Logstash for reindexing - https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06


(system) #10