Elasticsearch 6.0 Removal of mapping types - Alternatives

Elastic Stack Elasticsearch
1

Background

I migrating my ES index into ES version 6. I currenly stuck because ES6 removed the using on "_type" field.

Old Implementation (ES2)

My software has many users (>100K). Each user has at least one document in ES. So, the hierarchy looks like this:

INDEX  ->  TYPE      -> Document
myindex->  user-123  -> document-1

The key point here is with this structure I can easily remove all the document of specific user.

DELETE /myindex/user-123

(Delete all the document of specific user, with a single command)

The problem

"_type" is no longer supported by ES6.

Possible solution

Instead of using _type, use the index name as USER-ID. So my index will looks like:

"user-123" -> "static-name" -> document

Delete user is done by delete index (instead of delete type in previous implementation).

Questions:

  • My first worry is about the amount of index and performance: Having like 1M index is something that acceptable in terms of performance? don't forget I have to search on them frequently.
  • Most of my users has small amount of documents stored in ES. Is that make sense to hold a shard, which should be expensive, for < 10 documents?
  • My data architecture sounds reasonable for you?

Any other tip will be welcome!
Thanks.

2

No, that sounds like a terrible idea. Using separate indices for small users in Elasticsearch tends to scale and perform badly as each shard comes with overhead.

Instead add a new keyword field to each document that indicates the user the document belongs to, e.g. user_id. You can then filter on this when querying and use delete-by-query API to delete all documents belonging to a specific user in a single command.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.