Remove a Text field while being indexing over ES

Elastic Stack Elasticsearch
1

Hi Team

We are trying to ingest an index with 10 fields, while we see that the message field has very large data, so we want to either skip this field entirely from writing to ES, is this possible through any mapping setting.
We are using ES 6.8 Basic license, and the index is written by application directly, so we need a solution to do this change from ES ends only.
Please provide your suggestions.

Thanks
Flyboy

2

Hi Team,

Can any please post some suggestions

Thanks
flyboyacsrao

3

Please do not bump your topic after one day of inactivity. Keep in mind that this a completely voluntary forum. If you need answers within an SLA, think about getting commercial support.

You could use an ingest pipeline, which can be configured as an index setting by setting index.default.pipeline, to remove that field. However than this field would also vanish from the JSON.

You could alternatively not index that field in the mapping which would probably already save some space on disk.

4

Hi Alex

Thanks for the suggestion.,
Since the issue was in a PROD setup, i was in a hurry to find alternatives.
Currently we just changed the mapping for the index by adding:

"_source": {
"excludes": [
"description"
]},
after type and then at the column within properties, we set the column as

"description" : {
"enabled" : "false"

This setting saved us almost 80% space, earlier the index was 21 GB and after mapping change, it reduced to 3 GB.

We actually dont want to use this column for any thing, more over this was storing encrypted data.

Thanks
flyboyacsrao

5

sounds good! great improvement that should give you some headroom. I suppose the main question is, if you really need that column in the source or not, as it could be exposed to the outside.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.