Multi Fields Diadvantage

Hi Team,

I am parsing (in logstash using dissect filter) and indexing about 17 fields from a single log message (filebeat -> logstash ->ElasticSearch) . Logstash template by default places the parsed string fields as both text and keyword. obviously some fields like loglevel etc wont need to be of type text. I understand that log template can be changed ofcouse, however what are the implications if I dont (let all fields be both text and keyword even if not required)

Could My document size get bigger, or any performace issue ? Please explain.

Your index should be bigger and you should experience increased latency during writing and quering. However the document size is not afected.

Thanks for your reply. I could not get "Your index should be bigger However the document size is not afected." Request you to please elaborate a bit!!

Thanks

We have looked at this, though not deeply, since as far as we can tell, it would nearly double storage for the index, especially on logging with one giant string field. We are looking at using keyword only, but not checked its effects - it does change how/what you can search.

Text type is for full text search, which is often desirable for logs messages, as keywords are not broken down. On the flip side, text is kinda useless for other fields like log type, host name, daemon, etc.

1 Like

@saket_gupta the document contains only what you have sent. The mappings define how that data gets parsed and converted into tokens which are used for queries/aggregations.
If you define multiple fields the original payload doesn't change but the index would get bigger because you would have the same data analyzed in multiple ways. By default a string field gets mapped as text (standard tokenizer) and keyword.
The queries filter the data using the tokens and the result is extracted from the document. Only when you do aggregations you get the key value from the tokens.

1 Like

Thanks for your Help!!

@CSharpBender Thanks for your help. Yes I figured that out as well , after I Posted the Question. But thanks for your Reply

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.