I am parsing (in logstash using dissect filter) and indexing about 17 fields from a single log message (filebeat -> logstash ->ElasticSearch) . Logstash template by default places the parsed string fields as both text and keyword. obviously some fields like loglevel etc wont need to be of type text. I understand that log template can be changed ofcouse, however what are the implications if I dont (let all fields be both text and keyword even if not required)
Could My document size get bigger, or any performace issue ? Please explain.
We have looked at this, though not deeply, since as far as we can tell, it would nearly double storage for the index, especially on logging with one giant string field. We are looking at using keyword only, but not checked its effects - it does change how/what you can search.
Text type is for full text search, which is often desirable for logs messages, as keywords are not broken down. On the flip side, text is kinda useless for other fields like log type, host name, daemon, etc.
@saket_gupta the document contains only what you have sent. The mappings define how that data gets parsed and converted into tokens which are used for queries/aggregations.
If you define multiple fields the original payload doesn't change but the index would get bigger because you would have the same data analyzed in multiple ways. By default a string field gets mapped as text (standard tokenizer) and keyword.
The queries filter the data using the tokens and the result is extracted from the document. Only when you do aggregations you get the key value from the tokens.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.