# Data type for Log Message Fields, does keyword add overhead? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 5, 2022, 2:13pm UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043 "2022-05-05T14:13:54Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![scott\_stash](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Post date:** [May 5, 2022, 2:13pm UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/1 "2022-05-05T14:13:54Z")
Currently I'm indexing log messages into Elasticsearch. Generally the initial 'message' field is dissected and stored in the following fields: @timestamp(date), logLevel(keyword), logMessage(keyword/text). ```auto "logMessage": { "fields": { "keyword": { "ignore_above": 10000, "type": "keyword" } }, "type": "text" }, ``` The logMessage field has millions of variations and is not used for aggregations. We perform a lot of match\_phrase, wildcard and query\_string queries verse this field. Does saving this as a keyword cause issues? Or initial overhead? Should it be set to match\_text or text only? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 5, 2022, 2:22pm UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/2 "2022-05-05T14:22:23Z")
Storing this as `keyword` if you already is storing as `text` will use basically doube the disk space usage for each document. Since you do not need to use it for aggregations, you can map it as a text or as [match\_text\_only](https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html#match-only-text-field-type), but since you said you use `match_phrase`, it is better to keep it as a `text`. From the `match_text_only` documentation you have this information: > however queries that need positions such as the [`match_phrase` query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html) perform slower as they need to look at the `_source` document to verify whether a phrase matches. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [May 5, 2022, 3:20pm UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/3 "2022-05-05T15:20:40Z")
Issues with storing log messages in text or keyword fields was the primary reason for creating the wildcard field - see [Find strings within strings faster with the Elasticsearch wildcard field | Elastic Blog](https://www.elastic.co/blog/find-strings-within-strings-faster-with-the-new-elasticsearch-wildcard-field) ---
**Author:** ![scott\_stash](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Post date:** [May 5, 2022, 5:11pm UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/4 "2022-05-05T17:11:38Z")
Thanks everyone. Now I’m trying to decide on using text of wildcard. 90% of my queries on this field are probably match phrase where we want to match multiple words, example “server failed to respond to baseball fastball”. The remaining 10 are split between query string and wildcard. I’m having trouble getting wildcard searches to return results on the text field, not sure if it should be supported.(same query would work vs logMessage.keyword) Also my use of query string might be wrong, but generally I want to use it to do AND match phrase queries but it always returns log messages I don’t want due to the way it tokenizes the message. “server failed AND baseball fastball” returns a lot of false positives. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [May 5, 2022, 6:16pm UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/5 "2022-05-05T18:16:33Z")
A word-based index like the text field relies on matching word sequences using phrase queries or the more sophisticated JSON queries like ‘span’ or ‘interval’. Untokenised fields like keyword or wildcard fields are a single string token and you would instead use ‘wildcard’ or ‘regexp’ queries to match character sequences in those strings. ---
**Author:** ![scott\_stash](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@scott\_stash](https://discuss.elastic.co/u/scott_stash)\ **Post date:** [May 7, 2022, 1:49pm UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/6 "2022-05-07T13:49:29Z")
Thanks Mark. ```auto PUT message_test_index { "mappings": { "properties": { "@timestamp" : { "type" : "date" }, "messageWildcard" : { "type" : "wildcard" }, "messageKeyword" : { "type" : "keyword" }, "messageText" : { "type" : "text" }, "messageMatchOnlyText" : { "type" : "match_only_text" } } } } POST message_test_index/_doc { "@timestamp": "2022-01-01T12:10:30Z", "messageWildcard": "This is example of search query. Can anyone help me?", "messageKeyword": "This is example of search query. Can anyone help me?", "messageText": "This is example of search query. Can anyone help me?", "messageMatchOnlyText": "This is example of search query. Can anyone help me?" } GET message_test_index/_search { "query": {"match_phrase": { "messageWildcard": "is example of" }} } GET message_test_index/_search { "query": {"wildcard": { "messageWildcard": "*example of*" }} } GET message_test_index/_search { "query": {"regexp": { "messageWildcard": {"value": ".*is example of.*"} }} } GET message_test_index/_search { "query": {"match_phrase": { "messageText": "is example of" }} } GET message_test_index/_search { "query": {"wildcard": { "messageText": "*example of*" }} } GET message_test_index/_search { "query": {"regexp": { "messageText": {"value": ".*is example of.*"} }} } ``` Currently I have a service that reads/parses user generated search terms. Currently I have a keyword and text field but think I want wildcard. Most of the patterns are currently defined as match\_phrase searches, example "is example of" I'm not sure if it is possible to make "is example of" work as a search term verse the wildcard data type without adding \*'s at the start and end of the query. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [May 8, 2022, 6:23am UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/7 "2022-05-08T06:23:46Z")
Word-sequence searches on text fields assume the words to be found can appear _anywhere_ in the text. Character-sequence searches on keyword or wildcard fields assume the characters are rooted at the beginning and cover to the end of the string unless you use \* characters to un-anchor them. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 5, 2022, 6:24am UTC](https://discuss.elastic.co/t/data-type-for-log-message-fields-does-keyword-add-overhead/304043/8 "2022-06-05T06:24:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.