# Extract logs from the message field and create new separate runtime fields **URL:** \ **Category:** Kibana\ **Tags:** runtime-fields\ **Created:** [December 7, 2022, 10:24am UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670 "2022-12-07T10:24:20Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 7, 2022, 10:24am UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/1 "2022-12-07T10:24:20Z")
Hi everyone, I'm using ELK stack version 8.4.0 and i need to extract logs data from message field that is something like "ERRORS" "EXCEPTIONS" etc and i want a new field for every extracted value so that i can easily create a dashboard. Please check below logs, your help would be highly appreciated. Dec 7, 2022 @ 15:21:15.978 - @timestamp, column 3, row 1 2022-12-07 15:21:13,666 [INFO] [Thread-38] CommunicationController - Communication monitor is going to sleep for [30] seconds. - message, column 4, row 1 - openDetails, column 1, row 2 - select, column 2, row 2 Dec 7, 2022 @ 15:21:15.977 - @timestamp, column 3, row 2 2022-12-07 15:21:13,666 [WARN] [Thread-38] ChannelConnection - ECHO is required but it is not enabled for - message, column 4, row 2 - openDetails, column 1, row 3 - select, column 2, row 3 Dec 7, 2022 @ 15:21:15.976 - @timestamp, column 3, row 3 2022-12-07 15:21:13,666 [INFO] [Thread-38] ChannelConnection - Socket is idle for the past [180] seconds on - message, column 4, row 3 - openDetails, column 1, row 4 - select, column 2, row 4 Dec 7, 2022 @ 15:21:15.975 - @timestamp, column 3, row 4 2022-12-07 15:21:13,666 [WARN] [Thread-38] ChannelConnection - SIGN-ON is not enabled for MYHSSM ```auto type or paste code here ``` - message, column 4, row 4 - openDetails, column 1, row 5 - select, column 2, row 5 Dec 7, 2022 @ 15:21:15.974 - @timestamp, column 3, row 5 2022-12-07 15:21:13,666 [DEBUG] [Thread-38] ChannelConnection - Checking if SIGN-ON is required for MYHSM - message, column 4, row 5 - openDetails, column 1, row 6 - select, column 2, row 6 Dec 7, 2022 @ 15:21:15.973 - @timestamp, column 3, row 6 2022-12-07 15:21:13,666 [INFO] [Thread-38] ChannelConnection - Monitoring connection state of MYHSM - message, column 4, row 6 - openDetails, column 1, row 7 - select, column 2, row 7 Dec 7, 2022 @ 15:21:15.972 - @timestamp, column 3, row 7 2022-12-07 15:21:13,666 [INFO] [Thread-38] CommunicationController - [1] connection(s) are active on channel [MYHSM\_GATEWAY\_CLIENT\_C1]: [Client Channel [MYHSM\_GATEWAY\_CLIENT\_C1] on Socket - message, column 4, row 7 - openDetails, column 1, row 8 - select, column 2, row 8 Dec 7, 2022 @ 15:21:15.971 - @timestamp, column 3, row 8 - message, column 4, row 8 - openDetails, column 1, row 9 - select, column 2, row 9 Dec 7, 2022 @ 15:21:15.970 - @timestamp, column 3, row 9 2022-12-07 15:21:13,665 [INFO] [Thread-38] CommunicationController - CommunicationMonitor is resuming monitoring for host [MYHSM\_GATEWAY\_CLIENT] - message, column 4, row 9 - openDetails, column 1, row 10 - select, column 2, row 10 Dec 7, 2022 @ 15:21:00.965 - @timestamp, column 3, row 10 2022-12-07 15:20:59,028 [INFO] [Thread-33] ExecutionTimeLogger - Resuming logging of transaction execution times details & summary I want to extract a value like "Resuming logging of transaction execution times details" and want to have a field separate field named "resuming logging". Please help i'm stuck in it for last 3 days. thanks! ---
**Author:** ![Venkata\_Raja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkata_raja/32/114264_2.png) [@Venkata\_Raja](https://discuss.elastic.co/u/Venkata_Raja)\ **Post date:** [December 7, 2022, 2:05pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/2 "2022-12-07T14:05:34Z")
Hi, You can use [dissect](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/dissect-processor.html) processor to extract values from message field. See below configuration where i have written sample dissect processor for your log. ```auto "dissect": { "field": "message", "pattern": "%{timestamp} %{+timestamp},%{id} [%{logLevel}] [%{threadNumber}] %{comment} - %{rest}", "ignore_missing": true, "ignore_failure": true } ``` Result is ```auto "rest": "Resuming logging of transaction execution times details & summary", "logLevel": "INFO", "threadNumber": "Thread-33", "comment": "ExecutionTimeLogger", "id": "028", "message": "2022-12-07 15:20:59,028 [INFO] [Thread-33] ExecutionTimeLogger - Resuming logging of transaction execution times details & summary", "timestamp": "2022-12-0715:20:59" ``` ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 7, 2022, 2:20pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/3 "2022-12-07T14:20:08Z")
Hi, thank you so much for your reply. I'm new to this stack and started last week. I'm trying to execute this query in Dev-tools and on run time fields as well but getting below. Please see the screenshots. Also please share the runtime field query i.e Set Value so that i can create new field related to it. ![kibana_1](https://us1.discourse-cdn.com/elastic/original/3X/4/5/456007ee8cd1b28e07a00786d63bdba6aecefe4a.png) ![Kibana_2](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a847ceeaf63448e0e6e97736eefa284bd8a7112.png) ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 7, 2022, 2:35pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/4 "2022-12-07T14:35:33Z")
Please tell me if you need more information. thanks Expecting your response. ---
**Author:** ![Venkata\_Raja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkata_raja/32/114264_2.png) [@Venkata\_Raja](https://discuss.elastic.co/u/Venkata_Raja)\ **Post date:** [December 7, 2022, 2:46pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/5 "2022-12-07T14:46:58Z")
Hi, You need to create an ingest pipeline with the dissect processor i have provided , Refer similar example below. > **[Ingest pipelines | Elasticsearch Guide \[8.5\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html)** Once you are done with ingest pipeline creation , try to reindex logs to other index with this pipeline and add [`index.default_pipeline`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-default-pipeline) setting to your index template so that new logs will be automatically parsed with this pipeline. ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 7, 2022, 3:06pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/6 "2022-12-07T15:06:39Z")
Hi, Thanku again for the promt response. I want multiple fields like ERROR fields, exceptionfield,job1 filed etc under discover so that i can create kibana dashboards easily. Also i ran your provided script but getting below error ![invalid](https://us1.discourse-cdn.com/elastic/original/3X/5/1/51f3d67e566e78b629c232a8cef793292c9ae059.png) ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 7, 2022, 3:27pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/7 "2022-12-07T15:27:42Z")
Can you please show me how to create a new runtime field with any important logs from message field like errors, exceptions etc. Please share the syntax lets say i want to get logs like "deny transations" present in message field and want to create a new field like error Please share the syntax of Set value. Would be grateful. ![set](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c0847ec5cd27fd6cdef7739b0a515c011bfa5f84.png) ---
**Author:** ![jughosta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jughosta/32/107160_2.png) [@jughosta](https://discuss.elastic.co/u/jughosta)\ **Post date:** [December 8, 2022, 9:03am UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/8 "2022-12-08T09:03:56Z")
Hi @Ajmal_Khalil, If you would like to quickly extract parts of "message" field, then creating runtime fields like the following could help. ![Screenshot 2022-12-08 at 09.50.02](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00700861dad50145fd6cd587adc5dc7df10eae7e.png) For "type" field I defined: ```auto def value = doc["message"].value; if (value != null) { int startIndex = value.indexOf('['); int endIndex = value.indexOf(']'); if (startIndex > 0 && endIndex > 0) { emit(value.substring(startIndex + 1, endIndex)); return; } } emit(""); ``` For "details" field: ```auto def value = doc["message"].value; if (value != null) { int index = value.lastIndexOf(' - '); if (index > 0) { emit(value.substring(index + 3)); return; } } emit(""); ``` Here is another approach if you want to create separate fields per message type like "error" which was mentioned in your last comment: ```auto def value = doc["message"].value; if (value.contains('deny transations')) { emit(value); return; } emit(""); ``` ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 8, 2022, 10:39am UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/9 "2022-12-08T10:39:01Z")
Hi Jughosta, Thankyou so much for your reply. This is actually what i want to achieve, but getting below error with this script. ![Getting_this](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a51addcbfb3a6963cd4070224f6335e15ae1c283.png) To resolve this this i ran below query in Devtools ![dev](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4c4a2f5b541d2065ce74644de30b10675605af3d.png) After this i tried above query again but getting same error mentioned above. Please help. ---
**Author:** ![jughosta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jughosta/32/107160_2.png) [@jughosta](https://discuss.elastic.co/u/jughosta)\ **Post date:** [December 8, 2022, 10:57am UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/10 "2022-12-08T10:57:36Z")
When changing mapping, you might need to reindex your data [Explicit mapping | Elasticsearch Guide [8.5] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/explicit-mapping.html#update-mapping) What is your current mapping and name for a field which contains logs message? Maybe it already has a subfield with keyword type too. ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 8, 2022, 11:45am UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/11 "2022-12-08T11:45:37Z")
Hi Julia. thank for your quick response indeed. I'm not sure about the mapping, getting these logs directly from filebeat to elasticsearch and then kibana. I tried the above reindex technique but no luck. still getting the same error which i mentioned in my above reply. Please see the discover page screenshot where i'm getting default fields and logs against those fields. All i want is to pick a log info from message field and create a field against that and then view in dashboard. ![Field](https://us1.discourse-cdn.com/elastic/original/3X/3/4/34be18e40e47449b1b9b1d03ead0a159fd8ac7d2.png) I need custom fields for creation of custom dashboards like how may errors we are getting and how many exceptions etc. thanks ---
**Author:** ![jughosta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jughosta/32/107160_2.png) [@jughosta](https://discuss.elastic.co/u/jughosta)\ **Post date:** [December 8, 2022, 11:54am UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/12 "2022-12-08T11:54:42Z")
Try using `doc["message.keyword"].value` in scripts. ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 8, 2022, 12:07pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/13 "2022-12-08T12:07:54Z")
Seems it worked ![deny](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a98a598161297341c54ff4a3e2e122547bc9af18.png) but when i returned to discover screen there is no data and getting this error ![reolv](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a00d4a283a86801632456ad5e81c98ca76dc389c.png) I think we are close to resolve this, Please see screenshots ---
**Author:** ![jughosta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jughosta/32/107160_2.png) [@jughosta](https://discuss.elastic.co/u/jughosta)\ **Post date:** [December 8, 2022, 12:14pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/14 "2022-12-08T12:14:53Z")
2 things we can do here: - expand the time range via the time picker in the top right corner to see more results, - check details of the shards failure via the button in the bottom right corner. If the message field can be missing, then the script should be wrapped into an additional check: ```auto if (doc["message.keyword"].size() > 0) { // ... the rest return; } emit(""); ``` ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 8, 2022, 12:28pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/15 "2022-12-08T12:28:18Z")
On expanding the time range for last 30 days getting the logs for 3 seconds and then suddenly disappear. Attached is the screenshot ![last30days](https://us1.discourse-cdn.com/elastic/original/3X/6/0/60dbf6ddbc8b0bdc367e274ca1173dc09128e92d.png) On checking the error details getting this. Please check ![scri](https://us1.discourse-cdn.com/elastic/original/3X/1/8/18420ccb0293af0884e6c7db999fcdd4655ac924.png) Also this is the details of this error ```auto { "took": 339, "timed_out": false, "_shards": { "total": 2, "successful": 1, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "coreapilogs-2022.12.08", "node": "zctoeBqUQeiBItFNxr7Q8g", "reason": { "type": "script_exception", "reason": "runtime error", "script_stack": [ "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:469)", "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:463)", "value = doc[\"message.keyword\"].value;\r\n", " ^---- HERE" ], "script": "def value = doc[\"message.keyword\"].value; ...", "lang": "painless", "position": { "offset": 34, "start": 4, "end": 43 }, "caused_by": { "type": "illegal_state_exception", "reason": "A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!" } } } ] }, "hits": { "max_score": null, "hits": [] } } ``` ---
**Author:** ![jughosta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jughosta/32/107160_2.png) [@jughosta](https://discuss.elastic.co/u/jughosta)\ **Post date:** [December 8, 2022, 12:47pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/16 "2022-12-08T12:47:15Z")
Okay, looks like this field is not present in some documents. Have you tried changing the script as suggested in [previous comment](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/14)? ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 8, 2022, 1:13pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/17 "2022-12-08T13:13:20Z")
yes, i did but no luck. Please check ![yee](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bf8dbda3c0abea8aef8f76a3ffd6b2152c86a909.png) Please suggest more solutions... ---
**Author:** ![jughosta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jughosta/32/107160_2.png) [@jughosta](https://discuss.elastic.co/u/jughosta)\ **Post date:** [December 8, 2022, 1:25pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/18 "2022-12-08T13:25:05Z")
What does Discover show if the following script is defined? ```auto if (doc["message.keyword"].size() > 0) { def value = doc["message.keyword"].value; if (value.contains('deny transactions')) { emit(value); return; } } emit(""); ``` ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 8, 2022, 1:46pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/19 "2022-12-08T13:46:43Z")
Discover shows empty page(no logs) after doing this ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/2/220d8bed67935b38d0d9ff03c5e3692af572079a.png) ![noth](https://us1.discourse-cdn.com/elastic/original/3X/1/4/14a3c59913345882b31f4f9e5cdbc916241bb1c7.png) And getting this error ```auto { "took": 426, "timed_out": false, "_shards": { "total": 2, "successful": 1, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "coreapilogs-2022.12.08", "node": "zctoeBqUQeiBItFNxr7Q8g", "reason": { "type": "script_exception", "reason": "runtime error", "script_stack": [ "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:469)", "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:463)", "value = doc[\"message.keyword\"].value;\r\n", " ^---- HERE" ], "script": "def value = doc[\"message.keyword\"].value; ...", "lang": "painless", "position": { "offset": 34, "start": 4, "end": 43 }, "caused_by": { "type": "illegal_state_exception", "reason": "A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!" } } } ] }, "hits": { "max_score": null, "hits": [] } } ``` Please check ---
**Author:** ![Ajmal\_Khalil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajmal_khalil/32/106377_2.png) [@Ajmal\_Khalil](https://discuss.elastic.co/u/Ajmal_Khalil)\ **Post date:** [December 8, 2022, 6:58pm UTC](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670/20 "2022-12-08T18:58:03Z")
Hi, thankyou again for your help indeed. So i stopped the filebeat and run again, after that i'm able to add the "Transactions deny" value against it and a new field has been created but after that when i tried to create a new field like Transaction committed for a text message that is also present with the same logs like "Transactions deny" but this time getting below error. Please check both screenshots when you get time and let me know any solution. ![Transaction_commitederror](https://us1.discourse-cdn.com/elastic/original/3X/8/8/880fd098e820eb95298014b22589f1891b49a2c0.png) Getting this when i click on see full error ```auto Error: Conflict at e. (http://my_domain/55395/bundles/core/core.entry.js:1:276445) at f (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1458) at Generator._invoke (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1211) at Generator.next (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1821) at n (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:291404) at s (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:291615) ``` Thanks! [Next page](https://discuss.elastic.co/t/extract-logs-from-the-message-field-and-create-new-separate-runtime-fields/320670.md?page=2)