Extract logs from the message field and create new separate runtime fields

Ajmal_Khalil · December 7, 2022, 10:24am

Hi everyone,

I'm using ELK stack version 8.4.0 and i need to extract logs data from message field that is something like "ERRORS" "EXCEPTIONS" etc and i want a new field for every extracted value so that i can easily create a dashboard.
Please check below logs, your help would be highly appreciated.

Dec 7, 2022 @ 15:21:15.978

@timestamp, column 3, row 1

2022-12-07 15:21:13,666 [INFO] [Thread-38] CommunicationController - Communication monitor is going to sleep for [30] seconds.

message, column 4, row 1
openDetails, column 1, row 2
select, column 2, row 2

Dec 7, 2022 @ 15:21:15.977

@timestamp, column 3, row 2

2022-12-07 15:21:13,666 [WARN] [Thread-38] ChannelConnection - ECHO is required but it is not enabled for

message, column 4, row 2
openDetails, column 1, row 3
select, column 2, row 3

Dec 7, 2022 @ 15:21:15.976

@timestamp, column 3, row 3

2022-12-07 15:21:13,666 [INFO] [Thread-38] ChannelConnection - Socket is idle for the past [180] seconds on

message, column 4, row 3
openDetails, column 1, row 4
select, column 2, row 4

Dec 7, 2022 @ 15:21:15.975

@timestamp, column 3, row 4

2022-12-07 15:21:13,666 [WARN] [Thread-38] ChannelConnection - SIGN-ON is not enabled for MYHSSM

type or paste code here

message, column 4, row 4
openDetails, column 1, row 5
select, column 2, row 5

Dec 7, 2022 @ 15:21:15.974

@timestamp, column 3, row 5

2022-12-07 15:21:13,666 [DEBUG] [Thread-38] ChannelConnection - Checking if SIGN-ON is required for MYHSM

message, column 4, row 5
openDetails, column 1, row 6
select, column 2, row 6

Dec 7, 2022 @ 15:21:15.973

@timestamp, column 3, row 6

2022-12-07 15:21:13,666 [INFO] [Thread-38] ChannelConnection - Monitoring connection state of MYHSM

message, column 4, row 6
openDetails, column 1, row 7
select, column 2, row 7

Dec 7, 2022 @ 15:21:15.972

@timestamp, column 3, row 7

2022-12-07 15:21:13,666 [INFO] [Thread-38] CommunicationController - [1] connection(s) are active on channel [MYHSM_GATEWAY_CLIENT_C1]: [Client Channel [MYHSM_GATEWAY_CLIENT_C1] on Socket

message, column 4, row 7
openDetails, column 1, row 8
select, column 2, row 8

Dec 7, 2022 @ 15:21:15.971

@timestamp, column 3, row 8
message, column 4, row 8
openDetails, column 1, row 9
select, column 2, row 9

Dec 7, 2022 @ 15:21:15.970

@timestamp, column 3, row 9

2022-12-07 15:21:13,665 [INFO] [Thread-38] CommunicationController - CommunicationMonitor is resuming monitoring for host [MYHSM_GATEWAY_CLIENT]

message, column 4, row 9
openDetails, column 1, row 10
select, column 2, row 10

Dec 7, 2022 @ 15:21:00.965

@timestamp, column 3, row 10

2022-12-07 15:20:59,028 [INFO] [Thread-33] ExecutionTimeLogger - Resuming logging of transaction execution times details & summary

I want to extract a value like "Resuming logging of transaction execution times details" and want to have a field separate field named "resuming logging".

Please help i'm stuck in it for last 3 days. thanks!

Venkata_Raja · December 7, 2022, 2:05pm

Hi,

You can use dissect processor to extract values from message field.
See below configuration where i have written sample dissect processor for your log.

 "dissect": {
          "field": "message",
          "pattern": "%{timestamp} %{+timestamp},%{id} [%{logLevel}] [%{threadNumber}] %{comment} - %{rest}",
          "ignore_missing": true,
          "ignore_failure": true
        }

Result is

          "rest": "Resuming logging of transaction execution times details & summary",
          "logLevel": "INFO",
          "threadNumber": "Thread-33",
          "comment": "ExecutionTimeLogger",
          "id": "028",
          "message": "2022-12-07 15:20:59,028 [INFO] [Thread-33] ExecutionTimeLogger - Resuming logging of transaction execution times details & summary",
          "timestamp": "2022-12-0715:20:59"

Ajmal_Khalil · December 7, 2022, 2:20pm

Hi, thank you so much for your reply. I'm new to this stack and started last week.
I'm trying to execute this query in Dev-tools and on run time fields as well but getting below. Please see the screenshots. Also please share the runtime field query i.e Set Value so that i can create new field related to it.

Ajmal_Khalil · December 7, 2022, 2:35pm

Please tell me if you need more information. thanks
Expecting your response.

Venkata_Raja · December 7, 2022, 2:46pm

Hi,

You need to create an ingest pipeline with the dissect processor i have provided , Refer similar example below.

Once you are done with ingest pipeline creation , try to reindex logs to other index with this pipeline and add index.default_pipeline setting to your index template so that new logs will be automatically parsed with this pipeline.

Ajmal_Khalil · December 7, 2022, 3:06pm

Hi, Thanku again for the promt response. I want multiple fields like ERROR fields, exceptionfield,job1 filed etc under discover so that i can create kibana dashboards easily.
Also i ran your provided script but getting below error

Ajmal_Khalil · December 7, 2022, 3:27pm

Can you please show me how to create a new runtime field with any important logs from message field like errors, exceptions etc.
Please share the syntax lets say i want to get logs like "deny transations" present in message field and want to create a new field like error
Please share the syntax of Set value. Would be grateful.

jughosta · December 8, 2022, 9:03am

Hi @Ajmal_Khalil,

If you would like to quickly extract parts of "message" field, then creating runtime fields like the following could help.

For "type" field I defined:

def value = doc["message"].value;
if (value != null) {
    int startIndex = value.indexOf('[');
    int endIndex = value.indexOf(']');
    if (startIndex > 0 && endIndex > 0) {
        emit(value.substring(startIndex + 1, endIndex));
        return;
    }
}
emit("");

For "details" field:

def value = doc["message"].value;
if (value != null) {
    int index = value.lastIndexOf(' - ');
    if (index > 0) {
        emit(value.substring(index + 3));
        return;
    }
}
emit("");

Here is another approach if you want to create separate fields per message type like "error" which was mentioned in your last comment:

def value = doc["message"].value;
if (value.contains('deny transations')) {
    emit(value);
    return;
}
emit("");

Ajmal_Khalil · December 8, 2022, 10:39am

Hi Jughosta, Thankyou so much for your reply.

This is actually what i want to achieve, but getting below error with this script.

To resolve this this i ran below query in Devtools

After this i tried above query again but getting same error mentioned above. Please help.

jughosta · December 8, 2022, 10:57am

When changing mapping, you might need to reindex your data Explicit mapping | Elasticsearch Guide [8.5] | Elastic

What is your current mapping and name for a field which contains logs message? Maybe it already has a subfield with keyword type too.

Ajmal_Khalil · December 8, 2022, 11:45am

Hi Julia. thank for your quick response indeed.

I'm not sure about the mapping, getting these logs directly from filebeat to elasticsearch and then kibana.

I tried the above reindex technique but no luck. still getting the same error which i mentioned in my above reply.

Please see the discover page screenshot where i'm getting default fields and logs against those fields. All i want is to pick a log info from message field and create a field against that and then view in dashboard.

I need custom fields for creation of custom dashboards like how may errors we are getting and how many exceptions etc. thanks

jughosta · December 8, 2022, 11:54am

Try using doc["message.keyword"].value in scripts.

Ajmal_Khalil · December 8, 2022, 12:07pm

Seems it worked

but when i returned to discover screen there is no data and getting this error

I think we are close to resolve this, Please see screenshots

jughosta · December 8, 2022, 12:14pm

2 things we can do here:

expand the time range via the time picker in the top right corner to see more results,
check details of the shards failure via the button in the bottom right corner.

If the message field can be missing, then the script should be wrapped into an additional check:

if (doc["message.keyword"].size() > 0) {
  // ... the rest
  return;
}
emit("");

Ajmal_Khalil · December 8, 2022, 12:28pm

On expanding the time range for last 30 days getting the logs for 3 seconds and then suddenly disappear. Attached is the screenshot

On checking the error details getting this. Please check

Also this is the details of this error

{
  "took": 339,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 1,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "coreapilogs-2022.12.08",
        "node": "zctoeBqUQeiBItFNxr7Q8g",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:469)",
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:463)",
            "value = doc[\"message.keyword\"].value;\r\n",
            "                              ^---- HERE"
          ],
          "script": "def value = doc[\"message.keyword\"].value; ...",
          "lang": "painless",
          "position": {
            "offset": 34,
            "start": 4,
            "end": 43
          },
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
          }
        }
      }
    ]
  },
  "hits": {
    "max_score": null,
    "hits": []
  }
}

jughosta · December 8, 2022, 12:47pm

Okay, looks like this field is not present in some documents. Have you tried changing the script as suggested in previous comment?

Ajmal_Khalil · December 8, 2022, 1:13pm

yes, i did but no luck. Please check

Please suggest more solutions...

jughosta · December 8, 2022, 1:25pm

What does Discover show if the following script is defined?

if (doc["message.keyword"].size() > 0) {
  def value = doc["message.keyword"].value;
  if (value.contains('deny transactions')) {
      emit(value);
      return;
  }
}
emit("");

Ajmal_Khalil · December 8, 2022, 1:46pm

Discover shows empty page(no logs) after doing this

And getting this error

{
  "took": 426,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 1,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "coreapilogs-2022.12.08",
        "node": "zctoeBqUQeiBItFNxr7Q8g",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:469)",
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:463)",
            "value = doc[\"message.keyword\"].value;\r\n",
            "                              ^---- HERE"
          ],
          "script": "def value = doc[\"message.keyword\"].value; ...",
          "lang": "painless",
          "position": {
            "offset": 34,
            "start": 4,
            "end": 43
          },
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
          }
        }
      }
    ]
  },
  "hits": {
    "max_score": null,
    "hits": []
  }
}

Please check

Ajmal_Khalil · December 8, 2022, 6:58pm

Hi,

thankyou again for your help indeed.

So i stopped the filebeat and run again, after that i'm able to add the "Transactions deny" value against it and
a new field has been created but after that when i tried to create a new field like Transaction committed for
a text message that is also present with the same logs like "Transactions deny" but this time getting below error.
Please check both screenshots when you get time and let me know any solution.

Getting this when i click on see full error

Error: Conflict
    at e.<anonymous> (http://my_domain/55395/bundles/core/core.entry.js:1:276445)
    at f (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1458)
    at Generator._invoke (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1211)
    at Generator.next (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1821)
    at n (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:291404)
    at s (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:291615)

Thanks!

Topic		Replies	Views
Basic runtime fields to pull data from message field Elasticsearch runtime-fields	2	164	May 10, 2024
Extract field from message field with in logs Logstash	8	6422	March 15, 2019
Runtime field kibana Elasticsearch runtime-fields	2	712	March 1, 2023
Extracting substring from a field Kibana runtime	5	241	July 7, 2024
Filter Logs from Firewall Logstash	4	1849	April 22, 2019

Extract logs from the message field and create new separate runtime fields

Related topics