Extract logs from the message field and create new separate runtime fields

[Ajmal_Khalil]

Hi everyone,

I'm using ELK stack version 8.4.0 and i need to extract logs data from message field that is something like "ERRORS" "EXCEPTIONS" etc and i want a new field for every extracted value so that i can easily create a dashboard.
Please check below logs, your help would be highly appreciated.

Dec 7, 2022 @ 15:21:15.978

• @timestamp, column 3, row 1

2022-12-07 15:21:13,666 [INFO] [Thread-38] CommunicationController - Communication monitor is going to sleep for [30] seconds.

• message, column 4, row 1

• openDetails, column 1, row 2

• select, column 2, row 2

Dec 7, 2022 @ 15:21:15.977

• @timestamp, column 3, row 2

2022-12-07 15:21:13,666 [WARN] [Thread-38] ChannelConnection - ECHO is required but it is not enabled for

• message, column 4, row 2

• openDetails, column 1, row 3

• select, column 2, row 3

Dec 7, 2022 @ 15:21:15.976

• @timestamp, column 3, row 3

2022-12-07 15:21:13,666 [INFO] [Thread-38] ChannelConnection - Socket is idle for the past [180] seconds on

• message, column 4, row 3

• openDetails, column 1, row 4

• select, column 2, row 4

Dec 7, 2022 @ 15:21:15.975

• @timestamp, column 3, row 4

2022-12-07 15:21:13,666 [WARN] [Thread-38] ChannelConnection - SIGN-ON is not enabled for MYHSSM

type or paste code here

• message, column 4, row 4

• openDetails, column 1, row 5

• select, column 2, row 5

Dec 7, 2022 @ 15:21:15.974

• @timestamp, column 3, row 5

2022-12-07 15:21:13,666 [DEBUG] [Thread-38] ChannelConnection - Checking if SIGN-ON is required for MYHSM

• message, column 4, row 5

• openDetails, column 1, row 6

• select, column 2, row 6

Dec 7, 2022 @ 15:21:15.973

• @timestamp, column 3, row 6

2022-12-07 15:21:13,666 [INFO] [Thread-38] ChannelConnection - Monitoring connection state of MYHSM

• message, column 4, row 6

• openDetails, column 1, row 7

• select, column 2, row 7

Dec 7, 2022 @ 15:21:15.972

• @timestamp, column 3, row 7

2022-12-07 15:21:13,666 [INFO] [Thread-38] CommunicationController - [1] connection(s) are active on channel [MYHSM_GATEWAY_CLIENT_C1]: [Client Channel [MYHSM_GATEWAY_CLIENT_C1] on Socket

• message, column 4, row 7

• openDetails, column 1, row 8

• select, column 2, row 8

Dec 7, 2022 @ 15:21:15.971

• @timestamp, column 3, row 8

• message, column 4, row 8

• openDetails, column 1, row 9

• select, column 2, row 9

Dec 7, 2022 @ 15:21:15.970

• @timestamp, column 3, row 9

2022-12-07 15:21:13,665 [INFO] [Thread-38] CommunicationController - CommunicationMonitor is resuming monitoring for host [MYHSM_GATEWAY_CLIENT]

• message, column 4, row 9

• openDetails, column 1, row 10

• select, column 2, row 10

Dec 7, 2022 @ 15:21:00.965

• @timestamp, column 3, row 10

2022-12-07 15:20:59,028 [INFO] [Thread-33] ExecutionTimeLogger - Resuming logging of transaction execution times details & summary

I want to extract a value like "Resuming logging of transaction execution times details" and want to have a field separate field named "resuming logging".

Please help i'm stuck in it for last 3 days. thanks!

[Venkata_Raja]

Hi,

You can use dissect processor to extract values from message field.
See below configuration where i have written sample dissect processor for your log.

 "dissect": {
          "field": "message",
          "pattern": "%{timestamp} %{+timestamp},%{id} [%{logLevel}] [%{threadNumber}] %{comment} - %{rest}",
          "ignore_missing": true,
          "ignore_failure": true
        }

Result is

          "rest": "Resuming logging of transaction execution times details & summary",
          "logLevel": "INFO",
          "threadNumber": "Thread-33",
          "comment": "ExecutionTimeLogger",
          "id": "028",
          "message": "2022-12-07 15:20:59,028 [INFO] [Thread-33] ExecutionTimeLogger - Resuming logging of transaction execution times details & summary",
          "timestamp": "2022-12-0715:20:59"

[Ajmal_Khalil]

Hi, thank you so much for your reply. I'm new to this stack and started last week.
I'm trying to execute this query in Dev-tools and on run time fields as well but getting below. Please see the screenshots. Also please share the runtime field query i.e Set Value so that i can create new field related to it.

[Ajmal_Khalil]

Please tell me if you need more information. thanks
Expecting your response.

[Venkata_Raja]

Hi,

You need to create an ingest pipeline with the dissect processor i have provided , Refer similar example below.

Ingest pipelines | Elasticsearch Guide [8.5] | Elastic

Once you are done with ingest pipeline creation , try to reindex logs to other index with this pipeline and add index.default_pipeline setting to your index template so that new logs will be automatically parsed with this pipeline.

[Ajmal_Khalil]

Hi, Thanku again for the promt response. I want multiple fields like ERROR fields, exceptionfield,job1 filed etc under discover so that i can create kibana dashboards easily.
Also i ran your provided script but getting below error

[Ajmal_Khalil]

Can you please show me how to create a new runtime field with any important logs from message field like errors, exceptions etc.
Please share the syntax lets say i want to get logs like "deny transations" present in message field and want to create a new field like error
Please share the syntax of Set value. Would be grateful.

[jughosta (Julia Rechkunova)]

Hi @Ajmal_Khalil,

If you would like to quickly extract parts of "message" field, then creating runtime fields like the following could help.

For "type" field I defined:

def value = doc["message"].value;
if (value != null) {
    int startIndex = value.indexOf('[');
    int endIndex = value.indexOf(']');
    if (startIndex > 0 && endIndex > 0) {
        emit(value.substring(startIndex + 1, endIndex));
        return;
    }
}
emit("");

For "details" field:

def value = doc["message"].value;
if (value != null) {
    int index = value.lastIndexOf(' - ');
    if (index > 0) {
        emit(value.substring(index + 3));
        return;
    }
}
emit("");

Here is another approach if you want to create separate fields per message type like "error" which was mentioned in your last comment:

def value = doc["message"].value;
if (value.contains('deny transations')) {
    emit(value);
    return;
}
emit("");

[Ajmal_Khalil]

Hi Jughosta, Thankyou so much for your reply.

This is actually what i want to achieve, but getting below error with this script.

To resolve this this i ran below query in Devtools

After this i tried above query again but getting same error mentioned above. Please help.

[jughosta (Julia Rechkunova)]

When changing mapping, you might need to reindex your data Explicit mapping | Elasticsearch Guide [8.5] | Elastic

What is your current mapping and name for a field which contains logs message? Maybe it already has a subfield with keyword type too.

[Ajmal_Khalil]

Hi Julia. thank for your quick response indeed.

I'm not sure about the mapping, getting these logs directly from filebeat to elasticsearch and then kibana.

I tried the above reindex technique but no luck. still getting the same error which i mentioned in my above reply.

Please see the discover page screenshot where i'm getting default fields and logs against those fields. All i want is to pick a log info from message field and create a field against that and then view in dashboard.

I need custom fields for creation of custom dashboards like how may errors we are getting and how many exceptions etc. thanks

[jughosta (Julia Rechkunova)]

Try using doc["message.keyword"].value in scripts.

[Ajmal_Khalil]

Seems it worked

but when i returned to discover screen there is no data and getting this error

I think we are close to resolve this, Please see screenshots

[jughosta (Julia Rechkunova)]

2 things we can do here:

• expand the time range via the time picker in the top right corner to see more results,
• check details of the shards failure via the button in the bottom right corner.

If the message field can be missing, then the script should be wrapped into an additional check:

if (doc["message.keyword"].size() > 0) {
  // ... the rest
  return;
}
emit("");

[Ajmal_Khalil]

On expanding the time range for last 30 days getting the logs for 3 seconds and then suddenly disappear. Attached is the screenshot

On checking the error details getting this. Please check

Also this is the details of this error

{
  "took": 339,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 1,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "coreapilogs-2022.12.08",
        "node": "zctoeBqUQeiBItFNxr7Q8g",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:469)",
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:463)",
            "value = doc[\"message.keyword\"].value;\r\n",
            "                              ^---- HERE"
          ],
          "script": "def value = doc[\"message.keyword\"].value; ...",
          "lang": "painless",
          "position": {
            "offset": 34,
            "start": 4,
            "end": 43
          },
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
          }
        }
      }
    ]
  },
  "hits": {
    "max_score": null,
    "hits": []
  }
}

[jughosta (Julia Rechkunova)]

Okay, looks like this field is not present in some documents. Have you tried changing the script as suggested in previous comment?

[Ajmal_Khalil]

yes, i did but no luck. Please check

Please suggest more solutions...

[jughosta (Julia Rechkunova)]

What does Discover show if the following script is defined?

if (doc["message.keyword"].size() > 0) {
  def value = doc["message.keyword"].value;
  if (value.contains('deny transactions')) {
      emit(value);
      return;
  }
}
emit("");

[Ajmal_Khalil]

Discover shows empty page(no logs) after doing this

And getting this error

{
  "took": 426,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 1,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "coreapilogs-2022.12.08",
        "node": "zctoeBqUQeiBItFNxr7Q8g",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:469)",
            "org.elasticsearch.server@8.4.0/org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:463)",
            "value = doc[\"message.keyword\"].value;\r\n",
            "                              ^---- HERE"
          ],
          "script": "def value = doc[\"message.keyword\"].value; ...",
          "lang": "painless",
          "position": {
            "offset": 34,
            "start": 4,
            "end": 43
          },
          "caused_by": {
            "type": "illegal_state_exception",
            "reason": "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
          }
        }
      }
    ]
  },
  "hits": {
    "max_score": null,
    "hits": []
  }
}

Please check

[Ajmal_Khalil]

Hi,

thankyou again for your help indeed.

So i stopped the filebeat and run again, after that i'm able to add the "Transactions deny" value against it and
a new field has been created but after that when i tried to create a new field like Transaction committed for
a text message that is also present with the same logs like "Transactions deny" but this time getting below error.
Please check both screenshots when you get time and let me know any solution.

Getting this when i click on see full error

Error: Conflict
    at e.<anonymous> (http://my_domain/55395/bundles/core/core.entry.js:1:276445)
    at f (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1458)
    at Generator._invoke (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1211)
    at Generator.next (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:515:1821)
    at n (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:291404)
    at s (http://my_domain/55395/bundles/kbn-ui-shared-deps-npm/kbn-ui-shared-deps-npm.dll.js:364:291615)

Thanks!