Ingest pipeline created for filebeat log using grok pattern,but unable to run in dev tools

Elastic Stack Beats
,
1

Hello All,

I'm successfully getting output of my log pattern using grok pattern,but when trying to create log ingest pipeline that can be used in template later,I'm getting exception while running the ingest pipeline in dev tools.
Kindly suggest,how to rectify

image
image1873×873 50.3 KB

image
image1805×467 36.7 KB 

abcd02appl002_MONITORING_USECASES|abcnctd02appl002|MONITORING_USECASES|23653|/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log|2023-03-13 15:40:31|2023-03-13 15:40:34|0h 0min 3sec|2023-03-13 16:00:00|0.0|0.1|188360|33204|2796|perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties|/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl

%{WORD:UNIQUEID}\|%{WORD:SERVERNAME}\|%{WORD:USECASENAME}\|%{NUMBER:PID}\|%{UNIXPATH:LOGPATH}\|%{TIMESTAMP_ISO8601:STARTTIME}\|%{TIMESTAMP_ISO8601:ENDTIME}\|%{DATA:DURATION}\|%{TIMESTAMP_ISO8601:NEXTEVENT}\|%{NUMBER:CPUUSAGE}\|%{NUMBER:MEMORYUSAGE}\|%{NUMBER:VIRTUALMEMORY}\|%{NUMBER:RESIDENTMEMORY}\|%{NUMBER:SHAREDMEMORY}\|%{GREEDYDATA:COMMAND}\|%{GREEDYDATA:PERLFILE}

output:

{
  "PERLFILE": "/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl",
  "USECASENAME": "MONITORING_USECASES",
  "ENDTIME": "2023-03-13 15:40:34",
  "PID": "23653",
  "RESIDENTMEMORY": "33204",
  "SHAREDMEMORY": "2796",
  "SERVERNAME": "abcnctd02appl002",
  "STARTTIME": "2023-03-13 15:40:31",
  "UNIQUEID": "abcd02appl002_MONITORING_USECASES",
  "VIRTUALMEMORY": "188360",
  "CPUUSAGE": "0.0",
  "MEMORYUSAGE": "0.1",
  "COMMAND": "perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/globalconfiguration.properties",
  "NEXTEVENT": "2023-03-13 16:00:00",
  "DURATION": "0h 0min 3sec",
  "LOGPATH": "/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log"
}

PUT _ingest/pipeline/mis-usecases-ingest-pipeline
{
  "description": "Ingest Pipeline for incoming log documents from Filebeat agents ",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{WORD:UNIQUEID}\|%{WORD:SERVERNAME}\|%{WORD:USECASENAME}\|%{NUMBER:PID}\|%{UNIXPATH:LOGPATH}\|%{TIMESTAMP_ISO8601:STARTTIME}\|%{TIMESTAMP_ISO8601:ENDTIME}\|%{DATA:DURATION}\|%{TIMESTAMP_ISO8601:NEXTEVENT}\|%{NUMBER:CPUUSAGE}\|%{NUMBER:MEMORYUSAGE}\|%{NUMBER:VIRTUALMEMORY}\|%{NUMBER:RESIDENTMEMORY}\|%{NUMBER:SHAREDMEMORY}\|%{GREEDYDATA:COMMAND}\|%{GREEDYDATA:PERLFILE}"
        ],
        "on_failure": [
          {
            "set": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    }
  ]
}

Exception:Unable to understand

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "Failed to parse content to map"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Failed to parse content to map",
    "caused_by" : {
      "type" : "json_parse_exception",
      "reason" : "Unrecognized character escape '|' (code 124)\n at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 8, column: 30]"
    }
  },
  "status" : 400
}

How can I create proper log ingest pileline?

Thanx

2

I'm not sure you need to escape the |, have you tried without escaping it?

Also, if your message looks like it, it would be better to use the dissect processor or you can even use the csv processor setting the | as a separator.

3

Hello @leandrojmp ,
By escaping the | I don't get any data.I'm still not sure why In kibana dev tools I'm getting correct output with grok pattern created but while executing my ingest pipeline using PUT ,I'm getting the error.
I came to this solution after reading this statement in net:

Unrecognized character escape '|'" is most likely caused by the pipe character '|' in your grok pattern. The pipe character is a special character in regular expressions, and needs to be escaped with a backslash.
after adding \ I'm able to parse data in GROK Debugger in devtools,but in console getting error.

I'm not sure how disscet or csv would help here,would explore if that works.

Thanx

4

@PRASHANT_MEHTA

Try this note the triple quotes on the pattern...

PUT _ingest/pipeline/mis-usecases-ingest-pipeline
{
  "description": "Ingest Pipeline for incoming log documents from Filebeat agents ",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          """%{WORD:UNIQUEID}\|%{WORD:SERVERNAME}\|%{WORD:USECASENAME}\|%{NUMBER:PID}\|%{UNIXPATH:LOGPATH}\|%{TIMESTAMP_ISO8601:STARTTIME}\|%{TIMESTAMP_ISO8601:ENDTIME}\|%{DATA:DURATION}\|%{TIMESTAMP_ISO8601:NEXTEVENT}\|%{NUMBER:CPUUSAGE}\|%{NUMBER:MEMORYUSAGE}\|%{NUMBER:VIRTUALMEMORY}\|%{NUMBER:RESIDENTMEMORY}\|%{NUMBER:SHAREDMEMORY}\|%{GREEDYDATA:COMMAND}\|%{GREEDYDATA:PERLFILE}"""
        ],
        "on_failure": [
          {
            "set": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    }
  ]
}

POST _ingest/pipeline/mis-usecases-ingest-pipeline/_simulate
{
  "docs": [
    {
      "_source" : {
        "message" : "abcd02appl002_MONITORING_USECASES|abcnctd02appl002|MONITORING_USECASES|23653|/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log|2023-03-13 15:40:31|2023-03-13 15:40:34|0h 0min 3sec|2023-03-13 16:00:00|0.0|0.1|188360|33204|2796|perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties|/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl"
      }
    }
  ]
}

Results

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "PERLFILE": "/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl",
          "USECASENAME": "MONITORING_USECASES",
          "ENDTIME": "2023-03-13 15:40:34",
          "PID": "23653",
          "RESIDENTMEMORY": "33204",
          "message": "abcd02appl002_MONITORING_USECASES|abcnctd02appl002|MONITORING_USECASES|23653|/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log|2023-03-13 15:40:31|2023-03-13 15:40:34|0h 0min 3sec|2023-03-13 16:00:00|0.0|0.1|188360|33204|2796|perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties|/l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl",
          "SHAREDMEMORY": "2796",
          "SERVERNAME": "abcnctd02appl002",
          "STARTTIME": "2023-03-13 15:40:31",
          "UNIQUEID": "abcd02appl002_MONITORING_USECASES",
          "VIRTUALMEMORY": "188360",
          "CPUUSAGE": "0.0",
          "MEMORYUSAGE": "0.1",
          "COMMAND": "perl /l/app/MIS/monitoring/bin/cc_monitoring_usecases.pl --configpath /l/rool/MIS/configuration.properties",
          "NEXTEVENT": "2023-03-13 16:00:00",
          "DURATION": "0h 0min 3sec",
          "LOGPATH": "/l/logs/MIS/2023-03-13/monitor/cc_monitoring_usecases.log"
        },
        "_ingest": {
          "timestamp": "2023-03-14T15:33:16.145176498Z"
        }
      }
    }
  ]
}
5

Hello @stephenb ,

Thanx for your time to look into this.This works now. """ """ : this really helped.

Thanx

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.