Query regarding split function and conditionality

Elastic Stack Elasticsearch
1

Hi,

I am a fairly new user to elastic and trying to develop an ingest pipeline to process Cisco logs.

The format of the log is that the log.source.address is 1.1.1.1:

I am using the split function to delimit on : and put the IP address into a new field and the port into another one.

The code we are using is where source_ip_port is an array

"processors": [
{
"split": {
"if": "ctx.event.dataset == 'cisco.ios",
"field": "log.source.address",
"separator": ":",
"target_field": "source_ip_port",
"ignore_missing": false

However when I test this with example log I get the message that

Type =script exception error
Reason = runtime error
script_stack

ctx.event.dataset
^------HERE (this is highlighting the . after event)

event.dataset does exist in the log format table

I have searched around and can not find anyone else reporting this kind of error.

Any help would be greatly appreciated.

Cheers,
Ian

2

Hi @ianrobo

I don't know how your "event.dataset" and "log.source.address" fields are mapped but I made an example for you to see.
If your document is like the simulation doc below, it will work.

PUT _ingest/pipeline/example-split
{
  "description": "",
  "processors": [
    {
      "split": {
        "if": "ctx.event.dataset == 'cisco'",
        "field": "log.source.address",
        "separator": ":",
        "target_field":"new_field",
        "ignore_missing": false
      }
    }
  ]
}

POST /_ingest/pipeline/example-split/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "event": {
          "dataset": "cisco"
        },
        "log": {
          "source": {
            "address": "1.1.1.1:9200"
          }
        }
      }
    }
  ]
}
3

thanks a lot for that @RabBit_BR and instead of using the dataset one I used the tag field and that worked.

However in the next part of the code I set up the set command as below

"set": {
"if": "ctx.containsKey('source_ip_port')",
"field": "log.source.address",
"copy_from": "source_ip_port.0"

and now the the output is erroring on

"ctx.contains.key('source.ip.port)"
^------ HERE

Can you see what the problem is here ?

4

Enter the entire message to understand the error.
Usually you will see something like this (part example):

  "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "member method [java.util.Map, containsKfey/1] not found"
    }

If you can show your complete pipeline it will help to understand the problem.

5

thanks again and it shows null_pointer_exception

cannot access method/field [key] from a null def reference

The method is to to take the log source address, copy into a split field with 0 representing IP and 1 representing the port so then we can source on IP only.

hence the set uses the temp array field to do this

the whole code is here

PUT _ingest/pipeline/cisco_ios_syslog_split_log_source
{ 
  "description": "For Cisco IOS syslog events. Splits the log.source.address field into separate IP address and port fields. Stores address in log.source.address & port in log.source.port field.",
  "version": 2022032401,
  "on_failure": [
    {
      "set": {
        "description": "Set 'error.message'",
        "field": "error.message",
        "value": "Processor {{ _ingest.on_failure_processor_type }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}"
      }
    }
  ],
  "processors": [
    {
      "split": {
        "if": "ctx.tags == 'cisco-ios'",
        "field": "log.source.address",
        "separator": ":",
        "target_field": "source_ip_port",
        "ignore_missing": false
      }
    },
    { 
     "set": {
       "if": "ctx.containsKey('source_ip_port')",
       "field": "log.source.address",
       "copy_from": "source_ip_port.0"
      } 
    },
    { 
     "set": {
       "if": "ctx.containsKey('source_ip_port')",
       "field": "log.source.port",
       "copy_from": "source_ip_port.1"
      } 
    },
    { 
     "remove": {
       "if": "ctx.containsKey('source_ip_port')",
       "field": "source_ip_port"
      } 
    }
  ]
}

So I would appreciate any further help @RabBit_BR

Cheers,
Ian

6

I'm testing your pipeline and no errors occur.
How are you running?

7

I am testing it through the console using example log file to test against. I can not show the full error on here as it is in a secure environment.

8

Hi @ianrobo

Did you solve the problem?

One thing I noticed is that you search for the field "source.ip.port" but it doesn't exist. Wouldn't it be ctx.containsKey('source_ip_port')

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.