Ingest pipline - multiple fields processed by one porcessor

Elastic Stack Elasticsearch
1

Hi,

Is there any solution how I could process multiple filds with a single processor using ingest node. For e.g. I would like to process
source.ip
dst.ip
ip
client.ip

I would like t process all of them using geoip and output to

source.ip.geo
dst.ip.geo
ip.geo
client.ip.geo

2

I don't think you can. You need to run geoip processor 4 times I believe.

3

I have tried to run it twice but it is not accepting two instances of geoip

PUT _ingest/pipeline/geoip-info
{
"description": "Add geoip info",
"processors": [
{


  "geoip": {
    "field": "ip",
    "target_field": "client.geo",
    "ignore_failure": true
  },
  "geoip": {
    "field": "source.ip",
    "target_field": "sourceip.geo",
    "ignore_failure": true
  }
}


]
}
4

Ha right! I think you can use this then: https://www.elastic.co/guide/en/elasticsearch/reference/current/foreach-processor.html

5

Thank you - that's excellent

6

David, one more question. foreach needs to receive an array field. Supposedly I want to process two fields source.ip and dest.ip - how do I provide it for foreach processor?

7

Sorry. I was wrong. There is a more obvious way to make it work:

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "date pipeline ",
    "processors": [
      {
        "geoip": {
          "field": "dest",
          "target_field": "dest_geoip"
        }
      },
      {
        "geoip": {
          "field": "source",
          "target_field": "source_geoip"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_type": "_doc",
      "_id": "id",
      "_source": {
        "dest": "80.34.121.50",
        "source": "80.34.121.50"
      }
    }
  ]
}
8

David, Thank you for this again - it seams to be a solution. However, Packetbeat returns fields in dot notation like dest.ip source.ip - once I use those fields - pipeline is failing - is there any way to convert dot notation in to something else? It would have to be done on Packetbeat level - unless I can convince ingest node to process dot notation fields.

9

Have a look at the rename processor.

10

David,

Fields with dot notation cannot be processed by any processor according to documentation.

dot_expander processor is a solution. https://www.elastic.co/guide/en/elasticsearch/reference/6.6/dot-expand-processor.html

" Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor."

I have tried rename but it had problem with dot notation also.
Below is a final pipeline which allow multiple use of the same processor and also allow processing of fields with dot notations like source.ip dest.ip as they come from Packetbeat.

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "date pipeline ",
    "processors": [
      {
        "dot_expander": {
          "field": "source.ip"
        } 
      },
      {
        "dot_expander": {
          "field": "dest.ip"
        } 
      },
      {
        "geoip":{
          "field":"source.ip",
          "target_field":"source_geoip"
        }
      },
      {
        "geoip":{
          "field":"dest.ip",
          "target_field":"dest_geoip"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_type": "_doc",
      "_id": "id",
      "_source": {
        "dest.ip": "80.34.121.50",
        "source.ip": "191.12.41.50"
      }
    }
  ]
}
1 Like
11

Great! Thanks for sharing the final pipeline.

(I was not aware of the dot_expander processor :wink:)

1 Like
12

Hey - it was discovery for me as well :wink:

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.