Conversion type pipeline

Elastic Stack Elasticsearch
1

Hello,
I'm trying to write a pipeline to parse following part of data

ua="192.168.211.109:443"

I implement it as

 ua=\"%{IP:wkaddress}:%{NUMBER:port:integer}\"

but always I have wkaddress and port as Text field.
When I try its using simulate in develop tools port appears as integer but into the log it always is set up as text.
Someone have the same problem?

best regards

2

Have you checked your mappings? This looks more a Elasticsearch question but anyhow, I've done the following test and it worked as expected (7.9.1). From the DevTools console:

  • Create an index with a strict mapping with IP and port fields
  • Create a pipeline that processes a message like yours and extract the fields
  • Simulate the pipeline to be sure it works
  • Test the pipeline with a new document
  • Perform a IP range search to ensure the document is returned

Does this help?

PUT testing_ip
{
  "settings": {
    "index": {
      "number_of_shards": 1,  
      "number_of_replicas": 1
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "ip": { "type": "ip" },
      "port": { "type": "integer"}  
    }
  }
}

PUT _ingest/pipeline/parseip
{
  "description": "Parsing ua=\"ip:number\" message type",
  "processors": [
    {
      "kv": {
        "field": "message",
        "field_split": " ",
        "value_split": "="
      }
    },
    {
      "dissect": {
        "field": "ua",
        "pattern": "\"%{ip}:%{port}\""
      }
    },
    {
      "convert": {
        "field": "port",
        "type": "integer"
      }
    },
    {
      "remove": {
        "field": "message"
      }
    },
    {
      "remove": {
        "field": "ua"
      }
    }
  ]
}

POST /_ingest/pipeline/parseip/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "message": "ua=\"192.168.211.109:443\""
      }
    }
  ]
}

POST testing_ip/_doc?pipeline=parseip
{
  "message": "ua=\"192.168.211.109:443\""
}

GET testing_ip/_search
{
  "query": {
    "term": {
      "ip": "192.168.0.0/16"
    }
  }
}
3

Yes also in my solution into devTools it is works but it does not work in log parsing and I have example port as string

4

I'm afraid is hard to tell anything else with more details.

Could you share the relevant bits of your pipeline and the mapping of your index?

5

Es. this do not works

%{DATA:bodybytes}

 {
        "convert" : {
          "field" : "bodybytes",
          "type" : "long",
          "on_failure" : [
            {
              "set" : {
                "field" : "bodybytes",
                "value" : -1
              }
            }
          ]
        }
      }

in the same pipeline this works and wkhttpstatus is presented as number

uhs=\"%{DATA:wkhttpstatus}\"

 {
        "convert" : {
          "type" : "long",
          "on_failure" : [
            {
              "set" : {
                "value" : -1,
                "field" : "wkhttpstatus"
              }
            }
          ],
          "field" : "wkhttpstatus"
        }
      }
6

Moving this to the Elasticsearch forum since this is not Kibana related.

My suggestion would be to try to reproduce your parsing error in the simplest and reproducible environment, try to use the /_ingest/pipeline/{your_pipeline}/_simulate to get your pipeline working as you expect, then maybe create a new index with the mapping with dynamic: strict to avoid types that you don't want and test your documents on that index and see if that works.

Sorry but running out of ideas with the information available.

7

Do not worry thanks for your time
best regards

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.