Convert IP from integears

[arcsons (Jon L)]

Hi

I have two fields with integers. How can I convert it to IP ?

netconn_ipv4: 178,258,031
netconn_local_ipv4: 178,258,037

I guess I can do this with mappings but not sure.
I have tested with following without luck.

PUT /logs-carbon_black.observations-default-2025.02.09-000002
{
"mappings": {
"properties": {
"netconn_ipv4": {
"type": "ip"
},
"netconn_local_ipv4": {
"type": "ip"
}
}
}
}

{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "cannot create index with name [logs-carbon_black.observations-default-2025.02.09-000002], because it matches with template [logs] that creates data streams only, use create data stream api instead"
}
],
"type": "illegal_argument_exception",
"reason": "cannot create index with name [logs-carbon_black.observations-default-2025.02.09-000002], because it matches with template [logs] that creates data streams only, use create data stream api instead"
},
"status": 400
}

[RainTown (Kevin Maguire)]

The error message tells you the issue.

There’s an existing template / data streaming (fancy name for an alias to multiple indices, don’t shoot me for being imprecise), maybe that’s as it came with the system.

Your index name matches, so you cannot use it really.

Read the data stream section of the docs.

[stephenb (Stephen Brown)]

Hi @arcsons

What you are trying to do is going to lead to all sorts of issues!

Do not do that!

The right way to do this is add a custom ingest pipeline to convert those string to IPs

Looks at this..

Tutorial: Transform data with custom ingest pipelines | Fleet and Elastic...

In your

PUT _ingest/pipeline/logs@custom
{
  "processors": [
    {
      "pipeline": {
        "name": "mitre_attack_pipeline",
        "if": "ctx?.data_stream.dataset != null && (ctx?.data_stream.dataset == 'carbon_black.observations' || ctx?.data_stream.dataset == 'carbon_black.vulnerabilities')"
      }
    }
  ]
}

Just add a gsub processor to convert the , to .

The better way would be to add another pipeline called from this pipeline that does any cleanup you need to do...

You could also add it to the mitre_attack_pipeline

[stephenb (Stephen Brown)]

something like this

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "gsub": {
          "field": "netconn_ipv4",
          "pattern": ",",
          "replacement": "."
        }
      },
      {
        "gsub": {
          "field": "netconn_local_ipv4",
          "pattern": ",",
          "replacement": "."
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "netconn_ipv4": "178,258,031",
        "netconn_local_ipv4": "178,258,037"
      }
    }
  ]
}

# result
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "netconn_ipv4": "178.258.031",
          "netconn_local_ipv4": "178.258.037"
        },
        "_ingest": {
          "timestamp": "2025-02-18T19:21:20.182598305Z"
        }
      }
    }
  ]
}

[arcsons (Jon L)]

Hi Stephen,

Ok so something like this will work fine?

PUT _ingest/pipeline/convert_to_ip
{
  "description": "Convert comma-separated values to IP addresses",
  "processors": [
    {
      "gsub": {
        "field": "netconn_ipv4",
        "pattern": ",",
        "replacement": "."
      }
    },
    {
      "gsub": {
        "field": "netconn_local_ipv4",
        "pattern": ",",
        "replacement": "."
      }
    }
  ]
}

PUT /logs-carbon_black.observations-default-2025.02.09-000002/_settings
{
  "index.default_pipeline": "convert_to_ip"
}
´´´

[leandrojmp (Leandro Pereira)]

@arcsons

Ok so something like this will work fine?

No, this can break the integraiton.

You cannot change the index.default_pipeline nor the index.final_pipeline for any Elastic Agent integration.

All customizations needs to be done in the @custom ingest pipeline for the specific dataset of the integration or globally in the logs@custom ingest pipeline.

[stephenb (Stephen Brown)]

@arcsons

You already created a logs@custom when We work together on the enrichment.

You can just call the convert_to_ip from the logs@custom Just like you did the miter enrichment

Problems with enrichments

Assuming you have the above working this is how we will call your enrich pipeline We will follow the new framework here It refers to elastic agent but works with the data streams we created. Your Existing enrich pipeline PUT _ingest/pipeline/mitre_attack_pipeline { "description": "Pipeline to enrich MITRE ATT&CK fields", "processors": [ { "set": { "field": "pipeline_name", "value": "mitre_attack_pipeline" } }, { "enrich": { "policy_nam…

BTW @leandrojmp There is Not in integration for carbon black observations but I had him follow the framework.

[arcsons (Jon L)]

Ok thanks.
I will test that tomorrow morning.

[RainTown (Kevin Maguire)]

Out of curiosity, every day is a school day, why does the tool generate IPs like

network_ipv4: 178,258,031
netconn_local_ipv4: 178,258,037

I mean that is a pretty unusual “notation”, right. Is there some significance? Is 78,258,031 a shorthand for what is more usually written as 78.258.31.0/24? It’s a convention in a particular country/language/field ?

[arcsons (Jon L)]

Hi Rain,

Yeah you are right. I maybe should fix this directly in my code where I do my API call with python.

The output was actually in hexa, so Elastic though it was int.

So this code fixed it.

if 'netconn_ipv4' in json_data:
    json_data['netconn_ipv4'] = int_to_ip(json_data['netconn_ipv4'])
if 'netconn_local_ipv4' in json_data:
    json_data['netconn_local_ipv4'] = int_to_ip(json_data['netconn_local_ipv4'])

Thanks

[RainTown (Kevin Maguire)]

Good that you fixed it, one way or another. Personally I think the code fix was better way, but I also learned a bit about logs@custom too.

If that closes the topic, please accept one of the answers and good lick with your project.

[arcsons (Jon L)]

Good evening,

New problems.

I will post my idea which I think will solve it, give me few min. And Stephen maybe can verify it

Here is what I think will solve it:

1. GET log-carbon_black.observations-default-2025.02.09-000001/_mapping

2. Change this
        "netconn_ipv4": {
          "type": "long"
        },
        "netconn_local_ipv4": {
          "type": "long"
        },



PUT /log-carbon_black.observations-default-2025.02.09-000002
{
  "mappings": {
    "properties": {
      "netconn_ipv4": {
        "type": "ip"
      },
      "netconn_local_ipv4": {
        "type": "ip"
      },
      ### AND ALL OTHER ###
    }
  }
}


3.

POST /_reindex
{
  "source": {
    "index": "log-carbon_black.observations-default-2025.02.09-000001"
  },
  "dest": {
    "index": "log-carbon_black.observations-default-2025.02.09-000002"
  }
}


4. DELETE /log-carbon_black.observations-default-2025.02.09-000001
´´´

[arcsons (Jon L)]

Im close now.

I got it in JSON working in Elastic but the fields are strang. See the image below.

[stephenb (Stephen Brown)]

As you can see type is number not IP
So you have still not set the mapping correct And or there is a conflict in the data view? Because you have two indices with two different types of mappings this can cause these kind of issues

So please back up why are you trying to reindex the data?

I don't understand what current state you're in, so perhaps backup and tell us where you're at and what you're trying to accomplish.

I understand you're trying to get the IP but I don't know what you're doing and why

If you have two indices backing indices and they have a single data view that will be a conflict and cause issues like this.

You should probably really clean up the old data.

Create a new custom mapping for that field. Much like the ingest pipeline

Data streams | Fleet and Elastic Agent Guide [8.17] | Elastic

Add a logs@custon component template with the correct mapping

Data streams | Fleet and Elastic Agent Guide [8.17] | Elastic

Simply ingest new data with your fixed mapping

[arcsons (Jon L)]

When we were working together, the script cp.py produced output from two fields (netconn_ipv4 and netconn_local_ipv4) in hexadecimal format. In Elasticsearch, these fields were interpreted as integers. I modified cp.py to convert the hexadecimal values to IP addresses.

I haven't changed the pipelines, and I don't need to re-index since it's not in production yet. Now, I simply want these fields to be recognized as IP addresses and nothing else. Custom logs and parsing in Elasticsearch can be challenging, and it seems I'm not alone in this struggle if you search online. I apologize if my explanation isn't clear enough.

Ok I will try your suggestions, thanks.

[stephenb (Stephen Brown)]

@arcsons

Now, I simply want these fields to be recognized as IP addresses and nothing else.

Ok that helps. Remember, we are answering many questions... following each topic "Train of thought" can be challenging

@arcsons

Custom logs and parsing in Elasticsearch can be challenging, and it seems I'm not alone in this struggle if you search online.

Yes, it can... and there is a framework that can take a little getting used to ... and making assumption about how it works without reading the docs can sometimes make it worse, or using some other article etc...

@arcsons

I haven't changed the pipelines, and I don't need to re-index since it's not in production yet.

Ok good delete the existing data stream...

That will clean everything up.

Then start your data flow again... and report back.

The result of this will give us 1 of 2 path to follow.

1. It will work because you no longer have conflicting mappings in the data view (pretty sure this is the case)
2. We will add a custom mapping for those 2 fields (which is simple I will show you) and then we will clean up and start again.

For # 2 adding the custom mapping is simply this but test first and come back. This will force the 2 fields to be of the correct type

PUT _component_template/logs@custom
{
  "template": {
    "mappings": {
      "properties": {
        "netconn_ipv4": {
          "type": "ip"
        },
        "netconn_local_ipv4": {
          "type": "ip"
        }
      }
    }
  }
}

[arcsons (Jon L)]

Hey Stephen,

Everything works fine, thanks man

Case closed.

[stephenb (Stephen Brown)]

@arcsons Excellent!

Could help others if you could provide a couple of bullets on your end solutions, then we will marked as solved ... that will help others in the community.