Elastic-agent - Custom logs - no asn fields

Elastic Stack Elastic Agent
1

When using Elastic Agent, one can't use the GeoIP processor, presumably because it's built-in. It does the location lookup well, but the ASN details are missing and I've not found a way yet to toggle a setting for this.

So, what are my options? The returned JSON includes an IP field and the basic geoip fields are generated at root level, which is fine for logs with only one IP address (source).

I've tried copying the ip field to source.ip, this breaks geoip output entirely.

I guess it might be possible to add a scripted field in the data view, but I have no idea how to write the script for this, plus I guess that doing it at ingestion would be more efficient?

All I want to do is add the source.as.number and source.as.organization.name fields.

2

Hi @dmgeurts

Exactly which integration are you using?

Not really accurate the default behavior is the geo spatial data...

We can simple add on the ASN processor... Which integration are you using?

Did you already create your own ingest pipeline?

Share a bit more... Easy fix I think

If you already have pipeline add the asn part.. if you do not and it is an integration we can add it to the custom pipeline.

PUT _ingest/pipeline/geoip
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "source.ip",
        "target_field" : "source.ip.asn",
        "database_file" : "GeoLite2-ASN.mmdb"
      }
    },
        {
      "geoip" : {
        "field" : "source.ip",
        "target_field" : "source.ip.geo",
        "database_file" : "GeoLite2-City.mmdb"
      }
    }
  ]
}
3

Hi @stephenb ,

The Custom logs and Nginx integrations. In both, one can add processors, but using the geoip processor results in an error on the agent when the config is applied to a host. It states that geoip is not a valid processor.

I do get geoip fields added by both processors, it's just that I'd like to also have the as.number and as.organisation.name fields.

Not sure what you mean by custom pipeline.

I'm on ES v8.11

4

Hi @dmgeurts

Not....agent processor .... ingest pipeline which is composed of ingest processors. :slight_smile:
.. you should learn about the ingest pipelines.

Agent + Ingest Pipeline = Powerful

See here ...

Ngnix is is literally the example here

Kibana -> Stack Management - Ingest Pipeline

If you look at the logs-nginx.access-1.20.0 ingest pipeline this is all the processing for the integration.

so you could add any one of the following

  {
    "pipeline": {
      "name": "global@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Global pipeline for all data streams"
    }
  },
  {
    "pipeline": {
      "name": "logs@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Pipeline for all data streams of type `logs`"
    }
  },
  {
    "pipeline": {
      "name": "logs-nginx.integration@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Pipeline for all data streams of type `logs` defined by the `nginx` integration"
    }
  },
  {
    "pipeline": {
      "name": "logs-nginx.access@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Pipeline for the `nginx.access` dataset"
    }
  }
]

So if you want to add the ASN geoip for the the whole integration adding this custom pipeline will get called... and thus add the pipeline / ingest pipeline / ingest processors you want

Kibana -> Dev Tools (you can do via UI too but this is quicker)

PUT _ingest/pipeline/logs-nginx.integration@custom
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "source.ip",
        "target_field" : "source.ip.asn",
        "database_file" : "GeoLite2-ASN.mmdb"
      }
    }
  ]
}

That will be run / added as soon as you execute the command

1 Like
5

Thank you!

I found that the Nginx integration has the GeoIP information I was after, I had simply missed the field. But for the custom logs integration modifying the custom pipeline does what I needed it to do.

This has opened a whole new world of possibilities to me.

PUT _ingest/pipeline/logs-unifi.admin_activity@custom
{
  "description" : "Add geoip info",
  "processors" : [
    {
      "geoip" : {
        "field" : "ip",
        "target_field" : "source.as",
        "database_file" : "GeoLite2-ASN.mmdb",
        "properties": [
          "asn",
          "organization_name"
        ],
        "ignore_missing": true
      }
    }
  ]
}
1 Like
6

Cool yeah And I see you put that ignore missing in there. I should always put that in my samples. That's a must-have :slight_smile:

1 Like