APM 7.4 error with default pipeline [apm]

Elastic Observability APM
,
1

Today I have upgraded full stack to 7.4 in my test environment. I have also the APM server component installed. When I start it, all the documents are indexed for several minutes than it starts to log this error:

2019-10-02T16:29:11.633+0200 DEBUG [elasticsearch] elasticsearch/client.go:541 Bulk item insert failed (i=0, status=500): {"type":"exception","reason":"java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: '10.10.10.123:51122' is not an IP string literal.","caused_by":{"type":"illegal_argument_exception","reason":"java.lang.IllegalArgumentException: '10.10.10.123:51122' is not an IP string literal.","caused_by":{"type":"illegal_argument_exception","reason":"'10.10.10.123:51122' is not an IP string literal."}},"header":{"processor_type":"geoip"}}

furthermore, into Elasticsearch logs I'm facing this error:

[2019-10-02T16:31:29,629][DEBUG][o.e.a.b.TransportBulkAction] [saelklab1.crifnet.com] failed to execute pipeline [apm] for document [apm-7.4.0-transaction/_doc/null]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: '10.10.10.123:51122' is not an IP string literal.

Elastic's log should be relevant since it is indicating [apm] pipeline currently configured in this way:

{
  "apm" : {
    "description" : "Default enrichment for APM events",
    "processors" : [
      {
        "pipeline" : {
          "name" : "apm_user_agent"
        }
      },
      {
        "pipeline" : {
          "name" : "apm_user_geo"
        }
      }
    ]
  }
}

where apm_user_agent is

  "apm_user_agent" : {
    "description" : "Add user agent information for APM events",
    "processors" : [
      {
        "user_agent" : {
          "field" : "user_agent.original",
          "target_field" : "user_agent",
          "ignore_missing" : true
        }
      }
    ]
  }
}

and apm_user_geo is:

{
  "apm_user_geo" : {
    "description" : "Add user geo information for APM events",
    "processors" : [
      {
        "geoip" : {
          "database_file" : "GeoLite2-City.mmdb",
          "field" : "client.ip",
          "target_field" : "client.geo",
          "ignore_missing" : true
        }
      }
    ]
  }

another interesting behavior: if I restart the apm-server, it will index docs again when it will be started up but after several mins it will stop again.
my apm-client are are composed by java apm client.

2

Hi @rschirin,
for a quick fix you can disable the pipelines in the apm-server.yml.
What happens is that the GeoIp and the UserAgent pipelines are enabled now by default on incoming data, but it looks like some of the data are not in the right format.

We have to look into how this can happen.

3

hi @simitt, I cannot understand. in my apm-server.yml file I have not configured any pipeline (to be exact, pipeline block is commented).
furthermore, I have deleted with DELETE method pipelines indicated into the first post.

edit: with Elastic stack 7.4 and APM-Server 7.2 there is no issue. to be honest, I don't like to have different versions

4

You actually don't need to enable the processors, as from 7.3 on the apm pipeline is enabled by default, applying the GeoIP and UserAgent processors to apm data. If you want to disable that you can set pipeline: "_none" in the apm-server.yml.

I created an issue in the APM Server repo to ensure only information that can be processed by the default pipelines is indexed in the client.ip field.

1 Like
5

ok, so this should be a bug for apm v7.4?
I will try to disable explicitly the pipeline values in apm-server.yml.

thanks in advance!

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.