Could not index event to Elasticsearch

[lduvnjak]

Hey Everyone,

I'm having some issues trying to get APM logs to Elasticsearch going through Logstash.
Since it's a POC I'm using the Logstash JVM to test it out.

ELK version is 8.8.1, that includes the APM server. APM agent is LATEST which is 1.40.0 atm.
Prior to setting up the APM server and agent I installed the APM integration so it sets up Elasticsearch assets:

POST kbn:/api/fleet/epm/packages/apm/8.8.1
{ "force": true }

/etc/logstash/jvm.options

# Elastic APM

-javaagent:/opt/elastic/elastic-apm-agent.jar
-Delastic.apm.service_name=my-cool-service
-Delastic.apm.application_packages=org.example,org.another.example
-Delastic.apm.server_url=https://kibana:8200
-Delastic.apm.verify_server_cert=false

-Djavax.net.ssl.keyStore=/etc/logstash/certs/cert.pkcs12
-Djavax.net.ssl.keyStoreType=pkcs12
-Djavax.net.ssl.keyStorePassword=changeme
...

This is the Logstash configuration:

input {
  elastic_agent {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/logstash/certs/cert.crt"
    ssl_key => "/etc/logstash/certs/cert.key.pkcs8"
    ssl_verify_mode => "none"
    enrich => none
    type => "elastic-agent"
  }
}

filter {
  if [type] == "elastic-agent" {
    mutate {
      rename => {
        "[data_stream.type]" => "[data_stream][type]"
        "[data_stream.dataset]" => "[data_stream][dataset]"
        "[data_stream.namespace]" => "[data_stream][namespace]"
      }
    }
  }
}

output {
  if [type] == "elastic-agent" {
      elasticsearch {
        hosts => 'https://elk:9200'
        data_stream => true
        data_stream_auto_routing => "true"
        ssl => true
        cacert => "/etc/logstash/certs/ca.crt"
        user => "elastic"
        password => "password"
        manage_template => false
        action => "create"
      }
  }
}

Example error message is below (with stripped sensitive info):

[2023-07-28T11:58:16,154][WARN ][logstash.outputs.elasticsearch][main][21e1bb34aa5eb69fb9c83db3ff2aa8b820594d22795298305b910c20a2e7d0bc] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"metrics-apm.internal-default", :routing=>nil}, {"agent"=>{"name"=>"java", "ephemeral_id"=>"d749e386-2c73-4a7b-91c4-deb6ac6270df", "version"=>"1.40.0", "activation_method"=>"javaagent-flag"}, "observer"=>{"version"=>"8.8.1", "type"=>"apm-server", "hostname"=>"..."}, "processor"=>{"event"=>"metric", "name"=>"metric"}, "data_stream"=>{"type"=>"metrics", "namespace"=>"default", "dataset"=>"apm.internal"}, "_metric_descriptions"=>{"jvm.memory.heap.pool.committed"=>{}, "jvm.memory.heap.pool.used"=>{}}, "@timestamp"=>2023-07-28T11:58:05.006Z, "metricset.name"=>"app", "service"=>{"runtime"=>{"name"=>"Java", "version"=>"17.0.7"}, "name"=>"my-cool-service", "language"=>{"name"=>"Java", "version"=>"17.0.7"}, "node"=>{"name"=>"..."}}, "labels"=>{"name"=>"G1 Eden Space"}, "@version"=>"1", "jvm.memory.heap.pool.committed"=>49283072.0, "type"=>"elastic-agent", "process"=>{"title"=>"/usr/share/logstash/jdk/bin/java", "parent"=>{"pid"=>1}, "pid"=>4029}, "jvm.memory.heap.pool.used"=>13631488.0, "host"=>{"name"=>"...", "os"=>{"platform"=>"Linux"}, "hostname"=>"...", "architecture"=>"amd64", "ip"=>["..."]}}], :response=>{"create"=>{"_index"=>".ds-metrics-apm.internal-default-2023.07.28-000001", "_id"=>"nPJdnIkBy9LjXIlJIJYx", "status"=>400, "error"=>{"type"=>"strict_dynamic_mapping_exception", "reason"=>"[1:228] mapping set to strict, dynamic introduction of [type] within [_doc] is not allowed"}}}}

Anyone got any idea what's going on here and tips to fix it? if it's a bug I'll opet it on Github, that is also good information.

Thanks for any help in avdance.

Cheers,
Luka

[simitt (Silvia Mitter)]

My understanding is that the type specified in the pipeline is added to the apm event. However, we had a strict mapping defined for internal metrics, as they are expected to be known upfront, and therefore no unknown metrics should be indexed.

If it is an option for you, I recommend updating to 8.9, as the mapping was changed from strict to runtime, see changelog.

Otherwise you could make use of custom component templates to add the type field to your metrics attribute, OR set up an ES ingest pipeline with a drop processor to remove the field before indexing to ES.

[lduvnjak]

Thanks for the amazing reponse time. Upgrading is definitely an option, but that might not be the case with every deploy we have so I tested it with custom ingest pipelines for specific data streams that remove the fields type and @version since it was complaining about them.

For now, no more indexing errors are present in the logs, but the APM visualization still seems to be empty:

Is that because it's a Logstash JVM so it has no metrics to actually collect related to those supposed to display here or is it this an issue we should look into?

The indices do have documents in them so the first issue is definitely resolved (unless it breaks for not having @version metadata specified).

Cheers,
Luka

[Jonas_Kunz]

Is that because it's a Logstash JVM so it has no metrics to actually collect related to those supposed to display here or is it this an issue we should look into?

The screenshot you shared show transaction metrics. I would assume that because it's a logstash JVM, there are no transactions because it uses something different than the supported frameworks.

Could you go to the Metrics tab within your service at the top and check whether there are any JVM metrics showing up?

[vstol]

Hi,

sorry for the delayed response.

Yes, I can confirm there are JVM metrics visible under Metrics tab.

Please let me know if you need any further details or any other assistance.

Kind regards,
Vladimir