Hey Everyone,
I'm having some issues trying to get APM logs to Elasticsearch going through Logstash.
Since it's a POC I'm using the Logstash JVM to test it out.
ELK version is 8.8.1, that includes the APM server. APM agent is LATEST which is 1.40.0 atm.
Prior to setting up the APM server and agent I installed the APM integration so it sets up Elasticsearch assets:
POST kbn:/api/fleet/epm/packages/apm/8.8.1
{ "force": true }
/etc/logstash/jvm.options
# Elastic APM
-javaagent:/opt/elastic/elastic-apm-agent.jar
-Delastic.apm.service_name=my-cool-service
-Delastic.apm.application_packages=org.example,org.another.example
-Delastic.apm.server_url=https://kibana:8200
-Delastic.apm.verify_server_cert=false
-Djavax.net.ssl.keyStore=/etc/logstash/certs/cert.pkcs12
-Djavax.net.ssl.keyStoreType=pkcs12
-Djavax.net.ssl.keyStorePassword=changeme
...
This is the Logstash configuration:
input {
elastic_agent {
port => 5044
ssl => true
ssl_certificate => "/etc/logstash/certs/cert.crt"
ssl_key => "/etc/logstash/certs/cert.key.pkcs8"
ssl_verify_mode => "none"
enrich => none
type => "elastic-agent"
}
}
filter {
if [type] == "elastic-agent" {
mutate {
rename => {
"[data_stream.type]" => "[data_stream][type]"
"[data_stream.dataset]" => "[data_stream][dataset]"
"[data_stream.namespace]" => "[data_stream][namespace]"
}
}
}
}
output {
if [type] == "elastic-agent" {
elasticsearch {
hosts => 'https://elk:9200'
data_stream => true
data_stream_auto_routing => "true"
ssl => true
cacert => "/etc/logstash/certs/ca.crt"
user => "elastic"
password => "password"
manage_template => false
action => "create"
}
}
}
Example error message is below (with stripped sensitive info):
[2023-07-28T11:58:16,154][WARN ][logstash.outputs.elasticsearch][main][21e1bb34aa5eb69fb9c83db3ff2aa8b820594d22795298305b910c20a2e7d0bc] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"metrics-apm.internal-default", :routing=>nil}, {"agent"=>{"name"=>"java", "ephemeral_id"=>"d749e386-2c73-4a7b-91c4-deb6ac6270df", "version"=>"1.40.0", "activation_method"=>"javaagent-flag"}, "observer"=>{"version"=>"8.8.1", "type"=>"apm-server", "hostname"=>"..."}, "processor"=>{"event"=>"metric", "name"=>"metric"}, "data_stream"=>{"type"=>"metrics", "namespace"=>"default", "dataset"=>"apm.internal"}, "_metric_descriptions"=>{"jvm.memory.heap.pool.committed"=>{}, "jvm.memory.heap.pool.used"=>{}}, "@timestamp"=>2023-07-28T11:58:05.006Z, "metricset.name"=>"app", "service"=>{"runtime"=>{"name"=>"Java", "version"=>"17.0.7"}, "name"=>"my-cool-service", "language"=>{"name"=>"Java", "version"=>"17.0.7"}, "node"=>{"name"=>"..."}}, "labels"=>{"name"=>"G1 Eden Space"}, "@version"=>"1", "jvm.memory.heap.pool.committed"=>49283072.0, "type"=>"elastic-agent", "process"=>{"title"=>"/usr/share/logstash/jdk/bin/java", "parent"=>{"pid"=>1}, "pid"=>4029}, "jvm.memory.heap.pool.used"=>13631488.0, "host"=>{"name"=>"...", "os"=>{"platform"=>"Linux"}, "hostname"=>"...", "architecture"=>"amd64", "ip"=>["..."]}}], :response=>{"create"=>{"_index"=>".ds-metrics-apm.internal-default-2023.07.28-000001", "_id"=>"nPJdnIkBy9LjXIlJIJYx", "status"=>400, "error"=>{"type"=>"strict_dynamic_mapping_exception", "reason"=>"[1:228] mapping set to strict, dynamic introduction of [type] within [_doc] is not allowed"}}}}
Anyone got any idea what's going on here and tips to fix it? if it's a bug I'll opet it on Github, that is also good information.
Thanks for any help in avdance.
Cheers,
Luka