APM UI no loading data using kafka as apm server output

Elastic Observability APM
1

Hi folks,

Start off saying that my setup was working perfectly when I use elasticsearch as the apm server output.

The problem

I'm using apm version 6.6.0 and I'm trying to configure apm server to use kafka as output. I'm basically trying to follow this post but it did't work for me.

Let's review my config:

apm-server.yml

setup:
  kibana:
    host: "http://kibana:5601"
  dashboards:
    enabled: true
   ...
output.kafka:
  hosts: ["kafka:9092"]
  topics:
  - topic: 'apm-%{[processor.event]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

logstash

 input {
  ...
  kafka {
    bootstrap_servers => "kafka:9092"
    topics_pattern => "apm.*"
    codec => "json"
    tags => ["apm"]
    type => "log"
    client_id => "apm-server"
  }
} 

output {
  if [tags] == "apm" {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "apm-%{[beat][version]}-%{[processor][event]}-%{+YYYY.MM.dd}"
    }
  }
  else {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

kibana.yml
Note: I'm using this reference to override indexes found here

   server:
     name: kibana
     host: "0"
   elasticsearch:
     hosts: http://elasticsearch:9200
   apm:
      errorIndices: apm-*
      spanIndices: apm-*
      transactionIndices: apm-*
      onboardingIndices: apm-*

After I deploy the stack I run a k8s job to load the templates:

...
containers:
  - name: create-apm-template
    image: docker.elastic.co/apm/apm-server:6.6.0
    command: [ "apm-server", "setup", "--template",
      "-E", "output.kafka.enabled=false",
      "-E", 'output.elasticsearch.hosts=["elasticsearch.blah:9200"]' ]
  - name: force-kibana-new-documents
    image: "appropriate/curl"
    command: ["curl", "-XDELETE", "elasticsearch.blah:9200/apm-server-*"]
...

I go and check that the template is properly created:

GET _template/apm*

output here

As I have metrics beats enabled for kafka I check that topics are successfully created

image
image.png1094×572 91.5 KB

Also I see the entries in discovery

snippet

I had a similar problem described in this post (kudos for sqren). I'm trying to tail down the problem following the same steps. I can see the same errors in kibana here after loading the list of services, but in this time I can see the index as I loaded it manually as described above.

Any tips what's happing here?

Thanks!
Carlos M.

2

Hi Carlos,

it looks like your issue is related to a bug we are working on fixing atm.

The APM Server template you loaded is matching against apm-6.6.0 indices, but the actual created indices via logstash are apm-server-6.6.0.
If you change the index pattern in the template to apm-server-6.6.0 it should be properly applied to newly created indices. You can change the pattern by setting setup.template.pattern="apm-%{[beat.version]}-*" in your apm-server.yml.

3

Hi Silvia,

Thanks for the reply.

I did try to change the template pattern to match the index created via logstash but I still getting the same error and it did not create the index as expected (to match the index created in elasticsearch apm-server-*)

setup:
  kibana:
    host: "http://kibana:5601"
  dashboards:
    enabled: true
  template:
    pattern: "apm-server-%{[beat.version]}-*"
    overwrite: true

I also try the other way around, by changing the index name created via logstash to be apm-*, but it didn´t work too (which bring another question... why is not changing the index name??)

output {

  if [tags] == "apm" {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+YYYY.MM.dd}"
    }
  }
  else {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

A bit confused now, any tips?

Many thanks,
Carlos M.

4

Actually I was mistaken about the template loading, as this requires to have the ES output configured. You can manually load the template though.

If you want to change the index name instead, you can set logstash.index: 'apm' in the apm-server.yml.

5

Hi Silvia,

Thanks for the tips. I ended up adding extra settings to the index and added it manually as you described. I also made some changes to my logstash output setup.

To clarify, let me share all the configuration that I originally posted that is working for me after the changes.

apm-server.yml

... 
setup:
  kibana:
    ...
  dashboards:
    enabled: true
    index: "apm-server-*"
  template:
    enabled: false
  ...
  output.kafka:
  hosts: ["kafka:9092"]
  topics:
  - topic: 'apm-%{[processor.event]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
  ...

logstash-input

...
input {
  kafka {
    bootstrap_servers => "kafka:9092"
    topics_pattern => "apm.*"
    codec => "json"
    client_id => "apm-server"
  }
}
...

logstash-output

output {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
}

and finally the k8s job that loads the elasticsearch index manually:

...
name: create-apm-template
    image: docker.elastic.co/apm/apm-server:6.6.1
    command: [ "apm-server", "setup", "--template",
      "-E", "output.kafka.enabled=false",
      "-E", 'setup.template.name="apm-server"',
      "-E", 'setup.template.pattern="apm-server-*"',
      "-E", 'output.elasticsearch.hosts=["elasticsearch.blah.svc.cluster.local:9200"]']
...

Feel free to close the topic.

Thanks again Silvia!

Regards,
Carlos M.