# APM UI no loading data using kafka as apm server output

**URL:** <https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446>\
**Category:** APM\
**Created:** [February 21, 2019, 3:21pm UTC](https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446 "2019-02-21T15:21:26Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cmcornejocrespo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmcornejocrespo/32/37946_2.png) [@cmcornejocrespo](https://discuss.elastic.co/u/cmcornejocrespo)\
**Post date:** [February 21, 2019, 3:21pm UTC](https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446/1 "2019-02-21T15:21:27Z")

</div>

Hi folks,

Start off saying that my setup was working perfectly when I use elasticsearch as the apm server output.

**The problem**

I'm using apm version 6.6.0 and I'm trying to configure apm server to use kafka as output. I'm basically trying to follow this [post](https://www.elastic.co/blog/how-to-send-data-through-logstash-or-kafka-from-elastic-apm#introduce-kafka) but it did't work for me.

Let's review my config:

**apm-server.yml**

```
setup:
  kibana:
    host: "http://kibana:5601"
  dashboards:
    enabled: true
   ...
output.kafka:
  hosts: ["kafka:9092"]
  topics:
  - topic: 'apm-%{[processor.event]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

```

**logstash**

```
 input {
  ...
  kafka {
    bootstrap_servers => "kafka:9092"
    topics_pattern => "apm.*"
    codec => "json"
    tags => ["apm"]
    type => "log"
    client_id => "apm-server"
  }
} 

output {
  if [tags] == "apm" {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "apm-%{[beat][version]}-%{[processor][event]}-%{+YYYY.MM.dd}"
    }
  }
  else {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

```

**kibana.yml**  
Note: I'm using this reference to override indexes found [here](https://discuss.elastic.co/t/kibaba-apm-ui-setup-no-services-available/159919/2)

```
   server:
     name: kibana
     host: "0"
   elasticsearch:
     hosts: http://elasticsearch:9200
   apm:
      errorIndices: apm-*
      spanIndices: apm-*
      transactionIndices: apm-*
      onboardingIndices: apm-*

```

After I deploy the stack I run a k8s job to load the templates:

```
...
containers:
  - name: create-apm-template
    image: docker.elastic.co/apm/apm-server:6.6.0
    command: [ "apm-server", "setup", "--template",
      "-E", "output.kafka.enabled=false",
      "-E", 'output.elasticsearch.hosts=["elasticsearch.blah:9200"]' ]
  - name: force-kibana-new-documents
    image: "appropriate/curl"
    command: ["curl", "-XDELETE", "elasticsearch.blah:9200/apm-server-*"]
...

```

I go and check that the template is properly created:

```
GET _template/apm*

```

output [here](https://gist.github.com/cmcornejocrespo/b0c8b0c7fa31e78e3a1ce1d1dc2f9abe)

As I have metrics beats enabled for kafka I check that topics are successfully created

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/4/e4af0ca2aac0bb150e4b85662b4829af6f9c1f5b.png)

Also I see the entries in discovery

[snippet](https://gist.github.com/cmcornejocrespo/97f466616306fc563572b3f0528f398c)

I had a similar problem described in this [post](https://discuss.elastic.co/t/no-services-were-found/158722/3) (kudos for [sqren](https://discuss.elastic.co/u/sqren)). I'm trying to tail down the problem following the same steps. I can see the same errors in kibana [here](https://gist.github.com/cmcornejocrespo/f42949bfcff6e97dff0fd7ce1cd6b233) after loading the list of services, but in this time I can see the index as I loaded it manually as described above.

Any tips what's happing here?

Thanks!  
Carlos M.

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [February 21, 2019, 4:40pm UTC](https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446/2 "2019-02-21T16:40:44Z")

</div>

Hi Carlos,

it looks like your issue is related to a [bug](https://github.com/elastic/apm-server/issues/1792) we are working on fixing atm.

The APM Server template you loaded is matching against `apm-6.6.0` indices, but the actual created indices via logstash are `apm-server-6.6.0`.  
If you change the index pattern in the template to `apm-server-6.6.0` it should be properly applied to newly created indices. You can change the pattern by setting `setup.template.pattern="apm-%{[beat.version]}-*"` in your apm-server.yml.

---

<div class="post-metadata">

**Author:** ![cmcornejocrespo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmcornejocrespo/32/37946_2.png) [@cmcornejocrespo](https://discuss.elastic.co/u/cmcornejocrespo)\
**Post date:** [February 22, 2019, 11:03am UTC](https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446/3 "2019-02-22T11:03:38Z")

</div>

Hi Silvia,

Thanks for the reply.

I did try to change the template pattern to match the index created via logstash but I still getting the same error and it did not create the index as expected (to match the index created in elasticsearch apm-server-\*)

```
setup:
  kibana:
    host: "http://kibana:5601"
  dashboards:
    enabled: true
  template:
    pattern: "apm-server-%{[beat.version]}-*"
    overwrite: true

```

I also try the other way around, by changing the index name created via logstash to be apm-\*, but it didn´t work too (which bring another question... why is not changing the index name??)

```
output {

  if [tags] == "apm" {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+YYYY.MM.dd}"
    }
  }
  else {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}

```

A bit confused now, any tips?

Many thanks,  
Carlos M.

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [February 26, 2019, 2:14pm UTC](https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446/4 "2019-02-26T14:14:56Z")

</div>

Actually I was mistaken about the template loading, as this requires to have the ES output configured. You can [manually load the template](https://www.elastic.co/guide/en/apm/server/current/_manually_loading_template_configuration.html#load-template-manually) though.

If you want to change the index name instead, you can set [`logstash.index: 'apm'`](https://www.elastic.co/guide/en/apm/server/current/logstash-output.html#_accessing_metadata_fields) in the apm-server.yml.

---

<div class="post-metadata">

**Author:** ![cmcornejocrespo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmcornejocrespo/32/37946_2.png) [@cmcornejocrespo](https://discuss.elastic.co/u/cmcornejocrespo)\
**Post date:** [February 28, 2019, 5:47pm UTC](https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446/5 "2019-02-28T17:47:36Z")

</div>

Hi Silvia,

Thanks for the tips. I ended up adding extra settings to the index and added it manually as you [described](https://www.elastic.co/guide/en/apm/server/current/_manually_loading_template_configuration.html#load-template-manually). I also made some changes to my logstash output setup.

To clarify, let me share all the configuration that I originally posted that is working for me after the changes.

**apm-server.yml**

```
... 
setup:
  kibana:
    ...
  dashboards:
    enabled: true
    index: "apm-server-*"
  template:
    enabled: false
  ...
  output.kafka:
  hosts: ["kafka:9092"]
  topics:
  - topic: 'apm-%{[processor.event]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
  ...

```

**logstash-input**

```
...
input {
  kafka {
    bootstrap_servers => "kafka:9092"
    topics_pattern => "apm.*"
    codec => "json"
    client_id => "apm-server"
  }
}
...

```

**logstash-output**

```
output {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
}

```

and finally the k8s job that loads the elasticsearch index manually:

```
...
name: create-apm-template
    image: docker.elastic.co/apm/apm-server:6.6.1
    command: [ "apm-server", "setup", "--template",
      "-E", "output.kafka.enabled=false",
      "-E", 'setup.template.name="apm-server"',
      "-E", 'setup.template.pattern="apm-server-*"',
      "-E", 'output.elasticsearch.hosts=["elasticsearch.blah.svc.cluster.local:9200"]']
...

```

Feel free to close the topic.

Thanks again Silvia!

Regards,  
Carlos M.

---

<div class="post-metadata">

**Author:** ![simitt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simitt/32/106406_2.png) [@simitt](https://discuss.elastic.co/u/simitt)\
**Post date:** [March 1, 2019, 7:49am UTC](https://discuss.elastic.co/t/apm-ui-no-loading-data-using-kafka-as-apm-server-output/169446/6 "2019-03-01T07:49:14Z")

</div>