Issue with APM Server through Logstash

[digitalron (Ronald Tumulak)]

Hi, hope you can help me a bit on this one.

We have an Elastic APM cluster set up with custom dashboards and ingest pipelines on our Elasticsearch. All works good.

Now, we'd like to replicate the Elastic APM clusters on other data centers , but configure those to send data to our existing Elasticsearch cluster. We are thinking of configuring those APM clusters to send to Logstash first and have Logstash send to our Elasticsearch. That way, we can have some semblance of queue management in case APM traffic spikes.

Is it possible to configure Logstash as basically a pass-through/decorator? That way, we can take advantage of existing ingest rules in our Elasticsearch.

My attempts are creating errors:

Also, nothing appears in the Services tab of the APM page, though Traces appear in the Traces tab. The waterfall for the traces also appear. I've done everything from deleting, refreshing, flushing, refreshing indices and index patterns in the management page and re-running the APM setup to reload the Kibana objects.

My logstash.conf is below:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "apm-%{[@metadata][version]}-%{[processor][event]}-%{+YYYY.MM.dd}" 
  }
}

Logstash output configuration in apm-server.yml

output.logstash:
  hosts: ["logstash:5044"]
  worker: 2
  compression_level: 3
  escape_html: true
  pipelining: 2
  backoff.init: 1s
  backoff.max: 60s

[digitalron (Ronald Tumulak)]

Here are some additional info to help understand out intent.

In the diagram below, DATA CENTER 1 exists and is working. We have two other DCs and we want to put APM clusters there too. However, to ensure request resiliency, we want to send APM data from DCs 2 and 3 to their respective Logstash cluster first.

In our tests, putting Logstash in between the APM servers and Elasticsearch causes two things:

1. If the first APM transaction comes through Logstash, the transaction schema goes awry, so that nothing appears in the Services view, although a trace appeara in the Trace view. But the Trace view itself is also wrong as it shows zero transactions but only one trace even if multiple traces were actually sent. If the first transaction comes straight from an APM server, no problem occurs on the APM page.

2. Pipeline rules in Elasticsearch are not executed for APM data coming from Logstash

We'd really like to use Logstash as a pass-through proxy to avoid having to recreate pipeline ingest rules in multiple places. In theory, we should just put those in the ES cluster and work.

We are not keen to use Kafka to manage the request queue as it is not a core technology for us; we'd like to minimise the number of stacks we want to support. Besides, we see Logstash as our future standard log aggregator and become part of our Observability stack.

[digitalron (Ronald Tumulak)]

Nevermind. Went through the entire documentation and found I missed two config settings:

1. Manual loading of APM index template
2. Pipeline definition in logstash.conf <-- big "Doh!"

All is good now. This can be closed.