Monitor Elasticsearch HTTP in Golang [APM]

[RdrgPorto (Rodrigo Porto)]

Kibana version: 7.9.2

Elasticsearch version: 7.9.2

APM Server version: 7.9.2

APM Agent language and version: Golang 1.11.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker image on Kubernetes

Fresh install or upgraded from other version?: Fresh install

Is there anything special in your setup?: APM Server --> Kafka --> Logstash --> Elasticsearch

Hi, everyone

I have been working with APM and Golang in order to monitor HTTP connections to Elasticsearch.

On the code, I have tried to manage output requests by using HTTP wrapped client. Also, sending the method request (GET, POST, and so on) by using golang.org/x/net/context/ctxhttp library. Moreover, I have guaranteed of closing the response properly.

In despite of this configuration and even though the request is being properly processed, APM server does not show any information. In order to solve this issue I have had to open a specific transaction for every call

Here you are my code:

transaction := apm.DefaultTracer.StartTransaction("POST "+queryPath, "request")
defer transaction.End()
transaction.Result = "HTTP 2xx"
transaction.Context.SetLabel("type", "elastic POST")
transaction.Context.SetLabel("uri", config.ElasticsearchURL+queryPath)
ctx := apm.ContextWithTransaction(context.Background(), transaction)
apmRequest := request.WithContext(ctx)
response, err = ctxhttp.Do(ctx, HTTPClient, apmRequest)

Is it right ? Is there another way to configure it?

Thanks in advance,

Rodrigo

[axw (Andrew Wilkins)]

It's correct that you must first create a transaction for a span to be reported, though you don't necessarily need to create a transaction for every Elasticsearch request.

A transaction is something like an incoming HTTP request, or a periodic background task, while a span measures an operation within a transaction, such as a database request or external HTTP request.

If you can tell us a little more about how your application works (What initiates the Elasticsearch requests? Is it an HTTP service? Is it a periodic task?), then we may be able to provide some more guidance.

By the way, you should probably use apmelasticsearch for instrumenting the outgoing Elasticsearch requests, rather than apmhttp.

[RdrgPorto (Rodrigo Porto)]

Hi, @axw

My application executes scheduled requests in order to run queries on Elasticsearch indices.

I can not use "module/apmelasticsearch" because of our implementation is based on "net/http". Moreover, I do not use neither "go-elasticsearch" nor "olivere/elastic". It could be a future improvement but for the moment.

I have also tried to wrap the client like this:

span, ctx := apm.StartSpan(request.Context(), "elastic-request", "custom")
defer span.End()
var HTTPClient = apmhttp.WrapClient(http.DefaultClient)
response, err = HTTPClient.Do(request.WithContext(ctx))

But it does not work (that is, it does not show anything on APM) without creating a transaction.

Is there another easier way to create this transaction ?

Thanks in advance,

Rodrigo

[axw (Andrew Wilkins)]

@RdrgPorto

I can not use " module/apmelasticsearch " because of our implementation is based on " net/http ". Moreover, I do not use neither " go-elasticsearch " nor " olivere/elastic ". It could be a future improvement but for the moment.

It may still be possible, as apmelasticsearch instruments at the net/http level anyway. Anyway, it was just a side note, maybe it's not practical for you.

Is there another easier way to create this transaction ?

I can't say whether or not there's an easier way without seeing code, but if you're scheduling requests in a background goroutine, then most likely the right approach would be to start a transaction when the goroutine receives its signal to do its work. Something like:

go func() {
    for range time.Tick(time.Minute) {
      backgroundJob()
    }
}()

func backgroundJob() {
  tx := apm.DefaultTracer.StartTransaction("background job", "scheduled")
  defer tx.End()
  ctx := apm.ContextWithBackground(context.Background(), tx)
  results, err := queryElasticsearch(ctx)
  ...
}

func queryElasticsearch(ctx context.Context) {
  span, ctx := apm.StartSpan(ctx, "elastic-request", "app")
  defer span.End()

  request := ...
  var HTTPClient = apmhttp.WrapClient(http.DefaultClient)
  response, err = HTTPClient.Do(request.WithContext(ctx))
  ...
}

The transaction relates to the "scheduled job" (whatever that might be), which happens to involve an Elasticsearch request which will be represented as a span. Presumably your scheduled job does something other than just query Elasticsearch, so you might create additional spans representing those other tasks.

[RdrgPorto (Rodrigo Porto)]

Hi, @axw

Thanks a lot for your support .

Regards