Sending APM logs without using APM server

[Rajat_Panwar]

Hi,

We already have an inhouse developed APM tool which works on the same lines as aws X-Ray works. It has same structure of transaction, span etc.
There are many cases for us where the log size of APM can go beyond 4 MB (it contains 3rd part http request and response which we need to save). We have created a logic where we can still relay all these logs with least latency (under 70 ms) and save data in aws S# aswell.
Our current stack uses Redis, Mongodb, Inhouse developed tool, SQS. An its very reliable.

Now we want to take advantage of Elasticsearch APM as well as out of box dashboards and metrics.

My question is that can we bypass the need of APM server to ingest logs from the .net agent and push those to elastic search directly? I understand we will have to modify our log structure to what Elasticsearch can understand.

Thanks,
Rajat Panwar

[Sergey_Kleyman]

Hi @Rajat_Panwar and welcome to our forum!

By "logs from the .net agent" do you refer to .NET agent internal logging, as for example:

[2019-08-20 11:56:43][Trace] - {CurrentExecutionSegmentsContainer.transaction} Getting value... Thread: 1. Current value: (null).
[2019-08-20 11:56:43][Debug] - {HttpDiagnosticListenerImplBase} No current transaction, skip creating span for outgoing HTTP request

If so .NET agent doesn't send those logs to APM Server and I would like to clarify that these logs are for .NET agent's supportability purposes only. The data on which Elastic's APM solution is based is sent by the agents to APM Server via Intake API.

But if you would like to have .NET agent's internal logs in Elasticsearch for some purpose you can push them to Elasticsearch since these logs are just like any other regular logs.

[Rajat_Panwar]

Hi Sergey,

Thanks for your response, Probably I didn't explained myself well. What we are looking to is bypass APM server and push logs (not internal logs, but APM data generated by .NET agent) directly to Elasticsearch. And able to visualise that data in APM viewer in kibana.

Probably sending data directly to elasticsearch in OpenTrace format which we can generate without using elastic .NET APM agent. We have already created internal package which is able to instrument and generate APM data in OpenTracing format.

Thanks again.

[Sergey_Kleyman]

I see what you mean now but unfortunately pushing APM data directly to Elasticsearch is not supported.
The closest you can get to it is to implement a conversion from a format of data that you already generate to the format expected by APM Server's Intake API and send converted data to APM Server.

[axw (Andrew Wilkins)]

@Rajat_Panwar APM data all end up as documents in Elasticsearch, so in theory you could do this. As you said, you'll need to make sure you transform the data to the expected format. You can find some details on this in the docs: https://www.elastic.co/guide/en/apm/server/current/exploring-es-data.html.

There may be some assumptions in the UI based on conventions used by the agents, e.g. the format of our trace/transaction/span IDs. I don't know if this will be a problem - as Sergey said this isn't really a supported scenario, so you would need to experiment.

Finally, by not using the Elastic APM agents you'll miss out on some features like Central Configuration and Breakdown Metrics ("Time spent by span type"), which require agent-level support. That may be OK for you, just wanted to point out that there are features apart from the basic capture of trace data.