Parse logfiles (haproxy/nginx/apache) and send to APM

Elastic Observability APM
1

Hello,

is there a common way or guideline how to convert webserver access log information (like from a haproxy/nginx/apache service) into APM transactions/spans?

Would it be enough to include typical ECS common schema fields like transaction.id and then configure a filebeat to send the data to the apm-* index? Or is it necessary to use APM-specific APIs to push in the data?

Thanks for your help,
Wolfgang

2 Likes
2

Hey @wosc, welcome to the forum!

is there a common way or guideline how to convert webserver access log information (like from a haproxy/nginx/apache service) into APM transactions/spans?

I'm not aware of anything published on this.

Would it be enough to include typical ECS common schema fields like transaction.id and then configure a filebeat to send the data to the apm-* index? Or is it necessary to use APM-specific APIs to push in the data?

Sometimes you might hear us saying that APM data is "just another index". You do not need to go through the APM Server; it's enough that it has the right schema, and lives in the right indices.

You can find some sample Elasticsearch documents in the APM Server docs, which may help you get started: Transactions | APM User Guide [8.11] | Elastic.

Not all of those fields are required. Unfortunately I don't think we have documented which fields are required. At a minimum, you should set the following:

  • trace.id
  • transaction.id
  • transaction.name (value should ideally be something low cardinality, as it will be used for grouping)
  • transaction.type (value should be "request" for HTTP requests)
  • transaction.result (value will be used for grouping, e.g. we group all requests with status code 200-299 with the result "HTTP 2xx")
  • transaction.duration.us (transaction duration in microseconds)
  • host.hostname
  • host.name (may be the same as host.hostname)
  • processor.event (value should be "transaction" for HTTP requests)
  • service.name (e.g. "nginx")

Hope this helps.

1 Like
3

Hi @axw,

thank you, I was hoping you would say something like that! We had tried (rather blindly) to add some APM-looking fields, but couldn't get anything to show up in the APM UI so far. We'll give it another shot using the fields you mentioned, and hopefully report back successfully.

Thanks again,
Wolfgang

1 Like
4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.