Elastic Common Schema Fields for Reverse Proxies

Elastic Observability Logs
1

Hello,

we are currently migrating our NGINX logging format to the Elastic Common Schema. For some NGINX logging values I cannot find any fitting fields. In our case we use NGINX as a reverse proxy in Kubernetes. Most of the values we cannot match to ECS concern values about the upstream destination. Here are some examples where $value is the NGINX logging value:

  • $upstream_status: Its the status code received from the upstream server. I guess in most cases this will be the same as http.response.status_code. However, we would like to give it a separate field. Intuitively I would use a custom field like destination.status for that.
  • $upstream_response_time: It's the response time of the upstream server. Add this time to the time needed for NGINX to answer and you get event.duration. However, we would like to have this as a seperate field.
  • $upstream_connect_time: It's the time taken to establish the connection to upstream server.
  • $upstream_header_time: It's the time taken to receive the HTTP header of the upstream server's response.

Do I not see any matching fields? Are there plans to introduce new fields like this? How do you name custom fields when there is a fitting category like e.g. destination which we can use.

Best regards,
Lucas

2

@learnerbyheart
Hi Lucas, I'm don't think we have anything today that would be a good match, but thats what we have custom fields for!

Theres also a thread on github re: observer fields for proxies, load balancers, etc. that I need to get on adding in as an ECS RFC, you can check out the discussion here: [meta] Add support for proxies in ECS · Issue #938 · elastic/ecs · GitHub

The original source was F5 logs, but covering Nginx (and other solutions) is definitely a target. If you get a chance can you check the last post in that thread and see if you think it covers the requirements?

In the meantime you can always add custom nginx fields for the items not in ECS (you can customize the ECS fields as well, but we do recommend against that in case the final revisions change)

Thanks
/d

Dain Perkins

3

Hi @learnerbyheart - welcome to the community!

Like @Dain.Perkins mentioned, this is something we'd like to see defined formally in ECS, with the necessary fields along with usage guidance.

Here are the guidelines and best practices for creating custom fields, but a brief overview of the recommendations to add custom fields and avoid future ECS conflicts:

  • Add a custom top-level field to nest your custom fields; ECS avoids proper names as a convention. The proper name could be a company, project, or brand name: acme.nginx.upstream_status.
  • Another option to avoid a conflict is to use capitalization for your custom field names. ECS limits field naming to lowercase and underscores, so a custom field such as destination.UpstreamStatus is very safe from every colliding with a future ECS change.
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.