Ingesting Nginx access logs in custom log format

sai8274 · June 3, 2024, 11:40am

Hi All,

I had a question in regards to the nginx integration of elastic agent.

We are using a custom format for nginx logs that has some additional metrics, so the grok in default ingest policy that comes with nginx integration doesn't work.

I have built a custom ingest pipeline with custom grok that works, but I'm unable to figure out how to apply it.

If you check the image, there is the default logs-nginx.access-1.17.0 and custom ingest pipeline i.e. logs-nginx.access@custom that I made.

Since the default ingest pipeline cannot be removed and editing it is discouraged, my log parsing fails since it first goes through the default ingest pipeline which doesn't have the required grok.

Does anyone know how to handle this scenario? Can I make it go directly to my custom ingest pipeline instead of the default one?

stephenb · June 3, 2024, 2:37pm

Hi @sai8274 Welcome to the community...

asn Yes this is a bit of a "tricky" one...

What I would do is enable "preserve original event".

Then your custom pipeline can work against event.original which will still be available

Make sure you map all the fields correctly to the ECS field names

and you can clean up the errors etc...

Although not advised you can edit the provided but when an upgrade comes you will need to update again.. again etc...

BTW The latest version of the integration is 1.20.0 I believe

sai8274 · June 3, 2024, 3:52pm

@stephenb Thanks for the response.

It seems there are 2 ways to go about this.

Preserve original event as highlighted in your screenshot
Pro : I can update nginx integration without any worries.
Con : Compute would be double for each entry, as my log would go through 2 ingest pipelines.
Edit managed default ingest pipeline of Nginx integration
Pro : Compute would happen only once.
Con : After each update, I would need to modify the grok to match my custom log format.

Am I correct?