# Nginx logs with filebeat and parsing with pipeline

**URL:** https://discuss.elastic.co/t/nginx-logs-with-filebeat-and-parsing-with-pipeline/370413
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [November 12, 2024, 3:21pm UTC](https://discuss.elastic.co/t/nginx-logs-with-filebeat-and-parsing-with-pipeline/370413 "2024-11-12T15:21:49Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Enol](https://avatars.discourse-cdn.com/v4/letter/e/858c86/32.png) [@Enol](https://discuss.elastic.co/u/Enol)
#### Post date: [November 12, 2024, 3:21pm UTC](https://discuss.elastic.co/t/nginx-logs-with-filebeat-and-parsing-with-pipeline/370413/1 "2024-11-12T15:21:49Z")

</div>

Hello,I have filebeat installed in one server that have docker containers running inside. One of those container is a nginx container and I need to parse the log to see it well in elastic.

I have configured filebeat tu use a pipeline with this:

```auto
 pipelines:
   - pipeline: "filebeat-8.6.1-nginx-access-pipeline"
     when.contains:
        container.labels.com_docker_swarm_service_name: "nginx"

```

And I have created in the pipeline, the grok pattern:

```auto
"%{IP:client_ip} - - \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:method} %{URIPATHPARAM:request}(?:\\?%{DATA:query_params})? %{DATA:http_version})\" %{NUMBER:status_code} %{NUMBER:response_size} \"(?:%{DATA:referrer})\" \"(?:%{DATA:user_agent})\" \"%{IP:forwarded_ip}\" \\[%{WORD:cache_status}\\] %{NUMBER:response_time:float}"

```

If I try the pattern in the grok debbuger in Dev Tools in kibana, I get a success :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/d/ed37e4a9d1dd03c62dc3e1c0bc9ea9ac14e76a9a.png)

with this example trace:

```auto

10.0.1.36 - - [12/Nov/2024:14:58:14 +0000] "GET /actuator/prometheus HTTP/1.1" 403 9 "-" "Elastic-Metricbeat/8.6.1 (linux; amd64; 14f2f8d585f8c380945feee789771bd782cd6b2d; 2023-01-24 13:30:23 +0000 UTC)" "172.18.0.1" [-] 0.083

```

But in discover, when I look for some traces, I see there are grok pattern error:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/8/68eec757c88298bda628beb0c37adfb01a09a681.png)

Why if grok debugger sais Its ok, the traces are with error?

---

<div class="post-metadata">

### Author: ![Ljapunov](https://avatars.discourse-cdn.com/v4/letter/l/f4b2a3/32.png) [@Ljapunov](https://discuss.elastic.co/u/Ljapunov)
#### Post date: [November 13, 2024, 12:52pm UTC](https://discuss.elastic.co/t/nginx-logs-with-filebeat-and-parsing-with-pipeline/370413/2 "2024-11-13T12:52:00Z")

</div>

Please post the complete logstash config of the failing pipeline including your grok-pattern.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 13, 2024, 2:32pm UTC](https://discuss.elastic.co/t/nginx-logs-with-filebeat-and-parsing-with-pipeline/370413/3 "2024-11-13T14:32:00Z")

</div>

Perhaps test with the `_simulate` API in the Kibana Dev Tools it's typically the best way to debug.

> **[Simulate pipeline API | Elasticsearch Guide \[8.16\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html)**

Curious why you're not trying the nginx integration?

> **[Nginx module | Filebeat Reference \[8.16\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html)**

---

<div class="post-metadata">

### Author: ![Enol](https://avatars.discourse-cdn.com/v4/letter/e/858c86/32.png) [@Enol](https://discuss.elastic.co/u/Enol)
#### Post date: [November 21, 2024, 10:22am UTC](https://discuss.elastic.co/t/nginx-logs-with-filebeat-and-parsing-with-pipeline/370413/4 "2024-11-21T10:22:21Z")

</div>

Thanks, I could fix the error, I have changed the log format of the nginx and now works. Thanks

---

<div class="post-metadata">

### Author: ![Enol](https://avatars.discourse-cdn.com/v4/letter/e/858c86/32.png) [@Enol](https://discuss.elastic.co/u/Enol)
#### Post date: [November 21, 2024, 10:24am UTC](https://discuss.elastic.co/t/nginx-logs-with-filebeat-and-parsing-with-pipeline/370413/5 "2024-11-21T10:24:12Z")

</div>

Hi, thats because the nginx is inside a docker container and the logs are being harvested by filebeat