# Filebeat config for nginx logs

**URL:** <https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 4, 2017, 11:21am UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540 "2017-05-04T11:21:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![aarishramesh](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@aarishramesh](https://discuss.elastic.co/u/aarishramesh)\
**Post date:** [May 4, 2017, 11:21am UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540/1 "2017-05-04T11:21:55Z")

</div>

I am trying to forward nginx access & error logs from nginx as different indices to logstash. Below is the filebeat.yml config

```
filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/access.log
  document_type: nginx-access

  paths:
    - /var/log/nginx/error.log
  document_type: nginx-error

```

But on starting ngnix, I see only harvestor getting started for error logs only & not for access logs. Can someone help me with the issue ?

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [May 5, 2017, 10:11am UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540/2 "2017-05-05T10:11:44Z")

</div>

Hi @aarishramesh,

You are defining only 1 prospector, when you write the second `paths` field, the previous one gets overriden. Check [https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-filebeat-options.html](https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-filebeat-options.html) for a good example.

In your case I would do something like:

```
- input_type: log
  paths:
    - /var/log/nginx/access.log
  document_type: nginx-access

- input_type: log
  paths:
    - /var/log/nginx/error.log
  document_type: nginx-error

```

Let me know if that works for you!

---

<div class="post-metadata">

**Author:** ![aarishramesh](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@aarishramesh](https://discuss.elastic.co/u/aarishramesh)\
**Post date:** [May 5, 2017, 10:42am UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540/3 "2017-05-05T10:42:02Z")

</div>

Hi @exekias,

Thanks for the response.

I tried doing it but it does not work

This is my logs on starting filebeat

2017/05/05 10:37:14.567204 publish.go:295: INFO Publisher name: ip-172-31-1-152  
2017/05/05 10:37:14.567339 async.go:63: INFO Flush Interval set to: 1s  
2017/05/05 10:37:14.567372 async.go:64: INFO Max Bulk Size set to: 2048  
2017/05/05 10:37:14.567474 modules.go:93: ERR Not loading modules. Module directory not found: /usr/share/filebeat/bin/module  
2017/05/05 10:37:14.567536 beat.go:221: INFO filebeat start running.  
2017/05/05 10:37:14.567580 registrar.go:85: INFO Registry file set to: /usr/share/filebeat/bin/data/registry  
2017/05/05 10:37:14.567628 registrar.go:106: INFO Loading registrar data from /usr/share/filebeat/bin/data/registry  
2017/05/05 10:37:14.567772 registrar.go:123: INFO States Loaded from registrar: 2  
2017/05/05 10:37:14.567820 crawler.go:38: INFO Loading Prospectors: 1  
2017/05/05 10:37:14.567920 prospector\_log.go:65: INFO Prospector with previous states loaded: 1  
2017/05/05 10:37:14.568016 prospector.go:124: INFO Starting prospector of type: log; id: 13600542689130228667  
2017/05/05 10:37:14.568063 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1  
2017/05/05 10:37:14.568159 metrics.go:23: INFO Metrics logging every 30s  
2017/05/05 10:37:14.568239 registrar.go:236: INFO Starting Registrar  
2017/05/05 10:37:14.568293 sync.go:41: INFO Start sending events to output  
2017/05/05 10:37:14.568341 spooler.go:63: INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
2017/05/05 10:48:37.960489 sync.go:41: INFO Start sending events to output  
2017/05/05 10:48:37.960534 spooler.go:63: INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
2017/05/05 10:48:37.960731 log.go:91: INFO Harvester started for file: /var/log/nginx/access.log

I don't see any harvestor getting started for error.log. Can you please check this ?

Also I see two processes getting started for filebeat when i execute

```
sudo /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml &

root 13920 0.0 0.2 67992 2184 pts/2 S 10:37 0:00 sudo 
/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml
root 13921 0.0 0.4 12524 4976 pts/2 Sl 10:37 0:00 /usr/share/filebeat/bin/filebeat -e 
-c /etc/filebeat/filebeat.yml
```

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [May 5, 2017, 10:56am UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540/4 "2017-05-05T10:56:43Z")

</div>

Can you please share your full `filebeat.yml`?

---

<div class="post-metadata">

**Author:** ![aarishramesh](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@aarishramesh](https://discuss.elastic.co/u/aarishramesh)\
**Post date:** [May 5, 2017, 10:59am UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540/5 "2017-05-05T10:59:07Z")

</div>

@exekias Here is the full filebeat.yml

**filebeat.prospectors:**

**# Each - is a prospector. Most options can be set at the prospector level, so**  
**# you can use different prospectors for various configurations.**  
**# Below are the prospector specific configurations.**

- input\_type: log

- input\_type: log  
paths:

\*\* #- c:\programdata\elasticsearch\logs\*\*\*

\*\* # Exclude lines. A list of regular expressions to match. It drops the lines that are\*\*  
\*\* # matching any regular expression from the list.\*\*  
\*\* #exclude\_lines: ["^DBG"]\*\*

\*\* # Include lines. A list of regular expressions to match. It exports the lines that are\*\*  
\*\* # matching any regular expression from the list.\*\*  
\*\* #include\_lines: ["^ERR", "^WARN"]\*\*

\*\* # Exclude files. A list of regular expressions to match. Filebeat drops the files that\*\*  
\*\* # are matching any regular expression from the list. By default, no files are dropped.\*\*  
\*\* #exclude\_files: [".gz$"]\*\*

\*\* # Optional additional fields. These field can be freely picked\*\*  
\*\* # to add additional information to the crawled log files for filtering\*\*  
\*\* #fields:\*\*  
\*\* # level: debug\*\*  
\*\* # review: 1\*\*

\*\* ### Multiline options\*\*

\*\* # Mutiline can be used for log messages spanning multiple lines. This is common\*\*  
\*\* # for Java Stack Traces or C-Line Continuation\*\*

\*\* # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [\*\*  
\*\* #multiline.pattern: ^[\*\*

\*\* # Defines if the pattern set under pattern should be negated or not. Default is false.\*\*  
\*\* #multiline.negate: false\*\*

\*\* # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern\*\*  
\*\* # that was (not) matched before or after or as long as a pattern is not matched based on negate.\*\*  
\*\* # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash\*\*  
\*\* #multiline.match: after\*\*

**#================================ General =====================================**

**# The name of the shipper that publishes the network data. It can be used to group**  
**# all the transactions sent by a single shipper in the web interface.**  
**#name:**

**# The tags of the shipper are included in their own field with each**  
**# transaction published.**  
**#tags: ["service-X", "web-tier"]**

**# Optional fields that you can specify to add additional information to the**  
**# output.**  
**#fields:**  
**# env: staging**

**#================================ Outputs =====================================**

**# Configure what outputs to use when sending the data collected by the beat.**  
**# Multiple outputs may be used.**

**#-------------------------- Elasticsearch output ------------------------------**  
**#output.elasticsearch:**  
\*\* # Array of hosts to connect to.\*\*  
\*\* #hosts: ["localhost:9200"]\*\*

\*\* # Optional protocol and basic auth credentials.\*\*  
\*\* #protocol: "https"\*\*  
\*\* #username: "elastic"\*\*  
\*\* #password: "changeme"\*\*

**#----------------------------- Logstash output --------------------------------**  
**output.logstash:**  
\*\* # The Logstash hosts\*\*  
\*\* hosts: ["172.31.1.109:5044"]\*\*

\*\* # Optional SSL. By default is off.\*\*  
\*\* # List of root certificates for HTTPS server verifications\*\*  
\*\* #ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"]\*\*

\*\* # Certificate for SSL client authentication\*\*  
\*\* #ssl.certificate: "/etc/pki/client/cert.pem"\*\*

\*\* # Client Certificate Key\*\*  
\*\* #ssl.key: "/etc/pki/client/cert.key"\*\*

**#================================ Logging =====================================**

**# Sets log level. The default log level is info.**  
**# Available log levels are: critical, error, warning, info, debug**  
**#logging.level: debug**

**# At debug level, you can selectively enable logging only for some components.**  
**# To enable all selectors use ["\*"]. Examples of other selectors are "beat",**  
**# "publish", "service".**  
**#logging.selectors: ["\*"]**

---

<div class="post-metadata">

**Author:** ![Globule](https://avatars.discourse-cdn.com/v4/letter/g/8491ac/32.png) [@Globule](https://discuss.elastic.co/u/Globule)\
**Post date:** [May 15, 2017, 3:15pm UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540/6 "2017-05-15T15:15:35Z")

</div>

sorry, but you don't have any output configured...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2017, 3:25pm UTC](https://discuss.elastic.co/t/filebeat-config-for-nginx-logs/84540/7 "2017-06-12T15:25:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.