Filebeat config for nginx logs


(Aarish Ramesh) #1

I am trying to forward nginx access & error logs from nginx as different indices to logstash. Below is the filebeat.yml config

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/access.log
  document_type: nginx-access

  paths:
    - /var/log/nginx/error.log
  document_type: nginx-error

But on starting ngnix, I see only harvestor getting started for error logs only & not for access logs. Can someone help me with the issue ?


Filebeat config for nginx access & error logs
(Carlos Pérez Aradros) #2

Hi @aarishramesh,

You are defining only 1 prospector, when you write the second paths field, the previous one gets overriden. Check https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-filebeat-options.html for a good example.

In your case I would do something like:

- input_type: log
  paths:
    - /var/log/nginx/access.log
  document_type: nginx-access

- input_type: log
  paths:
    - /var/log/nginx/error.log
  document_type: nginx-error

Let me know if that works for you!


(Aarish Ramesh) #3

Hi @exekias,

Thanks for the response.

I tried doing it but it does not work

This is my logs on starting filebeat

2017/05/05 10:37:14.567204 publish.go:295: INFO Publisher name: ip-172-31-1-152
2017/05/05 10:37:14.567339 async.go:63: INFO Flush Interval set to: 1s
2017/05/05 10:37:14.567372 async.go:64: INFO Max Bulk Size set to: 2048
2017/05/05 10:37:14.567474 modules.go:93: ERR Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2017/05/05 10:37:14.567536 beat.go:221: INFO filebeat start running.
2017/05/05 10:37:14.567580 registrar.go:85: INFO Registry file set to: /usr/share/filebeat/bin/data/registry
2017/05/05 10:37:14.567628 registrar.go:106: INFO Loading registrar data from /usr/share/filebeat/bin/data/registry
2017/05/05 10:37:14.567772 registrar.go:123: INFO States Loaded from registrar: 2
2017/05/05 10:37:14.567820 crawler.go:38: INFO Loading Prospectors: 1
2017/05/05 10:37:14.567920 prospector_log.go:65: INFO Prospector with previous states loaded: 1
2017/05/05 10:37:14.568016 prospector.go:124: INFO Starting prospector of type: log; id: 13600542689130228667
2017/05/05 10:37:14.568063 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017/05/05 10:37:14.568159 metrics.go:23: INFO Metrics logging every 30s
2017/05/05 10:37:14.568239 registrar.go:236: INFO Starting Registrar
2017/05/05 10:37:14.568293 sync.go:41: INFO Start sending events to output
2017/05/05 10:37:14.568341 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/05/05 10:48:37.960489 sync.go:41: INFO Start sending events to output
2017/05/05 10:48:37.960534 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/05/05 10:48:37.960731 log.go:91: INFO Harvester started for file: /var/log/nginx/access.log

I don't see any harvestor getting started for error.log. Can you please check this ?

Also I see two processes getting started for filebeat when i execute

sudo /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml &

root     13920  0.0  0.2  67992  2184 pts/2    S    10:37   0:00 sudo 
/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml
root     13921  0.0  0.4  12524  4976 pts/2    Sl   10:37   0:00 /usr/share/filebeat/bin/filebeat -e 
-c /etc/filebeat/filebeat.yml

(Carlos Pérez Aradros) #4

Can you please share your full filebeat.yml?


(Aarish Ramesh) #5

@exekias Here is the full filebeat.yml

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:
    - /var/log/nginx/access.log
    document_type: nginx-access

  • input_type: log
    paths:

    • /var/log/nginx/error.log
      document_type: nginx-error

** #- c:\programdata\elasticsearch\logs***

** # Exclude lines. A list of regular expressions to match. It drops the lines that are**
** # matching any regular expression from the list.**
** #exclude_lines: ["^DBG"]**

** # Include lines. A list of regular expressions to match. It exports the lines that are**
** # matching any regular expression from the list.**
** #include_lines: ["^ERR", "^WARN"]**

** # Exclude files. A list of regular expressions to match. Filebeat drops the files that**
** # are matching any regular expression from the list. By default, no files are dropped.**
** #exclude_files: [".gz$"]**

** # Optional additional fields. These field can be freely picked**
** # to add additional information to the crawled log files for filtering**
** #fields:**
** # level: debug**
** # review: 1**

** ### Multiline options**

** # Mutiline can be used for log messages spanning multiple lines. This is common**
** # for Java Stack Traces or C-Line Continuation**

** # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [**
** #multiline.pattern: ^[**

** # Defines if the pattern set under pattern should be negated or not. Default is false.**
** #multiline.negate: false**

** # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern**
** # that was (not) matched before or after or as long as a pattern is not matched based on negate.**
** # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash**
** #multiline.match: after**

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
** # Array of hosts to connect to.**
** #hosts: ["localhost:9200"]**

** # Optional protocol and basic auth credentials.**
** #protocol: "https"**
** #username: "elastic"**
** #password: "changeme"**

#----------------------------- Logstash output --------------------------------
output.logstash:
** # The Logstash hosts**
** hosts: ["172.31.1.109:5044"]**

** # Optional SSL. By default is off.**
** # List of root certificates for HTTPS server verifications**
** #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]**

** # Certificate for SSL client authentication**
** #ssl.certificate: "/etc/pki/client/cert.pem"**

** # Client Certificate Key**
** #ssl.key: "/etc/pki/client/cert.key"**

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]


(Robin ) #6

sorry, but you don't have any output configured...


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.