Nginx logs in Kibana log streaming!

[sv_bcvt92 (Cuong Vu)]

Hi Guru's,
I'm setting up Laravel with Nginx.
And I expose Laravel access logs to Nginx logs as config below:

server {
        listen 80;
        listen [::]:80 ipv6only=on;
        server_name abcyxyz.ap-northeast-1.elb.amazonaws.com;
	
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        root /usr/share/nginx/laravel/public;
        index index.php index.html index.htm;

        location / {
                try_files $uri $uri/ /index.php?$query_string;
        }

        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass unix:/run/php-fpm/www.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }
}

And in /var/log/nginx/access.log I get:

But in Logs streaming I get:

As you can see in access log, logs are same format, but in Kibana, it's missing nginx.access in many records. Can anyone help me in this issue?

I need to query logs as http response code

Thanks

[kvch (Noémi Ványi)]

What's your NGINX version? Could you please share your Filebeat configuration and format it using </>?

[sv_bcvt92 (Cuong Vu)]

Hi,
After investigations, I realize that every record is duplicated. Now I cant explain why it is duplicated
My filebeat.yml
###################### Filebeat Configuration Example #########################
#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/*.log
 
#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  host: "abcxyz.compute.amazonaws.com:5601"

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["abcxyz.compute.amazonaws.com:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~


#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

[sv_bcvt92 (Cuong Vu)]

It's so weird!
I setup Filebeat in another Docker container with the same configurations (nginx, filebeat...), but logs aren't formated in Kibana

Just logs from 10.1.x.x are formated, 10.2.x.x are not

[kvch (Noémi Ványi)]

What is your NGINX version? I think you are trying to parse logs whose format is unsupported.
Our modules were tested with NGINX 1.10: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html