Nginx logs in Kibana log streaming!

Hi Guru's,
I'm setting up Laravel with Nginx.
And I expose Laravel access logs to Nginx logs as config below:

server {
        listen 80;
        listen [::]:80 ipv6only=on;
        server_name abcyxyz.ap-northeast-1.elb.amazonaws.com;
	
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        root /usr/share/nginx/laravel/public;
        index index.php index.html index.htm;

        location / {
                try_files $uri $uri/ /index.php?$query_string;
        }

        location ~ \.php$ {
                try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                fastcgi_pass unix:/run/php-fpm/www.sock;
                fastcgi_index index.php;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
        }
}

And in /var/log/nginx/access.log I get:

But in Logs streaming I get:

As you can see in access log, logs are same format, but in Kibana, it's missing nginx.access in many records. Can anyone help me in this issue?

I need to query logs as http response code

Thanks

What's your NGINX version? Could you please share your Filebeat configuration and format it using </>?

Hi,
After investigations, I realize that every record is duplicated. Now I cant explain why it is duplicated :slight_smile:
My filebeat.yml
###################### Filebeat Configuration Example #########################
#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/*.log
 
#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  host: "abcxyz.compute.amazonaws.com:5601"

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["abcxyz.compute.amazonaws.com:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~


#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

It's so weird!
I setup Filebeat in another Docker container with the same configurations (nginx, filebeat...), but logs aren't formated in Kibana


Just logs from 10.1.x.x are formated, 10.2.x.x are not

What is your NGINX version? I think you are trying to parse logs whose format is unsupported.
Our modules were tested with NGINX 1.10: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html