Data won't appear in elasticsearch/kibana

Elastic Stack Logstash
1

I need to parse the log files of nginx. But no data appear in elastichsearch/kibana.
Elasticsearch is up and running and receives data from beats like metricbeat, heartbeat.

nginx-pipeline.conf:

    input {
    beats {
        host => "0.0.0.0"
        port => "5043"
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
filter {
    grok {
         match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx]$
         remove_field => "message"
    }
    mutate {
#       rename => { "@timestamp" => "read_timestamp" }
        add_field => { "read_timestamp" => "@timestamp" }
    }
    date {
       match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
       remove_field => "[nginx][access][time]"
    }
    useragent {
       source => "[nginx][access][agent]"
       target => "[nginx][access][user_agent]"
       remove_field => "[nginx][access][agent]"
    }
    geoip {
       source => "[nginx][access][remote_ip]"
       target => "[nginx][access][geoip]"
    }
}
output {
#    stdout { codec => rubydebug }
    elasticsearch {
        hosts => "localhost:9200"
        user => elastic
        password => changeme
        manage_template => false
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
    }
    file {
        path => "/tmp/testlog"
    }
}

filebeat.yml:

     filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/nginx/access.log

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
 # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5043"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

I can see in /tmp/testfile that the data are processed but it seems that they aren't sent to elasticsearch.

What am I doing wrong?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.