Help you need to configure log files via filebeat and not logstash

Elastic Stack Beats
,
1

Now everything is configured through logstash, but I need to use filebeat, how can I do this? Here is the logstash conf file

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "nginx_logs" {
    grok {
      match => {
        "message" => '%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'
        "host" => "%{IPORHOST:host}"
      }
    }

    # Удаляем все остальные поля, кроме host и clientip
    mutate {
      remove_field => [
        "@timestamp", "_id", "_index", "_type", "_score",
        "extension", "geo.coordinates", "geo.dest", "geo.src",
        "geo.srcdest", "index", "ip", "machine.os", "machine.ram",
        "memory", "message", "phpmemory", "referer", "request",
        "response", "tags", "timestamp", "url", "utc_time"
      ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["10.0.1.160:9200"]
    user => "elastic"
    password => "changeme"
    index => "nginx-%{+YYYY.MM.dd}"
    document_type => "nginx_logs"
  }
}
2

Additional Information

logstash.yml

---
## Default Logstash configuration from Logstash base image.
## https://github.com/elastic/logstash/blob/master/docker/data/logstash/config/logstash-full.yml
#
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://10.0.1.160:9200" ]

## X-Pack security credentials
#
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme
#path.config: /u00/docker-elk-nginx-filebeat/logstash/config/logstash.conf
3

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log

output.logstash:
  enabled: true
  hosts: ["10.0.1.160:5044"]
  # FIXME: Cannot reach the hosts of Logstash

# output.elasticsearch:
#   hosts: ["http://elasticsearch:9200"]
#   username: "elastic"
#   password: "changeme"
setup.kibana:
  host: "http://10.0.1.160:5601"
  username: "elastic"
  password: "changeme"
4

kibana.yml

---
## Default Kibana configuration from Kibana base image.
## https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/templates/kibana_yml.template.js
#
server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://10.0.1.160:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true

## X-Pack security credentials
#
elasticsearch.username: elastic
elasticsearch.password: changeme
~
5

elasticsearch.yml

---
## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 10.0.1.160

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
xpack.license.self_generated.type: trial
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
6

If you need docker-compose.yml, tell me. Everything needs to be reworked so that nginx logs are sent via filebeat to elastic and you also need to create a config so that it does not send all the lines of the logs

7

We will install it in a test format in Rocky Linux 9. Elastic Stack in docker-compose

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.