Не приходят логи

[beren (Vladimir)]

Всем привет.

Пытаюсь связать filebeat с ELK 7.4

Создаю в /etc/logstash/conf.d/postfix.conf

input {
  beats {
    port => 5044
  }
}
output {
        elasticsearch {
            hosts    => "localhost:9200"
            index    => "postfix1-%{+YYYY.MM.dd}"
        }
}

В /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
      - /var/log/maillog*
output.logstash:
  hosts: ["192.168.199.146:5044"]
xpack.monitoring:
 enabled: true
  elasticsearch:
    hosts: ["http://192.168.199.146:9200"]

Но index не появляется в Kibana
В логах filebeat

WARN    beater/filebeat.go:152  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

Любая помощь =)

[Igor_Motov]

Я бы заменил output в logstash на stdout, чтобы убедиться, что в logstash что-то приходит. Это позволит нам понять где проблема - между filebeat и logstash или logstash и elasticsearch.

[beren (Vladimir)]

Теперь в /etc/logstash/conf.d/postfix.conf

Blockquote
input {
beats {
port => 5044
}
}

output {

   stdout { }

}
Перезапуск logstash, но ничего не пришло.

[Igor_Motov]

Если ничего - то, наверное, до него сообщения от filebeat не доходят, либо filebeat ничего из логов не подцепляет. В логах в этих что-нибудь появляется? Пользователь под которым filebeat запущен доступ к ним имеет? Порт 5044 не заблокирован? В логах filebeat еще что-нибудь есть?

[beren (Vladimir)]

В /etc/filebeat/filebeat.yml

Blockquote
filebeat.inputs:

• type: log
  enabled: true
  paths:
  - /var/log/maillog*
  output.logstash:
  hosts: ["192.168.199.146:5044"]

Порты не заблокированы, с filebeat 5044 телнетиться.

Вот полный лог filbeat

Blockquote
2019-10-11T03:30:30.292-0400 INFO instance/beat.go:607 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-10-11T03:30:30.292-0400 INFO instance/beat.go:615 Beat ID: 1a3f0e19-0b95-4d3e-881a-d37400468513
2019-10-11T03:30:30.292-0400 INFO [beat] instance/beat.go:903 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "1a3f0e19-0b95-4d3e-881a-d37400468513"}}}
2019-10-11T03:30:30.292-0400 INFO [beat] instance/beat.go:912 Build info {"system_info": {"build": {"commit": "f940c36884d3749901a9c99bea5463a6030cdd9c", "libbeat": "7.4.0", "time": "2019-09-27T07:45:44.000Z", "version": "7.4.0"}}}
2019-10-11T03:30:30.292-0400 INFO [beat] instance/beat.go:915 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.12.9"}}}
2019-10-11T03:30:30.293-0400 INFO [beat] instance/beat.go:919 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-10-10T06:15:26-04:00","containerized":false,"name":"mail.test.ru","ip":["127.0.0.1/8","::1/128","192.168.199.145/24","fe80::277c:61b3:ac2:bc8c/64"],"kernel_version":"3.10.0-957.el7.x86_64","mac":["00:0c:29:cb:1a:3d"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"b27e5adbf8a1485faffe0eeec83f47f2"}}}
2019-10-11T03:30:30.294-0400 INFO [beat] instance/beat.go:948 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/var/log", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 11533, "ppid": 10829, "seccomp": {"mode":"disabled"}, "start_time": "2019-10-11T03:30:29.400-0400"}}}
2019-10-11T03:30:30.294-0400 INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.4.0
2019-10-11T03:30:30.296-0400 INFO [publisher] pipeline/module.go:97 Beat name: mail.test.ru
2019-10-11T03:30:30.296-0400 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

А как понять под каким пользователем запускается filebeat ?

[Igor_Motov]

А как понять под каким пользователем запускается filebeat ?

Обычно под root-ом, но все зависит от того, как вы его сконфигурировали

Может @Dima_I сможет помочь с этим?

[beren (Vladimir)]

Спасибо. Проблема решена. На самом деле в лог ничего сегодня не писалось.

[Dima_I (Dima Ilchenko)]

На CentOS7/RHEL:
systemctl cat filebeat.service
ps uax | grep filebeat