Problem with Filebeat and Kibana

Hi,

I use Filebeat 7.4 and Kibana 7.3.2 . Filebeat receive cleanly the log. But he don't forwad to Kibana.
When I do : filebeat setup
I have this result :

Index setup finished.
 Loading dashboards (Kibana must be running and reachable)
 Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://172.17.2.9:5601/api/status fails: fail to execute the HTTP GET request: Get http://172.17.2.9:5601/api/status: dial tcp 172.17.2.9:5601: connect: connection refused. Response: .

Can you help me please ?
Best regards,

hey @Jonathan_Sieger

do you see anything in Kibana logs? can you connect to Kibana from where Beat is running? can you verify that Kibana is in fact listening on specified port on that URL?

In Kibana logs at this hours I saw nothing in stdout... The last log was yesterday.

Yesterday I had a problem like :

FATAL Error: Port 5601 is already in use. Another instance of Kibana may be running!
And the UI Kibana was not working. I resolve this issue.

netstat -paunt | grep 5601
tcp        0      0 127.0.0.1:5601          0.0.0.0:*               LISTEN      29963/node

I have nginx in the same VM for have authentication prompt.

I don't understand when you say "Can you connect to kibana from where beat is running" ? Service filebeat is running with error... :

oct. 09 14:07:15  filebeat[18899]: 2019-10-09T14:07:15.072+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...
oct. 09 14:07:45 filebeat[18899]: 2019-10-09T14:07:45.072+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...
oct. 09 14:07:53 filebeat[18899]: 2019-10-09T14:07:53.093+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:07:53 filebeat[18899]: 2019-10-09T14:07:53.093+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:07:53  filebeat[18899]: 2019-10-09T14:07:53.093+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:07:53  filebeat[18899]: 2019-10-09T14:07:53.093+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:07:53 filebeat[18899]: 2019-10-09T14:07:53.093+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:07:53 filebeat[18899]: 2019-10-09T14:07:53.093+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:08:00 filebeat[18899]: 2019-10-09T14:08:00.087+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:08:15 filebeat[18899]: 2019-10-09T14:08:15.072+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{

As per above error it is clear that your kibana server is not reachable from that machine where you installed filebeat and trying to rin filebeat setup.

Can you please try below command on machine where you run filebeat setup:

telnet 172.17.2.9 5601

If you are not able to telnet then check your firewall rule and kibana service.

Thanks.

@Jonathan_Sieger,

According to this output your kibana is running on localhost. You need to run it over internal IP (private ip) then it will be reachable from beat machine. If both machine are not reachable then contact your system admin team they will help you in reach ability of both machine.

Thanks.

Filebeat, ELK and Kibana are in the same machine, last week it was working. I upgrade 6.8 to 7.4. I just do : yum update filebeat, yum update elasticsearch.
Kibana service is correctly launched .

Telnet : telnet: connect to address 172.17.2.9: Connection refused

Firewalld is disable

@Jonathan_Sieger,

Run your kibana over 172.17.2.9 or change in your filebeat.yml configuration replace kibana IP 172.17.2.9 with 127.0.0.1.

Thanks.

I tried the twice IP in filebeat.yml.

In kibana.yml I have this : server.host: "127.0.0.1" and server.port: 5601
I modify the filebeat.yml from : setup.kibana: "http://172.17.2.9" to "http://127.0.0.1"
I restart kibana and filebeat service. And nothing have changed

@Jonathan_Sieger,

Can you please check filebeat service is running properly or not? If no can you please provide some error logs from filebeat?

Also please provide your filebeat.yml config.

Thanks.

systemctl status filebeat.service :

oct. 09 14:40:26  filebeat[28802]: 2019-10-09T14:40:26.209+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:40:33 inf-infvm-soc01.infocheops.local filebeat[28802]: 2019-10-09T14:40:33.140+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...
oct. 09 14:41:03  filebeat[28802]: 2019-10-09T14:41:03.140+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...
oct. 09 14:41:31l filebeat[28802]: 2019-10-09T14:41:31.223+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:41:33 filebeat[28802]: 2019-10-09T14:41:33.140+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...
oct. 09 14:42:03 filebeat[28802]: 2019-10-09T14:42:03.140+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...
oct. 09 14:42:16 filebeat[28802]: 2019-10-09T14:42:16.214+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:42:31 filebeat[28802]: 2019-10-09T14:42:31.227+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5...
oct. 09 14:42:33 filebeat[28802]: 2019-10-09T14:42:33.139+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...
oct. 09 14:43:03 filebeat[28802]: 2019-10-09T14:43:03.141+0200        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"...

And filebeat.yml  : 
    filebeat:
     inputs:
      - type: log
        paths:
         - /var/ossec/logs/alerts/alerts.json
        document_type: json
        json.message_key: log
        json.keys_under_root: true
        json.overwrite_keys: true
    #output.elasticsearch.hosts: ['http://127.0.0.1:9200']
    #output:
    # logstash:
       # The Logstash hosts
    #   hosts: ["172.17.2.9:5000"]
    #   ssl:
    #     certificate_authorities: ["/etc/filebeat/logstash.crt"
    output:
     elasticsearch:
      hosts: ["http://172.17.2.9:9200"]
      username: "USER"
      password: "PWD"

    setup.kibana.host: "http://127.0.0.1"
    setup.kibana.protocol: "http"
    setup.dashboard.enabled: true

@Jonathan_Sieger ,

In above part port is missing.

setup.kibana:
  host: "127.0.0.1:5601"

Thanks.

Nothing have changed

systemctl status filebeat.service : 
oct. 09 15:06:12 filebeat[4148]: 2019-10-09T15:06:12.055+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5f...
oct. 09 15:06:18 filebeat[4148]: 2019-10-09T15:06:18.973+0200        WARN        elasticsearch/client.go:535        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf5f...

@Jonathan_Sieger, did you restart the filebeat service after making changes in filebeat.yml file?
If yes, please provide log from filebeat because service status is not enough to troubleshoot the issue log may give some lead.

Thanks.

Hi,

I resolve my issue, the filebeat.yml was not properly configured with my last version of elasticsearch and my app Wazuh in Kibana. Thanks you for your help.

But after the upgrade, Elasticsearch and Kibana use the CPU at 122%.. I have 8 Go RAM and 2 vCPU. I add more CPU ?