Filebeat is not outputting logs to ElasticSearch, not even to Console

Elastic Stack Beats
1

@ElasticMaa @elastock @andrewkroh @steffens

Here is my filebeat.yml:

    filebeat.inputs:
       - type: log
        paths:
          - "/home/ubuntu/log/test_logs_json"
        enabled: true
        json.keys_under_root: true
        json.add_error_key: true
        json.message_key: log
        
        #setup.template:
        #enabled: true
        #name: test_template
        #pattern: 'test_template*'
    output.console:
            pretty: true
            #enabled: true
            #hosts: ['aws-vpc-elasticsearch-cluster']
            #index: "test_index"

    # Or
    output.elasticsearch:
             enabled: true
             hosts: ["https://vpc-elasticseacrh-cluster.amazonaws.com/"]
             index: "test_index"

And here is log file:

{"message": "adas asdsada asdsad"}
{"message": "new asdsada asdsad"}
{"message": "second asdsada asdsad"}
{"message": "third asdsada asdsad"}
{"message": "fourth asdsada asdsad"}
{"message": "fifth asdsada asdsad"}
{"message": "sixth asdsada asdsad"}
{"message": "seventh asdsada asdsad"}
{"message": "eighth asdsada asdsad"}

When I start or restart Filebeat service It does nothing and through one warning " filebeat is unable to load the ingest node", I think this warning is not the reason of filebeat not working.

2

Hi,

One thing I notice though is that your hosts field says vpc-elasticseacrh... instead of vpc-elasticsearch. If that doesn't work you might also want to double check whether you need a port number, though I think that may not be needed in this case. Failing both of those, can you share the Filebeat logs? (filebeat/logs/filebeat*) That should show where the problem is.

3

Thanks @faec

I had made my ElasticsSearch to public and using pipeline Filebeat >> Logstash >> AWS elasticsearch and its working if I make following settings:

elasticsearch {
   hosts => "https://search-es_domain.us-west-2.es.amazonaws.com:443"
   index => 'test_index_1'
   ssl => true
   ilm_enabled => false
}

But if I make the ilm_enabled to true, it does not work.

elasticsearch {
   hosts => "https://search-es_domain.us-west-2.es.amazonaws.com:443"
   index => 'test_index_1-%{+YYYY.MM.dd}'
   ssl => true
   ilm_enabled => true
}

Do I need to enable ILM, so that Logstash can create indices daily?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.