I have successfully installed ELK (Elasticsearch, Logstash and Kibana) on a Virtue Machine in Azure, and tested succesfully that it is able to read CSV file and link to Azure SQL Data, via some .conf file which links up the inputs and outputs.
However, when I try wth Filebeat, I have some questions.
Having installed successfully the Filebeat and done its configuration in the file /etc/filebeat/filebeat.yml, and setup the index management as below:
~ filebeat setup –index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["127.0.0.1:9200"]'
And to enable Filebeat System Module
~ filebeat modules enable system
~ sudo systemctl status filebeat
~ sudo systemctl start filebeat
~$ sudo systemctl enable filebeat
To execute the Logstash configuration file, I need to move inside the Logstash home directory (/usr/share/logstash/bin) and then run the following command:
./logstash --path.settings /etc/logstash/ --path.data /tmp/filebeat -f /etc/logstash/conf.d/localhost/logstash-filebeat.conf --config.reload.automatic
It gives me the output:
[Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
~$ curl -X GET http://localhost:9200/_cat/indices?v
gives that
filebeat-7.5.0-2019.12.15-000001 index was created status=Open, health=yellow, with docs.count=41093, which is always increasing.
It seems to be everything was properly setup, and I am able to retrieve data from the index.
(a) I was able to go Discover and get some information of the index from filebeat which was created.
(b) However, when I change the localhost to the Elastic Cloud in the configuration settings, I got this error:
[2019-12-18T08:38:56,525][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://6069ec8d285b49b68ce7bf35b9042c02.southeastasia.azure.elastic-cloud.com:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://6069ec8d285b49b68ce7bf35b9042c02.southeastasia.azure.elastic-cloud.com:9243/'"}