Set up dashboards for Logstash Output

diegz · July 18, 2021, 1:09pm

Hi community,

I would like to enable dashboards in kibana for metricbeat, heartbeat, filebeat and winlogbeat.
On the documentation it says to pass these commands when logstash is in output.
What are the settings when SSL is enabled on the whole stack?

heartbeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['localhost:9200'] \
  -E output.elasticsearch.username=heartbeat_internal \
  -E output.elasticsearch.password=YOUR_PASSWORD \
  -E setup.kibana.host=localhost:5601

legoguy1000 · July 18, 2021, 7:48pm

U can add additional settings to the command. See Configure SSL | Filebeat Reference [7.13] | Elastic. Adding something like

-E output.elasticsearch.ssl.certificate_authorities=["/etc/pki/root/ca.pem"] \
-E output.elasticsearch.ssl.certificate="/etc/pki/client/cert.pem" \
-E output.elasticsearch.ssl.key="/etc/pki/client/cert.key"`

diegz · July 19, 2021, 7:09am

@legoguy1000 Thanks for your answer.
I have already tried these settings without success


instance/beat.go:989	Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at https://node1.ad-it.fr:9300: Get "https://node1.ad-it.fr:9300": x509: certificate signed by unknown authority]

legoguy1000 · July 19, 2021, 10:11am

That's because you're using an untrusted or self signed certificate. You either need to copy the CA to the system running Filebeat and set the output.elasticsearch.ssl.certificate_authorities setting or set output.elasticsearch.ssl.verification_mode: none

diegz · July 19, 2021, 11:10am

Thanks for your answer @legoguy1000 .
Heartbeat is on the same host.

I tried with this settings :

heartbeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['node1.ad-it.fr:9200'] \
  -E output.elasticsearch.ssl.verification_mode: none \
  -E output.elasticsearch.ssl.certificate_authorities="/etc/heartbeat/certs/ca.crt" \
  -E setup.kibana.host=192.168.1.30:5601

ERROR	[esclientleg]	eslegclient/connection.go:261	error connecting to Elasticsearch at http://node1.ad-it.fr:9200: Get "http://node1.ad-it.fr:9200": EOF
2021-07-18T17:57:43.858+0200	ERROR	instance/beat.go:989	Exiting: couldn't connect to any of the configured Elasticsearch hosts. Errors: [error connecting to Elasticsearch at http://node1.ad-it.fr:9200: Get "http://node1.ad-it.fr:9200": EOF]

legoguy1000 · July 19, 2021, 11:32am

When set via the config file you use :, when set via the cli you use =. So those should be -E output.elasticsearch.ssl.verification_mode="none"

diegz · July 19, 2021, 11:36am

@legoguy1000
I succeeded with this configuration but the heartbeats dashboards are not present in kibana > dashboards.
I don't understand ...

  heartbeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.enabled=true \
  -E output.elasticsearch.hosts=['https://node1.ad-it.fr:9200'] \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password='password' \
  -E output.elasticsearch.ssl.verification_mode:none \
  -E output.elasticsearch.ssl.certificate_authorities=["/etc/heartbeat/certs/ca.crt"] \
  -E output.elasticsearch.ssl.certificate="/etc/heartbeat/certs/node1.crt" \
  -E output.elasticsearch.ssl.key="/etc/heartbeat/certs/node1.key" \
  -E setup.kibana.host="192.168.1.30:5601"

Output :

2021-07-18T18:23:38.483+0200	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2021-07-18T18:23:38.992+0200	INFO	[index-management.ilm]	ilm/std.go:170	ILM policy heartbeat successfully created.
2021-07-18T18:23:38.992+0200	INFO	[index-management]	idxmgmt/std.go:401	Set setup.template.name to '{heartbeat-7.13.3 {now/d}-000001}' as ILM is enabled.
2021-07-18T18:23:38.992+0200	INFO	[index-management]	idxmgmt/std.go:406	Set setup.template.pattern to 'heartbeat-7.13.3-*' as ILM is enabled.
2021-07-18T18:23:38.992+0200	INFO	[index-management]	idxmgmt/std.go:440	Set settings.index.lifecycle.rollover_alias in template to {heartbeat-7.13.3 {now/d}-000001} as ILM is enabled.
2021-07-18T18:23:38.992+0200	INFO	[index-management]	idxmgmt/std.go:444	Set settings.index.lifecycle.name in template to {heartbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-07-18T18:23:39.119+0200	INFO	template/load.go:228	Existing template will be overwritten, as overwrite is enabled.
2021-07-18T18:23:39.206+0200	INFO	template/load.go:131	Try loading template heartbeat-7.13.3 to Elasticsearch
2021-07-18T18:23:39.657+0200	INFO	template/load.go:123	template with name 'heartbeat-7.13.3' loaded.
2021-07-18T18:23:39.657+0200	INFO	[index-management]	idxmgmt/std.go:297	Loaded index template.
2021-07-18T18:23:40.738+0200	INFO	[index-management.ilm]	ilm/std.go:135	Index Alias heartbeat-7.13.3 successfully created.
Index setup finished.

legoguy1000 · July 19, 2021, 1:02pm

Try -E setup.dashboards.enabled=true

legoguy1000 · July 19, 2021, 1:04pm

Nm looks like they removed the built-in dashboards for heartbeat. See heartbeat 7.0.0 missing kibana dashboards · Issue #11802 · elastic/beats · GitHub.

diegz · July 20, 2021, 11:32am

@legoguy1000 Thanks for you answer.

Indeed dashboards are disabled for heartbeat so I downloaded on the github repo.
Now I want to activate them for filebeat and metricbeat with logstash but my license has expired. So I got an extend trial that I added via the GUI. I still have to activate it to make it take effect.
But I can't activate it with the ssl in place.
Could you help me?
I tried this command :

curl -X POST -cacert /etc/elasticsearch/certs/ca.crt -u elastic "https://node1.ad-it.fr:9200/_xpack/license/start_trial?acknowledge=true"

Output :

curl: (3) <url> malformed
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above

.

legoguy1000 · July 20, 2021, 11:53am

Add -k or --insecure to your curl command to ignore ssl verification. Even though you are providing the path to the CA, I suspect the something doesn't match like the fqdn isn't in the list of SANs...

diegz · July 20, 2021, 12:12pm

@legoguy1000
I tried this command, the output looks good but in GUI it still says "Your Trial license is inactive"
curl -XPUT -u elastic 'htContent-Type: application/json" -d @licence.json --insecure
Output :

{"acknowledged":true,"license_status":"valid"}

And when I want to activate filebeat dashboards with logtash I make :

`filebeat setup -e  
-E output.logstash.enabled=false \
-E output.elasticsearch.enabled=true \  
-E output.elasticsearch.hosts=['https://node1.ad-it.fr:9200'] \  
-E output.elasticsearch.username=elastic   
-E output.elasticsearch.password='password' \  
-E output.elasticsearch.ssl.verification_mode='none' \  
-E output.elasticsearch.ssl.certificate_authorities=["/etc/filebeat/config/certs/ca.crt"]  \
-E output.elasticsearch.ssl.certificate="/etc/filebeat/config/certs/node1.crt"  \
-E output.elasticsearch.ssl.key="/etc/elasticsearch/certs/node1.key" \  
-E setup.kibana.host="192.168.1.30:5601"  \
-E setup.dashboards.enabled=true`

Output :

2021-07-19T18:58:19.460+0200	ERROR	elasticsearch/elasticsearch.go:252	Error connecting to Elasticsearch at https://node1.ad-it.fr:9200: Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: could not extract license information from the server response: unknown state, received: 'invalid'
2021-07-19T18:58:19.460+0200	ERROR	instance/beat.go:906	Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://node1.ad-it.fr:9200: Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: could not extract license information from the server response: unknown state, received: 'invalid']
Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch https://node1.ad-it.fr:9200: Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: could not extract license information from the server response: unknown state, received: 'invalid']

I don't understand this line

cannot retrieve the elasticsearch license: could not extract license information from the server response: unknown state, received: 'invalid'

legoguy1000 · July 20, 2021, 1:21pm

Can you get to Kibana? What version of Elasticsearch? In the latest versions the /_xpack isn't needed. IDK if thats an issue. If you do curl -X GET -cacert /etc/elasticsearch/certs/ca.crt -u elastic "https://node1.ad-it.fr:9200/_license", what do you get?

diegz · July 20, 2021, 1:39pm

@legoguy1000 Tkanks for your answer.

I can get to Kibana and the elasticsearch version is 7.12.1

curl -X GET -cacert /etc/elasticsearch/certs/ca.crt -u elastic "https://node1.ad-it.fr:9200/_license" --insecure

license" : {
    "status" : "invalid",
    "uid" : "05362365-0fb0-4ccf-9e00-8a91de5e4f49",
    "type" : "trial",
    "issue_date" : "2021-07-20T00:00:00.000Z",
    "issue_date_in_millis" : 1626739200000,
    "expiry_date" : "2021-08-03T23:59:59.999Z",
    "expiry_date_in_millis" : 1628035199999,
    "max_nodes" : 1000,
    "issued_to" : "Axians (non-production environments)",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1

legoguy1000 · July 20, 2021, 1:42pm

Did you have a real license previously? If so you may not be able to downgrade to the trial. Are you using any feature that requires a paid license? or will the basic license work which includes a lot of the "xpack" features like security/ssl/...? See Start basic API | Elasticsearch Guide [7.13] | Elastic

diegz · July 20, 2021, 1:56pm

@legoguy1000
Before I had a trial license that I activated.
If you tell me that the basic license allows me to use the security features, ssl is enough for me.
But I thought I needed the trial license for these functions.

legoguy1000 · July 20, 2021, 3:03pm

Nope, they moved a lot of those x-pack features to the basic/free license. See Subscriptions | Elastic Stack Products & Support | Elastic if the basic license meets your needs.

diegz · July 21, 2021, 9:33am

Hi @legoguy1000
The license was not active because the clock on my server was not on time, now it's active.

Now it's kibana ssl parameters that I miss

filebeat setup -e   
-E output.logstash.enabled=false    \
-E output.elasticsearch.enabled=true \  
-E output.elasticsearch.hosts=['https://node1.ad-it.fr:9200']   \ 
-E output.elasticsearch.username=elastic \  
-E output.elasticsearch.password='password' \   
-E output.elasticsearch.ssl.verification_mode: none \   
-E output.elasticsearch.ssl.certificate_authorities=["/etc/elasticsearch/certs/ca.crt"] \   
-E output.elasticsearch.ssl.certificate="/etc/filebeat/config/certs/node1.crt"  \
-E output.elasticsearch.ssl.key="/etc/filebeat/config/certs/node1.key"  \
-E setup.kibana.host="https://kibana.ad-it.fr:5601"  \
-E setup.dashboards.enabled=true

Output :

ERROR	instance/beat.go:906	Exiting: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get https://kibana.ad-it.fr:5601/api/status: x509: certificate signed by unknown authority. Response: .
Exiting: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get https://kibana.ad-it.fr:5601/api/status: x509: certificate signed by unknown authority. Response: .

legoguy1000 · July 21, 2021, 10:07am

The ssl options for kibana are the same as elasticsearch, just set under the setup.kibana.ssl.xxx setting.

system · August 18, 2021, 12:08pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.