The following content has been translated using ChatGPT. Thank you for your understanding.
Hello, I'm currently learning Elasticsearch. I'm using Elasticsearch version 8.8.2. I have written the following pipeline in order to monitor Logstash in Kibana
And I'm using wsl2.
input{
file{
path => "/home/won/elastic/elasticsearch-8.8.2/logs/gc.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output{
elasticsearch{
hosts => ["https://localhost:9200"]
cacert => '/etc/logstash/config/certs/ca.crt'
index => "multipipe_pipe2"
}
}
However, I encountered the following error when using Logstash
[2023-07-19T15:53:41,311][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
By the way, I executed the 'Copy and save the certificate' and 'Configure the Elasticsearch output' steps from this link.
So, when I searched for a solution, I came across a suggestion to change 'xpack.security.enabled' from 'true' to 'false' in the elasticsearch.yml file. While this resolved the error in Logstash, it resulted in an error in Kibana, and I'm unable to access it.
elasticsearch.yml
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["won"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
kibana.yml
# This section was automatically generated during setup.
elasticsearch.hosts: ['https://127.0.0.1:9200']
# elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2ODk2NTgxODM4MDI6NVdJMFNrYXhSQUN2dVVFTDdVdlQwUQ
elasticsearch.ssl.certificateAuthorities: [/home/won/elastic/kibana-8.8.2/data/ca_1689658184301.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://127.0.0.1:9200'], ca_trusted_fingerprint: 957a6d50e80c63ad1b8a96095eaf5779a36ebace254fa88b412635c55e1b756a}]
elasticsearch.username: kibana
elasticsearch.password: qweasd
When 'xpack.security.enabled' is set to 'false', it results in an error
[2023-07-19T16:11:30.347+09:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]
[2023-07-19T16:11:38.084+09:00][INFO ][plugins-service] Plugin "cloudChat" is disabled.
[2023-07-19T16:11:38.085+09:00][INFO ][plugins-service] Plugin "cloudExperiments" is disabled.
[2023-07-19T16:11:38.085+09:00][INFO ][plugins-service] Plugin "cloudFullStory" is disabled.
[2023-07-19T16:11:38.086+09:00][INFO ][plugins-service] Plugin "cloudGainsight" is disabled.
[2023-07-19T16:11:38.103+09:00][INFO ][plugins-service] Plugin "profiling" is disabled.
[2023-07-19T16:11:38.174+09:00][INFO ][http.server.Preboot] http server running at http://localhost:5601
[2023-07-19T16:11:38.267+09:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
[2023-07-19T16:11:38.287+09:00][WARN ][config.deprecation] Kibana is configured to authenticate to Elasticsearch with the "kibana" user. Use a service account token instead.
[2023-07-19T16:11:38.288+09:00][WARN ][config.deprecation] The default mechanism for Reporting privileges will work differently in future versions, which will affect the behavior of this cluster. Set "xpack.reporting.roles.enabled" to "false" to adopt the future behavior before upgrading.
[2023-07-19T16:11:38.600+09:00][INFO ][plugins-system.standard] Setting up [136] plugins: [usageCollection,telemetryCollectionManager,telemetryCollectionXpack,taskManager,kibanaUsageCollection,cloud,translations,share,screenshotMode,newsfeed,savedObjectsFinder,monitoringCollection,licensing,mapsEms,globalSearch,globalSearchProviders,features,guidedOnboarding,banners,licenseApiGuard,customBranding,ftrApis,fieldFormats,expressions,screenshotting,dataViews,charts,esUiShared,customIntegrations,home,searchprofiler,painlessLab,management,cloudDataMigration,advancedSettings,spaces,security,telemetry,licenseManagement,snapshotRestore,lists,files,encryptedSavedObjects,eventLog,actions,notifications,grokdebugger,console,contentManagement,bfetch,data,watcher,unifiedFieldList,savedSearch,savedObjectsTagging,savedObjectsManagement,unifiedSearch,graph,embeddable,uiActionsEnhanced,presentationUtil,expressionShape,expressionRevealImage,expressionRepeatImage,expressionMetric,expressionImage,controls,fileUpload,ingestPipelines,eventAnnotation,ecsDataQualityDashboard,dataViewFieldEditor,visualizations,visTypeXy,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeTable,visTypeMetric,visTypeMarkdown,visTypeHeatmap,expressionXY,expressionTagcloud,expressionPartitionVis,visTypePie,expressionMetricVis,expressionLegacyMetricVis,expressionHeatmap,expressionGauge,visTypeGauge,dashboard,lens,maps,aiops,dashboardEnhanced,dataViewManagement,alerting,triggersActionsUi,transform,stackConnectors,stackAlerts,ruleRegistry,cases,timelines,sessionView,kubernetesSecurity,threatIntelligence,discover,reporting,canvas,fleet,osquery,indexManagement,rollup,remoteClusters,crossClusterReplication,indexLifecycleManagement,cloudSecurityPosture,cloudDefend,exploratoryView,observability,observabilityOnboarding,discoverEnhanced,dataVisualizer,ml,synthetics,securitySolution,infra,upgradeAssistant,monitoring,logstash,enterpriseSearch,apm,assetManager]
[2023-07-19T16:11:38.603+09:00][INFO ][plugins.taskManager] TaskManager is identified by the Kibana UUID: 632510bd-6785-46e7-a28c-d2fcb3dc462f
[2023-07-19T16:11:38.622+09:00][INFO ][custom-branding-service] CustomBrandingService registering plugin: customBranding
[2023-07-19T16:11:38.659+09:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-07-19T16:11:38.660+09:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-07-19T16:11:38.675+09:00][WARN ][plugins.security.config] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-07-19T16:11:38.676+09:00][WARN ][plugins.security.config] Session cookies will be transmitted over insecure connections. This is not recommended.
[2023-07-19T16:11:38.688+09:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-07-19T16:11:38.691+09:00][WARN ][plugins.actions] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-07-19T16:11:38.694+09:00][INFO ][plugins.notifications] Email Service Error: Email connector not specified.
[2023-07-19T16:11:38.845+09:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-07-19T16:11:38.885+09:00][WARN ][plugins.reporting.config] Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.
[2023-07-19T16:11:38.916+09:00][INFO ][plugins.cloudSecurityPosture] Registered task successfully [Task: cloud_security_posture-stats_task]
[2023-07-19T16:11:38.921+09:00][INFO ][plugins.alerting] Registering resources for context "observability.slo".
[2023-07-19T16:11:38.939+09:00][INFO ][plugins.alerting] Registering resources for context "observability.uptime".
[2023-07-19T16:11:38.954+09:00][INFO ][plugins.alerting] Registering resources for context "security".
[2023-07-19T16:11:38.973+09:00][INFO ][plugins.alerting] Registering resources for context "observability.logs".
[2023-07-19T16:11:38.974+09:00][INFO ][plugins.alerting] Registering resources for context "observability.metrics".
[2023-07-19T16:11:39.025+09:00][INFO ][plugins.alerting] Registering resources for context "observability.apm".
[2023-07-19T16:11:39.026+09:00][INFO ][plugins.assetManager] Asset manager plugin [tech preview] is NOT enabled
[2023-07-19T16:11:39.103+09:00][INFO ][plugins.screenshotting.config] Chromium sandbox provides an additional layer of protection, and is supported for Linux Ubuntu 22.04 OS. Automatically enabling Chromium sandbox.
[2023-07-19T16:11:39.159+09:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. write EPROTO 139889444902848:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
[2023-07-19T16:11:40.259+09:00][INFO ][plugins.screenshotting.chromium] Browser executable: /home/won/elastic/kibana-8.8.2/node_modules/@kbn/screenshotting-plugin/chromium/headless_shell-linux_x64/headless_shell
The ultimate goal is to monitor Logstash from Kibana.
logstash.yml
# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
xpack.monitoring.enabled: true
#xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.proxy: ["http://proxy:port"]
xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id:xxxxxxxxxx
#xpack.monitoring.elasticsearch.cloud_auth: logstash_system:password
# another authentication alternative is to use an Elasticsearch API key
#xpack.monitoring.elasticsearch.api_key: "id:api_key"
#xpack.monitoring.elasticsearch.ssl.certificate_authority: "/path/to/ca.crt"
#xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: xxxxxxxxxx
#xpack.monitoring.elasticsearch.ssl.truststore.path: path/to/file
#xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file
#xpack.monitoring.elasticsearch.ssl.keystore.password: password
#xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true
Could you please provide more specific details about the problem you're facing with Elasticsearch and Kibana? This will help me provide you with a more accurate solution or guide you to relevant documentation. Thank you.