I'm trying to test the connection between filebeat on server and elasticsearch running on another server "both are on the same network"
I'm getting this output when running this command ./filebeat test output
.....................................
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... ERROR 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [filebeat_admin] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [filebeat_admin] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}
my filebeat.yml as follow
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
# filestream is an input for collecting log messages from files.
- type: filestream
# Unique ID among all inputs, an ID is required.
id: my-filestream-id
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#prospector.scanner.exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "http://46.246.120.148:8907"
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]
hosts: ["https://10.4.4.37:9200"]
username: "filebeat_admin"
passwod: "passw0rd"
ssl:
enabled: true
ca_trusted_fingerprint: "080956518306f36a57445dbcbfcbab45fdcc060073f95a6acbfd5195fa0521ef"
# Protocol - either `http` (default) or `https`.
protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the filebeat.
#instrumentation:
# Set to true to enable instrumentation of filebeat.
#enabled: false
# Environment in which filebeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
and the result of my elasticsearch.log is
[2022-08-28T01:30:00,001][INFO ][o.e.x.s.SnapshotRetentionTask] [ogonz] starting SLM retention snapshot cleanup task
[2022-08-28T01:30:00,001][INFO ][o.e.x.m.MlDailyMaintenanceService] [ogonz] triggering scheduled [ML] maintenance tasks
[2022-08-28T01:30:00,007][INFO ][o.e.x.s.SnapshotRetentionTask] [ogonz] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2022-08-28T01:30:00,008][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [ogonz] Deleting expired data
[2022-08-28T01:30:00,010][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [ogonz] Successfully deleted [0] unused stats documents
[2022-08-28T01:30:00,010][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [ogonz] Completed deletion of expired ML data
[2022-08-28T01:30:00,011][INFO ][o.e.x.m.MlDailyMaintenanceService] [ogonz] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2022-08-28T08:41:06,454][WARN ][o.e.x.s.a.RealmsAuthenticator] [ogonz] Authentication to realm default_native failed - Password authentication failed for filebeat_admin
[2022-08-28T08:44:57,425][WARN ][o.e.x.s.a.RealmsAuthenticator] [ogonz] Authentication to realm default_native failed - Password authentication failed for filebeat_admin
[2022-08-28T08:56:49,270][INFO ][o.e.x.s.a.RealmsAuthenticator] [ogonz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2022-08-28T09:06:54,397][INFO ][o.e.n.Node ] [ogonz] stopping ...
[2022-08-28T09:06:54,407][INFO ][o.e.x.w.WatcherService ] [ogonz] stopping watch service, reason [shutdown initiated]
[2022-08-28T09:06:54,408][INFO ][o.e.x.w.WatcherLifeCycleService] [ogonz] watcher has stopped and shutdown
[2022-08-28T09:06:54,592][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ogonz] [controller/94025] [Main.cc@176] ML controller exiting
[2022-08-28T09:06:54,597][INFO ][o.e.x.m.p.NativeController] [ogonz] Native controller process has stopped - no new native processes can be started
[2022-08-28T09:06:54,799][INFO ][o.e.n.Node ] [ogonz] stopped
[2022-08-28T09:06:54,800][INFO ][o.e.n.Node ] [ogonz] closing ...
[2022-08-28T09:06:54,813][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [ogonz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-4374499027959750706/geoip-databases/CF15-O5dRla3zp4xSsWlwA/GeoLite2-Country.mmdb]
[2022-08-28T09:06:54,813][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [ogonz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-4374499027959750706/geoip-databases/CF15-O5dRla3zp4xSsWlwA/GeoLite2-ASN.mmdb]
[2022-08-28T09:06:54,813][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [ogonz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-4374499027959750706/geoip-databases/CF15-O5dRla3zp4xSsWlwA/GeoLite2-City.mmdb]
[2022-08-28T09:06:54,822][INFO ][o.e.n.Node ] [ogonz] closed
[2022-08-28T09:07:00,386][INFO ][o.e.n.Node ] [ogonz] version[8.3.3], pid[314798], build[deb/801fed82df74dbe537f89b71b098ccaff88d2c56/2022-07-23T19:30:09.227964828Z], OS[Linux/5.4.0-124-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.2/18.0.2+9-61]
[2022-08-28T09:07:00,389][INFO ][o.e.n.Node ] [ogonz] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-08-28T09:07:00,390][INFO ][o.e.n.Node ] [ogonz] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-8412760976280061372, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3978m, -Xmx3978m, -XX:MaxDirectMemorySize=2085617664, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=deb, --module-path=/usr/share/elasticsearch/lib, -Djdk.module.main=org.elasticsearch.server]
[2022-08-28T09:07:02,920][INFO ][c.a.c.i.j.JacksonVersion ] [ogonz] Package versions: jackson-annotations=2.13.2, jackson-core=2.13.2, jackson-databind=2.13.2.2, jackson-dataformat-xml=2.13.2, jackson-datatype-jsr310=2.13.2, azure-core=1.27.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2022-08-28T09:07:04,277][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [aggs-matrix-stats]
[2022-08-28T09:07:04,277][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [analysis-common]
[2022-08-28T09:07:04,277][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [constant-keyword]
[2022-08-28T09:07:04,277][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [data-streams]
[2022-08-28T09:07:04,277][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [frozen-indices]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [ingest-common]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [ingest-geoip]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [ingest-user-agent]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [kibana]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [lang-expression]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [lang-mustache]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [lang-painless]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [legacy-geo]
[2022-08-28T09:07:04,278][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [mapper-extras]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [mapper-version]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [old-lucene-versions]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [parent-join]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [percolator]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [rank-eval]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [reindex]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [repositories-metering-api]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [repository-azure]
[2022-08-28T09:07:04,279][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [repository-encrypted]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [repository-gcs]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [repository-s3]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [repository-url]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [runtime-fields-common]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [search-business-rules]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [searchable-snapshots]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [snapshot-based-recoveries]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [snapshot-repo-test-kit]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [spatial]
[2022-08-28T09:07:04,280][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [systemd]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [transform]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [transport-netty4]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [unsigned-long]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [vector-tile]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [vectors]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [wildcard]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-aggregate-metric]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-analytics]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-async]
[2022-08-28T09:07:04,281][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-async-search]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-autoscaling]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-ccr]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-core]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-deprecation]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-enrich]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-eql]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-fleet]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-graph]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-identity-provider]
[2022-08-28T09:07:04,282][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-ilm]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-logstash]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-ml]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-monitoring]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-ql]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-rollup]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-security]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-shutdown]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-sql]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-stack]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-text-structure]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-voting-only-node]
[2022-08-28T09:07:04,283][INFO ][o.e.p.PluginsService ] [ogonz] loaded module [x-pack-watcher]
[2022-08-28T09:07:04,284][INFO ][o.e.p.PluginsService ] [ogonz] no plugins loaded
[2022-08-28T09:07:06,090][INFO ][o.e.e.NodeEnvironment ] [ogonz] using [1] data paths, mounts [[/ (/dev/sda2)]], net usable_space [268.8gb], net total_space [294.2gb], types [ext4]
[2022-08-28T09:07:06,091][INFO ][o.e.e.NodeEnvironment ] [ogonz] heap size [3.8gb], compressed ordinary object pointers [true]
[2022-08-28T09:07:06,204][INFO ][o.e.n.Node ] [ogonz] node name [ogonz], node ID [CF15-O5dRla3zp4xSsWlwA], cluster name [elasticsearch], roles [transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen, ml, data_hot]
[2022-08-28T09:07:08,667][INFO ][o.e.x.s.Security ] [ogonz] Security is enabled
[2022-08-28T09:07:08,872][INFO ][o.e.x.s.a.s.FileRolesStore] [ogonz] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2022-08-28T09:07:09,179][INFO ][o.e.x.s.InitialNodeSecurityAutoConfiguration] [ogonz] Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the `bin/elasticsearch-reset-password` tool to set the password for the elastic user.
[2022-08-28T09:07:09,250][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [ogonz] [controller/314829] [Main.cc@123] controller (64 bit): Version 8.3.3 (Build d2d2e518384d45) Copyright (c) 2022 Elasticsearch BV
[2022-08-28T09:07:09,636][INFO ][o.e.t.n.NettyAllocator ] [ogonz] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-08-28T09:07:09,657][INFO ][o.e.i.r.RecoverySettings ] [ogonz] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-08-28T09:07:09,684][INFO ][o.e.d.DiscoveryModule ] [ogonz] using discovery type [multi-node] and seed hosts providers [settings]
[2022-08-28T09:07:10,585][INFO ][o.e.n.Node ] [ogonz] initialized
[2022-08-28T09:07:10,586][INFO ][o.e.n.Node ] [ogonz] starting ...
[2022-08-28T09:07:10,595][INFO ][o.e.x.s.c.f.PersistentCache] [ogonz] persistent cache index loaded
[2022-08-28T09:07:10,595][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [ogonz] deprecation component started
[2022-08-28T09:07:10,682][INFO ][o.e.t.TransportService ] [ogonz] publish_address {10.4.4.37:9300}, bound_addresses {[::]:9300}
[2022-08-28T09:07:10,995][INFO ][o.e.b.BootstrapChecks ] [ogonz] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-28T09:07:11,011][WARN ][o.e.c.c.ClusterBootstrapService] [ogonz] this node is locked into cluster UUID [0cU4ZYabTQOY25nrWQlQVw] but [cluster.initial_master_nodes] is set to [ogonz]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts
[2022-08-28T09:07:11,148][INFO ][o.e.c.s.MasterService ] [ogonz] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {ogonz}{CF15-O5dRla3zp4xSsWlwA}{vkxqvMeSQzyXVV6M8lhicw}{ogonz}{10.4.4.37}{10.4.4.37:9300}{cdfhilmrstw} completing election], term: 4, version: 121, delta: master node changed {previous [], current [{ogonz}{CF15-O5dRla3zp4xSsWlwA}{vkxqvMeSQzyXVV6M8lhicw}{ogonz}{10.4.4.37}{10.4.4.37:9300}{cdfhilmrstw}]}
[2022-08-28T09:07:11,198][INFO ][o.e.c.s.ClusterApplierService] [ogonz] master node changed {previous [], current [{ogonz}{CF15-O5dRla3zp4xSsWlwA}{vkxqvMeSQzyXVV6M8lhicw}{ogonz}{10.4.4.37}{10.4.4.37:9300}{cdfhilmrstw}]}, term: 4, version: 121, reason: Publication{term=4, version=121}
[2022-08-28T09:07:11,232][INFO ][o.e.h.AbstractHttpServerTransport] [ogonz] publish_address {10.4.4.37:9200}, bound_addresses {[::]:9200}
[2022-08-28T09:07:11,233][INFO ][o.e.n.Node ] [ogonz] started {ogonz}{CF15-O5dRla3zp4xSsWlwA}{vkxqvMeSQzyXVV6M8lhicw}{ogonz}{10.4.4.37}{10.4.4.37:9300}{cdfhilmrstw}{ml.machine_memory=8342769664, xpack.installed=true, ml.max_jvm_size=4173332480}
[2022-08-28T09:07:11,687][INFO ][o.e.l.LicenseService ] [ogonz] license [4f41da91-8a68-4c5d-b237-6c09c3117ea4] mode [basic] - valid
[2022-08-28T09:07:11,687][INFO ][o.e.x.s.a.Realms ] [ogonz] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-08-28T09:07:11,690][INFO ][o.e.g.GatewayService ] [ogonz] recovered [10] indices into cluster_state
[2022-08-28T09:07:11,722][ERROR][o.e.i.g.GeoIpDownloader ] [ogonz] exception during geoip databases update
org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:134) ~[ingest-geoip-8.3.3.jar:8.3.3]
at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274) [ingest-geoip-8.3.3.jar:8.3.3]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102) [ingest-geoip-8.3.3.jar:8.3.3]
at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48) [ingest-geoip-8.3.3.jar:8.3.3]
at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-8.3.3.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:769) [elasticsearch-8.3.3.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-8.3.3.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-08-28T09:07:12,197][INFO ][o.e.c.r.a.AllocationService] [ogonz] current.health="GREEN" message="Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.ds-ilm-history-5-2022.08.22-000001][0]]])." previous.health="RED" reason="shards started [[.ds-ilm-history-5-2022.08.22-000001][0]]"
[2022-08-28T09:07:12,273][INFO ][o.e.i.g.DatabaseNodeService] [ogonz] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[2022-08-28T09:07:12,379][INFO ][o.e.i.g.DatabaseNodeService] [ogonz] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2022-08-28T09:07:13,049][INFO ][o.e.i.g.DatabaseNodeService] [ogonz] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2022-08-28T09:08:55,045][WARN ][o.e.x.s.a.RealmsAuthenticator] [ogonz] Authentication to realm default_native failed - Password authentication failed for filebeat_admin
if I try to use curl with same filebeat_admin I'm getting this result
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u filebeat_admin https://10.4.4.37:9200
Enter host password for user 'filebeat_admin':
{
"name" : "ogonz",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "0cU4ZYabTQOY25nrWQlQVw",
"version" : {
"number" : "8.3.3",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "801fed82df74dbe537f89b71b098ccaff88d2c56",
"build_date" : "2022-07-23T19:30:09.227964828Z",
"build_snapshot" : false,
"lucene_version" : "9.2.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
