Connect to Elasticsearch remotely

Elastic Stack Elasticsearch
1

Hello community,

I have a problem. I have installed on ubuntu 20.04 an ELK stack 7.15.2. My goal is to connect remotely to Elasticsearch to ingest data from a web form into Elasticsearch and my problem is that I cannot connect to my Elasticsearch remotely. Every port needed is open also here and also on the remote machine which is also running ubuntu 20.04. I tried everything that I found here. I will share my configs:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1


# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#discovery.seed_hosts: ["0.0.0.0", "[::]"]
#
#
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: true
discovery.type: single-node

The command that I want to execute for testing is this :

curl -XGET -u elastic 'elasticseachIP/_cat/health?v&pretty'

When I execute this on localhost is work obviously. Bu when I execute this outside of localhost is giving me connection refused:

Enter host password for user 'elastic':
curl: (28) Failed to connect to xxx.xxx.xxx port 9200: Connection timed out

Every help is welcomed, thank you!
Alex

2

Can you share your Elasticsearch logs?

3

yes, sure

[2021-11-23T08:26:10,842][INFO ][o.e.x.m.p.NativeController] [elk-v1] Native controller process has stopped - no new native processes can be started
[2021-11-23T08:26:11,594][INFO ][o.e.n.Node               ] [elk-v1] stopped
[2021-11-23T08:26:11,595][INFO ][o.e.n.Node               ] [elk-v1] closing ...
[2021-11-23T08:26:11,604][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [elk-v1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-10253682917825219796/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-Country.mmdb]
[2021-11-23T08:26:11,604][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [elk-v1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-10253682917825219796/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-ASN.mmdb]
[2021-11-23T08:26:11,604][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [elk-v1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-10253682917825219796/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-City.mmdb]
[2021-11-23T08:26:11,608][INFO ][o.e.n.Node               ] [elk-v1] closed
[2021-11-23T08:26:17,203][INFO ][o.e.n.Node               ] [elk-v1] version[7.15.2], pid[2256], build[default/deb/93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c/2021-11-04T14:04:42.515624022Z], OS[Linux/5.11.0-1021-azure/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.1/17.0.1+12]
[2021-11-23T08:26:17,210][INFO ][o.e.n.Node               ] [elk-v1] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-11-23T08:26:17,210][INFO ][o.e.n.Node               ] [elk-v1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-10097663164985396688, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3979m, -Xmx3979m, -XX:MaxDirectMemorySize=2086666240, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-11-23T08:26:19,885][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [aggs-matrix-stats]
[2021-11-23T08:26:19,886][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [analysis-common]
[2021-11-23T08:26:19,886][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [constant-keyword]
[2021-11-23T08:26:19,887][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [frozen-indices]
[2021-11-23T08:26:19,888][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [ingest-common]
[2021-11-23T08:26:19,889][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [ingest-geoip]
[2021-11-23T08:26:19,889][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [ingest-user-agent]
[2021-11-23T08:26:19,890][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [kibana]
[2021-11-23T08:26:19,890][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [lang-expression]
[2021-11-23T08:26:19,890][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [lang-mustache]
[2021-11-23T08:26:19,891][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [lang-painless]
[2021-11-23T08:26:19,891][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [mapper-extras]
[2021-11-23T08:26:19,892][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [mapper-version]
[2021-11-23T08:26:19,892][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [parent-join]
[2021-11-23T08:26:19,893][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [percolator]
[2021-11-23T08:26:19,893][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [rank-eval]
[2021-11-23T08:26:19,893][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [reindex]
[2021-11-23T08:26:19,894][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [repositories-metering-api]
[2021-11-23T08:26:19,894][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [repository-encrypted]
[2021-11-23T08:26:19,894][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [repository-url]
[2021-11-23T08:26:19,894][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [runtime-fields-common]
[2021-11-23T08:26:19,894][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [search-business-rules]
[2021-11-23T08:26:19,895][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [searchable-snapshots]
[2021-11-23T08:26:19,895][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [snapshot-repo-test-kit]
[2021-11-23T08:26:19,895][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [spatial]
[2021-11-23T08:26:19,896][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [systemd]
[2021-11-23T08:26:19,896][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [transform]
[2021-11-23T08:26:19,897][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [transport-netty4]
[2021-11-23T08:26:19,897][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [unsigned-long]
[2021-11-23T08:26:19,897][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [vector-tile]
[2021-11-23T08:26:19,898][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [vectors]
[2021-11-23T08:26:19,898][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [wildcard]
[2021-11-23T08:26:19,898][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-aggregate-metric]
[2021-11-23T08:26:19,898][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-analytics]
[2021-11-23T08:26:19,898][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-async]
[2021-11-23T08:26:19,899][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-async-search]
[2021-11-23T08:26:19,899][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-autoscaling]
[2021-11-23T08:26:19,900][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-ccr]
[2021-11-23T08:26:19,900][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-core]
[2021-11-23T08:26:19,900][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-data-streams]
[2021-11-23T08:26:19,901][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-deprecation]
[2021-11-23T08:26:19,901][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-enrich]
[2021-11-23T08:26:19,901][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-eql]
[2021-11-23T08:26:19,902][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-fleet]
[2021-11-23T08:26:19,902][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-graph]
[2021-11-23T08:26:19,905][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-identity-provider]
[2021-11-23T08:26:19,905][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-ilm]
[2021-11-23T08:26:19,905][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-logstash]
[2021-11-23T08:26:19,905][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-ml]
[2021-11-23T08:26:19,905][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-monitoring]
[2021-11-23T08:26:19,905][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-ql]
[2021-11-23T08:26:19,906][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-rollup]
[2021-11-23T08:26:19,907][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-security]
[2021-11-23T08:26:19,908][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-shutdown]
[2021-11-23T08:26:19,909][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-sql]
[2021-11-23T08:26:19,909][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-stack]
[2021-11-23T08:26:19,911][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-text-structure]
[2021-11-23T08:26:19,911][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-voting-only-node]
[2021-11-23T08:26:19,912][INFO ][o.e.p.PluginsService     ] [elk-v1] loaded module [x-pack-watcher]
[2021-11-23T08:26:19,913][INFO ][o.e.p.PluginsService     ] [elk-v1] no plugins loaded
[2021-11-23T08:26:19,958][INFO ][o.e.e.NodeEnvironment    ] [elk-v1] using [1] data paths, mounts [[/ (/dev/root)]], net usable_space [23.4gb], net total_space [28.9gb], types [ext4]
4

Is that it?

5

I will post it in more blocks

[2021-11-23T08:26:19,960][INFO ][o.e.e.NodeEnvironment    ] [elk-v1] heap size [3.8gb], compressed ordinary object pointers [true]
[2021-11-23T08:26:20,055][INFO ][o.e.n.Node               ] [elk-v1] node name [elk-v1], node ID [Lq6LiQlbSGO946ePDPw4Sw], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-11-23T08:26:24,979][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elk-v1] [controller/2446] [Main.cc@122] controller (64 bit): Version 7.15.2 (Build 65497bb5299534) Copyright (c) 2021 Elasticsearch BV
[2021-11-23T08:26:25,588][INFO ][o.e.x.s.a.Realms         ] [elk-v1] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2021-11-23T08:26:25,596][INFO ][o.e.x.s.a.s.FileRolesStore] [elk-v1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-11-23T08:26:26,279][INFO ][o.e.i.g.LocalDatabases   ] [elk-v1] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2021-11-23T08:26:26,284][INFO ][o.e.i.g.DatabaseRegistry ] [elk-v1] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-10097663164985396688/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw]
[2021-11-23T08:26:26,962][INFO ][o.e.t.NettyAllocator     ] [elk-v1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-11-23T08:26:27,032][INFO ][o.e.d.DiscoveryModule    ] [elk-v1] using discovery type [single-node] and seed hosts providers [settings]
[2021-11-23T08:26:27,545][INFO ][o.e.g.DanglingIndicesState] [elk-v1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-11-23T08:26:28,009][INFO ][o.e.n.Node               ] [elk-v1] initialized
[2021-11-23T08:26:28,010][INFO ][o.e.n.Node               ] [elk-v1] starting ...
[2021-11-23T08:26:28,356][INFO ][o.e.x.s.c.f.PersistentCache] [elk-v1] persistent cache index loaded
[2021-11-23T08:26:28,483][INFO ][o.e.t.TransportService   ] [elk-v1] publish_address {10.0.0.6:9300}, bound_addresses {[::]:9300}
[2021-11-23T08:26:29,422][INFO ][o.e.c.c.Coordinator      ] [elk-v1] cluster UUID [hqWA7XQLSJequXXVNAhn9w]
[2021-11-23T08:26:29,605][INFO ][o.e.c.s.MasterService    ] [elk-v1] elected-as-master ([1] nodes joined)[{elk-v1}{Lq6LiQlbSGO946ePDPw4Sw}{9FmcpKNFR8OykERq3xnmlw}{10.0.0.6}{10.0.0.6:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 24, version: 693, delta: master node changed {previous [], current [{elk-v1}{Lq6LiQlbSGO946ePDPw4Sw}{9FmcpKNFR8OykERq3xnmlw}{10.0.0.6}{10.0.0.6:9300}{cdfhilmrstw}]}
[2021-11-23T08:26:29,832][INFO ][o.e.c.s.ClusterApplierService] [elk-v1] master node changed {previous [], current [{elk-v1}{Lq6LiQlbSGO946ePDPw4Sw}{9FmcpKNFR8OykERq3xnmlw}{10.0.0.6}{10.0.0.6:9300}{cdfhilmrstw}]}, term: 24, version: 693, reason: Publication{term=24, version=693}
[2021-11-23T08:26:29,900][INFO ][o.e.h.AbstractHttpServerTransport] [elk-v1] publish_address {10.0.0.6:9200}, bound_addresses {[::]:9200}
[2021-11-23T08:26:29,901][INFO ][o.e.n.Node               ] [elk-v1] started
[2021-11-23T08:26:31,281][ERROR][o.e.x.s.a.e.ReservedRealm] [elk-v1] failed to retrieve password hash for reserved user [kibana_system]
org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-7] is unavailable
        at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:148) ~[x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getReservedUserInfo(NativeUsersStore.java:493) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.getUserInfo(ReservedRealm.java:220) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.esnative.ReservedRealm.doAuthenticate(ReservedRealm.java:96) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:188) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:105) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$10(AuthenticationService.java:488) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:103) [x-pack-core-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:543) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForApiKey$5(AuthenticationService.java:413) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.ApiKeyService.authenticateWithApiKeyIfPresent(ApiKeyService.java:440) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForApiKey(AuthenticationService.java:387) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$checkForBearerToken$3(AuthenticationService.java:371) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.TokenService.tryAuthenticateToken(TokenService.java:393) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.checkForBearerToken(AuthenticationService.java:367) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:349) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:274) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:152) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:137) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:75) [x-pack-security-7.15.2.jar:7.15.2]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:275) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:343) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:196) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:348) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:413) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:330) [elasticsearch-7.15.2.jar:7.15.2]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.15.2.jar:7.15.2]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:17) [transport-netty4-client-7.15.2.jar:7.15.2]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
6

Sorry for long blocks but the limit is 13k :frowning:

at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:47) [transport-netty4-client-7.15.2.jar:7.15.2]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2021-11-23T08:26:31,293][INFO ][o.e.x.s.a.AuthenticationService] [elk-v1] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]
[2021-11-23T08:26:31,295][INFO ][o.e.l.LicenseService     ] [elk-v1] license [5b3210a6-68d6-4316-b42b-73d0e6b4c932] mode [trial] - valid
[2021-11-23T08:26:31,296][INFO ][o.e.x.s.a.Realms         ] [elk-v1] license mode is [trial], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2021-11-23T08:26:31,296][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [elk-v1] Active license is now [TRIAL]; Security is enabled
[2021-11-23T08:26:31,305][INFO ][o.e.g.GatewayService     ] [elk-v1] recovered [19] indices into cluster_state
[2021-11-23T08:26:31,320][WARN ][o.e.l.LicenseService     ] [elk-v1] License [will expire] on [Friday, December 17, 2021].
# If you have a new license, please update it. Otherwise, please reach out to
# your support contact.
#
# Commercial plugins operate with reduced functionality on license expiration:
# - security
#  - Cluster health, cluster stats and indices stats operations are blocked
#  - All data operations (read and write) continue to work
# - watcher
#  - PUT / GET watch APIs are disabled, DELETE watch API continues to work
#  - Watches execute and write to the history
#  - The actions of the watches don't execute
# - monitoring
#  - The agent will stop collecting cluster and indices metrics
#  - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
# - graph
#  - Graph explore APIs are disabled
# - ml
#  - Machine learning APIs are disabled
# - logstash
#  - Logstash will continue to poll centrally-managed pipelines
# - beats
#  - Beats will continue to poll centrally-managed configuration
# - deprecation
#  - Deprecation APIs are disabled
# - upgrade
#  - Upgrade API is disabled
# - sql
#  - SQL support is disabled
# - rollup
#  - Creating and Starting rollup jobs will no longer be allowed.
#  - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.
# - transform
#  - Creating, starting, updating transforms will no longer be allowed.
#  - Stopping/Deleting existing transforms continue to function.
# - analytics
#  - Aggregations provided by Analytics plugin are no longer usable.
# - ccr
#  - Creating new follower indices will be blocked
#  - Configuring auto-follow patterns will be blocked
#  - Auto-follow patterns will no longer discover new leader indices
#  - The CCR monitoring endpoint will be blocked
#  - Existing follower indices will continue to replicate data
7 
[2021-11-23T08:26:32,571][INFO ][o.e.i.g.GeoIpDownloader  ] [elk-v1] updating geoip databases
[2021-11-23T08:26:32,571][INFO ][o.e.i.g.GeoIpDownloader  ] [elk-v1] fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]
[2021-11-23T08:26:33,283][INFO ][o.e.i.g.GeoIpDownloader  ] [elk-v1] geoip database [GeoLite2-ASN.mmdb] is up to date, updated timestamp
[2021-11-23T08:26:33,479][INFO ][o.e.x.t.t.TransformTask  ] [elk-v1] [endpoint.metadata_current-default-1.1.1] updating state for transform to [{"task_state":"started","indexer_state":"stopped","checkpoint":1,"progress":{"docs_indexed":0,"docs_processed":0},"should_stop_at_checkpoint":false}].
[2021-11-23T08:26:33,946][INFO ][o.e.i.g.DatabaseRegistry ] [elk-v1] downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/elasticsearch-10097663164985396688/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-Country.mmdb.tmp.gz]
[2021-11-23T08:26:33,947][INFO ][o.e.i.g.DatabaseRegistry ] [elk-v1] downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-10097663164985396688/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-ASN.mmdb.tmp.gz]
[2021-11-23T08:26:33,955][INFO ][o.e.i.g.DatabaseRegistry ] [elk-v1] downloading geoip database [GeoLite2-City.mmdb] to [/tmp/elasticsearch-10097663164985396688/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-City.mmdb.tmp.gz]
[2021-11-23T08:26:34,231][INFO ][o.e.i.g.GeoIpDownloader  ] [elk-v1] geoip database [GeoLite2-City.mmdb] is up to date, updated timestamp
[2021-11-23T08:26:34,579][INFO ][o.e.i.g.DatabaseRegistry ] [elk-v1] successfully reloaded changed geoip database file [/tmp/elasticsearch-10097663164985396688/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-Country.mmdb]
[2021-11-23T08:26:34,654][INFO ][o.e.x.t.t.TransformPersistentTasksExecutor] [elk-v1] [endpoint.metadata_current-default-1.1.1] successfully completed and scheduled task in node operation
[2021-11-23T08:26:34,756][INFO ][o.e.i.g.DatabaseRegistry ] [elk-v1] successfully reloaded changed geoip database file [/tmp/elasticsearch-10097663164985396688/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-ASN.mmdb]
[2021-11-23T08:26:35,372][INFO ][o.e.x.s.a.AuthorizationService] [elk-v1] Took [72ms] to resolve [13] indices for action [indices:data/read/get] and user [kibana_system]
[2021-11-23T08:26:35,685][INFO ][o.e.i.g.GeoIpDownloader  ] [elk-v1] geoip database [GeoLite2-Country.mmdb] is up to date, updated timestamp
[2021-11-23T08:26:36,143][INFO ][o.e.c.r.a.AllocationService] [elk-v1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[test][0], [.ds-ilm-history-5-2021.11.17-000001][0], [.kibana-event-log-7.15.2-000001][0]]]).
[2021-11-23T08:26:36,690][INFO ][o.e.i.g.DatabaseRegistry ] [elk-v1] successfully reloaded changed geoip database file [/tmp/elasticsearch-10097663164985396688/geoip-databases/Lq6LiQlbSGO946ePDPw4Sw/GeoLite2-City.mmdb]
8

No worries, things look ok there from what I can see.

It is binding to publish_address {10.0.0.6:9200}, so is the host you are trying to connect from on the same network?

1 Like
9

My elk instance is running on 20.67.29.XXX and I am tryin to connect from 51.145.xxx to elk (20.67.29.XXX ). This is my firewall in elk:

 ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx HTTP                 ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
3306                       ALLOW       Anywhere
9200                       ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx HTTP (v6)            ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
3306 (v6)                  ALLOW       Anywhere (v6)
9200 (v6)                  ALLOW       Anywhere (v6)
10

Ahh ok, well Elasticsearch doesn't see the 20.X.X.X address on the host. Is it attached to an interface on that host?

11

I did not configure that :confused: But doing a ifconfig I see 10.0.0.6:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.6  netmask 255.255.XXX  broadcast 10.0.0.XXXX
12

Ok that's the gap then.

1 Like
13

Aaa ok, So my public ip 20.67.XXX is corelate it with the private one 10.0.0.6?
I tested the curl on 10.0.06 remotely and is working.
Is there a possibility to access the public ip instead of private one ?

14

You will need to ask your network admin about how to set that up, it's outside the scope of what we can do here sorry.

15

Yes sure, thank you for your help and time

1 Like
16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.