Connect to remote elasticsearch cluster

Elastic Stack Elasticsearch
1

I have an elasticsearch cluster running on an EC2 server. I get a variety of different error mesages when I try to connect.

Currently, in the elasticsearch.yml file all the transport items are commented out but I have tried:

network.host: 0.0.0.0
and
network.host: ec2-xx-xxx-xxx.aws.instance.com

In my flask app the code is as follows:

from datetime import datetime
from flask import Flask, jsonify, request
from elasticsearch import Elasticsearch

#es = Elasticsearch()http://34.245.51.240/
es = Elasticsearch(['ec2-34-xxx-xx-240.eu-west-1.compute.amazonaws.com','9200'])
#es = Elasticsearch(['34.245.51.240','9200'])

application = Flask(__name__)

@application.route('/', methods=['GET'])
def index():
    #results = es.get(index='contents', doc_type='title', id='my-new-slug')
    #return jsonify(results['_source'])
    doc = {
    'author': 'kimchy',
    'text': 'Elasticsearch: cool. bonsai cool.',
    'timestamp': datetime.now(),
    }
    res = es.index(index="test-index", doc_type='tweet', id=1, body=doc)
    print(res['res'])
    return res


#application.run(port=5000, debug=True)
if __name__ == '__main__':
    application.debug = True
    application.run()

I have googled multiple times and tried every possible configuration that I can find.

What is the correct way to achieve this?

Thank you

2

What are the elasticsearch logs when you start it?

3

thank you for your interest! Here are the logs in 2 parts:

bin/elasticsearch
[2018-12-13T15:06:38,320][INFO ][o.e.e.NodeEnvironment ] [gJL14rf] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [4.8gb], net total_space [7.6gb], types [ext4]
[2018-12-13T15:06:38,335][INFO ][o.e.e.NodeEnvironment ] [gJL14rf] heap size [1015.6mb], compressed ordinary object pointers [true]
[2018-12-13T15:06:38,345][INFO ][o.e.n.Node ] [gJL14rf] node name derived from node ID [gJL14rfqSFSq4f8n36ozxA]; set [node.name] to override
[2018-12-13T15:06:38,348][INFO ][o.e.n.Node ] [gJL14rf] version[6.5.2], pid[24992], build[default/tar/9434bed/2018-11-29T23:58:20.891072Z], OS[Linux/4.4.0-1072-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]
[2018-12-13T15:06:38,348][INFO ][o.e.n.Node ] [gJL14rf] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.Sjg8BGUS, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home/ubuntu/Downloads/elasticsearch-6.5.2, -Des.path.conf=/home/ubuntu/Downloads/elasticsearch-6.5.2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2018-12-13T15:06:41,849][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [aggs-matrix-stats]
[2018-12-13T15:06:41,849][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [analysis-common]
[2018-12-13T15:06:41,850][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [ingest-common]
[2018-12-13T15:06:41,850][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [lang-expression]
[2018-12-13T15:06:41,850][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [lang-mustache]
[2018-12-13T15:06:41,851][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [lang-painless]
[2018-12-13T15:06:41,851][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [mapper-extras]
[2018-12-13T15:06:41,851][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [parent-join]
[2018-12-13T15:06:41,851][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [percolator]
[2018-12-13T15:06:41,851][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [rank-eval]
[2018-12-13T15:06:41,851][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [reindex]
[2018-12-13T15:06:41,851][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [repository-url]
[2018-12-13T15:06:41,852][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [transport-netty4]
[2018-12-13T15:06:41,852][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [tribe]
[2018-12-13T15:06:41,854][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-ccr]
[2018-12-13T15:06:41,854][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-core]
[2018-12-13T15:06:41,854][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-deprecation]

4

[2018-12-13T15:06:41,855][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-graph]
[2018-12-13T15:06:41,855][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-logstash]
[2018-12-13T15:06:41,856][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-ml]
[2018-12-13T15:06:41,856][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-monitoring]
[2018-12-13T15:06:41,856][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-rollup]
[2018-12-13T15:06:41,856][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-security]
[2018-12-13T15:06:41,856][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-sql]
[2018-12-13T15:06:41,857][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-upgrade]
[2018-12-13T15:06:41,857][INFO ][o.e.p.PluginsService ] [gJL14rf] loaded module [x-pack-watcher]
[2018-12-13T15:06:41,864][INFO ][o.e.p.PluginsService ] [gJL14rf] no plugins loaded
[2018-12-13T15:06:50,498][INFO ][o.e.x.s.a.s.FileRolesStore] [gJL14rf] parsed [0] roles from file [/home/ubuntu/Downloads/elasticsearch-6.5.2/config/roles.yml]
[2018-12-13T15:06:51,580][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [gJL14rf] [controller/25042] [Main.cc@109] controller (64 bit): Version 6.5.2 (Build 767566e25172d6) Copyright (c) 2018 Elasticsearch BV
[2018-12-13T15:06:52,396][DEBUG][o.e.a.ActionModule ] [gJL14rf] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-12-13T15:06:52,835][INFO ][o.e.d.DiscoveryModule ] [gJL14rf] using discovery type [zen] and host providers [settings]
[2018-12-13T15:06:54,487][INFO ][o.e.n.Node ] [gJL14rf] initialized
[2018-12-13T15:06:54,496][INFO ][o.e.n.Node ] [gJL14rf] starting ...
[2018-12-13T15:06:54,781][INFO ][o.e.t.TransportService ] [gJL14rf] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-12-13T15:06:54,901][WARN ][o.e.b.BootstrapChecks ] [gJL14rf] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-12-13T15:06:57,994][INFO ][o.e.c.s.MasterService ] [gJL14rf] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {gJL14rf}{gJL14rfqSFSq4f8n36ozxA}{MgmuOUa2Qn2PB7heuOlBag}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=2095804416, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-12-13T15:06:58,012][INFO ][o.e.c.s.ClusterApplierService] [gJL14rf] new_master {gJL14rf}{gJL14rfqSFSq4f8n36ozxA}{MgmuOUa2Qn2PB7heuOlBag}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=2095804416, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {gJL14rf}{gJL14rfqSFSq4f8n36ozxA}{MgmuOUa2Qn2PB7heuOlBag}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=2095804416, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-12-13T15:06:58,073][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [gJL14rf] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-12-13T15:06:58,074][INFO ][o.e.n.Node ] [gJL14rf] started
[2018-12-13T15:06:58,858][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [gJL14rf] Failed to clear cache for realms []
[2018-12-13T15:06:58,956][INFO ][o.e.l.LicenseService ] [gJL14rf] license [30aa663e-c285-497c-8dc2-bed1ed186a8b] mode [basic] - valid
[2018-12-13T15:06:58,976][INFO ][o.e.g.GatewayService ] [gJL14rf] recovered [1] indices into cluster_state
[2018-12-13T15:06:59,395][INFO ][o.e.c.r.a.AllocationService] [gJL14rf] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.kibana_1][0]] ...]).

5

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

Anyway here your server is starting to listen on 127.0.0.1:9200. Which means that you can connect to it only from localhost.
Change network.host to make it listen on your IP address instead of localhost.

6

Thank you for your guidance. Your proposal does not work/ It seems the problem is the ip address cannot be bound to. Logs below. I think I have formatted incorrectly again but I did try.

indent preformatted text by 4 spaces

[2018-12-13T18:05:05,024][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [gJL14rf] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to bind to [9300-9400]]; nested: BindException[Cannot assign requested address];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.2.jar:6.5.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.2.jar:6.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.2.jar:6.5.2]
Caused by: org.elasticsearch.transport.BindTransportException: Failed to bind to [9300-9400]
at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:583) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:548) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:132) ~[?:?]
at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:92) ~[?:?]
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4ServerTransport.java:41) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:62) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:255) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:62) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.node.Node.start(Node.java:720) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:269) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-6.5.2.jar:6.5.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.2.jar:6.5.2]
... 6 more
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
at sun.nio.ch.Net.bind(Net.java:425) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130) ~[?:?]
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019) ~[?:?]
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) ~[?:?]
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2018-12-13T18:05:05,694][INFO ][o.e.n.Node ] [gJL14rf] stopping ...
[2018-12-13T18:05:05,701][INFO ][o.e.n.Node ] [gJL14rf] stopped
[2018-12-13T18:05:05,702][INFO ][o.e.n.Node ] [gJL14rf] closing ...
[2018-12-13T18:05:05,735][INFO ][o.e.n.Node ] [gJL14rf] closed
[2018-12-13T18:05:05,737][INFO ][o.e.x.m.j.p.NativeController] [gJL14rf] Native controller process has stopped - no new native processes can be started

7

formatted code here, I hope:

   [o.e.b.ElasticsearchUncaughtExceptionHandler] [gJL14rf] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to bind to [9300-9400]]; nested: BindException[Cannot assign requested address];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.5.2.jar:6.5.2]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.5.2.jar:6.5.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.5.2.jar:6.5.2]
Caused by: org.elasticsearch.transport.BindTransportException: Failed to bind to [9300-9400]
	at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:583) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:548) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:132) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:92) ~[?:?]
	at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4ServerTransport.java:41) ~[?:?]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:62) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.transport.TransportService.doStart(TransportService.java:255) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:62) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.node.Node.start(Node.java:720) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:269) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-6.5.2.jar:6.5.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.5.2.jar:6.5.2]
	... 6 more
Caused by: java.net.BindException: Cannot assign requested address
	at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:433) ~[?:?]
	at sun.nio.ch.Net.bind(Net.java:425) ~[?:?]
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) ~[?:?]
	at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130) ~[?:?]
	at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) ~[?:?]
	at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) ~[?:?]
	at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019) ~[?:?]
	at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) ~[?:?]
	at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[?:?]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[?:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2018-12-13T17:58:46,826][INFO ][o.e.n.Node               ] [gJL14rf] stopping ...
[2018-12-13T17:58:46,832][INFO ][o.e.n.Node               ] [gJL14rf] stopped
[2018-12-13T17:58:46,832][INFO ][o.e.n.Node               ] [gJL14rf] closing ...
[2018-12-13T17:58:46,857][INFO ][o.e.n.Node               ] [gJL14rf] closed
[2018-12-13T17:58:46,859][INFO ][o.e.x.m.j.p.NativeController] [gJL14rf] Native controller process has stopped - no new native processes can be started

figured it out, thank you!
Richard

8

So? Did it work at the end? If so may be share what the error was in case someone else read this thread in the future?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.