I am trying to host an Elasticsearch Application on an AWS Server. For this purpose I am using Elastic Enterprise Search. I configured the API connector as follows:
The application itself is running with npm under port 3000. I enabled both ports in the AWS Security Group.
After starting both Elastic Search and Enterprise Search, after npm start I can also connect to http://ec2-18-159-135-25.eu-central-1.compute.amazonaws.com:3000, so that works. However, once I make an API call over the application, I get the following error:
Moreover, when I type curl http://localhost:3002/api/as/v1/engines/sentence-similarity/search.json in the console, I get the following error:
{"error":"You need to sign in or sign up before continuing."}
Therefore I assume, that there is a problem with the connection to Elasticsearch. When I was hosting the application on my local machine, I never encountered this problem.
My ideas, how to possibly fix the problem:
I had to reinstall Elastic Search and Enterprise Search on the AWS
Server. Possibly, something with the authentification went wrong, and
I cannot connect to my Elasticsearch engine.
There are two .yml files, one for Elastic Search and one for Enterprise Search, which I can configure. Possibly, I have to adjust these, when I host the application not on my local machine.
In any case, I am a beginner in deployment and I need some help for the details. Thank you very much in advance!
There are two .yml files, one for Elastic Search and one for Enterprise Search, which I can configure. Possibly, I have to adjust these, when I host the application not on my local machine.
Full disclaimer, I also mostly work locally/in dev, but this sounds like the most likely culprit to me! On production builds, you definitely have to update the elasticsearch.yml and enterprise-search.yml files. On dev builds, there are a lot of fallbacks built in (the config/yml files assume localhost:9200 and localhost:3002, for example).
Off the top of my head, for enterprise-search.yml you need at least the following configs:
Thank you very much for your reply. Unfortunately I am totally lost here and the documentation confuses me a lot. Can someone help me? I assume, that with the information that I have posted it should potentially possible to configure everything. I would be so grateful, I am working on that since a long time now
At the moment, my files are configured as follows:
secret_management.encryption_keys: ['']
allow_es_settings_modification: true
elasticsearch.username: elastic
elasticsearch.password: ''
ent_search.auth.source: standard
Also, I feel like when I start elasticsearch it wants to connect me to a new instance instead of my existing one. I for example checked, that when I log into the app search interface, there are no engines installed.
It looks like you have 2 of the 3. Maybe try adding action.auto_create_index just in case. Oh, and can you note what version you're on? I'm assuming latest / 7.9.2?
enterprise-search.yml looks like it's missing elasticsearch.host - can you confirm you have it and that it's set to the correct URL? You don't have to paste it here (obviously, for security reasons - also I'd suggest replacing your elasticsearch.password with a '' as well!), just verbally confirm that it's there.
Also a quick reminder - don't forget to hard stop & restart both Elasticsearch and Enterprise Search whenever you change a config yml file! They don't restart automatically and the changes won't take if you don't manually restart.
Yes, elasticsearch.host should be whatever AWS server address you are hosting Elasticsearch itself on. I'm not totally sure what the setup is on your specific AWS instance (for example on Elastic Cloud, they're different URLs entirely), but if you're running Elasticsearch and Enterprise Search on the same server/box, then it's the same URL with a different port (likely aws-server-adress:9200, not 3000).
Do I have to open port 9200 as well?
Yes, if your Elasticsearch instance is set to run on port 9200. I believe this is the default, so unless you've changed the default port in your elasticsearch.yml settings, you must expose this port publicly for Enterprise Search to be able to connect to it.
Thank you a lot for your help. Unfortunately, I am still not making progress. Especially one thing is not clear to me:
I can manage to host Elasticsearch and Enterprise Search on my local machine. When I installed Enterprise Search, I had to login on localhost:3002/login and create an engine. This is the engine I want to use, it is called "similarity-search". Now I installed Elasticsearch and Enterprise search on the AWS server. Again, after installing Enterprise Search I can login at localhost:3002/login, but this is another instance (or I don't know if that's the correct term here). Anyways, the engine "similarity-search" does not exist there. How can I use the instance and engine that already exists? I would have assumed, that there is an individual username or something, but the username to log in there is enterprise_search in both cases.
A solution to me would have been, to just create a fresh engine on the AWS server. But this is not possible either, since I can only communicate over the terminal and in order to create an engine I need credentials, which I cannot see without the user interface of App Search. That's I want to connect to localhost:3002/login from my local machine, i.e. by aws-server:3002/login within my browser. But this leads to the error:
Failed to connect to ec2-XXXXX.eu-central-1.compute.amazonaws.com port 3002: Connection refused
I opened the following ports in my AWS security group:
Not answering to your exact problem, but why not just starting enterprise search from cloud.elastic.co in the same AWS region? You will have everything running in few seconds.
On the one hand I unfortunately don't have the authority to decide that yet - apart from that, we are using Elasticsearch in a different way than it is built for. We interact with our own API and don't upload any documents for indexing. I am not sure if that is easily transferrable to the cloud. Anyways, it is working on my local machine. I don't understand how it can be so difficult to get it to run on an AWS server
Super sorry for the delay on responding Spadel, it's been a busy week!
I'm a little confused by your references back to your local machine. Just to make sure we're on the same page: Engines you create on localhost will not transfer to AWS. This is because they're stored on completely different Elasticsearch instances/"databases".
Anyways, the engine "similarity-search" does not exist there. How can I use the instance and engine that already exists?
I really wouldn't try. Treat your localhost and your AWS instance (presumably production or staging) as separate instances. This is a general recommended developer best practice.
I would have assumed, that there is an individual username or something, but the username to log in there is enterprise_search in both cases.
That's correct. enterprise_search is the default user on all Enterprise Search instances. You can create more users from there, but that is the default "owner"/admin. Please note that their passwords should however differ per-instance.
Creating a fresh engine on the AWS server is the correct approach. I'm confused as to why you can't access App Search's interface. Does going to the :3002 port of your AWS server simply not work? Can you confirm that Enterprise Search is actually running?
I'm going to be honest also, a lot of your issues sound like they're to do with the AWS platform (access/ports etc.), which our company doesn't specialize in - could I recommend contacting AWS's support to get help? (If you do so, I would make sure to be very clear also that you're using Elasticsearch from Elastic Inc., and that you're not using AWS Elasticsearch which is a completely separate product not owned by us).
I finally figured out, that it is for sure the settings in the config .yml files. I was able to experiment with it and make it work, but it is incredibly unstable and crashes the AWS machine. However, this happens only, when I start enterprise-search after elasticsearch. I am pretty sure, that I still haven't configured the yml files correctly. Can someone please take a look?
Here are the relevant information:
Private DNS: ip-172-31-14-57.eu-central-1.compute.internal
Private IPs: 172.31.14.57
Public DNS (IPv4): ec2-3-122-100-12.eu-central-1.compute.amazonaws.com
IPv4 Public IP: 3.122.100.12
Could someone please have a look and see what I am doing wrong? I would appreciate a lot, I ve spent countless hours trying to fix this whole projects.
The first thing that jumps out is your security settings for elasticsearch. xpack.security.enabled needs to be set to true as per our documentation. Feel free to share the error you're hitting when you enable security.
You can validate that elasticsearch is working by doing a:
Note - you may need to change that to be https since your elasticsearch settings also show that you are running with SSL enabled.
For your enterprise_search.yml, you don't need to provide ent_search.external_url, ent_search.listen_host, or ent_search.listen_port, all of those will default in an expected manner. Again, make sure your elasticsearch.host matches up with http vs https for if you're using SSL.
Once you try these changes, please share what your error is, and what your resulting configurations are, and we can continue trying to assist you.
as stated in the documentation, I receive the following error:
uncaught exception in thread [main]
java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.keystore.path] setting
at org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport.<init>(SecurityNetty4HttpServerTransport.java:51)
at org.elasticsearch.xpack.security.Security.lambda$getHttpTransports$25(Security.java:1018)
at org.elasticsearch.node.Node.newHttpTransport(Node.java:1145)
at org.elasticsearch.node.Node.<init>(Node.java:556)
at org.elasticsearch.node.Node.<init>(Node.java:277)
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /home/ubuntu/elasticsearch-7.9.2/logs/elasticsearch.log
[2020-10-20T17:57:21,273][INFO ][o.e.x.m.p.NativeController] [ip-172-31-14-57] Native controller process has stopped - no new native processes can be started
Makes sense - that's because you have xpack.security.http.ssl.enabled: true set. You could change that to xpack.security.http.ssl.enabled: false, or you can read about how to create and configure your SSL keys/keystore/truststore/certificates etc: https://www.elastic.co/guide/en/elasticsearch/reference/current/ssl-tls.html
ERROR: [1] bootstrap checks failed
[1]: HTTPS is required in order to use the API key service; please enable HTTPS using the [xpack.security.http.ssl.enabled] setting or disable the API key service using the [xpack.security.authc.api_key.enabled] setting
But according to the documentation, xpack.security.authc.api_key.enabled should be true.
Ah, looks like you're trying to run it in "production mode". It's definitely a best-practice security-wise to use SSL when transmitting secure data (like API keys). When you're ready for this to be a production system, you absolutely should enable SSL. But that will require you setting up your keys/certs/trustores/etc.
When not running in "production mode", the ERROR that you're currently getting is just a WARN message, and doesn't block elasticsearch from starting.
As a side-note - getting a production-worthy setup, from scratch, is a non-trivial task. Especially when you're new to the stack. It's totally possible, and totally supported, we'll continue to assist along the way. But as @dadoonetsuggested above, the turnkey approach to this is to use Elastic Cloud. This takes away all of the manual infrastructure overhead.
Ahh very interesting, by adding discovery.type: single-node I can run Elasticsearch indeed.
You said, that configuring the enterprise_search.yml with the correct elasticsearch.host would be enough and the rest would work by default. Unfortunately that's not the case, if I don't configure the file the way I did, it will run on the localhost and I can't access it from outside the server. And if I do configure it the way I did, I get back to the initial problem - after running enterprise-search some time, the server freezes and after a couple minutes the enterprise-search process is killed.
@spadel can you share the logs/errors you get when you start the server with your hostname configured, and you observe the "server freezes and after a couple minutes the enterprise-search process is killed"?
I was finally able to make it work. The reason for the freeze was, that there was not enough RAM available. Our server had 4GB and we increased it to 32GB for the moment and it works fine.
Moreover we are running it with xpack.security.enabled set as false, since our application is only for demonstration purposes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
On production builds, you definitely have to update the 
