Elastic Search Error after altering/modifying the elasticsearch.yml file

Elastic Stack Elasticsearch
1

Hi,

I have installed Elasticsearch version 7.17.7 on ubuntu and I have noticed an issue whenever I alter or modify the elasticsearch.yml file within /etc/elasticsearch,

In the elasticsearch.yml file, I want to bind the IP address 192.168.52.88 and specify the port as 9200, like so:

# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 192.168.52.88
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#

The current problem that I am facing is that when I put the following values and restart Elasticsearch using the command "systemctl restart elasticsearch.service", an error shows which is:

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.

and when I look at the logs using the "systemctl status elasticsearch.service" command, the following logs appear:

elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-28 02:34:03 UTC; 2min 44s ago
       Docs: https://www.elastic.co
    Process: 584892 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 584892 (code=exited, status=1/FAILURE)
        CPU: 428ms

Jun 28 02:34:03 va-kibana systemd-entrypoint[584938]:         at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
Jun 28 02:34:03 va-kibana systemd-entrypoint[584938]:         at java.base/java.nio.file.Files.createDirectory(Files.java:700)
Jun 28 02:34:03 va-kibana systemd-entrypoint[584938]:         at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:134)
Jun 28 02:34:03 va-kibana systemd-entrypoint[584938]:         at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:171)
Jun 28 02:34:03 va-kibana systemd-entrypoint[584938]:         at java.base/java.nio.file.Files.createTempDirectory(Files.java:1017)
Jun 28 02:34:03 va-kibana systemd-entrypoint[584938]:         at org.elasticsearch.tools.launchers.Launchers.createTempDirectory(Launchers.java:55)
Jun 28 02:34:03 va-kibana systemd-entrypoint[584938]:         at org.elasticsearch.tools.launchers.TempDirectory.main(TempDirectory.java:43)
Jun 28 02:34:03 va-kibana systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jun 28 02:34:03 va-kibana systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jun 28 02:34:03 va-kibana systemd[1]: Failed to start Elasticsearch.

Okay so this is the first issue.

The thing that I am unable to wrap my head around is why cant Elasticsearch start running normally again after I have commented my previous changes which would look like this:

# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.52.88
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#

To summarize:

  1. I want to know what is causing the error after the changes (uncommenting the IP address & port).

  2. I want to know why reverting the changes (commenting the IP address & port) still causes the error to pop up even though I have restarted the elasticsearch.service.

2

When you have a systemd error you need to look at the system log to have some hints about what is the cause, you need to look at /var/log/syslog or /var/log/messages, depending on the system.

Some errors will not appear when using journalctl or systemctl status, you will only be able to see it looking at the system log.

3 
Jun 26 03:46:41 va-kibana logstash[783]: [2023-06-26T03:46:41,565][WARN ][logstash.outputs.elasticsearch][main][f78b13f4c0ade3e4371f28110b83a375db706503f97162e4ebebf83fd7d06eec] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.17.7-2023.06.26", :routing=>nil, :pipeline=>"filebeat-7.17.7-system-syslog-pipeline"}, {"ecs"=>{"version"=>"1.12.0"}, "tags"=>["beats_input_codec_plain_applied"], "@version"=>"1", "@timestamp"=>2023-06-26T03:46:40.143Z, "host"=>{"hostname"=>"va-kibana", "os"=>{"platform"=>"ubuntu", "version"=>"22.04.1 LTS (Jammy Jellyfish)", "codename"=>"jammy", "family"=>"debian", "type"=>"linux", "name"=>"Ubuntu", "kernel"=>"5.15.0-56-generic"}, "containerized"=>false, "mac"=>["50:6b:8d:e3:64:af"], "architecture"=>"x86_64", "ip"=>["192.168.52.88", "fe80::526b:8dff:fee3:64af"], "name"=>"va-kibana", "id"=>"ca1a3e6099984d4ebde073c9d262e9dc"}, "log"=>{"offset"=>1368684, "file"=>{"path"=>"/var/log/syslog"}}, "fileset"=>{"name"=>"syslog"}, "agent"=>{"ephemeral_id"=>"34f582c0-178c-4a09-aef7-b1555ce957bb", "version"=>"7.17.7", "hostname"=>"va-kibana", "type"=>"filebeat", "name"=>"va-kibana", "id"=>"a2ad6a9f-3526-4974-acce-be8dceae8b14"}, "service"=>{"type"=>"system"}, "event"=>{"dataset"=>"system.syslog", "timezone"=>"+00:00", "module"=>"system"}, "input"=>{"type"=>"log"}, "message"=>"Jun 18 01:34:30 va-kibana logstash[783]: [2023-06-18T01:34:30,360][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>\"http://localhost:9200/\", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>\"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)\"}"}], :response=>{"index"=>{"_index"=>"filebeat-7.17.7-2023.06.26", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [filebeat-7.17.7-system-syslog-pipeline] does not exist"}}}}
Jun 26 03:46:41 va-kibana logstash[783]: [2023-06-26T03:46:41,565][WARN ][logstash.outputs.elasticsearch][main][f78b13f4c0ade3e4371f28110b83a375db706503f97162e4ebebf83fd7d06eec] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.17.7-2023.06.26", :routing=>nil, :pipeline=>"filebeat-7.17.7-system-syslog-pipeline"}, {"ecs"=>{"version"=>"1.12.0"}, "tags"=>["beats_input_codec_plain_applied"], "@version"=>"1", "@timestamp"=>2023-06-26T03:46:40.137Z, "host"=>{"hostname"=>"va-kibana", "os"=>{"platform"=>"ubuntu", "version"=>"22.04.1 LTS (Jammy Jellyfish)", "codename"=>"jammy", "family"=>"debian", "type"=>"linux", "name"=>"Ubuntu", "kernel"=>"5.15.0-56-generic"}, "containerized"=>false, "mac"=>["50:6b:8d:e3:64:af"], "architecture"=>"x86_64", "ip"=>["192.168.52.88", "fe80::526b:8dff:fee3:64af"], "name"=>"va-kibana", "id"=>"ca1a3e6099984d4ebde073c9d262e9dc"}, "log"=>{"offset"=>1337661, "file"=>{"path"=>"/var/log/syslog"}}, "fileset"=>{"name"=>"syslog"}, "agent"=>{"ephemeral_id"=>"34f582c0-178c-4a09-aef7-b1555ce957bb", "version"=>"7.17.7", "hostname"=>"va-kibana", "type"=>"filebeat", "name"=>"va-kibana", "id"=>"a2ad6a9f-3526-4974-acce-be8dceae8b14"}, "service"=>{"type"=>"system"}, "event"=>{"dataset"=>"system.syslog", "timezone"=>"+00:00", "module"=>"system"}, "message"=>"Jun 18 01:32:25 va-kibana logstash[783]: [2023-06-18T01:32:25,235][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>\"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)\", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)}", "input"=>{"type"=>"log"}}], :response=>{"index"=>{"_index"=>"filebeat-7.17.7-2023.06.26", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"pipeline with id [filebeat-7.17.7-system-syslog-pipeline] does not exist"}}}}

This is the log shown in /var/log/syslog.

4

Do note that I have reinstalled Elasticsearch which is now working fine while Kibana is giving me some problems with the error:

× kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-28 06:49:20 UTC; 19min ago
       Docs: https://www.elastic.co
    Process: 597711 ExecStart=/usr/share/kibana/bin/kibana --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid --deprecation.skip_de>
   Main PID: 597711 (code=exited, status=1/FAILURE)
        CPU: 14.835s

Jun 28 06:49:20 va-kibana systemd[1]: kibana.service: Scheduled restart job, restart counter is at 4.
Jun 28 06:49:20 va-kibana systemd[1]: Stopped Kibana.
Jun 28 06:49:20 va-kibana systemd[1]: kibana.service: Consumed 14.835s CPU time.
Jun 28 06:49:20 va-kibana systemd[1]: kibana.service: Start request repeated too quickly.
Jun 28 06:49:20 va-kibana systemd[1]: kibana.service: Failed with result 'exit-code'.
Jun 28 06:49:20 va-kibana systemd[1]: Failed to start Kibana.

And when I try using curl http://localhost5601/, it gives me the error:

curl: (7) Failed to connect to localhost port 5601 after 0 ms: Connection refused

Note: I have checked the kibana.log file at /var/log/kibana/kibana.log and there are no logs present.

Hope you can enlighten me on this matter,

Best Regards,
Ismail.

5

This is error is from Logstash, not Elasticsearch, you need to look for Elasticsearch errors.

Again, this is a systemd errors, you need to look at /var/log/syslog, also share your kibana.yml file.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.