copy that...thanks...
Hi Rich,
I wouldn't be able to provide yaml file info on a security standpoint.. I can provide you with the errors presented.
rob
There really isn't anything private in the .yml file with the exception of IP addresses, potentially. You could simply replace anything you think was private with "XXXXXXX".
Our suspicion is that you modified the elasticsearch.yml file and left it in a state in which there are now syntax errors. For example,
in 'reader', line 120, column 1:
Configuration------------------- ...
^
There should be no line in the file that begins with Configuration------- (like your 120th line of your file). All lines that have section headers have the # character at the beginning of the line so that the rest of the line is commented out.
In fact, if you look at my elasticsearch.yml file that I run for the node running on my local macbook, nearly every single line (except for the last one) is commented out because I'm just taking the default values of almost every setting:
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: false
1 Like
Ok, I'll do that.
Rob
Rich,
Here you go:
======================== Elasticsearch Configuration =========================
NOTE: Elasticsearch comes with reasonable defaults for most settings.
Before you set out to tweak and tune the configuration, make sure you
understand what are you trying to accomplish and the consequences.
The primary way of configuring a node is via this file. This template lists
the most important settings you may want to configure for a production cluster.
Please consult the documentation for further information on configuration options:
https://www.elastic.co/guide/en/elasticsearch/reference/index.html
---------------------------------- Cluster -----------------------------------
Use a descriptive name for your cluster:
cluster.name: xxx
------------------------------------ Node ------------------------------------
Use a descriptive name for the node:
node.name: xxx
Add custom attributes to the node:
#node.attr.rack: xx
----------------------------------- Paths ------------------------------------
Path to directory where to store the data (separate multiple locations by comma):
path.data: /DataDisk/data
Path to log files:
path.logs: /DataDisk/logs
----------------------------------- Memory -----------------------------------
Lock the memory on startup:
bootstrap.memory_lock: false
#---------------------------------------------------------------------Other bootstrap checks
#---------------------------------------------------------------------
#bootstrap.system_call_filters: false
#--------------------------------------------------------------------
Make sure that the heap size is set to about half the memory available
on the system and that the owner of the process is allowed to use this
limit.
Elasticsearch performs poorly when the system is swapping the memory.
---------------------------------- Network -----------------------------------
Set the bind address to a specific IP (IPv4 or IPv6):
network.host: xxxxx
Set a custom port for HTTP:
http.port: xxx
For more information, consult the network module documentation.
--------------------------------- Discovery ----------------------------------
Pass an initial list of hosts to perform discovery when new node is started:
The default list of hosts is ["xxxx", "[::1]"]
discovery.zen.ping.unicast.hosts: xxxxx"]
Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
information, consult the zen discovery module documentation.
discovery.zen.minimum_master_nodes: 2
---------------------------------- Gateway -----------------------------------
Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3
For more information, consult the gateway module documentation.
---------------------------------- Various -----------------------------------
Require explicit names when deleting indices:
#action.destructive_requires_name: true
#Selection of Master Node
node.master: true
node.data: false
node.ingest: false
#No ingest node available for monitoring data
x-pack.monitoring.exporters.my_local:
type: local
use_ingest: false
#---------------------------------------------------------------------------------
#---------------------------------------Audit-------------------------------------
xpack.security.audit.enabled: true
xpack.security.audit.index.settings:
index:
number_of_shards: 1
number_of_replicas: 1
xpack.security.audit.outputs: [index, logfile]
xpack.security.audit.index.rollover: daily
xpack.security.audit.logfile.events.include: [anonymous_access_denied,
authentication_failed, access_granted, access_denied]
----------------------------------------------------------- SSL X-Pack
Configuration---------------------------------------------------------------
xpack.ssl.keystore.path: /etc/elasticsearch/x-
pack/certs/xxxx
xpack.ssl.keystore.password: password
xpack.ssl.keystore.key_password: password
xpack.ssl.truststore.path: /etc/elasticsearch/x-
pack/certs/xxxx
xpack.ssl.truststore.password: password
xpack.ssl.verification_mode: none
xpack..security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
--------------------------------------------------------------- SSL X-Pack
Configuration -----------------------------------------------------------------
Anonymous access for Kibana
#xpack.security.authc:
anonymous:
username: _es_anonymous_user
roles: Anonymous_RO
authz_exception: false
#xpack.security.authc:
anonymous:
username: _es_anonymous_user
username: xxxxxx
roles: kibana_user_For_ES_Anonymous, AllIndexReadOnly
roles: AllIndexReadOnly, MaccessRole
authz_exception: false
authz_exception: true
#xpack.security.authc:
realms:
native1:
type: native
order: 0
enabled: true
active_directory:
type: active_directory
order: 1
enabled: true
domain_name:xxxxxxxx
url: xxxxxxxxx
unmapped_groups_as_roles: true
ssl:
certificate_authorities: [ "/etc/elasticsearch/x-
pack/ca/xxxxxx" ]
When you get a chance, take a look at this:
Rob
You seem to have comment lines that are not commented out with a # at the start.
Hi Rob - just compare your file with the one I posted. Only lines that have directives, such as:
setting_name:value
should be on lines without # at the beginning. Every other line that has plain text descriptions or section headings need to be properly commented out with # at the beginning of the line.
1 Like
Rich,
I had zoomed out on the file. Back to normal. See below:
======================== Elasticsearch Configuration =========================
NOTE: Elasticsearch comes with reasonable defaults for most settings.
Before you set out to tweak and tune the configuration, make sure you
understand what are you trying to accomplish and the consequences.
The primary way of configuring a node is via this file. This template lists
the most important settings you may want to configure for a production cluster.
Please consult the documentation for further information on configuration options:
https://www.elastic.co/guide/en/elasticsearch/reference/index.html
---------------------------------- Cluster -----------------------------------
Use a descriptive name for your cluster:
cluster.name: xxx
------------------------------------ Node ------------------------------------
Use a descriptive name for the node:
node.name: xxx
Add custom attributes to the node:
#node.attr.rack: xx
----------------------------------- Paths ------------------------------------
Path to directory where to store the data (separate multiple locations by comma):
path.data: /DataDisk/data
Path to log files:
path.logs: /DataDisk/logs
----------------------------------- Memory -----------------------------------
Lock the memory on startup:
bootstrap.memory_lock: false
#---------------------------------------------------------------------Other bootstrap checks
#---------------------------------------------------------------------
#bootstrap.system_call_filters: false
#--------------------------------------------------------------------
Make sure that the heap size is set to about half the memory available
on the system and that the owner of the process is allowed to use this
limit.
Elasticsearch performs poorly when the system is swapping the memory.
---------------------------------- Network -----------------------------------
Set the bind address to a specific IP (IPv4 or IPv6):
network.host: xxxxx
Set a custom port for HTTP:
http.port: xxx
For more information, consult the network module documentation.
--------------------------------- Discovery ----------------------------------
Pass an initial list of hosts to perform discovery when new node is started:
The default list of hosts is ["xxxx", "[::1]"]
discovery.zen.ping.unicast.hosts: xxxxx"]
Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
information, consult the zen discovery module documentation.
discovery.zen.minimum_master_nodes: 2
---------------------------------- Gateway -----------------------------------
Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3
For more information, consult the gateway module documentation.
---------------------------------- Various -----------------------------------
Require explicit names when deleting indices:
#action.destructive_requires_name: true
#Selection of Master Node
node.master: true
node.data: false
node.ingest: false
#No ingest node available for monitoring data
x-pack.monitoring.exporters.my_local:
type: local
use_ingest: false
#---------------------------------------------------------------------------------
#---------------------------------------Audit-------------------------------------
xpack.security.audit.enabled: true
xpack.security.audit.index.settings:
index:
number_of_shards: 1
number_of_replicas: 1
xpack.security.audit.outputs: [index, logfile]
xpack.security.audit.index.rollover: daily
xpack.security.audit.logfile.events.include: [anonymous_access_denied,
authentication_failed, access_granted, access_denied]
----------------------------------------------------------- SSL X-Pack
Configuration---------------------------------------------------------------
xpack.ssl.keystore.path: /etc/elasticsearch/x-
pack/certs/xxxx
xpack.ssl.keystore.password: password
xpack.ssl.keystore.key_password: password
xpack.ssl.truststore.path: /etc/elasticsearch/x-
pack/certs/xxxx
xpack.ssl.truststore.password: password
xpack.ssl.verification_mode: none
xpack..security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
--------------------------------------------------------------- SSL X-Pack
Configuration -----------------------------------------------------------------
Anonymous access for Kibana
#xpack.security.authc:
anonymous:
username: _es_anonymous_user
roles: Anonymous_RO
authz_exception: false
#xpack.security.authc:
anonymous:
username: _es_anonymous_user
username: xxxxxx
roles: kibana_user_For_ES_Anonymous, AllIndexReadOnly
roles: AllIndexReadOnly, MaccessRole
authz_exception: false
authz_exception: true
#xpack.security.authc:
realms:
native1:
type: native
order: 0
enabled: true
active_directory:
type: active_directory
order: 1
enabled: true
domain_name:xxxxxxxx
url: xxxxxxxxx
unmapped_groups_as_roles: true
ssl:
certificate_authorities: [ "/etc/elasticsearch/x-
pack/ca/xxxxxx" ]
Can you let me know what I need to comment and not comment?
Thanks,
Rob
Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.
Or use markdown style like:
```
CODE
```
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
Comment out:
- Every line that does not have a
setting:valuepair on that line - Any line where that particular
setting:valuepair is not desired to be set
Again Rob, please look at the example elasticsearch.yml file I posted earlier. It should be obvious.
Rich,
Here you go:
======================== Elasticsearch Configuration =========================
NOTE: Elasticsearch comes with reasonable defaults for most settings.
Before you set out to tweak and tune the configuration, make sure you
understand what are you trying to accomplish and the consequences.
The primary way of configuring a node is via this file. This template lists
the most important settings you may want to configure for a production cluster.
Please consult the documentation for further information on configuration options:
https://www.elastic.co/guide/en/elasticsearch/reference/index.html
---------------------------------- Cluster -----------------------------------
Use a descriptive name for your cluster:
cluster.name: xxx
------------------------------------ Node ------------------------------------
Use a descriptive name for the node:
node.name: xxx
Add custom attributes to the node:
#node.attr.rack: xx
----------------------------------- Paths ------------------------------------
Path to directory where to store the data (separate multiple locations by comma):
path.data: /DataDisk/data
Path to log files:
path.logs: /DataDisk/logs
----------------------------------- Memory -----------------------------------
Lock the memory on startup:
bootstrap.memory_lock: false
#---------------------------------------------------------------------Other bootstrap checks
#---------------------------------------------------------------------
#bootstrap.system_call_filters: false
#--------------------------------------------------------------------
Make sure that the heap size is set to about half the memory available
on the system and that the owner of the process is allowed to use this
limit.
Elasticsearch performs poorly when the system is swapping the memory.
---------------------------------- Network -----------------------------------
Set the bind address to a specific IP (IPv4 or IPv6):
network.host: xxxxx
Set a custom port for HTTP:
http.port: xxx
For more information, consult the network module documentation.
--------------------------------- Discovery ----------------------------------
Pass an initial list of hosts to perform discovery when new node is started:
The default list of hosts is ["xxxx", "[::1]"]
discovery.zen.ping.unicast.hosts: xxxxx"]
Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
information, consult the zen discovery module documentation.
discovery.zen.minimum_master_nodes: 2
---------------------------------- Gateway -----------------------------------
Block initial recovery after a full cluster restart until N nodes are started:
#gateway.recover_after_nodes: 3
For more information, consult the gateway module documentation.
---------------------------------- Various -----------------------------------
Require explicit names when deleting indices:
#action.destructive_requires_name: true
#Selection of Master Node
node.master: true
node.data: false
node.ingest: false
#No ingest node available for monitoring data
x-pack.monitoring.exporters.my_local:
type: local
use_ingest: false
#---------------------------------------------------------------------------------
#---------------------------------------Audit-------------------------------------
xpack.security.audit.enabled: true
xpack.security.audit.index.settings:
index:
number_of_shards: 1
number_of_replicas: 1
xpack.security.audit.outputs: [index, logfile]
xpack.security.audit.index.rollover: daily
xpack.security.audit.logfile.events.include: [anonymous_access_denied,
authentication_failed, access_granted, access_denied]
----------------------------------------------------------- SSL X-Pack
Configuration---------------------------------------------------------------
xpack.ssl.keystore.path: /etc/elasticsearch/x-
pack/certs/xxxx
xpack.ssl.keystore.password: password
xpack.ssl.keystore.key_password: password
xpack.ssl.truststore.path: /etc/elasticsearch/x-
pack/certs/xxxx
xpack.ssl.truststore.password: password
xpack.ssl.verification_mode: none
xpack..security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
--------------------------------------------------------------- SSL X-Pack
Configuration -----------------------------------------------------------------
Anonymous access for Kibana
#xpack.security.authc:
anonymous:
username: _es_anonymous_user
roles: Anonymous_RO
authz_exception: false
#xpack.security.authc:
anonymous:
username: _es_anonymous_user
username: xxxxxx
roles: kibana_user_For_ES_Anonymous, AllIndexReadOnly
roles: AllIndexReadOnly, MaccessRole
authz_exception: false
authz_exception: true
#xpack.security.authc:
realms:
native1:
type: native
order: 0
enabled: true
active_directory:
type: active_directory
order: 1
enabled: true
domain_name:xxxxxxxx
url: xxxxxxxxx
unmapped_groups_as_roles: true
ssl:
certificate_authorities: [ "/etc/elasticsearch/x-
pack/ca/xxxxxx" ]
Let me know.
Thanks,
Robert
Robert, at first blush this looks no different and it looks as if you haven't followed our advice with respect to how the file is to be formatted and how it should be posted here so that it is readable. It is hard to tell if there's a fundamental miscommunication or if there's something technologically getting in our way. As I've said, you need to have # on every line except those in which you are overriding the default setting of a particular configuration parameter. Therefore, an example elasticsearch.yml file looks something like this in an actual text editor:
In this example, only two settings that are visible on the screen are being set: cluster.name and node.name.
The way that you have posted yours makes it look like your file is 98% incorrect - but again, I cannot know for certain simply because of the way you posted it with the bad formatting.
I think it is impractical for me (or any of us) to fix your file, line by line. Instead, follow our guidance on what the file needs to look like based on our information. If fixing your current file seems daunting, perhaps the easier thing for you would be to start over with a fresh installation (which will give you a new, untouched elasticsearch.yml file) and then you can carefully add in your desired changes.
OK, it's done. Do I create a secret gist or public gist?
Rob
As you wish. Secret is fine.
I want make sure you get it.
Rob
You edited things before posting it.
For example this line is wrong:
discovery.zen.ping.unicast.hosts: xxxxx"]
Those ones as well:
# certificate_authorities: [ "/etc/elasticsearch/x-
pack/ca/xxxxxx" ]
Understood. That data had to be blanked out. Project info.
Rob