Elastic wont start after adding realms configuration

AClerk · June 1, 2020, 6:03am

Hi All,
Added realm configuration to elasticsearch.yml.
After that the service wont start.
This is the added part →

#
# ---------------------------------- Realms ------------------------------------
#
 xpack:
   security:
     authc:
       realms:
         active_directory:
           active_directory_1:
             order: 1
			 enabled: true
			 domain_name: DOMAIN.local
             url: ldap://IP1:389, ldap://IP2:389, ldap://IP3:389, ldap://IP4:389, ldap://IP5:389, ldap://IP6:389
			 load_balance:
               type: round_robin
	       native:
            native_defulat:
             order: 0
			 enabled: true

Anything wrong here?

Thanks!

ikakavas · June 1, 2020, 6:23am

Your logs would contain all the necessary information as to why it fails to start.

AClerk · June 1, 2020, 6:34am

Hi @ikakavas
I searched the logs, couldn't find any specific error.
Should I look in a specific log?
Thanks!

ikakavas · June 1, 2020, 6:42am

Try

journalctl -u elasticsearch.service

AClerk · June 1, 2020, 7:02am

Not sure what this means...

Jun 01 16:57:11 AU01ELS00V01 systemd[1]: Stopping Elasticsearch...
Jun 01 16:57:17 AU01ELS00V01 systemd[1]: Stopped Elasticsearch.
Jun 01 16:57:30 AU01ELS00V01 systemd[1]: Starting Elasticsearch...
Jun 01 16:57:42 AU01ELS00V01 elasticsearch[226091]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: Exception in thread "main" 2020-06-01 16:57:53,166 main ERROR No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: MarkedYAMLException[while parsing a block mapping
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: in 'reader', line 17, column 1:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: cluster.name: cluster_xxx
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: ^
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: expected <block end>, but found BlockMappingStart
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: in 'reader', line 109, column 2:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: xpack:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: ^
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at [Source: sun.nio.ch.ChannelInputStream@565f390; line: 105, column: 36]]; nested: ParserException[while parsing a block mapping
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: in 'reader', line 17, column 1:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: cluster.name: cluster_xxx
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: ^
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: expected <block end>, but found BlockMappingStart
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: in 'reader', line 109, column 2:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: xpack:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: ^
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: ];
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1097)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1070)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.cli.Command.main(Command.java:90)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while parsing a block mapping
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: in 'reader', line 17, column 1:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: cluster.name: cluster_xxx
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: ^
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: expected <block end>, but found BlockMappingStart
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: in 'reader', line 109, column 2:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: xpack:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: ^
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at [Source: sun.nio.ch.ChannelInputStream@565f390; line: 105, column: 36]
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException.from(MarkedYAMLException.java:27)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:343)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:52)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:645)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Jun 01 16:57:53 AU01ELS00V01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 01 16:57:53 AU01ELS00V01 systemd[1]: Failed to start Elasticsearch.
Jun 01 16:57:53 AU01ELS00V01 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 01 16:57:53 AU01ELS00V01 systemd[1]: elasticsearch.service failed.

ikakavas · June 1, 2020, 8:41am

Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: MarkedYAMLException[while parsing a block mapping

It means that you modified your elasticsearch.yml in a way that it is not valid YML any more and elasticsearch can't parse it. Please go through your file and fix the format ( there are a number of tools, editors etc that can help you with it )

AClerk · June 2, 2020, 12:11am

@ikakavas
Thanks for the input.

The only code I added is above, in the first post. Couldn't find anything wrong with it.
What tool do you recommend for yml editing?
Thanks!

ikakavas · June 2, 2020, 5:56am

The only code I added is above, in the first post. Couldn't find anything wrong with it.

The logs seem to indicate an error around :

line 17, column 1:
Jun 01 16:57:53 AU01ELS00V01 elasticsearch[226091]: cluster.name: cluster_xxx

What tool do you recommend for yml editing?

Your standard editor should be fine if it doesn't add extra spaces, tabs, new lines etc. I don't have something specific to suggest I'm afraid, I use the tools I use for everything else. If you search for "yaml formatting" or "yaml linting" you will get enough options to select from.

AClerk · June 2, 2020, 6:28am

OK,
I have changed the syntax, as I want realm chain.
This is the syntax I have now →

 xpack.security.authc.realms:

 native.native_defulat:
       order: 0
       enabled: true

 active_directory.active_directory_1:
       order: 1
       enabled: true
       domain_name: DOMAIN.local
       url: ["ldaps://server-name:636", "ldaps://server-name:636", "ldaps://server-name:636", "ldaps://server-name:636"]
       load_balance.type: failover
       ssl.certificate_authorities: [ "pem.pem" ]

Still not able to start the node.
Another question that is troubling me. It says in the documentation

ssl.certificate_authorities and ssl.truststore.path cannot be used at the same time.

Is that^ at in the entire yml file or just under xpack.security.authc.realms

This is the log from the last update (part1)

-- Logs begin at Sat 2020-05-30 22:42:59 AEST, end at Tue 2020-06-02 16:05:23 AEST. --
...
Jun 02 16:03:59 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:03:59.182+1000        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://10.150.120.9:9200)) with 2 reconnect attempt(s)
Jun 02 16:04:10 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:04:10.374+1000        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":9393030,"time":{"ms":51}},"total":{"ticks":26635220,"time":{"ms":122},"value":26635220},"user":{"ticks":17242190,"time":{"ms":71}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":13},"info":{"ephemeral_id":"fd66180f-8ea8-4d23-be51-e42d9ba20262","uptime":{"ms":3722820126}},"memstats":{"gc_next":16645136,"memory_alloc":10351728,"memory_total":2048290436208},"runtime":{"goroutines":80}},"filebeat":{"events":{"active":-2,"added":69,"done":71},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":71,"batches":12,"total":71},"read":{"bytes":4669},"write":{"bytes":85702}},"pipeline":{"clients":5,"events":{"active":0,"published":69,"retry":3,"total":69},"queue":{"acked":71}}},"registrar":{"states":{"current":41,"update":71},"writes":{"success":12,"total":12}},"system":{"load":{"1":0.6,"15":0.55,"5":0.57,"norm":{"1":0.0375,"15":0.0344,"5":0.0356}}}}}}
Jun 02 16:04:12 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:04:12.127+1000        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(http://10.150.120.9:9200)): Get http://10.150.120.9:9200: dial tcp 10.150.120.9:9200: connect: connection refused
Jun 02 16:04:13 AU01ELS00V01 systemd-journal[3553]: Suppressed 696 messages from /system.slice/metricbeat.service
-- Subject: Messages from a service have been suppressed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: man:journald.conf(5)
-- 
-- A service has logged too many messages within a time period. Messages
-- from the service have been dropped.
-- 
-- Note that only messages from the service in question have been
-- dropped, other services' messages are unaffected.
-- 
-- The limits controlling when messages are dropped may be configured
-- with RateLimitInterval= and RateLimitBurst= in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Jun 02 16:04:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:13.812+1000        DEBUG        [cfgfile]        cfgfile/reload.go:197        Scan for new config files
Jun 02 16:04:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:13.812+1000        DEBUG        [cfgfile]        cfgfile/cfgfile.go:193        Load config from file: /etc/metricbeat/modules.d/system.yml
Jun 02 16:04:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:13.812+1000        DEBUG        [cfgfile]        cfgfile/reload.go:216        Number of module configs found: 3
Jun 02 16:04:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:13.812+1000        DEBUG        [reload]        cfgfile/list.go:62        Starting reload procedure, current runners: 3
Jun 02 16:04:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:13.812+1000        DEBUG        [reload]        cfgfile/list.go:80        Start list: 0, Stop list: 0
Jun 02 16:04:18 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:18.998+1000        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":96562280,"time":{"ms":191}},"total":{"ticks":150952790,"time":{"ms":191},"value":150952790},"user":{"ticks":54390510}},"handles":{"limit":{"hard":4096,"soft":1024},"open":7},"info":{"ephemeral_id":"e0ca6585-ec39-48a2-b83f-8d949ab1c068","uptime":{"ms":19357233114}},"memstats":{"gc_next":13172112,"memory_alloc":10002040,"memory_total":11699444800200},"runtime":{"goroutines":38}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":19,"batches":2,"failed":19,"total":38},"read":{"bytes":6},"write":{"bytes":4871,"errors":1}},"pipeline":{"clients":3,"events":{"active":0,"published":19,"retry":38,"total":19},"queue":{"acked":19}}},"metricbeat":{"system":{"cpu":{"events":1,"success":1},"filesystem":{"events":5,"success":5},"fsstat":{"events":1,"success":1},"load":{"events":1,"success":1},"memory":{"events":1,"success":1},"network":{"events":2,"success":2},"process":{"events":5,"success":5},"process_summary":{"events":1,"success":1},"socket_summary":{"events":1,"success":1},"uptime":{"events":1,"success":1}}},"system":{"load":{"1":0.66,"15":0.55,"5":0.59,"norm":{"1":0.0413,"15":0.0344,"5":0.0369}}}}}}
Jun 02 16:04:21 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:04:21.187+1000        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://10.150.120.9:9200)) with 3 reconnect attempt(s)
Jun 02 16:04:22 AU01ELS00V01 systemd[1]: Stopped Elasticsearch.
-- Subject: Unit elasticsearch.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit elasticsearch.service has finished shutting down.
Jun 02 16:04:22 AU01ELS00V01 polkitd[97204]: Unregistered Authentication Agent for unix-process:231868:1981375723 (system bus name :1.24806, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Jun 02 16:04:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:23.813+1000        DEBUG        [cfgfile]        cfgfile/reload.go:197        Scan for new config files
Jun 02 16:04:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:23.814+1000        DEBUG        [cfgfile]        cfgfile/cfgfile.go:193        Load config from file: /etc/metricbeat/modules.d/system.yml
Jun 02 16:04:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:23.814+1000        DEBUG        [cfgfile]        cfgfile/reload.go:216        Number of module configs found: 3
Jun 02 16:04:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:23.814+1000        DEBUG        [reload]        cfgfile/list.go:62        Starting reload procedure, current runners: 3
Jun 02 16:04:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:23.814+1000        DEBUG        [reload]        cfgfile/list.go:80        Start list: 0, Stop list: 0
Jun 02 16:04:33 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:33.814+1000        DEBUG        [cfgfile]        cfgfile/reload.go:197        Scan for new config files
Jun 02 16:04:33 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:33.815+1000        DEBUG        [cfgfile]        cfgfile/cfgfile.go:193        Load config from file: /etc/metricbeat/modules.d/system.yml
Jun 02 16:04:33 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:33.815+1000        DEBUG        [cfgfile]        cfgfile/reload.go:216        Number of module configs found: 3
Jun 02 16:04:33 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:33.815+1000        DEBUG        [reload]        cfgfile/list.go:62        Starting reload procedure, current runners: 3
Jun 02 16:04:33 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:33.815+1000        DEBUG        [reload]        cfgfile/list.go:80        Start list: 0, Stop list: 0
Jun 02 16:04:39 AU01ELS00V01 polkitd[97204]: Registered Authentication Agent for unix-process:231913:1981383374 (system bus name :1.24807 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Jun 02 16:04:39 AU01ELS00V01 systemd[1]: Starting Elasticsearch...

Thanks!

AClerk · June 2, 2020, 6:29am

part2

-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit elasticsearch.service has begun starting up.
Jun 02 16:04:40 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:04:40.372+1000        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":9393070,"time":{"ms":44}},"total":{"ticks":26635320,"time":{"ms":107},"value":26635320},"user":{"ticks":17242250,"time":{"ms":63}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":13},"info":{"ephemeral_id":"fd66180f-8ea8-4d23-be51-e42d9ba20262","uptime":{"ms":3722850126}},"memstats":{"gc_next":16438448,"memory_alloc":9738664,"memory_total":2048297097056},"runtime":{"goroutines":80}},"filebeat":{"events":{"added":84,"done":84},"harvester":{"open_files":3,"running":3}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":84,"batches":8,"total":84},"read":{"bytes":3365},"write":{"bytes":87743}},"pipeline":{"clients":5,"events":{"active":0,"published":84,"retry":15,"total":84},"queue":{"acked":84}}},"registrar":{"states":{"current":41,"update":84},"writes":{"success":8,"total":8}},"system":{"load":{"1":0.47,"15":0.54,"5":0.55,"norm":{"1":0.0294,"15":0.0338,"5":0.0344}}}}}}
Jun 02 16:04:43 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:04:43.765+1000        ERROR        pipeline/output.go:100        Failed to connect to backoff(elasticsearch(http://10.150.120.9:9200)): Get http://10.150.120.9:9200: dial tcp 10.150.120.9:9200: connect: connection refused
Jun 02 16:04:43 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:43.815+1000        DEBUG        [cfgfile]        cfgfile/reload.go:197        Scan for new config files
Jun 02 16:04:43 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:43.816+1000        DEBUG        [cfgfile]        cfgfile/cfgfile.go:193        Load config from file: /etc/metricbeat/modules.d/system.yml
Jun 02 16:04:43 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:43.816+1000        DEBUG        [cfgfile]        cfgfile/reload.go:216        Number of module configs found: 3
Jun 02 16:04:43 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:43.816+1000        DEBUG        [reload]        cfgfile/list.go:62        Starting reload procedure, current runners: 3
Jun 02 16:04:43 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:43.816+1000        DEBUG        [reload]        cfgfile/list.go:80        Start list: 0, Stop list: 0
Jun 02 16:04:48 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:48.999+1000        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":96562290,"time":{"ms":10}},"total":{"ticks":150952800,"time":{"ms":10},"value":150952800},"user":{"ticks":54390510}},"handles":{"limit":{"hard":4096,"soft":1024},"open":7},"info":{"ephemeral_id":"e0ca6585-ec39-48a2-b83f-8d949ab1c068","uptime":{"ms":19357263113}},"memstats":{"gc_next":13172112,"memory_alloc":10431136,"memory_total":11699445229296},"runtime":{"goroutines":38}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"pipeline":{"clients":3,"events":{"active":0}}},"system":{"load":{"1":0.48,"15":0.54,"5":0.55,"norm":{"1":0.03,"15":0.0338,"5":0.0344}}}}}}
Jun 02 16:04:51 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:04:51.391+1000        INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log
Jun 02 16:04:52 AU01ELS00V01 elasticsearch[231919]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 02 16:04:53 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:53.817+1000        DEBUG        [cfgfile]        cfgfile/reload.go:197        Scan for new config files
Jun 02 16:04:53 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:53.817+1000        DEBUG        [cfgfile]        cfgfile/cfgfile.go:193        Load config from file: /etc/metricbeat/modules.d/system.yml
Jun 02 16:04:53 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:53.817+1000        DEBUG        [cfgfile]        cfgfile/reload.go:216        Number of module configs found: 3
Jun 02 16:04:53 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:53.817+1000        DEBUG        [reload]        cfgfile/list.go:62        Starting reload procedure, current runners: 3
Jun 02 16:04:53 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:53.817+1000        DEBUG        [reload]        cfgfile/list.go:80        Start list: 0, Stop list: 0
Jun 02 16:04:59 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:59.000+1000        DEBUG        [processors]        processing/processors.go:183        Publish event: {
...
Jun 02 16:04:59 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:59.009+1000        DEBUG        [system.fsstat]        fsstat/fsstat.go:86        filesystem: /mnt/nfs_snapshots total=2083241590784, used=1592164089856, free=491077500928
Jun 02 16:04:59 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:59.009+1000        DEBUG        [system.fsstat]        fsstat/fsstat.go:86        filesystem: / total=53660876800, used=17173524480, free=36487352320
Jun 02 16:04:59 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:59.009+1000        DEBUG        [system.fsstat]        fsstat/fsstat.go:86        filesystem: /boot total=1063256064, used=183042048, free=880214016
Jun 02 16:04:59 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:59.009+1000        DEBUG        [system.fsstat]        fsstat/fsstat.go:86        filesystem: /mnt/data total=3170527019008, used=364502388736, free=2806024630272
Jun 02 16:04:59 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:59.009+1000        DEBUG        [system.fsstat]        fsstat/fsstat.go:86        filesystem: /home total=41851293696, used=33832960, free=41817460736
Jun 02 16:04:59 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:04:59.009+1000        DEBUG        [processors]        processing/processors.go:183        Publish event: {
...
Jun 02 16:05:01 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:05:01.392+1000        INFO        log/harvester.go:253        Harvester started for file: /var/log/elasticsearch/gc.log
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: Exception in thread "main" 2020-06-02 16:05:03,152 main ERROR No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: ElasticsearchParseException[null-valued setting found for key [xpack.security.authc.realms] found at line number [109], column number [29]]
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:694)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:670)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1093)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1070)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.cli.Command.main(Command.java:90)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jun 02 16:05:03 AU01ELS00V01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 02 16:05:03 AU01ELS00V01 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit elasticsearch.service has failed.
-- 
-- The result is failed.
Jun 02 16:05:03 AU01ELS00V01 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 02 16:05:03 AU01ELS00V01 systemd[1]: elasticsearch.service failed.
Jun 02 16:05:03 AU01ELS00V01 polkitd[97204]: Unregistered Authentication Agent for unix-process:231913:1981383374 (system bus name :1.24807, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Jun 02 16:05:09 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:05:09.394+1000        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(elasticsearch(http://10.150.120.9:9200)) with 4 reconnect attempt(s)
Jun 02 16:05:10 AU01ELS00V01 filebeat[127642]: 2020-06-02T16:05:10.374+1000        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":9393120,"time":{"ms":47}},"total":{"ticks":26635400,"time":{"ms":86},"value":26635400},"user":{"ticks":17242280,"time":{"ms":39}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":15},"info":{"ephemeral_id":"fd66180f-8ea8-4d23-be51-e42d9ba20262","uptime":{"ms":3722880127}},"memstats":{"gc_next":16436784,"memory_alloc":13946856,"memory_total":2048302280008,"rss":266240},"runtime":{"goroutines":90}},"filebeat":{"events":{"added":32,"done":32},"harvester":{"open_files":5,"running":5,"started":2},"input":{"log":{"files":{"truncated":2}}}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":30,"batches":5,"total":30},"read":{"bytes":1947},"write":{"bytes":30995}},"pipeline":{"clients":5,"events":{"active":0,"filtered":2,"published":30,"retry":1,"total":32},"queue":{"acked":30}}},"registrar":{"states":{"current":41,"update":32},"writes":{"success":7,"total":7}},"system":{"load":{"1":0.47,"15":0.54,"5":0.54,"norm":{"1":0.0294,"15":0.0338,"5":0.0338}}}}}}
Jun 02 16:05:13 AU01ELS00V01 systemd-journal[3553]: Suppressed 728 messages from /system.slice/metricbeat.service
-- Subject: Messages from a service have been suppressed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: man:journald.conf(5)
-- 
-- A service has logged too many messages within a time period. Messages
-- from the service have been dropped.
-- 
-- Note that only messages from the service in question have been
-- dropped, other services' messages are unaffected.
--

AClerk · June 2, 2020, 6:30am

part3

its controlling when messages are dropped may be configured
-- with RateLimitInterval= and RateLimitBurst= in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Jun 02 16:05:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:13.820+1000        DEBUG        [cfgfile]        cfgfile/reload.go:197        Scan for new config files
Jun 02 16:05:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:13.820+1000        DEBUG        [cfgfile]        cfgfile/cfgfile.go:193        Load config from file: /etc/metricbeat/modules.d/system.yml
Jun 02 16:05:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:13.820+1000        DEBUG        [cfgfile]        cfgfile/reload.go:216        Number of module configs found: 3
Jun 02 16:05:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:13.820+1000        DEBUG        [reload]        cfgfile/list.go:62        Starting reload procedure, current runners: 3
Jun 02 16:05:13 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:13.821+1000        DEBUG        [reload]        cfgfile/list.go:80        Start list: 0, Stop list: 0
Jun 02 16:05:18 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:18.999+1000        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":96562490,"time":{"ms":198}},"total":{"ticks":150953000,"time":{"ms":198},"value":150953000},"user":{"ticks":54390510}},"handles":{"limit":{"hard":4096,"soft":1024},"open":6},"info":{"ephemeral_id":"e0ca6585-ec39-48a2-b83f-8d949ab1c068","uptime":{"ms":19357293113}},"memstats":{"gc_next":15914352,"memory_alloc":9577552,"memory_total":11699467760632},"runtime":{"goroutines":38}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":19,"batches":2,"failed":19,"total":38},"read":{"bytes":6},"write":{"bytes":4899,"errors":1}},"pipeline":{"clients":3,"events":{"active":0,"published":19,"retry":38,"total":19},"queue":{"acked":19}}},"metricbeat":{"system":{"cpu":{"events":1,"success":1},"filesystem":{"events":5,"success":5},"fsstat":{"events":1,"success":1},"load":{"events":1,"success":1},"memory":{"events":1,"success":1},"network":{"events":2,"success":2},"process":{"events":6,"success":6},"process_summary":{"events":1,"success":1},"socket_summary":{"events":1,"success":1}}},"system":{"load":{"1":0.4,"15":0.53,"5":0.53,"norm":{"1":0.025,"15":0.0331,"5":0.0331}}}}}}
Jun 02 16:05:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:23.821+1000        DEBUG        [cfgfile]        cfgfile/reload.go:197        Scan for new config files
Jun 02 16:05:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:23.821+1000        DEBUG        [cfgfile]        cfgfile/cfgfile.go:193        Load config from file: /etc/metricbeat/modules.d/system.yml
Jun 02 16:05:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:23.821+1000        DEBUG        [cfgfile]        cfgfile/reload.go:216        Number of module configs found: 3
Jun 02 16:05:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:23.822+1000        DEBUG        [reload]        cfgfile/list.go:62        Starting reload procedure, current runners: 3
Jun 02 16:05:23 AU01ELS00V01 metricbeat[29305]: 2020-06-02T16:05:23.822+1000        DEBUG        [reload]        cfgfile/list.go:80        Start list: 0, Stop list: 0

ikakavas · June 2, 2020, 6:41am

This is not valid yaml.

Jun 02 16:05:03 AU01ELS00V01 elasticsearch[231919]: ElasticsearchParseException[null-valued setting found for key [xpack.security.authc.realms] found at line number [109], column number [29]]

Please put in the time to actually try and format your configuration file correctly, otherwise we will be just running in circles. Don't leave empty lines between keys, don't use arbitrary number of empty spaces or TABs for indentation. Use 2 or 4 (consistently) instead.

This applies to the same context only, i.e. under the same ssl.

Finally, run journalctl -u elasticsearch.service as opposed to journalctl so that you get logs from the elasticsearch service only and not everything else as above.

AClerk · June 2, 2020, 6:58am

Hey
Removed spaces/tabs/empty lines
I am following examples from the documentations, as far as to my understanding.
yml validator say the file is valid

What is null-valued setting found for key [xpack.security.authc.realms]?
The settings are there...

JJun 02 16:54:42 AU01ELS00V01 systemd[1]: Starting Elasticsearch...
Jun 02 16:54:52 AU01ELS00V01 elasticsearch[233047]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: Exception in thread "main" 2020-06-02 16:55:03,069 main ERROR No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: ElasticsearchParseException[null-valued setting found for key [xpack.security.authc.realms] found at line number [109], column number [29]]
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:694)
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:670)
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1093)
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1070)
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Jun 02 16:55:03 AU01ELS00V01 elasticsearch[233047]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Jun 02 16:55:03 AU01ELS00V01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 02 16:55:03 AU01ELS00V01 systemd[1]: Failed to start Elasticsearch.
Jun 02 16:55:03 AU01ELS00V01 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 02 16:55:03 AU01ELS00V01 systemd[1]: elasticsearch.service failed.

ikakavas · June 2, 2020, 7:27am

How exactly does it look like ?

It means that you have

xpack.security.authc.realms:

then an empty space, and then another line at the same indentation - which in yaml means that xpack.security.authc.realms: has no value.

You do, but it's just the formatting that is off. Maybe something wrong with your editor or how you copy and paste.

AClerk · June 4, 2020, 4:00am

Hey @ikakavas
Below are the current configuration I have and the error I get.
No spaces anywhere.
Any advice?
Any other syntax error?
Is there any example I can follow for realm chain?

xpack.security.authc.realms:
native.native_defulat:
order:0
enabled:true
active_directory.active_directory_1:
order:1
enabled:true
domain_name:DOMAIN.local
url:["ldaps://servername.local:636","ldaps://servername:636","ldaps://servername.local:636","ldaps://servername.local:636","ldaps://servername.local:636","ldaps://servername.local:636"]
load_balance.type:failover
ssl.certificate_authorities:["pem.pem"]

This is the error

Jun 04 13:51:01 AU01ELS00V01 systemd[1]: Starting Elasticsearch...
Jun 04 13:51:12 AU01ELS00V01 elasticsearch[8958]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: Exception in thread "main" 2020-06-04 13:51:23,331 main ERROR No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 i
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: ElasticsearchParseException[null-valued setting found for key [xpack.security.authc.realms] found at line number [108], column number [29]]
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.common.settings.Settings.validateValue(Settings.java:694)
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:670)
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:620)
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.common.settings.Settings.access$400(Settings.java:82)
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1093)
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1070)
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:83)
Jun 04 13:51:23 AU01ELS00V01 elasticsearch[8958]: at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
Jun 04 13:51:23 AU01ELS00V01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jun 04 13:51:23 AU01ELS00V01 systemd[1]: Failed to start Elasticsearch.
Jun 04 13:51:23 AU01ELS00V01 systemd[1]: Unit elasticsearch.service entered failed state.
Jun 04 13:51:23 AU01ELS00V01 systemd[1]: elasticsearch.service failed.

Thanks!

ikakavas · June 4, 2020, 4:39am

Yaml is space indentation dependent, you should not remove your spaces . I asked you to remove extra/arbitrary spaces, not all of them

As you can see in many places in our docs too, it needs to be

xpack.security.authc.realms:
  native.native_defulat:
    order:0
    enabled:true
  active_directory.active_directory_1:
    order:1
    enabled:true
    domain_name:DOMAIN.local
    url: ["ldaps://servername.local:636","ldaps://servername:636","ldaps://servername.local:636","ldaps://servername.local:636","ldaps://servername.local:636","ldaps://servername.local:636"]
    load_balance.type:failover
    ssl.certificate_authorities:["pem.pem"]

AClerk · June 4, 2020, 5:09am

ikakavas:

xpack.security.authc.realms:
  native.native_defulat:
    order:0
    enabled:true
  active_directory.active_directory_1:
    order:1
    enabled:true
    domain_name:DOMAIN.local
    url: ["ldaps://servername.local:636","ldaps://servername:636","ldaps://servername.local:636","ldaps://servername.local:636","ldaps://servername.local:636","ldaps://servername.local:636"]
    load_balance.type:failover
    ssl.certificate_authorities:["pem.pem"]

Thanks a lot!
That helped.
Also need a space after ':'.
e.g. instead of enabled:true --> enabled: true.

system · July 2, 2020, 5:09am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.