Logstash service getting restarted after trying to set keystore on `v7.14`

Elastic Stack Logstash
1

Hi Team,

I am trying to set logstash keystore but having hard time.

I have refer below link but still its not working. (not creating keystore password from below)

I am creating keystore as,

 echo y | /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create 
 /bin/cat /root/es_password | /usr/share/logstash/bin/logstash-keystore add --path.settings /etc/logstash es_pwd --stdin
 chown logstash:root /etc/logstash/logstash.keystore 
 chmod 0600 /etc/logstash/logstash.keystore

keystore file.

[root@ip-10-10-10-242 ~]# ls -l /etc/logstash/logstash.keystore 
-rw-r--r--. 1 logstash logstash 716 Sep 19 20:50 /etc/logstash/logstash.keystore
[root@ip-10-10-10-242 ~]#

key is getting listed.

[root@ip-10-10-10-242 ~]# /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash/ list
Using JAVA_HOME defined java: /opt/jre1.8.0_221
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-1.17.3/lib/bundler/rubygems_integration.rb:200: warning: constant Gem::ConfigMap is deprecated

es_pwd
[root@ip-10-10-10-242 ~]#

I have below /etc/logstash/conf.d/logstash.conf file.

i) -rw-r--r--. 1 logstash logstash 8316 Sep 19 20:11 /etc/logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
  }
}
filter {
if [log_type] == "access_server" and [app_id] == "pa"
  {
    grok { match => { "message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}(?::?%{SECOND})\| %{USERNAME:exchangeId}\| %{DATA:trackingId}\| %{NUMBER:RoundTrip:int}%{SPACE}ms\| %{NUMBER:ProxyRoundTrip:int}%{SPACE}ms\| %{NUMBER:UserInfoRoundTrip:int}%{SPACE}ms\| %{DATA:Resource}\| %{DATA:subject}\| %{DATA:authmech}\| %{DATA:scopes}\| %{IPV4:Client}\| %{WORD:method}\| %{DATA:Request_URI}\| %{INT:response_code}\| %{DATA:failedRuleType}\| %{DATA:failedRuleName}\| %{DATA:APP_Name}\| %{DATA:Resource_Name}\| %{DATA:Path_Prefix}" } }
    mutate {
             replace => {
               "[type]" => "access_server"
             }
           }
  }
}
output {
  if [log_type] == "access_server" {
  elasticsearch {
    hosts => ['http://10.10.10.242:9200']
        user => elastic
    password => "${es_pwd}"
     index => "access"
     template => "/root/access_template.json"
     template_name => "access"
     template_overwrite => "false"
      }
 }
 elasticsearch {
    hosts => ['http://10.10.10.242:9200']
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
    user => elastic
    password => "${es_pwd}"
  }
}

index template file is copied to /root as used in above setting with logstash ownership.

[root@ip-10-10-10-242 ~]# ls -l /root/access_template.json 
-rw-r--r--. 1 logstash logstash 840 Sep 19 15:42 /root/access_template.json
[root@ip-10-10-10-242 ~]#

ii) /etc/logstash/logstash.yml file.

-rw-r--r--. 1 logstash logstash 8220 Sep 19 20:11 /etc/logstash/logstash.yml

node.name: logstash_1
path.data: /var/lib/logstash

path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: ${es_pwd}
xpack.monitoring.elasticsearch.hosts: ['http://10.10.10.242:9200']

After above configuration, logstash service is getting restarted after some time and hence no data is indexed yet.

I am getting below error in /var/log/logstash/logstash-plain.log.

[2021-09-19T21:30:29,805][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2021-09-19T21:30:31,740][INFO ][org.reflections.Reflections] Reflections took 232 ms to scan 1 urls, producing 120 keys and 417 values 
[2021-09-19T21:30:33,752][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://10.10.10.242:9200"]}
[2021-09-19T21:30:33,858][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@10.10.10.242:9200/]}}
[2021-09-19T21:30:33,910][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@10.10.10.242:9200/"}
[2021-09-19T21:30:34,033][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-09-19T21:30:34,036][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Configuration is data stream compliant but due backwards compatibility Logstash 7.x will not assume writing to a data-stream, default behavior will change on Logstash 8.0 (set `data_stream => true/false` to disable this warning)
[2021-09-19T21:30:34,107][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2021-09-19T21:30:34,299][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0xd5430b4 run>"}
[2021-09-19T21:30:36,829][ERROR][logstash.outputs.elasticsearch] Invalid setting for elasticsearch output plugin:

  output {
    elasticsearch {
      # This setting must be a path
      # File does not exist or cannot be opened /root/access_template.json
      template => "/root/access_template.json"
      ...
    }
  }
[2021-09-19T21:30:36,849][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
[2021-09-19T21:30:36,990][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:404:in `block in converge_state'"]}
[2021-09-19T21:30:36,992][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>2.68}
[2021-09-19T21:30:37,016][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

testing logstash pipeline config.

[FATAL] 2021-09-19 21:39:55.737 [LogStash::Runner] runner - The given configuration is invalid. Reason: Unable to configure plugins: Cannot evaluate `${es_pwd}`. Replacement variable `es_pwd` is not defined in a Logstash secret store or an environment entry and there is no default value given.
[FATAL] 2021-09-19 21:39:55.758 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

even after creating keystore and adding key to , its not recognising es_pwd.

With this setup logstash service is getting restarted frequently and index access-000001 is not created.

Thanks,

2

That's what it's complaining about. I think it needs to be an unquoted value.

3

Hi @warkolm,

Thanks for your reply.

I have removed quotes but it still seems to have issue. There is nothing after template =>

Sep 19 22:36:07 ip-10-10-10-242.e logstash[28931]: [2021-09-19T22:36:07,415][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", [A-Za-z_], \"-\", [0-9], \"[\", \"{\" at line 117, column 18 (byte 4726) after output {\n if [log_type] == \"access_server\" {\n elasticsearch {\n hosts => ['http://10.10.10.242:9200']\n user => elastic\n password => \"${es_pwd}\"\n index => \"access_server\"\n template => ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:391:in `block in converge_state'"]}

Thanks,

4

after removing quotes, checking the pipeline config file. Its not showing template file path in below output.

/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/logstash.conf

FATAL] 2021-09-19 22:49:22.070 [LogStash::Runner] runner - The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", [A-Za-z_], "-", [0-9], "[", "{" at line 117, column 18 (byte 4726) after output {
   if [log_type] == "access_server" {
  elasticsearch {
    hosts => ['http://10.10.10.242:9200']
        user => elastic
    password => "${es_pwd}"
     index => "access_server"
     template => 
[FATAL] 2021-09-19 22:49:22.083 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
[root@ip-10-10-10-242 ~]#

With quotes around the template =>, above logstash pipeline configuration verification command complaints about keystore.

[FATAL] 2021-09-19 22:56:57.865 [LogStash::Runner] runner - The given configuration is invalid. Reason: Unable to configure plugins: Cannot evaluate `${es_pwd}`. Replacement variable `es_pwd` is not defined in a Logstash secret store or an environment entry and there is no default value given.
5

Pretty sure you need the quotes

# File does not exist or cannot be opened /root/access_template.json
      template => "/root/access_template.json"

What is results of command below, error seems to indicate logstash user can not access

ls -l /root/access_template.json

6

@stephenb, Thanks for your reply.

here is the output.

7

Next question would be is the contents of that file valid.

What if you take out that template line in your logstash config does it pass the test and config?

8

This won't work, the /root directory has reading permissions only to the root user, it doesn't matter that the logstash user has permission in the file, it does not have permission to the directory.

Put the file in other directory where logstash can read, do not change the /root directory permissions.

1 Like
9

Hi @stephenb,

Thanks for your reply.

Contents of one of the index template.

access_template.json

{
  "version": 2,
  "priority": 600,
  "template": {
    "settings": {
      "index.number_of_shards": 1,
      "index.number_of_replicas": 0,
      "index.lifecycle.name": "testpolicy",      
      "index.lifecycle.rollover_alias": "access" 
    },
    "mappings": {
      "_source": {
        "enabled": true,
        "includes": [],
        "excludes": []
      },
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": true,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic_templates": []
    },
    "aliases": {
      "aliases": {
        "alias1": {
          "aliases": "access"
        }
      }
    }
  },
  "index_patterns": [
    "access*"
  ]
}

Similarly there are other template files for other indices (as i think i cannot mentioned multiple index patterns in single template), all have same version, priority.

After removing that line i see indices getting created.

green  open   access       gdIODalpQLuXXUEIfbZ_iw   1   0       7876            0      5.7mb          5.7mb
green  open   directory    yq4B7OMCQjaCeYZtWljQBQ   1   0       6727            0      4.3mb          4.3mb

but does that setting not required?

Not sure why, config file validation check command still says this,

[FATAL] 2021-09-19 22:56:57.865 [LogStash::Runner] runner - The given configuration is invalid. Reason: Unable to configure plugins: Cannot evaluate `${es_pwd}`. Replacement variable `es_pwd` is not defined in a Logstash secret store or an environment entry and there is no default value given.

Thanks

10

You need to see the --path.settings as well otherwise it does not know where the keystore is.

/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/logstash.conf --path.settings /etc/logstash/

11

So A) as @leandrojmp advised don't put that access_template.json file in the root directory, put in you /etc/logstash/config directory for example.

B) No if you just load the template by hand into elasticsearch and the indices match the index template pattern in your case "access*" you do not need to load each time from logstash... you can but not required, sort of depends on you approach and automation.

12

Thanks @leandrojmp, I have cleanup and ran deployment again by changing the path to /home/

13

Hi @stephenb,

As said above, i ran again the deployment by removing the template line from logstash pipeline conf file, because of which now logstash service is running fine from some time but data is not getting indexed. I think default logstash template is getting used.

/var/log/elasticsearch/elastic.log file,

[2021-09-20T01:05:54,243][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [portal-api] for index patterns [logstash-*]
[2021-09-20T01:05:54,311][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [core-api] for index patterns [logstash-*]
[2021-09-20T01:05:54,386][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [core-spa] for index patterns [logstash-*]
[2021-09-20T01:05:54,464][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [logstash] for index patterns [logstash-*]
[2021-09-20T01:05:54,571][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [ob-api] for index patterns [logstash-*]
[2021-09-20T01:05:54,716][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [ob-spa] for index patterns [logstash-*]
[2021-09-20T01:05:54,765][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [developer-spa] for index patterns [logstash-*]
[2021-09-20T01:05:54,938][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [directory] for index patterns [logstash-*]
[2021-09-20T01:05:55,203][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [next-api] for index patterns [logstash-*]
[2021-09-20T01:05:55,352][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [consent-spa] for index patterns [logstash-*]
[2021-09-20T01:05:55,533][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [access] for index patterns [logstash-*]
[2021-09-20T01:05:55,705][INFO ][o.e.c.m.MetadataIndexTemplateService] [elasticsearch_1] adding template [federate] for index patterns [logstash-*]

In the GET _index_template from kibana, it shows below,

 "name" : "access_template",
      "index_template" : {
        "index_patterns" : [
          "access*"

From the GET /_template output, it shows below

 "access" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"

Thanks,

14

I have lost track a bit ...

Did you manually PUT your template into Elasticsearch via Kibana Dev Tools? If yes and you set the output below it should be working.

If you use this in the elasticsearch output section it should be writing to that index and matching your index template.

index => "access_server"

You need to be a little careful as there are templates (legacy) and index_template (new)

15

Hi @stephenb,

I loaded template via PUT method through automation tool.

I have index line below (slightly different than what you given)

  if [log_type] == "access_server" {
  elasticsearch {
    hosts => ['http://10.10.10.242:9200']
        user => elastic
    password => "${es_pwd}"
     index => "access"
     template_name => "access"
     template_overwrite => "false"
      }
 }

Thanks,

16

And what is happening What is the resulting index?

If you loaded The template manually you don't need to name it again in the logstash It's one or the other but generally not both.

Try taking out

template_name => "access"

Perhaps since you put access* in your index template and the name in the output is just access it's looking for an index with an additional character in other words it's not matching the index pattern and if it does not match the template won't be applied.

I think you're close now you just need to be careful with your names and patterns

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.