# \[FATAL\]\[logstash.runner\]

**URL:** <https://discuss.elastic.co/t/fatal-logstash-runner/100416>\
**Category:** Logstash\
**Created:** [September 13, 2017, 8:59pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416 "2017-09-13T20:59:26Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![19831d72452f0be1a755](https://avatars.discourse-cdn.com/v4/letter/1/a88e57/32.png) [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\
**Post date:** [September 13, 2017, 8:59pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/1 "2017-09-13T20:59:26Z")

</div>

Hello.

I'm using Logstash 5.5 Elastic 5.5 Kibana 5.5 X-Pack 5.5  
logstash - ssl - elastic

Here's the error:

[FATAL][logstash.runner] An unexpected error occurred! {:error=\>#\<ArgumentError: Setting "[xpack.monitoring.elasticsearch.ssl.ca](http://xpack.monitoring.elasticsearch.ssl.ca)" must be a String. Received: ["/etc/logstash/ca.cert.pem"] (Array)\>, :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:228:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:403:in`validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:413:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:191:in`set'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:64:in `set_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:83:in`merge'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:83:in`merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:135:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:244:in`execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:209:in`run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in`(root)'"]}

Config logstash.yml (grep -v "^ #"):

xpack.monitoring.enabled: true  
xpack.monitoring.elasticsearch.url: "[https://test:9200](https://test:9200)"  
xpack.monitoring.elasticsearch.username: "logstash\_internal"  
xpack.monitoring.elasticsearch.password: "test"  
[xpack.monitoring.elasticsearch.ssl.ca](http://xpack.monitoring.elasticsearch.ssl.ca): ["/etc/logstash/ca.cert.pem"]  
path.data: /var/lib/logstash  
path.config: /etc/logstash/conf.d  
path.logs: /var/log/logstash

Config /etc/logstash/conf.d/logstash.conf

output {  
elasticsearch {  
user =\> "logstash\_internal"  
password =\> "test"

```
    #ssl => true
    #cacert => "/etc/logstash/ca.cert.pem"

    document_type => "nginx_logs"

    hosts => ["https://test:9200"]
    #hosts => ["test:9200"]

    index => "logstash-%{+YYYY.MM.dd}"

```

}  
}

Pls HELP!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 13, 2017, 9:07pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/2 "2017-09-13T21:07:39Z")

</div>

> [@19831d72452f0be1a755](#):
>
> xpack.monitoring.elasticsearch.ssl.ca: ["/etc/logstash/ca.cert.pem"]

That is not a string, it is an array, so remove the brackets.

---

<div class="post-metadata">

**Author:** ![19831d72452f0be1a755](https://avatars.discourse-cdn.com/v4/letter/1/a88e57/32.png) [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\
**Post date:** [September 13, 2017, 9:13pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/3 "2017-09-13T21:13:03Z")

</div>

Thanks, but new error :

[2017-09-14T00:11:58,390][WARN][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=\>#\<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections\>}  
[2017-09-14T00:11:58,390][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error\_message=\>"No Available connections", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will\_retry\_in\_seconds=\>16}  
[2017-09-14T00:12:02,872][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[https://logstash\_internal:xxxxxx@logstashtest:9200/](https://logstash_internal:xxxxxx@logstashtest:9200/), :path=\>"/"}  
[2017-09-14T00:12:02,943][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"[https://logstash\_internal:xxxxxx@logstashtest:9200/](https://logstash_internal:xxxxxx@logstashtest:9200/)", :error\_type=\>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=\>"Elasticsearch Unreachable: [[https://logstash\_internal:xxxxxx@logstashtest:9200/](https://logstash_internal:xxxxxx@logstashtest:9200/)][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}  
[2017-09-14T00:12:07,946][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[https://logstash\_internal:xxxxxx@logstashtest:9200/](https://logstash_internal:xxxxxx@logstashtest:9200/), :path=\>"/"}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 13, 2017, 9:13pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/4 "2017-09-13T21:13:27Z")

</div>

> [@19831d72452f0be1a755](#):
>
> Perhaps Elasticsearch is unreachable or down?

Did you check that?

---

<div class="post-metadata">

**Author:** ![19831d72452f0be1a755](https://avatars.discourse-cdn.com/v4/letter/1/a88e57/32.png) [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\
**Post date:** [September 13, 2017, 9:14pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/5 "2017-09-13T21:14:46Z")

</div>

[https://logstashtest:9200/](https://logstashtest:9200/)

{  
"name" : "7VVpZ3c",  
"cluster\_name" : "elasticsearch",  
"cluster\_uuid" : "0JjM1\_-2TMGXExMW\_P\_kFA",  
"version" : {  
"number" : "5.5.1",  
"build\_hash" : "19c13d0",  
"build\_date" : "2017-07-18T20:44:24.823Z",  
"build\_snapshot" : false,  
"lucene\_version" : "6.6.0"  
},  
"tagline" : "You Know, for Search"  
}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 13, 2017, 9:15pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/6 "2017-09-13T21:15:57Z")

</div>

> [@19831d72452f0be1a755](#):
>
> [https://logstashtest:9200/](https://logstashtest:9200/)

That does not match the URL you have in your config.

---

<div class="post-metadata">

**Author:** ![19831d72452f0be1a755](https://avatars.discourse-cdn.com/v4/letter/1/a88e57/32.png) [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\
**Post date:** [September 13, 2017, 9:18pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/7 "2017-09-13T21:18:53Z")

</div>

sorry, i change before send

conf :  
...  
xpack.monitoring.elasticsearch.url: "[https://logstashtest:9200](https://logstashtest:9200)"  
...

and:

...  
hosts =\> ["[https://logstashtest:9200](https://logstashtest:9200)"]  
...

---

<div class="post-metadata">

**Author:** ![19831d72452f0be1a755](https://avatars.discourse-cdn.com/v4/letter/1/a88e57/32.png) [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\
**Post date:** [September 13, 2017, 9:32pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/8 "2017-09-13T21:32:59Z")

</div>

I have deleted comments on the lines:  
ssl =\> true  
cacert =\> "/etc/logstash/ca.cert.pem"

output {  
elasticsearch {  
user =\> "logstash\_internal"  
password =\> "test"

```
    ssl => true
    cacert => "/etc/logstash/ca.cert.pem"

    document_type => "nginx_logs"

    hosts => ["https://logstashtest:9200"]
    #hosts => ["logstashtest:9200"]

    index => "logstash-%{+YYYY.MM.dd}"

```

}

Now hangs on stage:

[2017-09-14T00:28:20,731][INFO][logstash.pipeline] Pipeline .monitoring-logstash started  
[2017-09-14T00:28:20,770][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[https://logstash\_internal:xxxxxx@logstashtest:9200/](https://logstash_internal:xxxxxx@logstashtest:9200/)]}}  
[2017-09-14T00:28:20,773][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[https://logstash\_internal:xxxxxx@logstashtest:9200/](https://logstash_internal:xxxxxx@logstashtest:9200/), :path=\>"/"}  
[2017-09-14T00:28:20,887][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[https://logstash\_internal:xxxxxx@logstashtest:9200/](https://logstash_internal:xxxxxx@logstashtest:9200/)"}  
[2017-09-14T00:28:20,891][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}  
[2017-09-14T00:28:20,973][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>50001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "norms"=\>false}, "dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "include\_in\_all"=\>false}, "@version"=\>{"type"=\>"keyword", "include\_in\_all"=\>false}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[2017-09-14T00:28:21,013][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[https://logstashtest:9200](https://logstashtest:9200)"]}  
[2017-09-14T00:28:21,459][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>1, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>125}  
[2017-09-14T00:28:22,076][INFO][logstash.pipeline] Pipeline main started  
[2017-09-14T00:28:22,218][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 13, 2017, 9:34pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/9 "2017-09-13T21:34:02Z")

</div>

That's not hung, it's running.

---

<div class="post-metadata">

**Author:** ![19831d72452f0be1a755](https://avatars.discourse-cdn.com/v4/letter/1/a88e57/32.png) [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\
**Post date:** [September 13, 2017, 9:38pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/10 "2017-09-13T21:38:40Z")

</div>

Thanks. It's work! =)

It was necessary to remove the parentheses of the array []  
Did according to your instructions  
[https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html](https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html)  
[xpack.monitoring.elasticsearch.ssl.ca](http://xpack.monitoring.elasticsearch.ssl.ca): ["/path/to/ca.crt"]

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 13, 2017, 9:40pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/11 "2017-09-13T21:40:15Z")

</div>

We'll get that updated 🙂

---

<div class="post-metadata">

**Author:** ![19831d72452f0be1a755](https://avatars.discourse-cdn.com/v4/letter/1/a88e57/32.png) [@19831d72452f0be1a755](https://discuss.elastic.co/u/19831d72452f0be1a755)\
**Post date:** [September 13, 2017, 9:42pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/12 "2017-09-13T21:42:13Z")

</div>

Thank you for your prompt help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2017, 9:54pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner/100416/13 "2017-10-11T21:54:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.