# Unable to push logs to elasticsearch after upgrade
**URL:** https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738
**Category:** Elasticsearch
**Created:** [July 5, 2018, 2:00pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738 "2018-07-05T14:00:01Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@kavirayani\_shiva](https://discuss.elastic.co/u/kavirayani_shiva)
#### Post date: [July 5, 2018, 2:00pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/1 "2018-07-05T14:00:01Z")
I have upgraded the elasticsearch to 5.6.9 version and the logstash is still at the 2.4.1. After the upgrade the logstash is unable to push the logs to the elasticsearch. below are the errors i see in the logstash.log
{:timestamp=\>"2018-07-05T13:10:17.121000+0000", :message=\>"Cannot get new connection from pool.", :class=\>"Elasticsearch::Transport::Transport::Error", :backtrace=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:249:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in`perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/sniffer.rb:32:in `hosts'", "org/jruby/ext/timeout/Timeout.java:147:in`timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:79:in`reload\_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:72:in `sniff!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in`start\_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in`start\_sniffing!'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:in`start\_sniffing!'"], :level=\>:error}
{:timestamp=\>"2018-07-05T13:10:17.614000+0000", :message=\>"Cannot get new connection from pool.", :class=\>"Elasticsearch::Transport::Transport::Error", :backtrace=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:249:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in`perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in`bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http\_client.rb:53:in `non_threadsafe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in`bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in`bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in`submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in`multi\_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in`multi\_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output\_delegator.rb:130:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in`multi\_receive'",
It says that the elasticsearch is down but i see the elasticsearch up and running
[cloud\_1IF-1IF\_FACTORY root@LOG-0-1 logstash]# curl localhost:9200/\_cluster/health?pretty
{
"cluster\_name" : "logstash",
"status" : "yellow",
"timed\_out" : false,
"number\_of\_nodes" : 1,
"number\_of\_data\_nodes" : 1,
"active\_primary\_shards" : 6,
"active\_shards" : 6,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 6,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
"active\_shards\_percent\_as\_number" : 50.0
}
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 5, 2018, 8:11pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/2 "2018-07-05T20:11:16Z")
What does your Logstash config look like?
---
### Author:  [@kavirayani\_shiva](https://discuss.elastic.co/u/kavirayani_shiva)
#### Post date: [July 6, 2018, 1:16pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/3 "2018-07-06T13:16:00Z")
grok {
patterns\_dir =\> "/etc/logstash/patterns"
match =\> { "message" =\> "(?m)%{TANUKI\_LOG}"}
add\_tag =\> ["parsed"]
}
date {
match =\> ["tanuki\_timestamp", "yyyy/MM/dd HH:mm:ss"]
}
if "\_grokparsefailure" not in [tags] {
mutate {
replace =\> ["message", "%{tanuki\_message}"]
remove\_field =\> ["tanuki\_message","tanuki\_timestamp"]
}
}
}
}
mutate {
uppercase =\> ["level"]
}
}
}
# END TEMPLATE '/etc/puppet/modules/ntc\_profile\_centrallog/templates/filter.apache.erb'
# START TEMPLATE: '/etc/puppet/modules/ntc\_profile\_centrallog/templates/filter.syslog.erb'
filter {
if [type] == "syslog" and "parsed" not in [tags]{
grok {
patterns\_dir =\> "/etc/logstash/patterns"
# Use special hostname pattern which allows underscores in hostnames, to match switches
# Some example syslog messages:
# Normal: Sep 8 15:26:51 BSC-0-1 crmd[3467]: notice: State transition S\_IDLE -\> S\_POLICY\_ENGINE [input=I\_PE\_CALC cause=C\_TIMER\_POPPED origin=crm\_timer\_popped]
# HP DSW: Sep 8 14:37:37 corasat-4IF-HMMS\_LOC1-DSW-1 %%10ARP/5/ARP\_DUPLICATE\_IPADDR\_DETECT(l): -DevIP=172.20.0.1; Detected an IP address conflict.
# Cisco ASW: Sep 8 14:40:40 asw-1 3329: corasat-4IF-HMMS\_LOC1-ASW-1: Sep 8 14:36:52.996: %SYS-5-CONFIG\_I: Configured from 172.20.0.42 by snmp
match =\> { "message" =\> "(?:%{SYSLOGTIMESTAMP:syslog\_timestamp}|%{TIMESTAMP\_ISO8601:syslog\_timestamp}) (?:.\*%{NTC\_SYSLOG\_HOSTNAME\_SWITCH:syslog\_hostname}:?|%{NTC\_SYSLOG\_HOSTNAME:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?🙂 %{GREEDYDATA:syslog\_message}" }
add\_tag =\> ["parsed"]
add\_field =\> ["received\_at", "%{@timestamp}"]
add\_field =\> ["received\_from", "%{host}"]
}
syslog\_pri { }
date {
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
}
if "\_grokparsefailure" not in [tags] {
mutate {
replace =\> ["host", "%{syslog\_hostname}"]
replace =\> ["message", "%{syslog\_message}"]
}
}
# # PLATO-2325: because of a bug in JRuby disabling this.
# # Once upgrade to Logstash 5 we can try to re-enable this.
# dns {
# reverse =\> ["host"]
# action =\> "replace"
# hit\_cache\_size =\> 20
# failed\_cache\_size =\> 20
# }
# PLATO-2325 Manually parsing the hostnames of the switches.
# REMOVE WHEN RE-ENABLING DNS ABOVE
mutate {
gsub =\> [
"host", "172.20.0.3", "ASW-1",
"host", "172.20.0.3", "ASW-2",
"host", "172.20.0.1", "DSW-1",
"host", "172.20.0.2", "DSW-2",
"host", "172.19.1.12", "DSW-1.dmavlan",
"host", "172.19.1.13", "DSW-2.dmavlan"
]
}
if "\_grokparsefailure" not in [tags] {
mutate {
# Remove cruft from switches hostnames
add\_field =\> ["level", "%{syslog\_severity}"]
remove\_field =\> ["syslog\_hostname", "syslog\_message", "syslog\_timestamp"]
}
}
mutate {
gsub =\> [
"level", "debug", "DEBUG",
"level", "informational", "INFO",
"level", "notice", "NOTICE",
"level", "warning", "WARN",
"level", "error", "ERROR",
"level", "alert", "ALERT",
"level", "critical", "CRIT",
"level", "emergency", "EMERG"
]
}
}
}
# END TEMPLATE '/etc/puppet/modules/ntc\_profile\_centrallog/templates/filter.syslog.erb'
# START TEMPLATE: '/etc/puppet/modules/ntc\_profile\_centrallog/templates/filter.removerawmessage.erb'
filter {
if "\_grokparsefailure" not in [tags] and [raw\_message] {
mutate {
remove\_field =\> ["raw\_message"]
}
}
}
# END TEMPLATE '/etc/puppet/modules/ntc\_profile\_centrallog/templates/filter.removerawmessage.erb'
output {
elasticsearch {
hosts =\> ["localhost"]
# sniffing =\> true
}
}Preformatted text
---
### Author:  [@kavirayani\_shiva](https://discuss.elastic.co/u/kavirayani_shiva)
#### Post date: [July 6, 2018, 1:18pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/4 "2018-07-06T13:18:25Z")
Actually we have many rules which are specified out of which the lostash output to elasticsearch plugin is configured as below
output {
elasticsearch {
hosts =\> ["localhost"]
sniffing =\> true
}
}
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 6, 2018, 1:37pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/5 "2018-07-06T13:37:50Z")
Please format your code, logs or configuration files using `` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) and not the citation button. It will make your post more readable.
Or use markdown style like:
````
```
CODE
```
````
This is the icon to use if you are not using markdown format:
There's a live preview panel for exactly this reasons.
Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.
---
### Author:  [@kavirayani\_shiva](https://discuss.elastic.co/u/kavirayani_shiva)
#### Post date: [July 6, 2018, 1:42pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/6 "2018-07-06T13:42:39Z")
Thanks done the same
---
### Author:  [@A\_B](https://discuss.elastic.co/u/A_B)
#### Post date: [July 6, 2018, 1:44pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/7 "2018-07-06T13:44:48Z")
> [@kavirayani\_shiva](#):
>
> output {
> elasticsearch {
> hosts =\> ["localhost"]
> sniffing =\> true
> }
> }
I'm pretty sure you have to specify the Elasticsearch port as well as the hostname. That would be the first thing I would try.
---
### Author:  [@kavirayani\_shiva](https://discuss.elastic.co/u/kavirayani_shiva)
#### Post date: [July 6, 2018, 1:46pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/8 "2018-07-06T13:46:31Z")
I tried that indeed and the problem still persists
---
### Author:  [@kavirayani\_shiva](https://discuss.elastic.co/u/kavirayani_shiva)
#### Post date: [July 6, 2018, 1:50pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/9 "2018-07-06T13:50:24Z")
Ãlso to add on my elasticcluster is healthy enough and is serving the requests as well
curl -X GET [http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health)
{"cluster\_name":"logstash","status":"yellow","timed\_out":false,"number\_of\_nodes":1,"number\_of\_data\_nodes":1,"active\_primary\_shards":10,"active\_shards":10,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":10,"delayed\_unassigned\_shards":0,"number\_of\_pending\_tasks":0,"number\_of\_in\_flight\_fetch":0,"task\_max\_waiting\_in\_queue\_millis":0,"active\_shards\_percent\_as\_number":50.0}
---
### Author:  [@A\_B](https://discuss.elastic.co/u/A_B)
#### Post date: [July 6, 2018, 2:00pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/10 "2018-07-06T14:00:13Z")
Do you still have the same errors in `logstash.log`?
---
### Author:  [@kavirayani\_shiva](https://discuss.elastic.co/u/kavirayani_shiva)
#### Post date: [July 6, 2018, 2:10pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/11 "2018-07-06T14:10:35Z")
yes i see the same logs in the logs as well
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [July 6, 2018, 4:14pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/12 "2018-07-06T16:14:25Z")
No. It's not correct. For example you have:
output {
elasticsearch {
hosts =\> ["localhost"]
sniffing =\> true
}
}
While I'm expecting something like:
```auto
output {
elasticsearch {
hosts => ["localhost"]
sniffing => true
}
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 3, 2018, 4:14pm UTC](https://discuss.elastic.co/t/unable-to-push-logs-to-elasticsearch-after-upgrade/138738/13 "2018-08-03T16:14:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.