# Lots of 429s, can't connect **URL:** \ **Category:** Logstash\ **Created:** [June 10, 2019, 7:53pm UTC](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047 "2019-06-10T19:53:38Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![topher](https://avatars.discourse-cdn.com/v4/letter/t/9de053/32.png) [@topher](https://discuss.elastic.co/u/topher)\ **Post date:** [June 10, 2019, 7:53pm UTC](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047/1 "2019-06-10T19:53:38Z")
A little over a week ago, I updated from 7.0 to 7.1.1. I didn't realize it until late last week, but going back through logs I see I am getting a ton of 429 errors that started about the same time. Now it sends messages saying it mostly can't connect and almost nothing makes it to elasticsearch from logstash. Sample messages include: [2019-06-10T15:48:59,203][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://10.226.1.93:9200/](http://10.226.1.93:9200/)"} [2019-06-10T15:48:59,207][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://10.226.1.96:9200/](http://10.226.1.96:9200/)"} [2019-06-10T15:49:03,504][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://10.226.1.99:9200/](http://10.226.1.99:9200/)][Manticore::SocketTimeout] Read timed out {:url=\>[http://10.226.1.99:9200/](http://10.226.1.99:9200/), :error\_message=\>"Elasticsearch Unreachable: [[http://10.226.1.99:9200/](http://10.226.1.99:9200/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} [2019-06-10T15:49:03,504][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [[http://10.226.1.99:9200/](http://10.226.1.99:9200/)][Manticore::SocketTimeout] Read timed out", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>8} [2019-06-10T15:49:04,212][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://10.226.1.99:9200/](http://10.226.1.99:9200/)"} [2019-06-10T15:49:08,257][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://10.226.1.95:9200/](http://10.226.1.95:9200/)][Manticore::SocketTimeout] Read timed out {:url=\>[http://10.226.1.95:9200/](http://10.226.1.95:9200/), :error\_message=\>"Elasticsearch Unreachable: [[http://10.226.1.95:9200/](http://10.226.1.95:9200/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} [2019-06-10T15:49:08,258][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error\_message=\>"Elasticsearch Unreachable: [[http://10.226.1.95:9200/](http://10.226.1.95:9200/)][Manticore::SocketTimeout] Read timed out", :class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will\_retry\_in\_seconds=\>16} [2019-06-10T15:49:08,269][WARN][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [[http://10.226.1.96:9200/](http://10.226.1.96:9200/)][Manticore::SocketTimeout] Read timed out {:url=\>[http://10.226.1.96:9200/](http://10.226.1.96:9200/), :error\_message=\>"Elasticsearch Unreachable: [[http://10.226.1.96:9200/](http://10.226.1.96:9200/)][Manticore::SocketTimeout] Read timed out", :error\_class=\>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"} Any assistance would be appreciated! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 10, 2019, 10:03pm UTC](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047/2 "2019-06-10T22:03:06Z")
What do your Elasticsearch logs show? ---
**Author:** ![topher](https://avatars.discourse-cdn.com/v4/letter/t/9de053/32.png) [@topher](https://discuss.elastic.co/u/topher)\ **Post date:** [June 11, 2019, 1:37pm UTC](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047/3 "2019-06-11T13:37:54Z")
Lots more interesting stuff, apparently. [2019-06-11T00:11:33,123][WARN][o.e.x.m.e.l.LocalExporter] [FL-JAX-SECELKELS1] unexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring.exporter.ExportException: RemoteTransportException[[FL-JAX-SECELKELS6][10.226.1.99:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[FL-JAX-SECELKELS5][10.226.1.92:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[FL-JAX-SECELKELS5][10.226.1.92:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of processing of [2425608][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[.monitoring-es-7-2019.06.11][0]] containing [index {[.monitoring-es-7-2019.06.11][\_doc][ZuS7RGsBU6B\_O6Kvx0a0], source[{"cluster\_uuid":"8Le75XanSxuXyvR2IvzoYw","timestamp":"2019-06-11T04:11:25.398Z","interval\_ms":10000,"type":"node\_stats","source\_node":{"uuid":"IKx-4rGjR2qCf-IPXbVvWw","host":"10.226.1.94","transport\_address":"10.226.1.94:9300","ip":"10.226.1.94","name":"FL-JAX-SECELKELS1","timestamp":"2019-06-11T04:11:25.397Z"},"node\_stats":{"node\_id":"IKx-4rGjR2qCf-IPXbVvWw","node\_master":false,"mlockall":false,"indices":{"docs":{"count":7000135709},"store":{"size\_in\_bytes":4558644814452},"indexing":{"index\_total":41772,"index\_time\_in\_millis":27982,"throttle\_time\_in\_millis":0},"search":{"query\_total":10184,"query\_time\_in\_millis":4068},"query\_cache":{"memory\_size\_in\_bytes":2160,"hit\_count":182,"miss\_count":145,"evictions":0},"fielddata":{"memory\_size\_in\_bytes":11064,"evictions":0},"segments":{"count":655,"memory\_in\_bytes":8021958571,"terms\_memory\_in\_bytes":5668982758,"stored\_fields\_memory\_in\_bytes":2160722024,"term\_vectors\_memory\_in\_bytes":0,"norms\_memory\_in\_bytes":0,"points\_memory\_in\_bytes":189215633,"doc\_values\_memory\_in\_bytes":3038156,"index\_writer\_memory\_in\_bytes":0,"version\_map\_memory\_in\_bytes":0,"fixed\_bit\_set\_memory\_in\_bytes":91472},"request\_cache":{"memory\_size\_in\_bytes":39439,"evictions":0,"hit\_count":1883,"miss\_count":547}},"os":{"cpu":{"load\_average":{"1m":0.0,"5m":0.01,"15m":0.05}},"cgroup":{"cpuacct":{"control\_group":"/","usage\_nanos":49194604672373},"cpu":{"control\_group":"/","cfs\_period\_micros":100000,"cfs\_quota\_micros":-1,"stat":{"number\_of\_elapsed\_periods":0,"number\_of\_times\_throttled":0,"time\_throttled\_nanos":0}},"memory":{"control\_group":"/","limit\_in\_bytes":"9223372036854771712","usage\_in\_bytes":"64726368256"}}},"process":{"open\_file\_descriptors":7656,"max\_file\_descriptors":65535,"cpu":{"percent":0}},"jvm":{"mem":{"heap\_used\_in\_bytes":9970663680,"heap\_used\_percent":37,"heap\_max\_in\_bytes":26773815296},"gc":{"collectors":{"young":{"collection\_count":1161,"collection\_time\_in\_millis":55812},"old":{"collection\_count":2,"collection\_time\_in\_millis":115}}}},"thread\_pool":{"generic":{"threads":12,"queue":0,"rejected":0},"get":{"threads":0,"queue":0,"rejected":0},"management":{"threads":5,"queue":0,"rejected":0},"search":{"threads":13,"queue":0,"rejected":0},"watcher":{"threads":0,"queue":0,"rejected":0},"write":{"threads":8,"queue":0,"rejected":0}},"fs":{"total":{"total\_in\_bytes":10994540609536,"free\_in\_bytes":6289274310656,"available\_in\_bytes":6289274310656},"io\_stats":{"total":{"operations":749123,"read\_operations":148205,"write\_operations":600918,"read\_kilobytes":49172880,"write\_kilobytes":367900076}}}}}]}], target allocation id: ZBiDS4RhRKymNm4kKGsEGQ, primary term: 1 on EsThreadPoolExecutor[name = FL-JAX-SECELKELS5/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@119b39bf[Running, pool size = 8, active threads = 8, queued tasks = 200, completed tasks = 148727]]]; at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.1.1.jar:7.1.1] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) ~[?:?] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) [x-pack-monitoring-7.1.1.jar:7.1.1] at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:108) [x-pack-monitoring-7.1.1.jar:7.1.1] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:659) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:464) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:459) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:74) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:937) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:895) [elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) [elasticsearch-7.1.1.jar:7.1.1] Any tips on uploading a .txt file so I can share a full entry? Above is abbreviated. If that's not enough, let me know. 🙂 ---
**Author:** ![topher](https://avatars.discourse-cdn.com/v4/letter/t/9de053/32.png) [@topher](https://discuss.elastic.co/u/topher)\ **Post date:** [June 11, 2019, 4:09pm UTC](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047/4 "2019-06-11T16:09:23Z")
Based on other things I've found online, I commented out the xpack.monitoring options from the various yml files, restarted logstash and one of the ES nodes in my cluster, now the tremendous flows of errors in logs have stopped.... but I'm still not seeing data in kibana under Discover for my logstash filter. (Worth noting, the filebeat instances I have running do seem to be passing traffic directly to ES just fine throughout.) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 9, 2019, 4:09pm UTC](https://discuss.elastic.co/t/lots-of-429s-cant-connect/185047/5 "2019-07-09T16:09:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.