# Elastic cluster "Received ban for the parent \[channel closed\]", Apache Metron "listener timeout after waiting for"
**URL:** https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535
**Category:** Elasticsearch
**Created:** [December 14, 2020, 5:23am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535 "2020-12-14T05:23:22Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)
#### Post date: [December 14, 2020, 5:23am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/1 "2020-12-14T05:23:22Z")
Hello, we using ES cluster with 3 nodes, each datanode. We faced with problem that sometimes we can't write into our cluster using Apache Metron :
```auto
java.io.IOException: listener timeout after waiting for [60000] ms at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:660) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:219) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:191) at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:396) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:382) at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:197) at org.apache.metron.elasticsearch.bulk.ElasticsearchBulkDocumentWriter.write(ElasticsearchBulkDocumentWriter.java:89) at org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:105) at org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:123) at org.apache.metron.writer.BulkWriterComponent.applyShouldFlush(BulkWriterComponent.java:179) at org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:99) at org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:303) at org.apache.storm.daemon.executor$fn __10219$tuple_action_fn__ 10221.invoke(executor.clj:745) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10138.invoke(executor.clj:473) at org.apache.storm.disruptor$clojure_handler$reify__4115.onEvent(disruptor.clj:41) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:509) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:487) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:74) at org.apache.storm.daemon.executor$fn __10219$fn__ 10232$fn__10287.invoke(executor.clj:868) at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748)
```
At this time at cluster nodes we get this messages:
```auto
[2020-12-14T10:00:04,292][DEBUG][o.e.t.TaskCancellationService] [h1-es03] Received ban for the parent [MT3BSgtaQBWux8BJDBSsHg:53280489] on the node [Qshtg7-TQIyxeiccpkmlIA], reason: [channel closed]
[2020-12-14T10:00:04,463][DEBUG][o.e.t.TaskCancellationService] [h1-es03] Received ban for the parent [MT3BSgtaQBWux8BJDBSsHg:53280512] on the node [Qshtg7-TQIyxeiccpkmlIA], reason: [channel closed]
[2020-12-14T10:00:04,523][DEBUG][o.e.t.TaskCancellationService] [h1-es03] Received ban for the parent [MT3BSgtaQBWux8BJDBSsHg:53280620] on the node [Qshtg7-TQIyxeiccpkmlIA], reason: [channel closed]
[2020-12-14T10:00:04,631][DEBUG][o.e.t.TaskCancellationService] [h1-es03] Received ban for the parent [MT3BSgtaQBWux8BJDBSsHg:53280799] on the node [Qshtg7-TQIyxeiccpkmlIA], reason: [channel closed]
[2020-12-14T10:00:04,689][DEBUG][o.e.t.TaskCancellationService] [h1-es03] Received ban for the parent [MT3BSgtaQBWux8BJDBSsHg:53280902] on the node [Qshtg7-TQIyxeiccpkmlIA], reason: [channel closed]
[2020-12-14T10:00:04,857][DEBUG][o.e.t.TaskCancellationService] [h1-es03] Received ban for the parent [MT3BSgtaQBWux8BJDBSsHg:53281046] on the node [Qshtg7-TQIyxeiccpkmlIA], reason: [channel closed]
```
And also:
```auto
org.elasticsearch.transport.TransportException: failure to send
...
Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks
```
Have someone else faced with it? Help us to understand what's going on, please!
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 14, 2020, 6:41am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/2 "2020-12-14T06:41:54Z")
Which version of Elasticsearch are you using? What does your configuration look like?
---
### Author:  [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)
#### Post date: [December 14, 2020, 6:48am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/3 "2020-12-14T06:48:02Z")
Elasticsearch version 7.9.1. How I can upload file with settings? Seems like I can attach only images to my messages
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 14, 2020, 7:02am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/4 "2020-12-14T07:02:56Z")
> [@ThreatInter](#):
>
> we can't write into our cluster using Apache Metron :
Does this client enable [TCP keepalives](https://github.com/elastic/elasticsearch/issues/65213), and is your OS configured to send them promptly? If not, that would explain this.
---
### Author:  [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)
#### Post date: [December 14, 2020, 7:22am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/5 "2020-12-14T07:22:39Z")
Elastic nodes have this settings:
net.ipv4.tcp\_keepalive\_intvl = 60
net.ipv4.tcp\_keepalive\_probes = 10
net.ipv4.tcp\_keepalive\_time = 300
Client in case Apache Metron - Elastic is Elastic isn't it?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 14, 2020, 7:40am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/6 "2020-12-14T07:40:47Z")
Those settings sound good for the Elasticsearch nodes but you also need keepalives on the connection from Metron and Elasticsearch. I have no experience with Metron so can't tell you how to do that.
Another possible explanation is that Metron is configured to time out requests after 60 seconds, but the failing request simply needs longer.
---
### Author:  [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)
#### Post date: [December 14, 2020, 7:42am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/7 "2020-12-14T07:42:21Z")
Ok, will try. Thank you for your answers.
Oh, we have trobles between ES nodes too. How we can deal with it?
```auto
[2020-12-14T12:42:43,885][DEBUG][o.e.a.s.TransportSearchAction]
...
org.elasticsearch.transport.TransportException: failure to send
...
Caused by: org.elasticsearch.tasks.TaskCancelledException: The parent task was cancelled, shouldn't start any child tasks
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 14, 2020, 7:54am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/8 "2020-12-14T07:54:05Z")
How have you secured the cluster? Are you using any third party plugins?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 14, 2020, 7:55am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/9 "2020-12-14T07:55:45Z")
> [@ThreatInter](#):
>
> Oh, we have trobles between ES nodes too. How we can deal with it?
The message you quote is a `DEBUG` message and can therefore be ignored. It indicates that a search was cancelled because the client disconnected, which is the expected behaviour.
---
### Author:  [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)
#### Post date: [December 14, 2020, 7:58am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/10 "2020-12-14T07:58:45Z")
Just time of this DEBUG messages and errors from Metron is the same\*. So I thought that they linked some way.
\*We see this DEBUG messages and Metron errors about the same time
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 14, 2020, 8:04am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/11 "2020-12-14T08:04:06Z")
> [@ThreatInter](#):
>
> So I thought that they linked some way.
Yes, they are. They indicate that the client disconnected.
---
### Author:  [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)
#### Post date: [December 14, 2020, 10:35am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/12 "2020-12-14T10:35:13Z")
We set Metron's storms settings same as ES nodes:
```auto
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 10
net.ipv4.tcp_keepalive_time = 300
```
And it doesn't help. We again get errors during writing to ES.
Now we also have this message:
```auto
[2020-12-14T15:48:36,265][DEBUG][o.e.m.j.JvmGcMonitorService] [h1-es01] [gc][430695] overhead, spent [191ms] collecting in the last [1s]
```
One of nodes left cluster
```auto
[2020-12-14T15:42:33,493][DEBUG][o.e.c.s.MasterService] [h1-es02] executing cluster state update for [node-left[{h1-es01}{MT3BSgtaQBWux8BJDBSsHg}{-bqqYp4XQbSAio6SjrVMlw}{h1-es01ip}{h1-es01ip}{dimr} reason: followers ch
eck retry count exceeded]]
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 14, 2020, 11:16am UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/13 "2020-12-14T11:16:03Z")
> [@ThreatInter](#):
>
> We set Metron's storms settings same as ES nodes:
That may not be enough, the client needs to specifically request keepalives on each connection too. One way to check this is to run `sudo netstat -anto` and verify that the connections from the client do have a keepalive timer.
Also did you address this:
> [@DavidTurner](#):
>
> Another possible explanation is that Metron is configured to time out requests after 60 seconds, but the failing request simply needs longer.
---
### Author:  [@ThreatInter](https://discuss.elastic.co/u/ThreatInter)
#### Post date: [December 14, 2020, 12:16pm UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/14 "2020-12-14T12:16:32Z")
looks like connection haven't keepalive timer
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 11, 2021, 12:16pm UTC](https://discuss.elastic.co/t/elastic-cluster-received-ban-for-the-parent-channel-closed-apache-metron-listener-timeout-after-waiting-for/258535/15 "2021-01-11T12:16:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.