# .NET APM agent An existing connection was forcibly closed by the remote host

**URL:** <https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942>\
**Category:** APM\
**Tags:** dotnet\
**Created:** [May 27, 2019, 3:52pm UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942 "2019-05-27T15:52:20Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Futerkowiec](https://avatars.discourse-cdn.com/v4/letter/f/edb3f5/32.png) [@Futerkowiec](https://discuss.elastic.co/u/Futerkowiec)\
**Post date:** [May 27, 2019, 3:52pm UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942/1 "2019-05-27T15:52:20Z")

</div>

**Kibana version** : 7.1.0

**Elasticsearch version** : 7.1.0

**APM Server version** : 7.1.0

**APM Agent language and version** : .NET Agent version 0.0.2.0

**Browser version** : Firefox 67.0

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: all components were downloaded from official elastic page.

**Fresh install or upgraded from other version?** Fresh install

**Is there anything special in your setup?** all components are working on my local machine.

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: I am trying to get any events from apm agent into ES to see how does it work. I run elasticsearch, kibana, apm server and apm agent on my local machine.  
To see if configuration is correct and how APM documents looks like in kibana I wrote small program and after i run method

```
Agent.Tracer.CaptureTransaction("test_name", "test_type", () =>
{
	string rt;
	WebRequest request = WebRequest.Create("http://localhost:9200/_count");
	WebResponse response = request.GetResponse();
	Stream dataStream = response.GetResponseStream();
	StreamReader reader = new StreamReader(dataStream);
	rt = reader.ReadToEnd();
	Console.WriteLine(rt);
	reader.Close();
	response.Close();
	Thread.Sleep(1000);
});

```

no events appear in index and elasticsearch logged following error:

```
[2019-05-27T16:47:57,120][WARN][o.e.h.AbstractHttpServerTransport] [PCName] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/0:0:0:0:0:0:0:1:9200, remoteAddress=/0:0:0:0:0:0:0:1:61717}
java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
    at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.read(Unknown Source) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[?:?]
    at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_211]

```

In apm-server logs i did not find anything except few lines after i started server which contains info about established connection to elasticsearch.  
To see if elasticsearch works properly, i used metricbeat to colect some events and it worked.

I would like to ask for some advice what I did wrong.  
Here are configs for apm-server and elasticsearch:

```
apm-server:
	host: "localhost:8200"
output.elasticsearch:
	hosts: ["localhost:9200"]

```

elasticsearch config:

```
bootstrap.memory_lock: false
cluster.name: elasticsearch
http.port: 9200
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
node.name: PCNAME
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
transport.tcp.port: 9300
xpack.license.self_generated.type: basic
xpack.security.enabled: false
network.host: 0.0.0.0
cluster.initial_master_nodes: PCNAME

```

if anything else is needed let me know.  
Thank you in advance!

---

<div class="post-metadata">

**Author:** ![GregKalapos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregkalapos/32/37205_2.png) [@GregKalapos](https://discuss.elastic.co/u/GregKalapos)\
**Post date:** [May 27, 2019, 5:25pm UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942/2 "2019-05-27T17:25:33Z")

</div>

Hi @Futerkowiec,

I'm not totally sure this is related to the agent. Seems like something between elasticsearch and the apm-server.

I'd first make sure the agent can send the date to the APM Server. For this you can set the loglevel setting to `Debug`, it's described here: [https://www.elastic.co/guide/en/apm/agent/dotnet/current/config-reporter.html#config-log-level](https://www.elastic.co/guide/en/apm/agent/dotnet/current/config-reporter.html#config-log-level)

E.g. one easy way is to set the `ELASTIC_APM_LOG_LEVEL` environment variable to `Debug`.

If we don't see anything suspicious in those logs, we should check what happens between elasticsearch and the apm-server.

---

<div class="post-metadata">

**Author:** ![Futerkowiec](https://avatars.discourse-cdn.com/v4/letter/f/edb3f5/32.png) [@Futerkowiec](https://discuss.elastic.co/u/Futerkowiec)\
**Post date:** [May 28, 2019, 3:36pm UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942/3 "2019-05-28T15:36:58Z")

</div>

Hi @GregKalapos,

Thanks for quick response!  
After your messege here is what I did.  
I set loglevel in agent, apm-server and elastic to debug. Unfortunately i did not find anything usefull in apm-server or in elasticseach log. I am not sure where can I find logs from agent. If they should appear in console while catching exception in CaptureTransaction() method then I did not see anything there.  
In addition, whenever I start apm-server, elasticsearch get document in "apm-7.1.0-onboarding-2019.05.28" index. Moreover I checked in Kibana status of apm-server and it showed it is configured correctly.  
I noticed one more thing. Whenever I terminate apm-server elasticsearch log the same warning as when I run my program

```
[2019-05-28T12:25:10,715][WARN][o.e.h.AbstractHttpServerTransport] [PCName] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/0:0:0:0:0:0:0:1:9200, remoteAddress=/0:0:0:0:0:0:0:1:61004}
java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
    at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.read(Unknown Source) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[?:?]
    at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_211]

```

Do you have any other idea what can I check?

---

<div class="post-metadata">

**Author:** ![Sergey\_Kleyman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sergey_kleyman/32/39333_2.png) [@Sergey\_Kleyman](https://discuss.elastic.co/u/Sergey_Kleyman)\
**Post date:** [May 28, 2019, 6:11pm UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942/4 "2019-05-28T18:11:48Z")

</div>

Hi @Futerkowiec,

Let's try to narrow down which leg of communication has issues: agent \<-\> apm-server or apm-server \<-\> elasticsearch. Could you please try out the steps listed at [Common problems | APM Server Reference [7.15] | Elastic](https://www.elastic.co/guide/en/apm/server/current/common-problems.html#no-data-indexed)  
especially the part

> To see if the agent can connect to the APM Server, send requests to the instrumented service and look for lines containing `[request]` in the APM Server logs.

Please run your test app instrumented by the agent and see if you have lines containing `[request]` in the APM Server logs.

---

<div class="post-metadata">

**Author:** ![Sergey\_Kleyman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sergey_kleyman/32/39333_2.png) [@Sergey\_Kleyman](https://discuss.elastic.co/u/Sergey_Kleyman)\
**Post date:** [May 28, 2019, 6:18pm UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942/5 "2019-05-28T18:18:30Z")

</div>

Another question regarding the application you have instrumented by the agent. Is the excerpt containing `Agent.Tracer.CaptureTransaction` the whole program? Namely does this program terminate right after the call to `Agent.Tracer.CaptureTransaction` returns? If so please add a short sleep (let's say 10 seconds) after `Agent.Tracer.CaptureTransaction` returns but before the end of the program and see if running a modified program causes lines containing `[request]` to appear in the APM Server logs.

```
Agent.Tracer.CaptureTransaction("test_name", "test_type", () =>
{
	// ...
});

Thread.Sleep(1000);

```

Since the agent sends data to APM Server asynchronously it's possible that the program terminates before agent has a chance to send anything to APM Server.

---

<div class="post-metadata">

**Author:** ![Futerkowiec](https://avatars.discourse-cdn.com/v4/letter/f/edb3f5/32.png) [@Futerkowiec](https://discuss.elastic.co/u/Futerkowiec)\
**Post date:** [May 29, 2019, 8:39am UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942/6 "2019-05-29T08:39:52Z")

</div>

Hi @Sergey_Kleyman ,  
Thank you for your advice!  
Indeed that was the problem. After I added sleep after CaptureTransaction(), everything worked.

Thank you so much for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 19, 2019, 4:40am UTC](https://discuss.elastic.co/t/net-apm-agent-an-existing-connection-was-forcibly-closed-by-the-remote-host/182942/7 "2019-06-19T04:40:02Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.