.NET APM agent An existing connection was forcibly closed by the remote host

Kibana version: 7.1.0

Elasticsearch version: 7.1.0

APM Server version: 7.1.0

APM Agent language and version: .NET Agent version 0.0.2.0

Browser version: Firefox 67.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: all components were downloaded from official elastic page.

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? all components are working on my local machine.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): I am trying to get any events from apm agent into ES to see how does it work. I run elasticsearch, kibana, apm server and apm agent on my local machine.
To see if configuration is correct and how APM documents looks like in kibana I wrote small program and after i run method

Agent.Tracer.CaptureTransaction("test_name", "test_type", () =>
{
	string rt;
	WebRequest request = WebRequest.Create("http://localhost:9200/_count");
	WebResponse response = request.GetResponse();
	Stream dataStream = response.GetResponseStream();
	StreamReader reader = new StreamReader(dataStream);
	rt = reader.ReadToEnd();
	Console.WriteLine(rt);
	reader.Close();
	response.Close();
	Thread.Sleep(1000);
});

no events appear in index and elasticsearch logged following error:

[2019-05-27T16:47:57,120][WARN ][o.e.h.AbstractHttpServerTransport] [PCName] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/0:0:0:0:0:0:0:1:9200, remoteAddress=/0:0:0:0:0:0:0:1:61717}
java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
    at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.read(Unknown Source) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[?:?]
    at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_211]

In apm-server logs i did not find anything except few lines after i started server which contains info about established connection to elasticsearch.
To see if elasticsearch works properly, i used metricbeat to colect some events and it worked.

I would like to ask for some advice what I did wrong.
Here are configs for apm-server and elasticsearch:

apm-server:
	host: "localhost:8200"
output.elasticsearch:
	hosts: ["localhost:9200"]

elasticsearch config:

bootstrap.memory_lock: false
cluster.name: elasticsearch
http.port: 9200
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
node.name: PCNAME
path.data: C:\ProgramData\Elastic\Elasticsearch\data
path.logs: C:\ProgramData\Elastic\Elasticsearch\logs
transport.tcp.port: 9300
xpack.license.self_generated.type: basic
xpack.security.enabled: false
network.host: 0.0.0.0
cluster.initial_master_nodes: PCNAME

if anything else is needed let me know.
Thank you in advance!

Hi @Futerkowiec,

I'm not totally sure this is related to the agent. Seems like something between elasticsearch and the apm-server.

I'd first make sure the agent can send the date to the APM Server. For this you can set the loglevel setting to Debug, it's described here: https://www.elastic.co/guide/en/apm/agent/dotnet/current/config-reporter.html#config-log-level

E.g. one easy way is to set the ELASTIC_APM_LOG_LEVEL environment variable to Debug.

If we don't see anything suspicious in those logs, we should check what happens between elasticsearch and the apm-server.

Hi @GregKalapos,

Thanks for quick response!
After your messege here is what I did.
I set loglevel in agent, apm-server and elastic to debug. Unfortunately i did not find anything usefull in apm-server or in elasticseach log. I am not sure where can I find logs from agent. If they should appear in console while catching exception in CaptureTransaction() method then I did not see anything there.
In addition, whenever I start apm-server, elasticsearch get document in "apm-7.1.0-onboarding-2019.05.28" index. Moreover I checked in Kibana status of apm-server and it showed it is configured correctly.
I noticed one more thing. Whenever I terminate apm-server elasticsearch log the same warning as when I run my program

[2019-05-28T12:25:10,715][WARN ][o.e.h.AbstractHttpServerTransport] [PCName] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/0:0:0:0:0:0:0:1:9200, remoteAddress=/0:0:0:0:0:0:0:1:61004}
java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
    at sun.nio.ch.SocketDispatcher.read(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) ~[?:?]
    at sun.nio.ch.IOUtil.read(Unknown Source) ~[?:?]
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[?:?]
    at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:347) ~[netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_211]

Do you have any other idea what can I check?

Hi @Futerkowiec,

Let's try to narrow down which leg of communication has issues: agent <-> apm-server or apm-server <-> elasticsearch. Could you please try out the steps listed at Common problems | APM Server Reference [7.15] | Elastic
especially the part

To see if the agent can connect to the APM Server, send requests to the instrumented service and look for lines containing [request] in the APM Server logs.

Please run your test app instrumented by the agent and see if you have lines containing [request] in the APM Server logs.

Another question regarding the application you have instrumented by the agent. Is the excerpt containing Agent.Tracer.CaptureTransaction the whole program? Namely does this program terminate right after the call to Agent.Tracer.CaptureTransaction returns? If so please add a short sleep (let's say 10 seconds) after Agent.Tracer.CaptureTransaction returns but before the end of the program and see if running a modified program causes lines containing [request] to appear in the APM Server logs.

Agent.Tracer.CaptureTransaction("test_name", "test_type", () =>
{
	// ...
});

Thread.Sleep(1000);

Since the agent sends data to APM Server asynchronously it's possible that the program terminates before agent has a chance to send anything to APM Server.

Hi @Sergey_Kleyman ,
Thank you for your advice!
Indeed that was the problem. After I added sleep after CaptureTransaction(), everything worked.

Thank you so much for your help!

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.