DotNetAgent: No data has been received from agents yet

Kibana version: 7.8.0

Elasticsearch version: 7.8.0

APM Server version: 7.8.0

APM Agent language and version: .NET (1.6.0)

Browser version: Version 78.0.3904.97 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Search Website

Fresh install or upgraded from other version? Fresh Install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

All Default

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I installed ES, Kibana and APM Server on my local machine to set up APM. I've created a small Console App to test the Agent(.NET). Now when I go to Set Up Instructions on Kibana its able to check the status of APM server but for Agent status it says: No data has been received from agents yet

Provide logs and/or server output (if relevant):

APM Server Logs:
---------------------------
Further, in APM Server logs, I could see its able to receive some events from Agents

2020-07-14T09:21:59.964+0530    DEBUG   [agentcfg]      agentcfg/cache.go:58    Cache size 1. Added ID MyConsoleApp.
2020-07-14T09:21:59.972+0530    INFO    [request]       middleware/log_middleware.go:97 request ok  {"request_id": "285e472e-
d17f-41e3-bddc-2a20a81ed306", "method": "GET", "URL": "/config/v1/agents?service.name=MyConsoleApp", "content_length": 0, "re
mote_address": "127.0.0.1", "user-agent": "elasticapm-dotnet/1.6.0 System.Net.Http/4.7.3062.0 .NET_Framework/4.7.3620.0", "re
sponse_code": 200}

http://localhost:8200/config/v1/agents?service.name=MyConsoleApp, this doesnot return anything and also if you see the Content_Length is 0 in APM Server logs.

Kibana Logs:
-----------------
{"type":"log","@timestamp":"2020-07-14T03:51:59Z","tags":["info","plugins","apm"],"pid":32128,"message":"Config was not found for MyConsoleApp/undefined"}
{"type":"response","@timestamp":"2020-07-14T03:51:59Z","tags":["access:apm"],"pid":32128,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","content-length":"45","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":404,"responseTime":118,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 118ms - 9.0B"}
{"type":"response","@timestamp":"2020-07-14T03:53:11Z","tags":[],"pid":32128,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"623","origin":"http://localhost:5601","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"http://localhost:5601/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":41,"contentLength":9},"message":"POST /api/ui_metric/report 200 41ms - 9.0B"}
{"type":"response","@timestamp":"2020-07-14T03:53:22Z","tags":[],"pid":32128,"method":"post","statusCode":200,"req":{"url":"/api/ui_metric/report","method":"post","headers":{"host":"localhost:5601","connection":"keep-alive","content-length":"116","origin":"http://localhost:5601","kbn-version":"7.8.0","user-agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"http://localhost:5601/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":200,"responseTime":37,"contentLength":9},"message":"POST /api/ui_metric/report 200 37ms - 9.0B"}
{"type":"log","@timestamp":"2020-07-14T03:53:31Z","tags":["error","plugins","taskManager","taskManager"],"pid":32128,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}

Hi @ImdotnetJunkie,

This answer might be relevant.

1 Like

Hey @Sergey_Kleyman, no luck with that. Please find logs from Agent and Server:

Agents Logs:

2020-07-15 17:19:12.4511|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigFetcher} Making HTTP request to APM Server... Request: Method: GET, RequestUri: 'http://127.0.0.1:8200/config/v1/agents?service.name=FlowTrader', Version: 1.1, Content: <null>, Headers:
{
  If-None-Match: "-"
}.
2020-07-15 17:19:12.4511|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Finished executing task. Task: {ID: 6989, Status: RanToCompletion, AsyncState: System.Action}
2020-07-15 17:19:12.5901|ERROR|3||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Trying to execute task inline... Task: {ID: 7017, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:12.5901|ERROR|3||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Adding task... Task: {ID: 7017, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:12.5901|ERROR|3||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Added task. Task: {ID: 7017, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:12.5901|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Starting to execute task... Task: {ID: 7017, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:12.5901|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigFetcher} Reading HTTP response body... Response: StatusCode: 304, ReasonPhrase: 'Not Modified', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Access-Control-Expose-Headers: Etag
  X-Content-Type-Options: nosniff
  Connection: close
  Cache-Control: must-revalidate, max-age=30
  Date: Wed, 15 Jul 2020 11:49:12 GMT
  ETag: "-"
}.

2020-07-15 17:19:12.5901|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Trying to execute task inline... Task: {ID: 7036, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:12.5901|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigResponseParser} Processing HTTP response...
+-> Response:
    StatusCode: 304, ReasonPhrase: 'Not Modified', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
    {
      Access-Control-Expose-Headers: Etag
      X-Content-Type-Options: nosniff
      Connection: close
      Cache-Control: must-revalidate, max-age=30
      Date: Wed, 15 Jul 2020 11:49:12 GMT
      ETag: "-"
    }
+-> Response body [length: 0]:
    
2020-07-15 17:19:12.5901|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigResponseParser} HTTP status code is Not Modified (304) which means the configuration has not changed since the previous fetch. Response:
    StatusCode: 304, ReasonPhrase: 'Not Modified', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
    {
      Access-Control-Expose-Headers: Etag
      X-Content-Type-Options: nosniff
      Connection: close
      Cache-Control: must-revalidate, max-age=30
      Date: Wed, 15 Jul 2020 11:49:12 GMT
      ETag: "-"
    }
2020-07-15 17:19:12.5901|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigFetcher} Waiting 30s... Wait time is taken from max-age directive in Cache-Control header in APM Server's response. dbgIterationsCount: 3.
2020-07-15 17:19:12.5901|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Finished executing task. Task: {ID: 7017, Status: RanToCompletion, AsyncState: System.Action}
2020-07-15 17:19:12.7141|ERROR|1|GUI|APMLogger|{AbstractConfigurationReader (EnvironmentConfigurationReader)} No transaction sample rate provided. Defaulting to '1'
2020-07-15 17:19:12.7581|ERROR|1|GUI|APMLogger|{Transaction.e654433c672c4dd0} New Transaction instance created: Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True}. IsSampled (True) is based on the given sampler (Sampler{ constant: True }). Start time: 2020-07-15 11:49:12.7191950 UTC (as timestamp: 1594813752719195)
2020-07-15 17:19:12.7581|ERROR|1|GUI|APMLogger|{Tracer} Starting Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True}

2020-07-15 17:19:15.7364|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} There's data to be sent. Batch size: 1. newEventQueueCount: 0. First event: Elastic.Apm.Metrics.MetricSet.
2020-07-15 17:19:15.7364|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 7073, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:15.7364|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Serialized item to send: Elastic.Apm.Metrics.MetricSet as {"samples":{"system.process.cpu.total.norm.pct":{"value":0.0033793457637204651},"system.cpu.total.norm.pct":{"value":0.041686353683471677},"system.process.memory.size":{"value":1454882816.0},"system.process.memory.rss.bytes":{"value":497528832.0},"system.memory.actual.free":{"value":533032960.0},"system.memory.total":{"value":17099870208.0}},"timestamp":1594813752019125}

2020-07-15 17:19:15.7564|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Starting to execute task... Task: {ID: 7082, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:15.7564|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Sent items to server:
    Elastic.Apm.Metrics.MetricSet
2020-07-15 17:19:15.7564|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Waiting for data to send... FlushInterval: 5s
2020-07-15 17:19:15.7915|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Finished executing task. Task: {ID: 7082, Status: RanToCompletion, AsyncState: System.Action}

2020-07-15 17:19:35.0884|ERROR|1|GUI|APMLogger|{AbstractConfigurationReader (EnvironmentConfigurationReader)} No transaction max spans provided. Defaulting to '500'
2020-07-15 17:19:35.0884|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} New Span instance created: Span{Id: 8b0c615da540cf91, TransactionId: e654433c672c4dd0, ParentId: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, Name: Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: exec, IsSampled: True}. Start time: 2020-07-15 11:49:35.0864320 UTC (as timestamp: 1594813775086432). Parent span: (null)
2020-07-15 17:19:35.0884|ERROR|1|GUI|APMLogger|{Transaction.e654433c672c4dd0} Starting Span{Id: 8b0c615da540cf91, TransactionId: e654433c672c4dd0, ParentId: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, Name: Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: exec, IsSampled: True}
2020-07-15 17:19:35.0884|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} Ended Span{Id: 8b0c615da540cf91, TransactionId: e654433c672c4dd0, ParentId: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, Name: Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: exec, IsSampled: True}. Start time: 2020-07-15 11:49:35.0864320 UTC (as timestamp: 1594813775086432), End time: 2020-07-15 11:49:35.0984330 UTC (as timestamp: 1594813775098433), Duration: 12.001ms
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} transform stack frames
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} End, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} CaptureSpan, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} Send, 318
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} <Initialize>b__2, 88
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} CriticalExecuteCommandSource, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} OnClick, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} OnClick, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} OnMouseLeftButtonUp, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InvokeHandler, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InvokeHandler, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InvokeHandlersImpl, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} ReRaiseEventAs, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} OnMouseUpThunk, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InvokeHandler, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InvokeHandler, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InvokeHandlersImpl, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} RaiseEventImpl, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} RaiseTrustedEvent, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} ProcessStagingArea, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} ProcessInput, 0
2020-07-15 17:19:35.2004|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} ReportInput, 0
2020-07-15 17:19:35.2284|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} ReportInput, 0
2020-07-15 17:19:35.2284|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} FilterMessage, 0
2020-07-15 17:19:35.2284|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InputFilterMessage, 0
2020-07-15 17:19:35.2284|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} WndProc, 0
2020-07-15 17:19:35.2644|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} DispatcherCallbackOperation, 0
2020-07-15 17:19:35.2644|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} InternalRealCall, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} TryCatchWhen, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} LegacyInvokeImpl, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} SubclassWndProc, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} DispatchMessage, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} DispatchMessage, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} PushFrameImpl, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} RunDispatcher, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} RunInternal, 0
2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{Span.8b0c615da540cf91} Main, 50


Further Agents Logs:

2020-07-15 17:19:35.2774|ERROR|1|GUI|APMLogger|{PayloadSenderV2} Enqueued Span. newEventQueueCount: 1. MaxQueueEventCount: 1000. Span: Span{Id: 8b0c615da540cf91, TransactionId: e654433c672c4dd0, ParentId: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, Name: Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: exec, IsSampled: True}.

2020-07-15 17:19:35.7915|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} There's data to be sent. Batch size: 1. newEventQueueCount: 0. First event: Span{Id: 8b0c615da540cf91, TransactionId: e654433c672c4dd0, ParentId: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, Name: Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: exec, IsSampled: True}.
2020-07-15 17:19:35.7915|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 10317, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Serialized item to send: Span{Id: 8b0c615da540cf91, TransactionId: e654433c672c4dd0, ParentId: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, Name: Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: exec, IsSampled: True} as {"duration":12.001,"id":"8b0c615da540cf91","name":"Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0","parent_id":"e654433c672c4dd0","stacktrace":[{"filename":"Elastic.Apm.Model.Span","function":"End","lineno":0,"module":"Elastic.Apm, Version=1.5.1.0, Culture=neutral, PublicKeyToken=ae7400d2c189cf22"},{"filename":"Elastic.Apm.Model.ExecutionSegmentCommon","function":"CaptureSpan","lineno":0,"module":"Elastic.Apm, Version=1.5.1.0, Culture=neutral, PublicKeyToken=ae7400d2c189cf22"},{"abs_path":"C:\\dev\\CP\\FlowTrader\\flowtrader_190620\\flowtrader\\src\\Execution\\Execution.Gui.OrderTicket\\ViewControllers\\OrderExecutionViewController.cs","filename":"Rbs.Markets.FICC.FlowTrader.Execution.Gui.OrderTicket.ViewControllers.OrderExecutionViewController","function":"Send","lineno":318,"module":"Rbs.Markets.FICC.FlowTrader.Execution.Gui.OrderTicket, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"},{"abs_path":"C:\\dev\\CP\\FlowTrader\\flowtrader_190620\\flowtrader\\src\\Execution\\Execution.Gui.OrderTicket\\ViewControllers\\OrderExecutionViewController.cs","filename":"Rbs.Markets.FICC.FlowTrader.Execution.Gui.OrderTicket.ViewControllers.OrderExecutionViewController+<>c__DisplayClass20_0","function":"<Initialize>b__2","lineno":88,"module":"Rbs.Markets.FICC.FlowTrader.Execution.Gui.OrderTicket, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"},PublicKeyToken=31bf3856ad364e35"},{"abs_path":"C:\\dev\\CP\\FlowTrader\\flowtrader_190620\\flowtrader\\src\\FlowTrader\\obj\\Debug\\App.g.cs","filename":"FlowTrader.App","function":"Main","lineno":50,"module":"FlowTrader, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"}],"timestamp":1594813775086432,"trace_id":"160336c35a8d56428df7c7845c11d7f3","transaction_id":"e654433c672c4dd0","type":"exec"}2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Finished executing task. Task: {ID: 10312, Status: RanToCompletion, AsyncState: System.Action}
2020-07-15 17:19:35.8345|ERROR|42||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 10333, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:35.8345|ERROR|42||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Adding task... Task: {ID: 10333, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:35.8345|ERROR|42||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Added task. Task: {ID: 10333, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Starting to execute task... Task: {ID: 10333, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Sent items to server:
    Span{Id: 8b0c615da540cf91, TransactionId: e654433c672c4dd0, ParentId: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, Name: Send-4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: exec, IsSampled: True}
2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 10338, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 10340, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Waiting for data to send... FlushInterval: 5s
2020-07-15 17:19:35.8345|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Finished executing task. Task: {ID: 10333, Status: RanToCompletion, AsyncState: System.Action}

2020-07-15 17:19:46.6535|ERROR|12||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 10935, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:46.6535|ERROR|18||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Trying to execute task inline... Task: {ID: 10941, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:46.6535|ERROR|60||APMLogger|{MetricsCollector} CollectAllMetrics started
2020-07-15 17:19:46.6535|ERROR|1|GUI|APMLogger|{Transaction.e654433c672c4dd0} Ended Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True}. Start time: 2020-07-15 11:49:12.7191950 UTC (as timestamp: 1594813752719195), End time: 2020-07-15 11:49:46.6555890 UTC (as timestamp: 1594813786655589), Duration: 33936.394ms

2020-07-15 17:19:46.6605|ERROR|60||APMLogger|{MetricsCollector} Start collecting process total CPU time
2020-07-15 17:19:46.6605|ERROR|60||APMLogger|{MetricsCollector} Collected process total CPU time - adding it to MetricSet
2020-07-15 17:19:46.6605|ERROR|60||APMLogger|{MetricsCollector} Start collecting total system CPU time
2020-07-15 17:19:46.6605|ERROR|18||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Adding task... Task: {ID: 10941, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:46.6765|ERROR|60||APMLogger|{MetricsCollector} Collected total system CPU time - adding it to MetricSet
2020-07-15 17:19:46.6765|ERROR|18||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Added task. Task: {ID: 10941, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:46.6765|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Starting to execute task... Task: {ID: 10941, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:46.6765|ERROR|60||APMLogger|{MetricsCollector} Start collecting process's working set and virtual memory size
2020-07-15 17:19:46.6765|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Trying to execute task inline... Task: {ID: 10947, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:46.6765|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Trying to execute task inline... Task: {ID: 10949, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:46.6765|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigFetcher} Making HTTP request to APM Server... Request: Method: GET, RequestUri: 'http://127.0.0.1:8200/config/v1/agents?service.name=FlowTrader', Version: 1.1, Content: <null>, Headers:
{
  If-None-Match: "-"
}.
2020-07-15 17:19:46.7015|ERROR|60||APMLogger|{MetricsCollector} Collected process's working set and virtual memory size - adding it to MetricSet
2020-07-15 17:19:46.7015|ERROR|60||APMLogger|{MetricsCollector} Start collecting total and free memory
2020-07-15 17:19:46.7015|ERROR|60||APMLogger|{MetricsCollector} Collected total and free memory - adding it to MetricSet
2020-07-15 17:19:46.7015|ERROR|60||APMLogger|{MetricsCollector} Skipping GcMetricsProvider - IsMetricAlreadyCaptured indicated false
2020-07-15 17:19:46.7015|ERROR|60||APMLogger|{PayloadSenderV2} Enqueued MetricSet. newEventQueueCount: 1. MaxQueueEventCount: 1000. MetricSet: Elastic.Apm.Metrics.MetricSet.
2020-07-15 17:19:46.7015|ERROR|60||APMLogger|{MetricsCollector} Metrics collected: MetricSample{system.process.cpu.total.norm.pct: 0.00679091325218726}, MetricSample{system.cpu.total.norm.pct: 0.0499272871017456}, MetricSample{system.process.memory.size: 1507192832}, MetricSample{system.process.memory.rss.bytes: 511496192}, MetricSample{system.memory.actual.free: 491151360}, MetricSample{system.memory.total: 17099870208}
2020-07-15 17:19:46.7115|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmCentralConfigFetcher)} Finished executing task. Task: {ID: 10941, Status: RanToCompletion, AsyncState: System.Action}
2020-07-15 17:19:46.7115|ERROR|1|GUI|APMLogger|{PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 2. MaxQueueEventCount: 1000. Transaction: Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True}.
2020-07-15 17:19:46.7695|INFO|36|EventLoopScheduler2|AgoraTransport<IOrderStatus>|Subscribing to Agora domains: 'EMarketPrice' topic*: RATES_OMS_STATE_UK/^C4f6ce194-faa7-48cf-b6e9-2e39661ad7a0\/.*

2020-07-15 17:19:46.8396|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigFetcher} Reading HTTP response body... Response: StatusCode: 304, ReasonPhrase: 'Not Modified', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Access-Control-Expose-Headers: Etag
  X-Content-Type-Options: nosniff
  Cache-Control: must-revalidate, max-age=30
  Date: Wed, 15 Jul 2020 11:49:46 GMT
  ETag: "-"
}.


APM-Server Logs:

2020-07-15T18:00:25.172+0530	INFO	[esclientleg]	eslegclient/connection.go:306	Attempting to connect to Elasticsearch version 7.8.0
2020-07-15T18:00:25.187+0530	INFO	[request]	middleware/log_middleware.go:97	not modified	{"request_id": "75700ec2-cfd3-4117-afc7-cbdbfc4bcdc2", "method": "GET", "URL": "/config/v1/agents?service.name=FlowTrader", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "elasticapm-dotnet/1.5.1 System.Net.Http/4.7.3062.0 .NET_Framework/4.7.3620.0", "response_code": 304}

Kibana Logs:

{"type":"log","@timestamp":"2020-07-15T12:31:25Z","tags":["info","plugins","apm"],"pid":32128,"message":"Config was not found for FlowTrader/undefined"}
{"type":"response","@timestamp":"2020-07-15T12:31:25Z","tags":["access:apm"],"pid":32128,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"127.0.0.1:5601","user-agent":"Go-http-client/1.1","content-length":"44","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":404,"responseTime":106,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 106ms - 9.0B"}
{"type":"log","@timestamp":"2020-07-15T12:31:39Z","tags":["error","plugins","taskManager","taskManager"],"pid":32128,"message":"Failed to poll for work: Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2020-07-15T12:31:55Z","tags":["info","plugins","apm"],"pid":32128,"message":"Config was not found for FlowTrader/undefined"}
{"type":"response","@timestamp":"2020-07-15T12:31:55Z","tags":["access:apm"],"pid":32128,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"127.0.0.1:5601","user-agent":"Go-http-client/1.1","content-length":"44","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":404,"responseTime":143,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 143ms - 9.0B"}

Hi @ImdotnetJunkie,

I see a single transaction in the log, and it ended at 17:19:46.6535

2020-07-15 17:19:46.6535|ERROR|1|GUI|APMLogger|{Transaction.e654433c672c4dd0} Ended Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True}. Start time: 2020-07-15 11:49:12.7191950 UTC (as timestamp: 1594813752719195), End time: 2020-07-15 11:49:46.6555890 UTC (as timestamp: 1594813786655589), Duration: 33936.394ms

It was immediately enqueued at 17:19:46.7115

2020-07-15 17:19:46.7115|ERROR|1|GUI|APMLogger|{PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 2. MaxQueueEventCount: 1000. Transaction: Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True}.

The next line is not from the agent and the line after that is the last line from the log with timestamp 17:19:46.8396.

If this is the full agent log then this means the process terminates before the agent has the chance to send the data. I definitely don't see that the agent tried to send the data - I only see it records it, but then I don't see the part when it'd send it (there should be a log line with the given transaction with {PayloadSenderV2} Sent items to server:, but that's not in the logs).

Either let the process hold up for the agent to reach the flushinterval or if you want to just try the agent, put it into a web application, you won't have this problem there.

Further Agent Logs:

2020-07-15 17:19:46.8396|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigResponseParser} Processing HTTP response...
+-> Response:
    StatusCode: 304, ReasonPhrase: 'Not Modified', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
    {
      Access-Control-Expose-Headers: Etag
      X-Content-Type-Options: nosniff
      Cache-Control: must-revalidate, max-age=30
      Date: Wed, 15 Jul 2020 11:49:46 GMT
      ETag: "-"
    }
+-> Response body [length: 0]:
    
2020-07-15 17:19:46.8396|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigResponseParser} HTTP status code is Not Modified (304) which means the configuration has not changed since the previous fetch. Response:
    StatusCode: 304, ReasonPhrase: 'Not Modified', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
    {
      Access-Control-Expose-Headers: Etag
      X-Content-Type-Options: nosniff
      Cache-Control: must-revalidate, max-age=30
      Date: Wed, 15 Jul 2020 11:49:46 GMT
      ETag: "-"
    }
2020-07-15 17:19:46.8396|ERROR|44|ElasticApmCentralConfigFetcher|APMLogger|{CentralConfigFetcher} Waiting 30s... Wait time is taken from max-age directive in Cache-Control header in APM Server's response. dbgIterationsCount: 4.

2020-07-15 17:19:51.6700|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} There's data to be sent. Batch size: 2. newEventQueueCount: 0. First event: Elastic.Apm.Metrics.MetricSet.
2020-07-15 17:19:51.6700|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 11841, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:51.6700|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Serialized item to send: Elastic.Apm.Metrics.MetricSet as {"samples":{"system.process.cpu.total.norm.pct":{"value":0.0067909132521872606},"system.cpu.total.norm.pct":{"value":0.049927287101745606},"system.process.memory.size":{"value":1507192832.0},"system.process.memory.rss.bytes":{"value":511496192.0},"system.memory.actual.free":{"value":491151360.0},"system.memory.total":{"value":17099870208.0}},"timestamp":1594813786704594}
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Serialized item to send: Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True} as {"context":{},"duration":33936.394,"id":"e654433c672c4dd0","sampled":true,"name":"4f6ce194-faa7-48cf-b6e9-2e39661ad7a0","span_count":{"dropped":0,"started":1},"timestamp":1594813752719195,"trace_id":"160336c35a8d56428df7c7845c11d7f3","type":"request"}
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Finished executing task. Task: {ID: 11836, Status: RanToCompletion, AsyncState: System.Action}
2020-07-15 17:19:51.7020|ERROR|29||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 11854, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:51.7020|ERROR|29||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Adding task... Task: {ID: 11854, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:51.7020|ERROR|29||APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Added task. Task: {ID: 11854, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Starting to execute task... Task: {ID: 11854, Status: WaitingToRun, AsyncState: System.Action}
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Sent items to server:
    Elastic.Apm.Metrics.MetricSet,
    Transaction{Id: e654433c672c4dd0, TraceId: 160336c35a8d56428df7c7845c11d7f3, ParentId: null, Name: 4f6ce194-faa7-48cf-b6e9-2e39661ad7a0, Type: request, IsSampled: True}
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 11858, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Trying to execute task inline... Task: {ID: 11860, Status: WaitingToRun, AsyncState: System.Action}, taskWasPreviouslyQueued: False
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Waiting for data to send... FlushInterval: 5s
2020-07-15 17:19:51.7020|ERROR|43|ElasticApmPayloadSenderV2|APMLogger|{SingleThreadTaskScheduler (thread: ElasticApmPayloadSenderV2)} Finished executing task. Task: {ID: 11854, Status: RanToCompletion, AsyncState: System.Action}

Ok, with the new log, now it seems that the agent sends the transaction to the APM Server.

Just to make sure we don't overlook something: @ImdotnetJunkie you write you see "No data has been received from agents yet" under setup instruction - just to be sure: have you also checked the APM UI, too? Is really no data there?

If there isn't then the next step would be to look at APM Server logs on debug log level.

APU UI in Kibana right ?

Yes, that's where it should show up.

Ok, then all I can say is that based on logs the agent sends the transaction to the APM Server and no errors seems to be reported back to the agent.

Maybe server logs on debug level contain more info.

APM Server Logs:

2020-07-15T22:01:47.214+0530	INFO	[pipelines]	pipeline/register.go:56	Registered Ingest Pipelines successfully.
2020-07-15T22:01:47.214+0530	INFO	[publisher_pipeline_output]	pipeline/output.go:152	Connection to backoff(elasticsearch(http://localhost:9200)) established
2020-07-15T22:02:47.008+0530	INFO	[request]	middleware/log_middleware.go:97	request ok	{"request_id": "ba450c60-13eb-48ea-bd80-eb2ad01e1b4e", "method": "GET", "URL": "/config/v1/agents?service.name=FlowTrader", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "elasticapm-dotnet/1.5.1 System.Net.Http/4.7.3062.0 .NET_Framework/4.7.3620.0", "response_code": 200}
2020-07-15T22:03:17.207+0530	INFO	[request]	middleware/log_middleware.go:97	not modified	{"request_id": "4431473c-573f-4b78-919d-6ebee0c82d13", "method": "GET", "URL": "/config/v1/agents?service.name=FlowTrader", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "elasticapm-dotnet/1.5.1 System.Net.Http/4.7.3062.0 .NET_Framework/4.7.3620.0", "response_code": 304}
2020-07-15T22:03:19.113+0530	INFO	[request]	middleware/log_middleware.go:97	request accepted	{"request_id": "4fca4767-2087-4fdd-b935-49a980fc57ec", "method": "POST", "URL": "/intake/v2/events", "content_length": 616, "remote_address": "127.0.0.1", "user-agent": "elasticapm-dotnet/1.5.1 System.Net.Http/4.7.3062.0 .NET_Framework/4.7.3620.0", "response_code": 202}
2020-07-15T22:03:20.115+0530	INFO	[publisher]	pipeline/retry.go:221	retryer: send unwait signal to consumer
2020-07-15T22:03:20.115+0530	INFO	[publisher]	pipeline/retry.go:225	  done
2020-07-15T22:03:21.225+0530	ERROR	[publisher_pipeline_output]	pipeline/output.go:181	failed to publish events: temporary bulk send failure
2020-07-15T22:03:21.225+0530	INFO	[publisher_pipeline_output]	pipeline/output.go:144	Connecting to backoff(elasticsearch(http://localhost:9200))
2020-07-15T22:03:21.225+0530	INFO	[publisher]	pipeline/retry.go:221	retryer: send unwait signal to consumer
2020-07-15T22:03:21.225+0530	INFO	[publisher]	pipeline/retry.go:225	  done

Are you looking for anything specific log lines ?

We should see the incoming transaction from the .NET APM Agent - I'd look for a line with that.

I could see Agent is sending the data as follows:

Agent Log:

2020-07-16 08:48:41.8927|ERROR|42|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Sent items to server:
    Span{Id: 4cfcc92c06d69696, TransactionId: 5d5b474711fc55ab, ParentId: 5d5b474711fc55ab, TraceId: 771972a0f1fbba4789530152f07b2dd4, Name: Send-b66c99a1-010a-4fcf-98f4-d32ca82a521b, Type: exec, IsSampled: True}

But I don't see anything picked up by Server:

2020-07-16T08:48:41.192+0530	INFO	[publisher_pipeline_output]	pipeline/output.go:152	Connection to backoff(elasticsearch(http://localhost:9200)) established
2020-07-16T08:48:41.192+0530	INFO	[publisher]	pipeline/retry.go:215	retryer: send wait signal to consumer
2020-07-16T08:48:41.192+0530	INFO	[publisher]	pipeline/retry.go:219	  done
2020-07-16T08:48:41.893+0530	INFO	[request]	middleware/log_middleware.go:97	request accepted	{"request_id": "62dfc256-b053-40f4-af9a-ff6b3d6681f1", "method": "POST", "URL": "/intake/v2/events", "content_length": 7610, "remote_address": "127.0.0.1", "user-agent": "elasticapm-dotnet/1.5.1 System.Net.Http/4.7.3062.0 .NET_Framework/4.7.3620.0", "response_code": 202}
2020-07-16T08:48:42.378+0530	ERROR	[publisher_pipeline_output]	pipeline/output.go:181	failed to publish events: temporary bulk send failure
2020-07-16T08:48:42.378+0530	INFO	[publisher_pipeline_output]	pipeline/output.go:144	Connecting to backoff(elasticsearch(http://localhost:9200))
2020-07-16T08:48:42.378+0530	INFO	[publisher]	pipeline/retry.go:215	retryer: send wait signal to consumer
2020-07-16T08:48:42.378+0530	INFO	[publisher]	pipeline/retry.go:219	  done

What do you want me check next?

@GregKalapos could you please me with next steps ?

Any update on this please ?

@ImdotnetJunkie

Here is more info on how to get more logs from APM Server. I'd try:

apm-server -e -d "*"

That'll give you all logs.

Also another comment:

On the agent side we see this:

2020-07-16 08:48:41.8927|ERROR|42|ElasticApmPayloadSenderV2|APMLogger|{PayloadSenderV2} Sent items to server:
    Span{Id: 4cfcc92c06d69696, TransactionId: 5d5b474711fc55ab, ParentId: 5d5b474711fc55ab, TraceId: 771972a0f1fbba4789530152f07b2dd4, Name: Send-b66c99a1-010a-4fcf-98f4-d32ca82a521b, Type: exec, IsSampled: True}

I think that log is not an Error log - that log is printed on debug level. Maybe the way you log puts an error there, but form the agent's perspective this is a debug log without any error.

This has worked now. Thanks a lot for the help.