Not Able to See logs on kibana

Kibana version:
Latest
Elasticsearch version:
Latest
APM Server version:
Latest
APM Agent language and version:
English
Browser version:
Latest
Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
[2022-03-25 11:02:50.630 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} Service name provided in configuration is WebService_RealTimeHub_ServicesAccessV2
[2022-03-25 11:02:50.658 +00:00][Info] - {AbstractConfigurationReader (FullFrameworkConfigReader)} The agent was started without a service version. The service version will be automatically discovered.
[2022-03-25 11:02:50.659 +00:00][Warning] - {AbstractConfigurationReader (FullFrameworkConfigReader)} Failed to discover service version, the service version will be omitted.
[2022-03-25 11:02:50.686 +00:00][Debug] - {SystemInfoHelper} No /proc/self/cgroup found - the agent will not report container id
[2022-03-25 11:02:50.839 +00:00][Debug] - {BackendCommUtils} Building HTTP client with BaseAddress: apmserver.com:8200/ for BackendCommComponentBase (PayloadSenderV2)...
[2022-03-25 11:02:50.928 +00:00][Debug] - {ExceptionUtils.DoSwallowingExceptions} ElasticApmPayloadSenderV2' (managed ID: 18) thread entry method entered [2022-03-25 11:02:50.956 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} GlobalLabels configuration option doesn't have a valid value - using default (empty map) [2022-03-25 11:02:50.960 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} MaxQueueEventCount configuration option doesn't have a valid value - using default: 1000 [2022-03-25 11:02:50.962 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} MaxBatchEventCount configuration option doesn't have a valid value - using default: 10 [2022-03-25 11:02:50.970 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} FlushInterval configuration option doesn't have a valid value - using default: 00:00:10 [2022-03-25 11:02:50.974 +00:00][Debug] - {PayloadSenderV2} Using the following configuration options: Events intake API absolute URL: apmserver.com:8200/intake/v2/events, FlushInterval: 10s, MaxBatchEventCount: 10, MaxQueueEventCount: 1000 [2022-03-25 11:02:51.005 +00:00][Debug] - {BackendCommComponentBase (PayloadSenderV2)} Posted Work loop to internal task scheduler [2022-03-25 11:02:51.023 +00:00][Debug] - {BackendCommComponentBase (PayloadSenderV2)} Waiting for work loop started event... [2022-03-25 11:02:51.027 +00:00][Debug] - {BackendCommComponentBase (PayloadSenderV2)} Signaling work loop started event... [2022-03-25 11:02:51.028 +00:00][Debug] - {BackendCommComponentBase (PayloadSenderV2)} Work loop started signaled [2022-03-25 11:02:51.032 +00:00][Debug] - {ExceptionUtils.DoSwallowingExceptions} BackendCommComponentBase.WorkLoop entered. Current thread: ElasticApmPayloadSenderV2' (managed ID: 18)
[2022-03-25 11:02:51.037 +00:00][Debug] - {BackendCommComponentBase (CentralConfigFetcher)} Disabled - exiting without initializing any members used by work loop
[2022-03-25 11:02:51.039 +00:00][Info] - {CentralConfigFetcher} Central configuration feature is DISABLED because CentralConfig option's value is False (default value is True)
[2022-03-25 11:02:53.288 +00:00][Warning] - {GcMetricsProvider} TraceEventSession initialization failed - GC metrics won't be collected+-> Exception: System.Runtime.InteropServices.COMException: Insufficient system resources exist to complete the requested service. (Exception from HRESULT: 0x800705AA)
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at Microsoft.Diagnostics.Tracing.Session.TraceEventSession.InsureStarted(EVENT_TRACE_PROPERTIES* properties)
at Microsoft.Diagnostics.Tracing.Session.TraceEventSession.get_Source()
at Elastic.Apm.Metrics.MetricsProvider.GcMetricsProvider..ctor(IApmLogger logger, IReadOnlyList`1 disabledMetrics)

[2022-03-25 11:02:53.293 +00:00][Debug] - {CgroupMetricsProvider} /proc/self/cgroup does not exist. Cgroup metrics will not be reported
[2022-03-25 11:02:53.295 +00:00][Info] - {MetricsCollector} Collecting metrics in 30000 milliseconds interval
[2022-03-25 11:02:53.311 +00:00][Debug] - {ElasticApmModule.#1} Found ASP.NET version: 4.8.4465.0
[2022-03-25 11:02:53.319 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} CaptureHeaders configuration option doesn't have a valid value - using default: True
[2022-03-25 11:02:53.324 +00:00][Debug] - {HttpDiagnosticListener} Current runtime is detected as Full Framework. RuntimeInformation.FrameworkDescription: .NET Framework 4.8.4470.0
[2022-03-25 11:02:53.354 +00:00][Debug] - {DiagnosticInitializer} Subscribed Elastic.Apm.DiagnosticListeners.HttpDiagnosticListenerFullFrameworkImpl to `System.Net.Http.Desktop' events source
[2022-03-25 11:02:53.355 +00:00][Debug] - {ElasticApmModule.#1} Initialized Agent singleton. .NET runtime: .NET Framework 4.8.4470.0; IIS: 10.0
[2022-03-25 11:02:53.460 +00:00][Debug] - {HttpDiagnosticListenerFullFrameworkImpl} No current transaction, skip creating span for outgoing HTTP request
[2022-03-25 11:02:53.517 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing started - starting trace...
[2022-03-25 11:02:53.518 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing started - starting trace...
[2022-03-25 11:02:53.563 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have elastic-apm-traceparent header - it means request doesn't have incoming distributed tracing data
[2022-03-25 11:02:53.563 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have elastic-apm-traceparent header - it means request doesn't have incoming distributed tracing data
[2022-03-25 11:02:53.591 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have valid incoming distributed tracing data - starting trace with new trace ID
[2022-03-25 11:02:53.621 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have valid incoming distributed tracing data - starting trace with new trace ID
[2022-03-25 11:02:53.655 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} No transaction sample rate provided. Defaulting to '1'
[2022-03-25 11:02:53.657 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} No transaction sample rate provided. Defaulting to '1'
[2022-03-25 11:02:53.672 +00:00][Debug] - {Tracer} Starting Transaction{Id: 30ed08bacc031592, TraceId: 406fec4d8d9690b22440e61d01c9c8a6, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Unknown, IsSampled: True}
[2022-03-25 11:02:53.673 +00:00][Debug] - {Tracer} Starting Transaction{Id: 05ccb8d135e98c41, TraceId: 1b6c1be78252b2483d52edb8a0c67b87, ParentId: null, Name: GET /, Type: request, Outcome: Unknown, IsSampled: True}
[2022-03-25 11:02:53.802 +00:00][Warning] - {PayloadSenderV2} Failed reading APM server info+-> Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elastic.Apm.ServerInfo.ApmServerInfoProvider.d__0.MoveNext()

[2022-03-25 11:02:53.919 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing started - starting trace...
[2022-03-25 11:02:53.921 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have elastic-apm-traceparent header - it means request doesn't have incoming distributed tracing data
[2022-03-25 11:02:53.922 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have valid incoming distributed tracing data - starting trace with new trace ID
[2022-03-25 11:02:53.922 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} No transaction sample rate provided. Defaulting to '1'
[2022-03-25 11:02:53.923 +00:00][Debug] - {Tracer} Starting Transaction{Id: db151a6c030d8d5d, TraceId: f2e4a6506e27181dc7c30e98214951e8, ParentId: null, Name: GET /, Type: request, Outcome: Unknown, IsSampled: True}
[2022-03-25 11:02:53.927 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing finished - ending trace...
[2022-03-25 11:02:53.953 +00:00][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 1. MaxQueueEventCount: 1000. Transaction: Transaction{Id: db151a6c030d8d5d, TraceId: f2e4a6506e27181dc7c30e98214951e8, ParentId: null, Name: GET /, Type: request, Outcome: Success, IsSampled: True}.
[2022-03-25 11:02:53.976 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing finished - ending trace...
[2022-03-25 11:02:53.978 +00:00][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 2. MaxQueueEventCount: 1000. Transaction: Transaction{Id: 05ccb8d135e98c41, TraceId: 1b6c1be78252b2483d52edb8a0c67b87, ParentId: null, Name: GET /, Type: request, Outcome: Success, IsSampled: True}.
[2022-03-25 11:02:54.472 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing finished - ending trace...
[2022-03-25 11:02:54.473 +00:00][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 3. MaxQueueEventCount: 1000. Transaction: Transaction{Id: 30ed08bacc031592, TraceId: 406fec4d8d9690b22440e61d01c9c8a6, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Success, IsSampled: True}.
[2022-03-25 11:03:04.075 +00:00][Warning] - {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (apmserver.com:8200/):
Transaction{Id: db151a6c030d8d5d, TraceId: f2e4a6506e27181dc7c30e98214951e8, ParentId: null, Name: GET /, Type: request, Outcome: Success, IsSampled: True},
Transaction{Id: 05ccb8d135e98c41, TraceId: 1b6c1be78252b2483d52edb8a0c67b87, ParentId: null, Name: GET /, Type: request, Outcome: Success, IsSampled: True},
Transaction{Id: 30ed08bacc031592, TraceId: 406fec4d8d9690b22440e61d01c9c8a6, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Success, IsSampled: True}+-> Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elastic.Apm.Report.PayloadSenderV2.d__32.MoveNext()

[2022-03-25 11:03:08.413 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing started - starting trace...
[2022-03-25 11:03:08.415 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have elastic-apm-traceparent header - it means request doesn't have incoming distributed tracing data
[2022-03-25 11:03:08.416 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have valid incoming distributed tracing data - starting trace with new trace ID
[2022-03-25 11:03:08.418 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} No transaction sample rate provided. Defaulting to '1'
[2022-03-25 11:03:08.419 +00:00][Debug] - {Tracer} Starting Transaction{Id: 5fdc39804feaa707, TraceId: 9161b74e08059b2b492e0dd5a2fb6b24, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Unknown, IsSampled: True}
[2022-03-25 11:03:08.421 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing finished - ending trace...
[2022-03-25 11:03:08.422 +00:00][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 1. MaxQueueEventCount: 1000. Transaction: Transaction{Id: 5fdc39804feaa707, TraceId: 9161b74e08059b2b492e0dd5a2fb6b24, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Success, IsSampled: True}.
[2022-03-25 11:03:08.641 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing started - starting trace...
[2022-03-25 11:03:08.644 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have elastic-apm-traceparent header - it means request doesn't have incoming distributed tracing data
[2022-03-25 11:03:08.646 +00:00][Debug] - {ElasticApmModule.#1} Incoming request doesn't have valid incoming distributed tracing data - starting trace with new trace ID
[2022-03-25 11:03:08.648 +00:00][Debug] - {AbstractConfigurationReader (FullFrameworkConfigReader)} No transaction sample rate provided. Defaulting to '1'
[2022-03-25 11:03:08.649 +00:00][Debug] - {Tracer} Starting Transaction{Id: f95023d98a119c18, TraceId: 6a9dcd252850d0a0fe677e2e7cc506b8, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Unknown, IsSampled: True}
[2022-03-25 11:03:08.649 +00:00][Debug] - {ElasticApmModule.#1} Incoming request processing finished - ending trace...
[2022-03-25 11:03:08.650 +00:00][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 2. MaxQueueEventCount: 1000. Transaction: Transaction{Id: f95023d98a119c18, TraceId: 6a9dcd252850d0a0fe677e2e7cc506b8, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Success, IsSampled: True}.
[2022-03-25 11:03:14.148 +00:00][Warning] - {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (apmserver.com:8200/):
Transaction{Id: 5fdc39804feaa707, TraceId: 9161b74e08059b2b492e0dd5a2fb6b24, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Success, IsSampled: True},
Transaction{Id: f95023d98a119c18, TraceId: 6a9dcd252850d0a0fe677e2e7cc506b8, ParentId: null, Name: GET /ServicesAccess.asmx/CheckService, Type: request, Outcome: Success, IsSampled: True}+-> Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elastic.Apm.Report.PayloadSenderV2.d__32.MoveNext()

@wajihrehman hey! do you mind giving us a little more info about what you would expect and what's not working? there's a bunch of questions in the template that went unanswered. the more information the better we can help you.

@dgieselaar i am expecting to see my logs in kibana dashboard. The problem is that same configuration working to all my projects but just for soap base webservices asmx. Trying to using the same configuration not working any soap base webservices project and getting error. Could you please help me why any thing missing in soap base services or add extra something. Also do you want any thing else.

webconfig

Error

[2022-03-25 10:18:09.696 +00:00][Warning] - {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (https://apmserver.com:8200/):
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet,
Elastic.Apm.Metrics.MetricSet+-> Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Elastic.Apm.Report.PayloadSenderV2.d__32.MoveNext()

@GregKalapos

Hey @wajihrehman,

what the logs say is that the .NET APM Agent was unable to send data to the APM Server:

[2022-03-25 11:03:14.148 +00:00][Warning] - {PayloadSenderV2} Failed sending events. Following events were not transferred successfully to the server (apmserver.com:8200/)

The configured serverurl seems to be http://apmserver.com:8200/ - that's strange. Could you maybe double check what server url is configured?

@GregKalapos yes i have been checked same url working in another projects. but just for this soap base webservices project not working. Tried and search alot but nothing found till now. Actually why it's failed to sending events. Please check it's from another project dashboard agent everything fine.

Unfortunately I'm don't really see what's the root cause is, but to me this looks like a connection issue. HTTP requests can't be sent from the agent to the server.

Is it maybe API key? Or HTTPS? Or something similar? I'd first check the connection - you can send an HTTP GET ot the exact same URL http://apmserver.com:8200/ - it'll respond if it's available.

Maybe another idea:
Could you maybe try to deploy a small dummy .NET application into the very same environment (same IIS, etc.)? The only thing this app should do is to send an HTTP GET to the server url and print the result. If that does not get HTTP200, then it's something with the connection - that'll show you the exact same exception which we already see in the logs.

@GregKalapos i have been hit directly my apm server url. can not give orginal url there. But see that is the response. when hitting directly on browser.

{
"build_date": "2020-11-09T18:38:48Z",
"build_sha": "d5f4e01307540517d964edfe4af9d31227e0ce91",
"version": "7.10.0"
}

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.