Multiline stack traces (.net) with whitespace

[Raman_Sawhney]

I have issues with Multiline stack traces (.net) with whitespace. logs are getting split and we are not able to troubleshoot the things its more time consuming.

I tried the fix from ES-stack website

Manage multiline messages | Filebeat Reference [7.9] | Elastic

but after applying the changes we are still getting the split logs with whitespace.

#############################################
# Filebeat Helm Chart Values
image:
  tag: 7.6.0

overrideConfig:
  filebeat.config:
    modules:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: true

  filebeat.autodiscover:
    providers:
    - type: kubernetes
      hints.enabled: true
      templates:
      - condition:
          equals:
            kubernetes.namespace: "*"
        config:
          - type: container
            paths:
            - /var/log/containers/*-${data.kubernetes.container.id}.log
            multiline.type: pattern
            multiline.pattern: '^[[:space:]]'
            multiline.negate: false
            multiline.match: after
  processors:
  - add_kubernetes_metadata:
      in_cluster: true
  - drop_event:
      when:
        or:
          - contains:
              kubernetes.namespace: "kube-system"
          - contains:
              kubernetes.namespace: "default"        
  - add_cloud_metadata:
  - add_fields:
      target: kubernetes
      fields:
        cluster_name: dev-{{env}}
  output:
    elasticsearch:
      hosts: ["{{es_host}}"]
      bulk_max_size: 20
      protocol: https
      enabled: true
      username: elastic
      password: {{password}}
      index: "dev-{{env}}-%{[kubernetes.pod.labels.service-name]:filebeat}-%{+xxxx.ww}"
    file:
      enabled: false

Please suggest what i can do to get the multiline logs combined.

[kvch (Noémi Ványi)]

Could you please share some example input logs and the desired output?

[Raman_Sawhney]

Sorry for late reply

[inf]:Serilog.AspNetCore.RequestLoggingMiddleware:[0HM30SF3H6IAF:00000001]:ssgdft-1887-483e-ba2c-2ee2fdfd64bb HTTP GET /v1/tenants/2000056/apps responded 200 in 1.6669 ms
[inf]:ABC.ServiceManagement.Common.Apps.ExceptionHandling.ExceptionHandler:[0HM30SF3H6IAJ:00000001]:052f2644-66ed-45fc-94bc-6da5bada0ef8 {"message":"Quota type 'maxAccumulatedDay' cannot be found for tenant 99"}
	ABC.ServiceManagement.Common.Apps.Exceptions.NotFoundException: Quota type 'maxAccumulatedDay' cannot be found for tenant 99
 ---> ABC.EntitlementService.Exceptions.CapacityTypeNotFoundException: Quota type 'maxAccumulatedDay' cannot be found for tenant 99
   at ABC.EntitlementService.Handlers.QuotaEntitlementHandler.GetTotalQuotaValueAsync(String tenantId, String capacityType, String productCode)
   at ABC.EntitlementService.Controllers.v1.QuotaEntitlementController.GetTotalQuotaValueAsync(String tenantId, String capacityType, String productCode) in /opt/source/service-management/ABC.EntitlementService/src/Controllers/v1/QuotaEntitlementController.cs:line 68
   --- End of inner exception stack trace ---
   at ABC.EntitlementService.Controllers.v1.QuotaEntitlementController.GetTotalQuotaValueAsync(String tenantId, String capacityType, String productCode) in /opt/source/service-management/ABC.EntitlementService/src/Controllers/v1/QuotaEntitlementController.cs:line 68
   at lambda_method(Closure , Object )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

[Raman_Sawhney]

I guess not only whitespace would help here i need to combine it with below things:

• whitespace
• keyword "at"
• keyword/special-character "---"

Probably this would be on the basis of pattren i have observed above.