Not able to get file logs from otel collector to elasticsearch using APM server

Elastic Stack Kibana
1

extensions:
health_check:
pprof:
endpoint: 0.0.0.0:1777
zpages:
endpoint: 0.0.0.0:55679

receivers:
filelog:
include: [/path/to log/.log]
operators:
- type: regex_parser
regex: '^(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P[A-Z]) (?P.)$'
timestamp:
parse_from: attributes.time
layout: '%Y-%m-%d %H:%M:%S'
severity:
parse_from: attributes.sev

processors:
batch:

exporters:
logging:
loglevel: debug

file:
path: /home/oteldest.log

otlp/elastic: # Elastic APM server https endpoint without the "https://" prefix
endpoint: *** :8200
headers:
# Elastic APM Server secret token
Authorization: "ApiKey ***"

elasticsearch/log:
endpoints: [http://***]
logs_index: mylogs
user: elastic
password: "changeme"
sending_queue:
enabled: true
num_consumers: 20
queue_size: 1000

service:

pipelines:

logs:
 receivers: [filelog]
 processors: [batch]
 exporters: [file, otlp/elastic]
2

Hi @Akshay_Ranka,

It's difficult to read your config without the indentation. Can you reformat as code using the code option?

Do you have any errors in your OTel collector logs? Or indeed in your APM server? The below troubleshooting resources might help:

  1. OTel collector: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/troubleshooting.md
  2. APM Server: Common problems | APM Server Reference [7.15] | Elastic
3 
type or paste code here
```extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679
 
receivers:
  filelog:
    include: [/home/ec2-user/yashaswi/logs/appltn.log]
 
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'

        severity:
          parse_from: attributes.sev
 
  filelog/two:
    include: [/home/ec2-user/yashaswi/logs/test.log]
 
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'

        severity:
          parse_from: attributes.sev
 
 
processors:
  batch:
 
 
exporters:
  logging:
    loglevel: debug
 
  file:
    path: /home/ec2-user/yashaswi/logs/oteldest.log
 
  otlp/elastic:    # Elastic APM server https endpoint without the "https://" prefix
    endpoint: ******:8200
    headers:
      # Elastic APM Server secret token
      Authorization: "ApiKey  ******"
 
  elasticsearch/log:
    endpoints: [http://*****:9200]
    logs_index: logsmylogs
    user: elastic
    password: "*******"
    sending_queue:
      enabled: true
      num_consumers: 20
      queue_size: 1000
 
service:
 
  pipelines:
 
    logs:
     receivers: [filelog]
     processors: [batch]
     exporters: [file, otlp/elastic]
 
    logs/two:
     receivers: [filelog/two]
     processors: [batch]
     exporters: [file, otlp/elastic]

I am not able to get the logs to elasticsearch through APM with this otel configuration, maybe I am approacing a different way. Please help me. 
Thanks,
Akshay
4

Thanks for sharing. Do you have any errors in your collector logs?

5

No Not facing the issue, actually how to bring service.name , host.name in the logs

6

To confirm, you are receiving logs but the issue is that the service.name and host.name fields are not populated? If the fields don't exist on your log entries, have you tried mapping these fields using a processor in your OTel exporter?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.