# Not able to get file logs from otel collector to elasticsearch using APM server

**URL:** <https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214>\
**Category:** Kibana\
**Created:** [November 29, 2023, 10:18am UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214 "2023-11-29T10:18:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Akshay\_Ranka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ranka/32/121946_2.png) [@Akshay\_Ranka](https://discuss.elastic.co/u/Akshay_Ranka)\
**Post date:** [November 29, 2023, 10:18am UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214/1 "2023-11-29T10:18:23Z")

</div>

extensions:  
health\_check:  
pprof:  
endpoint: 0.0.0.0:1777  
zpages:  
endpoint: 0.0.0.0:55679

receivers:  
filelog:  
include: [/path/to log/.log]  
operators:  
- type: regex\_parser  
regex: '^(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P[A-Z]_) (?P._)$'  
timestamp:  
parse\_from: attributes.time  
layout: '%Y-%m-%d %H:%M:%S'  
severity:  
parse\_from: attributes.sev

processors:  
batch:

exporters:  
logging:  
loglevel: debug

file:  
path: /home/oteldest.log

otlp/elastic: # Elastic APM server https endpoint without the "https://" prefix  
endpoint: \*\*\* :8200  
headers:  
# Elastic APM Server secret token  
Authorization: "ApiKey \*\*\*"

elasticsearch/log:  
endpoints: [http://\*\*\*]  
logs\_index: mylogs  
user: elastic  
password: "changeme"  
sending\_queue:  
enabled: true  
num\_consumers: 20  
queue\_size: 1000

service:

pipelines:

```
logs:
 receivers: [filelog]
 processors: [batch]
 exporters: [file, otlp/elastic]

```

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [November 29, 2023, 5:10pm UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214/2 "2023-11-29T17:10:28Z")

</div>

Hi @Akshay_Ranka,

It's difficult to read your config without the indentation. Can you reformat as `code` using the code option?

Do you have any errors in your OTel collector logs? Or indeed in your APM server? The below troubleshooting resources might help:

1. OTel collector: [https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/troubleshooting.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/troubleshooting.md)
2. APM Server: [Common problems | APM Server Reference [7.15] | Elastic](https://www.elastic.co/guide/en/apm/server/current/common-problems.html#no-data-indexed)

---

<div class="post-metadata">

**Author:** ![Akshay\_Ranka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ranka/32/121946_2.png) [@Akshay\_Ranka](https://discuss.elastic.co/u/Akshay_Ranka)\
**Post date:** [December 4, 2023, 6:06pm UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214/3 "2023-12-04T18:06:16Z")

</div>

````auto
type or paste code here
```extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679
 
receivers:
  filelog:
    include: [/home/ec2-user/yashaswi/logs/appltn.log]
 
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'

        severity:
          parse_from: attributes.sev
 
  filelog/two:
    include: [/home/ec2-user/yashaswi/logs/test.log]
 
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]*) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'

        severity:
          parse_from: attributes.sev
 
 
processors:
  batch:
 
 
exporters:
  logging:
    loglevel: debug
 
  file:
    path: /home/ec2-user/yashaswi/logs/oteldest.log
 
  otlp/elastic: # Elastic APM server https endpoint without the "https://" prefix
    endpoint: ****** :8200
    headers:
      # Elastic APM Server secret token
      Authorization: "ApiKey ******"
 
  elasticsearch/log:
    endpoints: [http:// ***** :9200]
    logs_index: logsmylogs
    user: elastic
    password: " *******"
    sending_queue:
      enabled: true
      num_consumers: 20
      queue_size: 1000
 
service:
 
  pipelines:
 
    logs:
     receivers: [filelog]
     processors: [batch]
     exporters: [file, otlp/elastic]
 
    logs/two:
     receivers: [filelog/two]
     processors: [batch]
     exporters: [file, otlp/elastic]

I am not able to get the logs to elasticsearch through APM with this otel configuration, maybe I am approacing a different way. Please help me. 
Thanks,
Akshay
````

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [December 5, 2023, 9:48am UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214/4 "2023-12-05T09:48:05Z")

</div>

Thanks for sharing. Do you have any errors in your collector logs?

---

<div class="post-metadata">

**Author:** ![Akshay\_Ranka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_ranka/32/121946_2.png) [@Akshay\_Ranka](https://discuss.elastic.co/u/Akshay_Ranka)\
**Post date:** [December 7, 2023, 9:06am UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214/5 "2023-12-07T09:06:41Z")

</div>

No Not facing the issue, actually how to bring service.name , host.name in the logs

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [December 7, 2023, 11:56am UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214/6 "2023-12-07T11:56:08Z")

</div>

To confirm, you are receiving logs but the issue is that the service.name and host.name fields are not populated? If the fields don't exist on your log entries, have you tried mapping these fields using a processor in your OTel exporter?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2024, 11:57am UTC](https://discuss.elastic.co/t/not-able-to-get-file-logs-from-otel-collector-to-elasticsearch-using-apm-server/348214/7 "2024-01-04T11:57:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.