# FileBeat not scrapping log lines

**URL:** <https://discuss.elastic.co/t/filebeat-not-scrapping-log-lines/355943>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 21, 2024, 8:21pm UTC](https://discuss.elastic.co/t/filebeat-not-scrapping-log-lines/355943 "2024-03-21T20:21:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![yevsh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yevsh1/32/132887_2.png) [@yevsh1](https://discuss.elastic.co/u/yevsh1)\
**Post date:** [March 21, 2024, 8:21pm UTC](https://discuss.elastic.co/t/filebeat-not-scrapping-log-lines/355943/1 "2024-03-21T20:21:57Z")

</div>

Hello!

I am new to `FileBeat`, trying to scrap `INFO` log lines from log files and send to kafka.

```auto
filebeat.inputs:
- type: log
  enabled: true
  tags:
    - myservice
  paths:
    - /usr/share/services/myservice/*.log
  include_lines: ['^INFO']
#tried this also
  processors:
  - drop_event:
      when:
        not:
          contains:
            log.level: "INFO"

output.kafka:
  hosts: ["localhost:9092"]
  topic: 'applogs'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

```

this is my log file:

```auto
22:07:05.897 [main] INFO c.a.a.r.k.KafkaCopyApplication - No active profile set, falling back to 1 default profile: "default"
22:07:10.281 [main] INFO c.a.aia.rda.kafkacopy.KafkaConsumer - TEST

```

but nothing scrapped, though log file is picked up

```auto
filebeat | {"log.level":"info","@timestamp":"2024-03-21T20:07:03.678Z","log.logger":"input.harvester","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).Run","file.name":"log/harvester.go","file.line":311},"message":"Harvester started for paths: [/usr/share/services/myservice/*.log]","service.name":"filebeat","input_id":"00748f4b-c724-41bb-bc41-adbc87004920","source_file":"/usr/share/services/myservice/thelog1.log","state_id":"native::88544381-132","finished":false,"os_id":"88544381-132","harvester_id":"e0405f9e-7f84-4b08-ab81-959cb00c5312","ecs.version":"1.6.0"}
filebeat | {"log.level":"info","@timestamp":"2024-03-21T20:07:23.603Z","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":36818944}}}},"cpu":{"system":{"ticks":90,"time":{"ms":10}},"total":{"ticks":210,"time":{"ms":20},"value":210},"user":{"ticks":120,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"351093e2-644d-490f-8bbc-1f1fcdde75d7","uptime":{"ms":210084},"version":"8.12.2"},"memstats":{"gc_next":36900512,"memory_alloc":18923368,"memory_total":63575088,"rss":102920192},"runtime":{"goroutines":31}},"filebeat":{"events":{"active":0,"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":3,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.09,"15":0.09,"5":0.11,"norm":{"1":0.0225,"15":0.0225,"5":0.0275}}}},"ecs.version":"1.6.0"}}

```

please

if i enable debug i just see:

```auto

filebeat | {"log.level":"debug","@timestamp":"2024-03-21T20:49:13.041Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).harvestExistingFile","file.name":"log/input.go","file.line":627},"message":"Harvester for file is still running: /usr/share/services/myservice/thelog1.log","service.name":"filebeat","input_id":"3cc3e3c7-8018-4f44-875b-b8682998d36e","source_file":"/usr/share/services/myservice/thelog1.log","state_id":"native::88544381-132","finished":false,"os_id":"88544381-132","old_source":"/usr/share/services/myservice/thelog1.log","old_finished":false,"old_os_id":"88544381-132","ecs.version":"1.6.0"}

```

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [March 23, 2024, 1:06pm UTC](https://discuss.elastic.co/t/filebeat-not-scrapping-log-lines/355943/2 "2024-03-23T13:06:44Z")

</div>

> [@yevsh1](#):
>
> `include_lines: ['^INFO']`

This regex requires each line to start with the word INFO to be scraped but your log lines start with timestamps

Your filter will then drop any lines picked up because you haven't parsed out the log level field from your log line yet

---

<div class="post-metadata">

**Author:** ![yevsh1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yevsh1/32/132887_2.png) [@yevsh1](https://discuss.elastic.co/u/yevsh1)\
**Post date:** [March 23, 2024, 2:43pm UTC](https://discuss.elastic.co/t/filebeat-not-scrapping-log-lines/355943/3 "2024-03-23T14:43:22Z")

</div>

Thanks!  
I removed the filter and changed the

include\_lines: [**'INFO'**]

It picks the line now.

Now have other issue,

kafka runs as docker (3 replicas) as well.

with  
`hosts: ["localhost:9092"]`

it didn't connect

When I changed to  
` hosts: ["host.docker.internal:9092"]`

it connected. it created a topic 'applogs'

but failed sending the data:

```auto

23T14:37:22.555Z","log.logger":"kafka","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/kafka.(*msgRef).dec","file.name":"kafka/client.go","file.line":406},"message":"Kafka publish failed with: dial tcp 127.0.0.1:9093: connect: connection refused","service.name":"filebeat","ecs.version":"1.6.0"}

```

it tries to send to port 9093

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 23, 2024, 4:18pm UTC](https://discuss.elastic.co/t/filebeat-not-scrapping-log-lines/355943/4 "2024-03-23T16:18:03Z")

</div>

> [@yevsh1](#):
>
> `hosts: ["localhost:9092"]`
> 
> it didn't connect

Using `localhost` **always** means that you are trying to connect to a server on the **same** host, so if your filebeat is not running on the same host/container then your kafka, then this is expected to not work.

> [@yevsh1](#):
>
> `23T14:37:22.555Z","log.logger":"kafka","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/kafka.(*msgRef).dec","file.name":"kafka/client.go","file.line":406},"message":"Kafka publish failed with: dial tcp 127.0.0.1:9093: connect: connection refused","service.name":"filebeat","ecs.version":"1.6.0"}`

I think this is related to your Kafka Cluster configuration, the clients use the `advertised.listeners` settting to connect, so you need to check what are the configured advertised listeners, it cannot be localhost by the same reasons mentioned before.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 20, 2024, 6:18pm UTC](https://discuss.elastic.co/t/filebeat-not-scrapping-log-lines/355943/5 "2024-04-20T18:18:13Z")

</div>

Closed on April 20, 2024, 6:18pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.