Filebeat exclude_lines

[Betorov]

Hi evreyone,
I wanted to use exclude_lines to exlude all line of a file exept the first one. (I want only the name of the file and his path i don't need his message).

filebeat.inputs:
- type: log
  enabled: true
  #close_eof: true
  #harvester_limit : 1
  exclude_lines: [.]  
  paths:
    - C:\Users\U1\Desktop\file_example\**\*
output.logstash:
  hosts: ["localhost:5044"]

My logstash does not receive any data . I think the reason is that filebeat exclude everything.

[Betorov]

I don't know how I delete the first post but here it is:
I do this to take the first value:

filebeat.inputs:
- type: log
  enabled: true
  #close_eof: true
  #harvester_limit : 1
  max_bytes : 1
  multiline.pattern: '.'
  multiline.match: after
  paths:
    - C:\Users\U1\Desktop\PE22KW-J1\*\**
output.logstash:
  hosts: ["localhost:5044"]

but I get this error (and the files are sent multiple times)

2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 335 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 91 bytes line
2021-06-24T12:16:11.811+0200    ERROR   [reader_line]   readfile/line.go:170    Error skipping until new line, err:reader closed
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 24 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 307 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 86 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 873 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 65 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 380 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 24 bytes line
2021-06-24T12:16:11.811+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 23 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 103 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 408 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 89 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 95 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 670 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 2987 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 47 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 29 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 1228 bytes line
2021-06-24T12:16:11.812+0200    WARN    [reader_line]   readfile/line.go:159    Exceeded 4 max bytes in line limit, skipped 216 bytes line
2021-06-24T12:16:11.812+0200    ERROR   [reader_line]   readfile/line.go:170    Error skipping until new line, err:reader closed

Someone know how to prevent filebeat from trying to send all file?

[jsoriano (Jaime Soriano)]

Hey @Betorov,

@Betorov

I want only the name of the file and his path i don't need his message

Filebeat is not thought for this use case, filebeat is thought to collect all the messages in the files, I don't think there is any way to do what you are trying to do.

But if you want to monitor files created in specific folders, you can give a try to auditbeat, that has features for this specific use case, as the file_integrity module.

[Betorov]

Thanks for your reply I am going to try it