Issue With reading file containing \r or ^M as line terminator

(Linoj Wilson) #1

Dear All,
I am trying to read some custom logs. I am using multi pattern as well. The file is containing only \r or ^M as line terminator. I tried with line terminator configuration auto and carriage_return as well. But the file is not getting read by file beat. Below is my configurations. Appreciated any help or inputs. Attached the snapshot of sample log file as well.

filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • D:\Softwares\ELK\logs\test\1*
      exclude_lines: ['(![\d\d-\d\d-\d\d \d\d:\d\d:\d\d]\sACCUMM COUNTER VALUES\s:\d*:\d*:\d*:\d*)']
      line_terminator: carriage_return

(Martin R.) #2

To me it looks like if you’re using an unreleased feature and configuration option.

Can you link to the reference that makes you think this is a valid configuration option?

Also please always include the version of the softwares you’re talking about in your post.

(Linoj Wilson) #3

I am using Elastic 6.7

(Linoj Wilson) #4

Thanks for the reply @martinr_ubi
I have seen the below configuration in some of the docs.

Characters which separate the lines. Valid values: auto, line_feed, vertical_tab, form_feed,

carriage_return, carriage_return_line_feed, next_line, line_separator, paragraph_separator.

#line_terminator: auto
So I tried with that option also.
Apart from that is there a way to read files with only \r as terminator?

(Martin R.) #5

Hey,

I don't know your context and what other options you could have to get around the issue or to fix the source files so the only option I can suggest is to try a snapshot build of filebeat. If you could change the app generating those file to use LF or CR+LF for newline and make them "compatible" with filebeat, I have to guess you wouldn't be here in the first place...

The line_terminator feature appear merged in master and I think it will be available in the snapshot builds. So you can try that and use that unreleased version of filebeat until a version of filebeat is released with that feature.

That's an imperfect solution I guess, even if it works, but from where I'm standing it's the only helpful suggestion I can make because I can't know what else you could do compared to that. If there exists a better workaround or solution, I'm not in a position to come up with it.

For testing purposes, we generate snapshot builds that you can find here. Please be aware that these are built on top of master and are not meant for production.

https://beats-ci.elastic.co/job/elastic+beats+master+multijob-package-linux/lastSuccessfulBuild/gcsObjects/

By the way I can't suggest a particular build, I don't know what OS, arch, platform, etc you need. If you're not sure which snapshot build to use/try, tell us exactly where you get the current filebeat package you're using and the name of it with all relevant info and we should be able to assist in choosing the right file from that link if you're unsure.

One last thing, if you do try this, the right config according to your description of your log file would have to be:

line_terminator: carriage_return

Since "auto" is the config that is equivalent to the current behavior of filebeat which appear to not work for you because it doesn't support CR only line endings..

(Linoj Wilson) #6

@martinr_ubi Thanks for your reply and insight martin. Will check some other work around and feasibility other than Filebeat.

The context is I am reading ATM generated logs and analyzing it. So the feasibility of asking them to put \n instead of \r is nill.