Logstash reading updated logs?

Elastic Stack Logstash
1

Hi all,

I would like to check if logstash is able to read updated logs instead of appended logs.

For example, my program generates an XML log over time but the XML data is updated into the log file which means there is no new lines in the log file, just minor changes. Can logstash read such information?
If so, what is the configuration?

In addition, the current XML log is a single line and logstash doesn't read the line until I forcefully append a new line to it. Is there a way for logstash to read the single line log file?

Thanks everyone

2

For example, my program generates an XML log over time but the XML data is updated into the log file which means there is no new lines in the log file, just minor changes. Can logstash read such information?

It depends. Exactly how is the file updated? Rewritten in place (same inode)? Brand new file each time?

In addition, the current XML log is a single line and logstash doesn't read the line until I forcefully append a new line to it. Is there a way for logstash to read the single line log file?

You mean the file doesn't end with a newline character? No, the file input doesn't support that.

I suppose you could use an exec input that just runs "cat filename.xml". If you're outputting the data to ES you could use the duplicate avoidance technique described in a recent elastic.co blog post.

3

The file is updated via changes to the file (exact same file name).
For example, there is a <Status>Started</Status> in the file and after some time gets change to <Status>Error</Status>

Could you give more details as to how to use an exec to run "cat" in the input section in the logstash.conf?
And yes I am outputting data to ES so could u refer me to the recent blog post?

Thanks so much for your help ^^

Cheers

4

The file is updated via changes to the file (exact same file name).

That description doesn't answer my question of whether the same inode is reused.

Could you give more details as to how to use an exec to run "cat" in the input section in the logstash.conf?

What kind of details do you need? It's a simple plugin with two options. Did you look at the documentation?

And yes I am outputting data to ES so could u refer me to the recent blog post?

I think it was posted in the last week. I'm sure you can find it.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.