Logstash reading updated logs?

[Aylwinns_Tan]

Hi all,

I would like to check if logstash is able to read updated logs instead of appended logs.

For example, my program generates an XML log over time but the XML data is updated into the log file which means there is no new lines in the log file, just minor changes. Can logstash read such information?
If so, what is the configuration?

In addition, the current XML log is a single line and logstash doesn't read the line until I forcefully append a new line to it. Is there a way for logstash to read the single line log file?

Thanks everyone

[magnusbaeck (Magnus Bäck)]

For example, my program generates an XML log over time but the XML data is updated into the log file which means there is no new lines in the log file, just minor changes. Can logstash read such information?

It depends. Exactly how is the file updated? Rewritten in place (same inode)? Brand new file each time?

In addition, the current XML log is a single line and logstash doesn't read the line until I forcefully append a new line to it. Is there a way for logstash to read the single line log file?

You mean the file doesn't end with a newline character? No, the file input doesn't support that.

I suppose you could use an exec input that just runs "cat filename.xml". If you're outputting the data to ES you could use the duplicate avoidance technique described in a recent elastic.co blog post.

[Aylwinns_Tan]

@magnusbaeck

It depends. Exactly how is the file updated? Rewritten in place (same inode)? Brand new file each time?

The file is updated via changes to the file (exact same file name).
For example, there is a <Status>Started</Status> in the file and after some time gets change to <Status>Error</Status>

@magnusbaeck

I suppose you could use an exec input that just runs "cat filename.xml". If you're outputting the data to ES you could use the duplicate avoidance technique described in a recent elastic.co blog post.

Could you give more details as to how to use an exec to run "cat" in the input section in the logstash.conf?
And yes I am outputting data to ES so could u refer me to the recent blog post?

Thanks so much for your help ^^

Cheers

[magnusbaeck (Magnus Bäck)]

The file is updated via changes to the file (exact same file name).

That description doesn't answer my question of whether the same inode is reused.

Could you give more details as to how to use an exec to run "cat" in the input section in the logstash.conf?

What kind of details do you need? It's a simple plugin with two options. Did you look at the documentation?

And yes I am outputting data to ES so could u refer me to the recent blog post?

I think it was posted in the last week. I'm sure you can find it.