Logstash output plugin not writing data sequentially

[kaushik.vankayala]

Hi There,

I am using logstash as means of backup tool for the logs of my application. The conf file i created is simply an input with a file plugin and an output with a file plugin.

So basically i have a shell script running which keeps updating the temp file (overwrites) with the content of application log file. The logstash will now copy the lines from input to output. There maybe a case where the log file of the appilcation is cleared of the entire content and starts appending new lines from the beginning.

So is the purpose of using the logstash to copy the unique lines off of the temp file to back up file. But the problem is......

input
{
	file
	{
		path => "C:\Users\Desktop\temp.txt"
		start_position => "beginning"
	}
}

output
{
	stdout { codec => line { format => "%{message}"} }
	file
	{
		codec => line { format => "%{message}"}
		path => "C:\Users\Desktop\backup.txt"
	}
}

.....While each line being copied into my output file, they are not in order.

How to make it copy sequentially?

Below is the input and output i got running the logstash conf as given above!

Input file;

Output file;

As can been the lines are pretty jumbled!

Please help.

Regards

Kaushik Vankayal

[magnusbaeck (Magnus Bäck)]

Make sure the number of pipeline workers is set to 1 (one).

[kaushik.vankayala]

It worked like a charm. Thanks @magnusbaeck.

I have one query though, if i have say 2-3 conf files will the number of workers affect both the conf files of the pipeline?

Can we have different workers for different conf files?

[magnusbaeck (Magnus Bäck)]

I have one query though, if i have say 2-3 conf files will the number of workers affect both the conf files of the pipeline?

Yes. Logstash doesn't care how you split your configuration into separate files. When the configuration file parser reads them they just get concatenated anyway.

Can we have different workers for different conf files?

Yes, if you run multiple Logstash pipelines.

[kaushik.vankayala]

For some reason, logstash is not writing the last line of a input file until there is some new lines occuring.

I tried to set flush_interval to zero but still it does not work. I also tried to print it to the console even there it is not showing the last line.

Can you please help?

Content of my input file

Hello
this is
a test to
file input plugin

Output of logstash

 [2018-08-30T23:35:16,132][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
 [2018-08-30T23:35:16,157][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xf2c0d4c sleep>"}
 [2018-08-30T23:35:16,173][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
 Hello
 this is
 a test to
 [2018-08-30T23:35:42,248][INFO ][logstash.outputs.file    ] Opening file {:path=>"C:/Users/M1045583/Desktop/backup.txt"}

Regards

Kaushik