Data is lost when outputting to file

[Please excuse my poor English...]

If output the file to the same file with a line codec of the file output, the data will be lost.

my logstash.conf

    input {
      beats	{
    	port => "5044"
      }
    }
     
    filter {
    	grok{
    		match => {
    			"message" => "^%{INT:num}$"
    		}
    	}

        if "_grokparsefailure" in [tags] {
            drop { }
        }

    	mutate { 
            convert => {"num" => "integer"} 
        }
    }

    output {
    	if [num] == 1111111111 {
    		file { 
    			path => "test/test.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}
    	}
    	else if [num] == 2222222222 {
    		file { 
    			path => "test/test.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}	
    	}
    }

A file with only 1111111111 or 222222222 was used as input.
Outputs to the same input file differ.

1st - 110,992 KB
2nd - 110,945 KB

Just by comparing the two, there is a difference in capacity by 47KB.

When the output file is divided.
logstash.conf

    input {
      beats	{
    	port => "5044"
      }
    }
     
    filter {
    	grok{
    		match => {
    			"message" => "^%{INT:num}$"
    		}
    	}

        if "_grokparsefailure" in [tags] {
            drop { }
        }

    	mutate { 
            convert => {"num" => "integer"} 
        }
    }

    output {
    	if [num] == 1111111111 {
    		file { 
    			path => "test/test1.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}
    	}
    	else if [num] == 2222222222 {
    		file { 
    			path => "test/test2.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}	
    	}
    }

1st - 55,665KB, 55,665KB
2nd - 55,665KB, 55,665KB

We can see that there is no difference in capacity between the two attempts.
The conclusion is that if you print the same file with line codec, the data will be lost.

Is it a bug?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.