# Data is lost when outputting to file

**URL:** <https://discuss.elastic.co/t/data-is-lost-when-outputting-to-file/258089>\
**Category:** Logstash\
**Created:** [December 9, 2020, 8:06am UTC](https://discuss.elastic.co/t/data-is-lost-when-outputting-to-file/258089 "2020-12-09T08:06:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![lon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lon/32/80472_2.png) [@lon](https://discuss.elastic.co/u/lon)\
**Post date:** [December 9, 2020, 8:06am UTC](https://discuss.elastic.co/t/data-is-lost-when-outputting-to-file/258089/1 "2020-12-09T08:06:00Z")

</div>

[Please excuse my poor English...]

If output the file to the same file with a line codec of the file output, the data will be lost.

my logstash.conf

```auto
    input {
      beats	{
    	port => "5044"
      }
    }
     
    filter {
    	grok{
    		match => {
    			"message" => "^%{INT:num}$"
    		}
    	}

        if "_grokparsefailure" in [tags] {
            drop { }
        }

    	mutate { 
            convert => {"num" => "integer"} 
        }
    }

    output {
    	if [num] == 1111111111 {
    		file { 
    			path => "test/test.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}
    	}
    	else if [num] == 2222222222 {
    		file { 
    			path => "test/test.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}	
    	}
    }

```

A file with only 1111111111 or 222222222 was used as input.  
Outputs to the same input file differ.

1st - 110,992 KB  
2nd - 110,945 KB

Just by comparing the two, there is a difference in capacity by 47KB.

When the output file is divided.  
logstash.conf

```auto
    input {
      beats	{
    	port => "5044"
      }
    }
     
    filter {
    	grok{
    		match => {
    			"message" => "^%{INT:num}$"
    		}
    	}

        if "_grokparsefailure" in [tags] {
            drop { }
        }

    	mutate { 
            convert => {"num" => "integer"} 
        }
    }

    output {
    	if [num] == 1111111111 {
    		file { 
    			path => "test/test1.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}
    	}
    	else if [num] == 2222222222 {
    		file { 
    			path => "test/test2.txt"
    			codec => line {
    				format => 'test1="%{num}",test2="%{num}"'
    			}
    		}	
    	}
    }

```

1st - 55,665KB, 55,665KB  
2nd - 55,665KB, 55,665KB

We can see that there is no difference in capacity between the two attempts.  
The conclusion is that if you print the same file with line codec, the data will be lost.

Is it a bug?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2021, 8:06am UTC](https://discuss.elastic.co/t/data-is-lost-when-outputting-to-file/258089/2 "2021-01-06T08:06:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.