# Multiline codec not expected bulk

**URL:** <https://discuss.elastic.co/t/multiline-codec-not-expected-bulk/318052>\
**Category:** Logstash\
**Created:** [November 2, 2022, 8:45pm UTC](https://discuss.elastic.co/t/multiline-codec-not-expected-bulk/318052 "2022-11-02T20:45:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [November 2, 2022, 8:45pm UTC](https://discuss.elastic.co/t/multiline-codec-not-expected-bulk/318052/1 "2022-11-02T20:45:40Z")

</div>

My case is that I have a separate pipelines for each file. Here for "Numbers\*" files, it doesn't work properly for me to put data into elastic. On input side I have files divided into 50\_000 lines and with a name that is assigned to the pipeline name. If I add files from different days, logstash packs me in one bulk and sends with wrong date. Are you able to point out where the error is?  
Every files has a date in the snapshot line. So it should close the file and get the next new. But why it doesn't work as expected. I've also tried option with "auto\_flush\_interval =\> 4", it didn't help.

NUMBERs\_AutoExport\_-a\_20221029044502.txt  
NUMBERs\_AutoExport\_-a\_20221023044502.txt

```auto
# snapshot,68843601,20221023044502
# NUMBERs
a,b,c
d,e,f
# Type2
foo,1,2,3
bar,4,5,6
# DN Blocks
224135896,224135897,,,,,,,,,,,
224135896,224135897,,,,,,,,,,,

```

```auto
input {
    file {
        path => "/opt/data/input/Numbers_*.txt""
        sincedb_path => "/dev/null"
        start_position => beginning
        codec => multiline { pattern => "^#" negate => true what => previous multiline_tag => "" }
    }
}
filter {
    mutate { remove_field => ["[event]", "log" ] }
     if [message] =~ /# snapshot/{
         dissect {
            mapping => {
                "[message]" => "# %{activity},%{val},%{time}"
            }
            remove_field => ["[message]"]
        }
        date {
                match => ["time", "yyyyMMddHHmmss"]
                timezone => "Europe/Paris"
                target => "timestamp"
            }
        ruby { code => '@@metadata = event.get("@timestamp")' }
        drop {}

    } else if "# NUMBERs" in [message] {
        mutate { add_field => { "eventType" => "NUMBERs" } }
        split { field => "message" }
        if [message] !~ /^#/ {
            csv { columns => ["c1", "c2", "c3"] }
        }  
 ruby { code => 'event.set("@timestamp", @@metadata)' }

    } else if "# Type2" in [message] {
        mutate { add_field => { "eventType" => "Type2" } }
        split { field => "message" }
    } else {
        mutate { add_field => { "eventType" => "Unrecognized" } }
    }
}

```

logsatsh.yml

```auto
log.level: info
config.reload.automatic: true
config.reload.interval: 30s
pipeline.ecs_compatibility: disabled
pipeline.workers: 48
pipeline.batch.size: 2000
pipeline.batch.delay: 50
pipeline.ordered: auto

```

![double](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b1308c2aa7a1746d98c58c1a064a7b67d2f2415f.jpeg)

---

<div class="post-metadata">

**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\
**Post date:** [November 3, 2022, 6:55am UTC](https://discuss.elastic.co/t/multiline-codec-not-expected-bulk/318052/2 "2022-11-03T06:55:11Z")

</div>

Can anyone look at this case? the same behavior was observed with "pipeline.workers: 1"

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 1, 2022, 6:55am UTC](https://discuss.elastic.co/t/multiline-codec-not-expected-bulk/318052/3 "2022-12-01T06:55:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.