# Data are cut into logstash

**URL:** <https://discuss.elastic.co/t/data-are-cut-into-logstash/87493>\
**Category:** Logstash\
**Created:** [May 29, 2017, 9:42pm UTC](https://discuss.elastic.co/t/data-are-cut-into-logstash/87493 "2017-05-29T21:42:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SunDog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sundog/32/18464_2.png) [@SunDog](https://discuss.elastic.co/u/SunDog)\
**Post date:** [May 29, 2017, 9:42pm UTC](https://discuss.elastic.co/t/data-are-cut-into-logstash/87493/1 "2017-05-29T21:42:22Z")

</div>

Hello,

I collect logs from 40 devices.  
Logs are read by filebeat from a text file, filebeat also adds a field with the device name (dev\_id).  
The contents of the file looks something like this:

23:53:15 SOME TEXT  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
00424183 27-03-17 20:08:28  
SOME TEXT  
4..8181 (Some Text)  
xxxxx/xxxxxxxxxxxxx 15.00 XXX  
C1: 5; C2: 3; C3: 3; C4: 2  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
\*434\*01:06:52 GO IN SERVICE COMMAND  
22:15:33 SOME TEXT  
22:14:32 SOME TEXT  
21:04:03 SOME TEXT  
21:00:11 -\> SOME TEXT

From time to time in elasticsearch inserts records without a field offset, dev\_id, in the field \_type value% {[metadata] [type]} is inserted.  
More often it happens on lines like \*923\* 00:04:57 SOME TEXT, but sometimes on others

config file:

input {  
beats {  
port =\> 5044  
codec =\> multiline {  
auto\_flush\_interval =\> 15  
pattern =\> "^%{TIME} "  
negate =\> true  
what =\> next  
}  
}  
}  
filter {  
grok {  
match =\> ["message", "%{TIME:transaction\_time}"]  
}  
grok {  
match =\> ["source", "%{POSINT:transaction\_date}"]  
}  
mutate {  
add\_field =\> {"timestamp =\> "%{transaction\_date} %{transaction\_time}"}  
}  
date {  
match =\> ["timestamp", "YYYYMMdd HH:mm:ss"]  
target =\> "@timestamp"  
}

```
    mutate {
      remove_field => ["year", "tags", "beat", "source", "type", "input_type", "timestamp", "transaction_date", "transaction_time"]

    }

```

}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 29, 2017, 11:02pm UTC](https://discuss.elastic.co/t/data-are-cut-into-logstash/87493/2 "2017-05-29T23:02:17Z")

</div>

You should use the mutiline functionality at the beat side, not ES.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 26, 2017, 11:02pm UTC](https://discuss.elastic.co/t/data-are-cut-into-logstash/87493/3 "2017-06-26T23:02:26Z")

</div>

Closed on June 26, 2017, 11:02pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.