# Logstash File plugin custom delimiter

**URL:** <https://discuss.elastic.co/t/logstash-file-plugin-custom-delimiter/50013>\
**Category:** Logstash\
**Created:** [May 13, 2016, 4:47pm UTC](https://discuss.elastic.co/t/logstash-file-plugin-custom-delimiter/50013 "2016-05-13T16:47:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![revendar](https://avatars.discourse-cdn.com/v4/letter/r/e56c9b/32.png) [@revendar](https://discuss.elastic.co/u/revendar)\
**Post date:** [May 13, 2016, 4:47pm UTC](https://discuss.elastic.co/t/logstash-file-plugin-custom-delimiter/50013/1 "2016-05-13T16:47:33Z")

</div>

I'm using logstash for quite a time. I tried using a custom delimiter in File plugin. I'm reading a static file. I see file plugin extracts 32KB data and passes it to tokenizer for splitting by delimiter.

> ```
> data = watched_file.file_read(32768)
> changed = true
> watched_file.buffer_extract(data).each do |line|
> listener.accept(line)
> @sincedb[watched_file.inode] += (line.bytesize + @delimiter_byte_size)
> end
> 
> ```

What happens when the last byte is not new line ( ie: part of a line ).

I have a custom gzip plugin similar to File plugin and my regex fails on the partial line and skips that. I lose an event in this case. Sample of my code.

> Zlib::GzipReader.open(gzfile) do |stream|  
> while line = stream.read(1024) do  
> offset += line.unpack("C\*").size  
> lineno += 1  
> @buffer.extract(line).each do |l|  
> if !exclude\_patterns.nil? and l.match(excludeRegex)  
> next  
> end  
> if l.match(includeRegex)  
> event = LogStash::Event.new("message" =\> @converter.convert(l))  
> totevents += 1  
> event["file"] = gzfile  
> event["offset"] = offset  
> event["lineno"] = lineno  
> decorate(event)  
> queue \<\< event  
> end  
> end  
> end  
> end

I have seen this on a custom delimiter which can happen on \n delimiter as well. Please enlighten me.

---

<div class="post-metadata">

**Author:** ![revendar](https://avatars.discourse-cdn.com/v4/letter/r/e56c9b/32.png) [@revendar](https://discuss.elastic.co/u/revendar)\
**Post date:** [June 27, 2016, 9:45am UTC](https://discuss.elastic.co/t/logstash-file-plugin-custom-delimiter/50013/2 "2016-06-27T09:45:12Z")

</div>

There is a bug in BufferedTokenizer for custom delimiter.  
In the extract method, the left over part of previous data chunk is added to entities array after splitting by delimiter.  
As a result when a data chunk has part of delimiter it just gets added to an event without being split.

```
def extract(data)
    data = @input.join + data # added
    @input.clear # added

    entities = data.split @delimiter, -1
    ...
    ...
end

```

Adding the first 2 lines fixes the issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:50am UTC](https://discuss.elastic.co/t/logstash-file-plugin-custom-delimiter/50013/3 "2017-07-06T04:50:49Z")

</div>