# Logstash parses log files again: is that recurrent?

**URL:** <https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812>\
**Category:** Logstash\
**Created:** [September 25, 2018, 11:36am UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812 "2018-09-25T11:36:34Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michel99\_7](https://avatars.discourse-cdn.com/v4/letter/m/97f17d/32.png) [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)\
**Post date:** [September 25, 2018, 11:36am UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812/1 "2018-09-25T11:36:34Z")

</div>

Hi all,

Our environment is the following:

- OS: Red Hat Enterprise Linux Server release 7.3
- Logstash version: 6.3.0
- ES version: 6.3.2

Our problem: after having stopped the Logstash process for modification purpose in the .conf file, we started the process again but noticed that log files already parsed are treated again; we are using a sincedb file which is not corrupted. The process was working well since a few months until now.

It's the second time we are facing such a problem. Last time, we had to reindex all log files.

Questions:

- does anybody already encountered such a problem?
- if yes, what did you do to come back to a normal situation? (to avoid parsing again all log files)

Many thanks in advance.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2018, 11:39am UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812/2 "2018-09-25T11:39:38Z")

</div>

What does your config look like? What type of storage are you reading from?

---

<div class="post-metadata">

**Author:** ![Michel99\_7](https://avatars.discourse-cdn.com/v4/letter/m/97f17d/32.png) [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)\
**Post date:** [September 25, 2018, 1:37pm UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812/3 "2018-09-25T13:37:25Z")

</div>

Hello Christian,

We are using the file input plugin. The log files are extracted from a mount point folder path. It is the same with the sincedb file.

```
file {
	path => "${LS_XXX_PATH}"
	sincedb_path => "${LS_XXX_SINCEDBPATH}"
	start_position => "beginning"
	add_field => {"Origin" => "XXX"}
        discover_interval => 5
	close_older => 60
	codec => multiline {
		pattern => "^(\[%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
		negate => true
		what => previous
	}
}

```

And the definition of the environment variables:  
LS\_XXX\_PATH=/data\_nfs/xxx/\*.log  
LS\_XXX\_SINCEDBPATH=$LS\_SINCEDB/XXX.db

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2018, 1:43pm UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812/4 "2018-09-25T13:43:48Z")

</div>

I know that there have been some [issues reading from network volumes](https://www.elastic.co/guide/en/logstash/6.4/plugins-inputs-file.html#_reading_from_remote_network_volumes) in the past, but am not sure what you are experiencing could be attributed to that. There seems to have been [some improvements added in Logstash 6.4](https://www.elastic.co/guide/en/logstash/6.4/logstash-6-4-0.html), but I will have to leave it to someone more knowledgeable about this to comment further.

---

<div class="post-metadata">

**Author:** ![Michel99\_7](https://avatars.discourse-cdn.com/v4/letter/m/97f17d/32.png) [@Michel99\_7](https://discuss.elastic.co/u/Michel99_7)\
**Post date:** [September 25, 2018, 2:06pm UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812/5 "2018-09-25T14:06:51Z")

</div>

Thank you, Christian.

I can see for example that the same file is parsed and referenced twice in the sincedb with a different minor version:  
12223334 0 38  
...  
12223334 0 39

Do you know what does this minor version (38 and 39) refer to?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 23, 2018, 2:06pm UTC](https://discuss.elastic.co/t/logstash-parses-log-files-again-is-that-recurrent/149812/6 "2018-10-23T14:06:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.