# Logstash log repeated acquisition **URL:** \ **Category:** Logstash\ **Created:** [November 23, 2023, 7:05am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821 "2023-11-23T07:05:15Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [November 23, 2023, 7:05am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/1 "2023-11-23T07:05:15Z")
1. My log is like this: [23/Nov/2023:14:12:37 +0800] | gateway | [http] | 195.161.250.29 | POST /gateway/xxx HTTP/1.1 | 9090 | 200 | 182 | sss67jhsd | 0 | gateway | - | - | - | - | Java/1.8.0\_362 2. Logstash(version: 7.4.2) configuration: ```auto input { file { path => ["/gateway/logs/access/*.log"] ignore_older => 86400 type => "acclog" max_open_files => 65536 sincedb_path => "/gateway/config/sincedb_path/sincedb" } } output { if [type] == "acclog" { kafka { retries => 2 bootstrap_servers => "kafka2:9092" topic_id => 'acclog' } } ``` 1. problem description: I clearly set ignore\_older =\> 86400, but there is always the problem of repeated collection of logs during use. How to solve this problem? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 24, 2023, 12:24pm UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/2 "2023-11-24T12:24:24Z")
> [@kubo\_Smith](#): > > I clearly set ignore\_older =\> 86400, but there is always the problem of repeated collection of logs during use. Can you provide more context about this? Share some examples of data? The `ignore_older` will make Logstash ignore any files that haven't been modified in the specified time, in this case 86400 seconds, but if this file is modified it will no longer ignored. Do you have files older than 86400 seconds that logstash did not ignore? Can you provide some evidence? Like the logs from those files and the result of the `stat /path/file.log` command on linux? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [November 24, 2023, 3:06pm UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/3 "2023-11-24T15:06:59Z")
> [@kubo\_Smith](#): > > ` sincedb_path => "/gateway/config/sincedb_path/sincedb"` Just to add, maaaaybe, the logstash user doesn't have right on the sincedb file, check permissions. ---
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [November 27, 2023, 7:48am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/4 "2023-11-27T07:48:08Z")
1. ooo If it's ATIME, his time is changing 2. In addition to this parameter ignore\_older is there any other way I can control the collection of only the day's logs? ---
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [November 27, 2023, 7:51am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/5 "2023-11-27T07:51:23Z")
permissions: -rw-r--r-- 1 logstash logstash 17M Nov 27 15:48 sincedb ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [November 27, 2023, 9:22am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/6 "2023-11-27T09:22:24Z")
Good, no issues with sincedb. How logs are named? Are those rollover or daily logs? Don't forget to respond to Leandro questions. ---
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [November 28, 2023, 1:54am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/7 "2023-11-28T01:54:20Z")
The daily log will be cut every day. The name of the log is 2023-11-28.log today and 2023-11-29.log tomorrow. ---
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [November 28, 2023, 1:56am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/8 "2023-11-28T01:56:29Z")
> [@kubo\_Smith](#): > > - ooo If it's ATIME, his time is changing > - In addition to this parameter ignore\_older is there any other way I can control the collection of only the day's logs? 1. ooo If it's ATIME, his time is changing 2. In addition to this parameter ignore\_older is there any other way I can control the collection of only the day's logs? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 28, 2023, 12:31pm UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/9 "2023-11-28T12:31:39Z")
> [@kubo\_Smith](#): > > ooo If it's ATIME, his time is changing You didn't share the return of the `stat` command nor any evidence of the duplication, can you share that? Like a screenshot of Kibana showing duplicate lines. `atime` is access time, it not necessarily mean that the file was changed. > [@kubo\_Smith](#): > > The name of the log is 2023-11-28.log today and 2023-11-29.log tomorrow. What does this mean? You have daily logs? Or are you renaming the log file? What will happen with the log `2023-11-28.log` when the day changes to `2023-11-29.log`? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 28, 2023, 12:40pm UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/10 "2023-11-28T12:40:28Z")
> [@kubo\_Smith](#): > > `/gateway/logs/access/` Also, is this path a network filesystem? ---
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [November 29, 2023, 1:55am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/11 "2023-11-29T01:55:16Z")
1. The result returned by the stat command： File: ‘application.2023-11-27.5bc6b9b99c.log’ Size: 15462033165 Blocks: 30199288 IO Block: 1048576 regular file Device: d4h/212d Inode: 129956680212 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 2000/ app) Gid: ( 2000/ app) Access: 2023-11-27 00:00:00.159117000 +0800 Modify: 2023-11-28 00:00:00.031329000 +0800 Change: 2023-11-28 00:00:00.199025000 +0800 Birth: - 2. yes,i have daily logs. 3. no network filesystem,It's local storage 4. is there any other way to collect only the daily log besides ignore\_older? ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [November 29, 2023, 4:50am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/12 "2023-11-29T04:50:02Z")
The sincedb database file should keep track which exist and with the correct rights, however from some reason files have been read again. Can you set `log.level: trace` in logstash.yml and restart LS? There should be info about sincedb. ---
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [November 29, 2023, 6:14am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/13 "2023-11-29T06:14:43Z")
> [@Rios](#): > > log.level: trace Ok, I set log.level: trace, but I should pay attention to those keywords in the log. ---
**Author:** ![kubo\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kubo_smith/32/127830_2.png) [@kubo\_Smith](https://discuss.elastic.co/u/kubo_Smith)\ **Post date:** [December 4, 2023, 3:13am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/14 "2023-12-04T03:13:24Z")
Hello, in addition, I would like to ask why I only collect the logs of the day except the parameter ignore\_older. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 4, 2023, 7:34am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/15 "2023-12-04T07:34:16Z")
Are you using live log tracking or you read closed file, finished for writing? I have dug the documentation, can you add next settings: ```auto input { file { start_position => "beginning" mode=> "read" ignore_older => "1 d" ... ``` Explanation: - **mode read** - If `read` is specified, these settings can be used: `ignore_older` (older files are not processed) - **start\_position** - Choose where Logstash starts initially reading files: at the beginning or at the end. The default behavior treats files like live streams and thus starts at the end. If you have old data you want to import, set this to _beginning_ . \* Default value is `"end"` - **ignore\_older** - Use the string notation, easier to read ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 1, 2024, 7:34am UTC](https://discuss.elastic.co/t/logstash-log-repeated-acquisition/347821/16 "2024-01-01T07:34:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.