# Logstash never picked up any logs. I mean it never parse any logs except 2 log lines for the entire day **URL:** \ **Category:** Logstash\ **Created:** [September 28, 2023, 12:36pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063 "2023-09-28T12:36:09Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Post date:** [September 28, 2023, 12:36pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/1 "2023-09-28T12:36:09Z")
I have using logstash 8.10.0 for windows and linux downloaded from [elastic.co](http://elastic.co) downloads. I could hardly see 2 events ingested by logstash. logstash-plain.log does not have any error after pipeline\_running. ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cbf19ecc8eb414cd058e9ab8eb1abc3584abbc08.png) configuration ```auto input { file { path => "C:/setup/log/abbmq1*" type => "abbmq1" start_position => "beginning" } filter { if [type] == "abbmq1" { grok { match => {"message" => "%{DATA:logtimeStamp} %{WORD:operation} %{GREEDYDATA:message}"} overwrite => ["message"] } mutate { remove_field => ["logtimeStamp"] } } } output { stdout {} } ``` please help. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 28, 2023, 12:39pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/2 "2023-09-28T12:39:53Z")
> [@ranjini](#): > > `C:/setup/log/abbmq1*` What does the message on those files looks like? Each file has many lines or just one line per file? ---
**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Post date:** [September 28, 2023, 12:40pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/3 "2023-09-28T12:40:51Z")
C:/setup/log/abbmq1.log the file. It has many log lines. I have checked it manually. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 28, 2023, 12:46pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/4 "2023-09-28T12:46:34Z")
And how are you running logstash? Which user? Does the user have permissions on the folder path? What do you have in Logstash logs? I do not use windows, so I'm not sure which user logstash uses if you run it as a service, but if the paht is correct and the permissions are also correct, it should've read the files. Can you restart logstash and share fresh logs? ---
**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Post date:** [September 28, 2023, 1:17pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/5 "2023-09-28T13:17:57Z")
@leandrojmp thanks for looking into this one. I would like to provide linux config details. ```auto [root@hostname]# cat .sincedb_2801e934a5da4805e0d8082e0c062138 10783905 0 66305 14164630 1695906657.855361 /opt/tomcat.log input { file { path => "/opt/tomcat/logs/cs.log" type => "cs" start_position => "beginning" } } filter { if [type] == "cs"{ grok { match => { "message" => [ "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{DATA:thread}\] %{SPACE}\[\] %{SPACE}\[%{URIPATH:uri_path}\]%{SPACE}\[\] %{GREEDYDATA:message}", "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{DATA:thread}\] %{SPACE}\[\] %{SPACE}\[\]%{SPACE}\[\] %{SPACE}\[\]%{GREEDYDATA:message}", "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}\[%{GREEDYDATA:thread}\] %{SPACE}\[\] %{SPACE}\[\]%{SPACE} \[\] %{SPACE}\[\]%{SPACE}%{DATA:logger}%{SPACE}\[\] - %{GREEDYDATA:message}", "^%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{SPACE}\[%{DATA:thread}\]%{SPACE}\[%{DATA:tenant}\]%{SPACE}\[\]%{SPACE}\[%{DATA:class}\]%{SPACE}\[\] es.CSUtil%{SPACE}\[\]%{SPACE}- This condition should not occur for the tenant:%{SPACE}%{GREEDYDATA:tenant_value}" ] } overwrite => ["message"] } } } I run logstash as service [root@hostname file]# cat /etc/systemd/system/logstash.service [Unit] Description=logstash service [Service] Type=simple User=root Group=root # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/logstash-8.10.0/bin/logstash "-f" "/usr/share/logstash/logstash-8.10.0/conf.d" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target ``` When never I restart one log line getting indexed. But when I run in debug mode it does emit all the logs. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 28, 2023, 1:33pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/6 "2023-09-28T13:33:44Z")
You need to share logs. I do not see anything wrong in your linux configuration, just the fact that you are running logstash as the root user in the service, which is not recommended and it may be a security issue. > [@ranjini](#): > > But when I run in debug mode it does emit all the logs. Not sure what you mean with that, can you explain? Also, if logstash already read lines of a file, it will not read those lines again, only new lines. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 28, 2023, 1:55pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/7 "2023-09-28T13:55:20Z")
1. Set your own sincedb or use `/dev/null` temporarily in sincedb\_path ```auto input { file { path => "/opt/tomcat/logs/cs.log" type => "cs" start_position => "beginning" sincedb_path => "/path/since.db" } } ``` 1. Remove `if [type] == "abbmq1" {`, just for test and I don't see the point if is only one grok. 2. Are paths OK? - 10783905 0 66305 14164630 1695906657.855361 **/opt/tomcat.log** - path =\> **"/opt/tomcat/logs/cs.log"** 1. Check rights on LS somefile.conf, does LS can read. You will see is it picked up in debug mode. 2. Set debug mode `log.level: debug` in logstash.yml ---
**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Post date:** [September 28, 2023, 3:11pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/8 "2023-09-28T15:11:59Z")
Thanks @leandrojmp I use latest logstash 8.10.0. When I stdout the logs I could see many log lines ( I mean event/document) But in ES I could see only one event How do I verify if the logs are been sent from logstash-\> ELK ---
**Author:** ![ranjini](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Post date:** [September 28, 2023, 3:47pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/9 "2023-09-28T15:47:55Z")
@Rios In debug mode, I could stdout the logs. Is there any other ways to see if logstash is sending the logs. Like sincedb file. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [September 28, 2023, 3:55pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/10 "2023-09-28T15:55:40Z")
Is this when you run in the service mode or the process? When you run as the command, LS run under that the logged in user. The service mode is under the logstash user. If is this case, that means your the name.conf file doesn't have proper rights. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 26, 2023, 3:55pm UTC](https://discuss.elastic.co/t/logstash-never-picked-up-any-logs-i-mean-it-never-parse-any-logs-except-2-log-lines-for-the-entire-day/344063/11 "2023-10-26T15:55:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.