# Logstash not picking the file after placing in the input path
**URL:** https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703
**Category:** Logstash
**Created:** [May 12, 2019, 3:51pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703 "2019-05-12T15:51:35Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 12, 2019, 3:51pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/1 "2019-05-12T15:51:35Z")
I have placed the config.properties file with input of the log file path but whenever a new file is put in that location it is not fetching the latest file my conf as below
input {
file {
path =\> "/mnt/storage/logs/\*"
type =\> "apache-access" # a type to identify those logs (will need this later)
start\_position =\> "beginning"
discover\_interval =\> 420
stat\_interval =\> 10
ignore\_older =\> 3000
}
}
---
### Author:  [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)
#### Post date: [May 13, 2019, 7:21am UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/2 "2019-05-13T07:21:19Z")
Have you tried restarting logstash?
Have you looked at /var/log/logstash/logstash-plain.log to see If there are any errors that you are not picking up?
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 8:17am UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/3 "2019-05-13T08:17:29Z")
Hi Jasonespo,
Have tried starting it but no luck
Also no ERRORS in the log file
---
### Author:  [@Jasonespo](https://discuss.elastic.co/u/Jasonespo)
#### Post date: [May 13, 2019, 8:25am UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/4 "2019-05-13T08:25:08Z")
Why don't you use Filebeat to ship your logs from a file as this is its purpose?
[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html)
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 8:29am UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/5 "2019-05-13T08:29:54Z")
Will use filebeat once the format of the logs and all is finalised and fine as this is for the production, planning to do it sometime this month but want this to work for the log file as well
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 13, 2019, 2:09pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/6 "2019-05-13T14:09:16Z")
Run with "--log.level trace" to see what filewatch thinks is happening.
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 7:46pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/7 "2019-05-13T19:46:54Z")
> [@Badger](#):
>
> log.level trace
Where do we set that ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 13, 2019, 7:55pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/8 "2019-05-13T19:55:50Z")
Add
```
--log.level trace
```
to the command line, or else find the line in /etc/logstash/logstash.yml that says
```
# log.level: info
```
and change it to
```
log.level: trace
```
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 8:10pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/9 "2019-05-13T20:10:44Z")
I am getting this error when i run with debug
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 9:56pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/10 "2019-05-13T21:56:14Z")
I am getting as below when i run debug
[2019-05-13T22:46:33,094][TRACE][filewatch.tailmode.handlers.grow] reading... {"iterations"=\>1, "amount"=\>15660, "filename"=\>"fty.txt"}
[2019-05-13T22:46:33,095][DEBUG][filewatch.tailmode.handlers.grow] read\_to\_eof: get chunk
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 9:57pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/11 "2019-05-13T21:57:12Z")
It also showed as below but it dint write into elastic search nor present in Kibana
[2019-05-13T22:46:33,098][DEBUG][logstash.inputs.file] Received line {:path=\>"/mnt/storage/logs/fty.txt", :text=\>"\\....etc
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 9:57pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/12 "2019-05-13T21:57:55Z")
Further Details are , not sure why is it not picking
[2019-05-13T22:51:24,983][DEBUG][logstash.outputs.file] Starting flush cycle
[2019-05-13T22:51:25,219][DEBUG][org.logstash.execution.PeriodicFlush] Pushing flush onto pipeline.
[2019-05-13T22:51:26,066][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=\>"ParNew"}
[2019-05-13T22:51:26,066][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=\>"ConcurrentMarkSweep"}
[2019-05-13T22:51:26,983][DEBUG][logstash.outputs.file] Starting flush cycle
[2019-05-13T22:51:28,983][DEBUG][logstash.outputs.file] Starting flush cycle
[2019-05-13T22:51:29,326][DEBUG][logstash.outputs.file] Starting stale files cleanup cycle {:files=\>{}}
[2019-05-13T22:51:29,326][DEBUG][logstash.outputs.file] 0 stale files found {:inactive\_files=\>{}}
[2019-05-13T22:51:30,219][DEBUG][org.logstash.execution.PeriodicFlush] Pushing flush onto pipeline.
[2019-05-13T22:51:30,983][DEBUG][logstash.outputs.file] Starting flush cycle
[2019-05-13T22:51:31,069][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=\>"ParNew"}
[2019-05-13T22:51:31,069][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=\>"ConcurrentMarkSweep"}
[2019-05-13T22:51:32,983][DEBUG][logstash.outputs.file] Starting flush cycle
[2019-05-13T22:51:33,175][TRACE][filewatch.tailmode.processor] Delayed Delete processing
[2019-05-13T22:51:33,175][TRACE][filewatch.tailmode.processor] Watched + Active restat processing
[2019-05-13T22:51:33,175][TRACE][filewatch.tailmode.processor] Rotation In Progress processing
[2019-05-13T22:51:33,175][TRACE][filewatch.tailmode.processor] Watched processing
[2019-05-13T22:51:33,175][TRACE][filewatch.tailmode.processor] Active - no change {"watched\_file"=\>"\"}
[2019-05-13T22:51:34,984][DEBUG][logstash.outputs.file] Starting flush cycle
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 13, 2019, 10:00pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/13 "2019-05-13T22:00:38Z")
> [@syed\_faheem\_Hussain](#):
>
> [2019-05-13T22:46:33,098][DEBUG][logstash.inputs.file] Received line {:path=\>"/mnt/storage/logs/fty.txt", :text=\>"\\....etc
If you go that line then logstash got an event. In which case your problem is not with the file input, it is elsewhere in the configuration. What does the rest of the configuration look like?
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 13, 2019, 10:16pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/14 "2019-05-13T22:16:33Z")
This is my config
input {
file {
path =\> "/mnt/storage/logs/\*"
type =\> "apache-access" # a type to identify those logs (will need this later)
start\_position =\> "beginning"
discover\_interval =\> 420
stat\_interval =\> 10
}
}
filter{
xml{
source =\> "message"
store\_xml =\> false
xpath =\> ["//RECORD/MESSAGEID/text()", "logeventid","//RECORD/CATEGORY/text()", "capability","//RECORD/TEXT/text()", "actual\_message","//RECORD/DATE/text()","message\_date","//RECORD/TIME/text()","message\_time","//RECORD/E2EDATA/text()","e2eData","//RECORD/SERVER/text()","server","//RECORD/PORT/text()","port"]
}
kv
{
source =\> "e2eData"
field\_split =\> ",="
remove\_field =\> ["E2E.threadID","E2E.busTxnHdr"]
}
if ([message] !~ "Access\_298") {
drop { }
}
mutate {
lowercase =\> ["capability"]
}
mutate {
add\_field =\> {
"message\_dateTime" =\> "%{message\_date}T%{message\_time}"
#2017-11-15T05:42:29.485
}
remove\_field =\> ["message\_date","message\_time","host"]
}
}
output {
stdout { codec =\> rubydebug }
file {
path =\> "/mnt/storage/logs/process.txt"
}
elasticsearch {
hosts =\> "http://:port"
index =\> "%{capability}"
}
}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 13, 2019, 11:45pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/15 "2019-05-13T23:45:54Z")
> [@syed\_faheem\_Hussain](#):
>
> if ([message] !~ "Access\_298") {
> drop { }
> }
You will ignore any records that do not contain the string "Access\_298". Are you sure there are records that contain that? Do you see them in the file+rubydebug output?
```
kv { source => "e2eData" field_split => ",=" remove_field => ["E2E.threadID","E2E.busTxnHdr"] }
```
I do not see anything that would create those 2 fields, so why do you try to remove them?
```
elasticsearch { hosts => "http://:port" index => "%{capability}" }
```
That hosts option does not look right to me. And if xpath failed to parse capability this might not work.
Are you able to share an example of the rubydebug with as little as possible redacted?
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 14, 2019, 2:31am UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/16 "2019-05-14T02:31:06Z")
Hi Badger,
Thanks for the reply
Regarding the host and port I have just given an example as I din want to expose the Actual host , my hosts look like [http://localhost:61000](http://localhost:61000)
And the string doesn't have that Access\_298 it have some other data
May I know which example are u looking for ?
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 14, 2019, 12:09pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/17 "2019-05-14T12:09:45Z")
Anything further help on this please ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 14, 2019, 12:30pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/18 "2019-05-14T12:30:23Z")
> [@syed\_faheem\_Hussain](#):
>
> And the string doesn't have that Access\_298 it have some other data
If it does not contain Access\_298 then you call a drop {} filter, which discards it, so it will not get indexed.
---
### Author:  [@syed\_faheem\_Hussain](https://discuss.elastic.co/u/syed_faheem_Hussain)
#### Post date: [May 14, 2019, 12:38pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/19 "2019-05-14T12:38:29Z")
Hi Badger,
We don't have those entries we have as below which should ideally get processed as this doesn't satisfy the above condition
\RS-RoBTESB\_MPA-CAPAD\_2983\
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [May 14, 2019, 1:10pm UTC](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703/20 "2019-05-14T13:10:04Z")
Then your test is backwards. You are dropping any events that _do not match_ Access\_298. Did you mean to write =~ rather than !~ perhaps?
[Next page](https://discuss.elastic.co/t/logstash-not-picking-the-file-after-placing-in-the-input-path/180703.md?page=2)