# File input to grok filter to elasticsearch output

**URL:** <https://discuss.elastic.co/t/file-input-to-grok-filter-to-elasticsearch-output/105126>\
**Category:** Logstash\
**Created:** [October 24, 2017, 9:15pm UTC](https://discuss.elastic.co/t/file-input-to-grok-filter-to-elasticsearch-output/105126 "2017-10-24T21:15:35Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cm47](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@cm47](https://discuss.elastic.co/u/cm47)\
**Post date:** [October 24, 2017, 9:15pm UTC](https://discuss.elastic.co/t/file-input-to-grok-filter-to-elasticsearch-output/105126/1 "2017-10-24T21:15:35Z")

</div>

Hi, I am trying to configure logstash to read lines from a custom file.

Logstash configuration file:  
input {  
file {  
path =\> "/home/centos/customlog/testfile.log"  
codec =\> line  
start\_position =\> "beginning"  
ignore\_older =\> 0  
}  
}

filter {  
grok {  
patterns\_dir =\> "/etc/logstash/patterns"  
match =\> { "message" =\> "%{DATETIME\_PATTERN:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:message}" }  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
user =\> "elastic"  
password =\> "omitted"  
index =\> "custom-log-test"  
}  
}

where DATETIME\_PATTERN is a custom pattern declared like so:  
DATETIME\_PATTERN %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[]%{HOUR}:%{MINUTE}:%{SECOND}

Here is part of the log trying to be read:  
2017-10-05 13:10:00 INFO Running Check Time issue  
2017-10-05 13:10:00 INFO No need to alert - there are no future times  
2017-10-05 13:20:00 INFO Running Check Time issue  
2017-10-05 13:20:00 ERROR Issue found - sending alert

Where am I going wrong here? For testing purposes, I have also tried with stdout output using rubydebug codec and still don't see anything happening in logstash logs when I write and quit the file (logstash logs are raised to debug level). I also tried without the line codec incase it was not correct however I do believe that is the right codec to use. I also tried with a trivial grok pattern like just the log level and greedy data afterward and creating a log to match that pattern with no success as well. Any direction would be greatly appreciated, thanks!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 25, 2017, 2:49am UTC](https://discuss.elastic.co/t/file-input-to-grok-filter-to-elasticsearch-output/105126/2 "2017-10-25T02:49:07Z")

</div>

It's probably a sincedb issue, so try setting that to `/dev/null`.

---

<div class="post-metadata">

**Author:** ![cm47](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@cm47](https://discuss.elastic.co/u/cm47)\
**Post date:** [October 25, 2017, 7:04pm UTC](https://discuss.elastic.co/t/file-input-to-grok-filter-to-elasticsearch-output/105126/3 "2017-10-25T19:04:22Z")

</div>

Thanks for your reply, Mark. Shouldn't setting start position to beginning negate that issue? Anyway, I tried setting the sincedb path to /dev/null and no change. Still not seeing anything in logstash logs when I write and quit the test log file.

The only thing remotely related to this whole process that I see in the logs is this single line:

[DEBUG][logstash.inputs.file] \_globbed\_files: /home/centos/customlog/testfile.log: glob is: []

Does this indicate something is wrong? Sorry, fairly inexperienced with ELK and just trying to figure out why I can't get something that should be very basic to work.

---

<div class="post-metadata">

**Author:** ![cm47](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@cm47](https://discuss.elastic.co/u/cm47)\
**Post date:** [October 25, 2017, 7:18pm UTC](https://discuss.elastic.co/t/file-input-to-grok-filter-to-elasticsearch-output/105126/4 "2017-10-25T19:18:06Z")

</div>

I was able to resolve the issue. For whatever reason, logstash did not like the log file being in the centos user path that I had posted in my configuration. I changed it to /etc/logstash/customlog/testfile.log and I was able to see lines from the file printed to stdout.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 22, 2017, 7:18pm UTC](https://discuss.elastic.co/t/file-input-to-grok-filter-to-elasticsearch-output/105126/5 "2017-11-22T19:18:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.