# Logstash not indexing data into elasticsearch
**URL:** https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290
**Category:** Logstash
**Created:** [October 25, 2017, 8:01pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290 "2017-10-25T20:01:43Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@cm47](https://discuss.elastic.co/u/cm47)
#### Post date: [October 25, 2017, 8:01pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290/1 "2017-10-25T20:01:43Z")
Hi there, I am trying to get custom log information indexed into elasticsearch using a grok filter.
Logstash configuration file:
input {
file {
path =\> "/etc/logstash/customlog/testfile.log"
codec =\> line
start\_position =\> "beginning"
ignore\_older =\> 0
sincedb\_path =\> "/dev/null"
}
}
filter {
grok {
patterns\_dir =\> "/etc/logstash/patterns"
match =\> { "message" =\> "%{DATETIME\_PATTERN:timestamp} %{LOGLEVEL:log-level} %{GREEDYDATA:message}" }
}
}
output {
elasticsearch {
hosts =\> ["localhost:9200"]
user =\> "elastic"
password =\> "omitted"
index =\> "custom-log-test"
}
#stdout { codec =\> rubydebug }
}
Log file example:
2017-10-05 13:10:00 INFO Running Check Time issue
2017-10-05 13:10:00 INFO No need to alert - there are no future times
2017-10-05 13:20:00 INFO Running Check Time issue
2017-10-05 13:20:00 ERROR Issue found - sending alert
I can confirm that logstash is receiving the lines from the file, this can be seen in the logstash logs like so:
[DEBUG][logstash.inputs.file] Received line {:path=\>"/etc/logstash/customlog/testfile.log", :text=\>"2017-10-05 13:40:00 INFO No need to alert - there are no future times"}
etc...
I have also tested my grok pattern against these log lines using the site here: [http://grokconstructor.appspot.com/do/match](http://grokconstructor.appspot.com/do/match) and can confirm that my pattern does successfully parse the line of text into the fields I specified.
For reference, here is how my custom pattern is defined:
DATETIME\_PATTERN %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[]%{HOUR}:%{MINUTE}:%{SECOND}
Any idea where this might be going wrong? After seeing that the input was received from the file, when I curl elasticsearch to list its indices, I am not seeing the "custom-log-test" index at all indicating the data never made it to elasticsearch.
Any tips for troubleshooting to diagnose the issue? Any help is greatly appreciated.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [October 26, 2017, 5:29am UTC](https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290/2 "2017-10-26T05:29:05Z")
Anything else in Logstash's log? If it has problems sending to ES I'm pretty sure it tells you about it.
---
### Author:  [@cm47](https://discuss.elastic.co/u/cm47)
#### Post date: [October 26, 2017, 8:28pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290/3 "2017-10-26T20:28:10Z")
Thanks for the reply, Magnus.
Nothing that is obvious. No errors or any messages indicating issues with elasticsearch. Here is relevant portion of the log (let me know if you need to see more). After I see the input, there are no messages regarding any sort of output to elasticsearch.
...
[2017-10-26T20:23:14,205][DEBUG][logstash.inputs.file] Received line {:path=\>"/etc/logstash/customlog/testfile.log", :text=\>"2017-10-05 13:30:00 INFO No need to alert - there are no future times"}
[2017-10-26T20:23:14,205][DEBUG][logstash.inputs.file] Received line {:path=\>"/etc/logstash/customlog/testfile.log", :text=\>"2017-10-05 13:40:00 INFO Running Check Time issue"}
[2017-10-26T20:23:14,205][DEBUG][logstash.inputs.file] Received line {:path=\>"/etc/logstash/customlog/testfile.log", :text=\>"2017-10-05 13:40:00 INFO No need to alert - there are no future times"}
[2017-10-26T20:23:14,206][DEBUG][logstash.inputs.file] writing sincedb (delta since last write = 826)
[2017-10-26T20:23:18,471][DEBUG][logstash.pipeline] Pushing flush onto pipeline
[2017-10-26T20:23:18,811][DEBUG][logstash.pipeline] Pushing flush onto pipeline
[2017-10-26T20:23:18,833][DEBUG][logstash.inputs.metrics] Metrics input: received a new snapshot
...
Does "pushing flush onto pipeline" mean it is being sent to elasticsearch?
If this is the case, what are potential reasons I might not being seeing the index created when I query ES like so?
curl -XGET -u elastic:omitted 'localhost:9200/\_cat/indices?v&pretty' | grep cust
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [October 27, 2017, 3:33am UTC](https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290/4 "2017-10-27T03:33:52Z")
> Does "pushing flush onto pipeline" mean it is being sent to elasticsearch?
No.
> If this is the case, what are potential reasons I might not being seeing the index created when I query ES like so?
I'm not sure. If you enable the stdout output, are you seeing event being dumped to the log?
---
### Author:  [@cm47](https://discuss.elastic.co/u/cm47)
#### Post date: [October 27, 2017, 2:39pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290/5 "2017-10-27T14:39:52Z")
When I uncomment stdout line in the config file, and comment the elasticsearch output, I do not see anything in the logstash output after a LS restart. Still only just the logs posted above showing input, sincedb write, then nothing.
So logstash isn't sending output at all. Any tips for diagnosing this further? Would it indicate an issue with the grok pattern? (even though it worked when I tested the pattern on the site mentioned above)
Thanks, I appreciate your assistance.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 24, 2017, 2:40pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-data-into-elasticsearch/105290/6 "2017-11-24T14:40:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.