Having trouble with basic guide, doesn't appear to read any logs

Elastic Stack Logstash
#1

hi everyone, I'm trying to setup logstash and am following this guide https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html

However, I am not seeing any output from logstash (should I?) when I run the command.

There doesn't appear to be anything in elasticsearch either.

What could I be doing wrong? here is my output: https://gist.github.com/chrisan/f2a02f2f1d33947ae1b13f26cdbd4005

It is just sitting on "Pipeline main started"

#2

See my response here: Logstash not indexing the input data to elastic search

#3

Thank you for the reply @magnusbaeck !

I do get the correct output with rubydebug

https://gist.github.com/chrisan/7642d3c6a7c5c1b853495fccaef027c0 (I trimmed logstash-tutorial.log down to 10 rows)

Next I added back the elasticsearch output and ran the command with --debug https://gist.github.com/chrisan/6b1ad44ff4f1825582679b85f33755bd

It just keeps outputting the last few lines over and over.

I killed it after about 10 minutes and checked ES again. Still no indicies were created or searches worked. Out of curiosity I added a new line to logstash-tutorial.log and gave it today's date

83.149.9.216 - - [10/Oct/2016:05:13:43 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard.jpg HTTP/1.1" 200 321631 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

I briefly saw something different in the logstash debug output and now I have a new index for today as well as 1 record

$ curl 'localhost:9200/_cat/indices?v'health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open logstash-2016.10.11 5 1 1 0 19kb 19kb

From my original gist I have ignore_older => 0 and start_position => beginning per the elastic.co guide

Any ideas from here?

#4

Are you deleting the sincedb file in between the runs?

#5

I was just following the guide https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html

I did not know about sincedb, thank you for pointing that out. I deleted that from my home directory and re-ran and now I have 12 records. Which I assume are 1 from my test of adding a new date to the log, 10 from the original trimmed tutorial file, and a repeat of that 1 I added because I deleted the sincedb

Thanks for your help! On to more advanced stuff :slight_smile:

#6