# How Do I Verify Logstash is Loading Data?

**URL:** <https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579>\
**Category:** Logstash\
**Created:** [August 9, 2016, 10:10am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579 "2016-08-09T10:10:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![hisaotsu](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@hisaotsu](https://discuss.elastic.co/u/hisaotsu)\
**Post date:** [August 9, 2016, 10:10am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579/1 "2016-08-09T10:10:31Z")

</div>

I just set up Elastic Search, Kibana and Logstash.  
I am trying to load up the historical apache log to Elastic Search via Logstash.

I have the below configuration for Logstash, however, it seems to stop after showing the message below. Elastic Search is working since I can PUT/GET and DELETE data from Sense.

Questions are:  
(1) How do I confirm my Logstash is loading data to Elastic Search? (No apparent log on ES log right now.)  
(2) What's wrong with my configuration that possibly prevents loading data?

`// Last Message in log for LogStash

{:timestamp=\>"2016-08-09T18:53:08.436000+0900", :message=\>"Grok compile", :field=\>"message", :patterns=\>[], :level=\>:info}  
{:timestamp=\>"2016-08-09T18:53:08.627000+0900", :message=\>"Starting pipeline", :id=\>"main", :pipeline\_workers=\>16, :batch\_size=\>125, :batch\_delay=\>5, :max\_inflight=\>8000, :level=\>:info}  
{:timestamp=\>"2016-08-09T18:53:08.648000+0900", :message=\>"Pipeline main started"}

// Start up command  
logstash hisaotsu$ bin/logstash -f ./ls-apache.conf --verbose --log ls.log -w 16  
Sending logstash logs to ls.log.

// Configuration File  
logstash hisaotsu$ cat ls-apache.conf  
input{  
file{  
path =\> "/mydir/es/log\_dropbox/access\_log\_\*"  
start\_position =\> "beginning"  
ignore\_older =\> 0  
sincedb\_path =\> "/mydir/mydir/es/"  
}  
}

filter {  
grok {  
match =\> { "message" =\> "%{COMBINEDAPACHELOG}" }  
break\_on\_match =\> false  
tag\_on\_failure =\> ["\_message\_parse\_failure"]  
}  
date {  
match =\> ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]  
locale =\> en  
}  
geoip {  
source =\> ["clientip"]  
}  
grok {  
match =\> { "request" =\> "^/%{WORD:first\_path}/%{GREEDYDATA}$" }  
tag\_on\_failure =\> ["\_request\_parse\_failure"]  
}  
useragent {  
source =\> "agent"  
target =\> "useragent"  
}  
}

output {  
elasticsearch {}  
stdout { codec =\> rubydebug }  
}  
`

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 10, 2016, 3:10am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579/2 "2016-08-10T03:10:41Z")

</div>

> [@hisaotsu](#):
>
> I have the below configuration for Logstash, however, it seems to stop after showing the message below. Elastic Search is working since I can PUT/GET and DELETE data from Sense.

Are you adding new data to the file?  
It looks like LS is waiting for that.

---

<div class="post-metadata">

**Author:** ![hisaotsu](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@hisaotsu](https://discuss.elastic.co/u/hisaotsu)\
**Post date:** [August 10, 2016, 3:24am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579/3 "2016-08-10T03:24:40Z")

</div>

Thank you for your reply.  
If I specify a file, LS seems to load it up.  
However, when I specify the file path with a wildcard character, it doesn't.  
There is a file under the directory, which should be new data.

Adding another file into the directory did not change the behavior.

I will try out some more and update if I find anything.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 10, 2016, 5:41am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579/4 "2016-08-10T05:41:32Z")

</div>

> sincedb\_path =\> "/mydir/mydir/es/"

This may or may not be related to your problem, but `sincedb_path` should be a file and not a directory.

---

<div class="post-metadata">

**Author:** ![hisaotsu](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@hisaotsu](https://discuss.elastic.co/u/hisaotsu)\
**Post date:** [August 10, 2016, 6:51am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579/5 "2016-08-10T06:51:00Z")

</div>

Thank you, I will fix that as well.

---

<div class="post-metadata">

**Author:** ![hisaotsu](https://avatars.discourse-cdn.com/v4/letter/h/ee7513/32.png) [@hisaotsu](https://discuss.elastic.co/u/hisaotsu)\
**Post date:** [September 5, 2016, 1:00am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579/6 "2016-09-05T01:00:18Z")

</div>

It turns out that number of files in the directory and file size themselves were too large for my environment to see updates right away. I made a new directory, copied some files and then started Logstash again. This seems to work for me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:39am UTC](https://discuss.elastic.co/t/how-do-i-verify-logstash-is-loading-data/57579/7 "2017-07-06T04:39:59Z")

</div>