# Reports - Old Data - Oh My

**URL:** <https://discuss.elastic.co/t/reports-old-data-oh-my/1947>\
**Category:** Logstash\
**Created:** [June 4, 2015, 4:22pm UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947 "2015-06-04T16:22:42Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![bdunbar](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@bdunbar](https://discuss.elastic.co/u/bdunbar)\
**Post date:** [June 4, 2015, 4:22pm UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/1 "2015-06-04T16:22:42Z")

</div>

We host 16 apache servers behind varnish. I have users (Marketing) who are not technical. They're not happy with the current stats software we have installed. I'm giving different webstats apps a try.

I've got ELK installed, and _I_ like it well enough so far.

What I'd like to do is ..

- Generate reports: Suzy from Marketing logs in and clicks a button 'Visits per hour' or 'GEO-IP of people from the UK'. Like that.

- We have all logs from 2014 saved on disk. Can I get these inserted into ELK?

I am going to go search documentation, but if anyone wants to chime in with pointers, tips or etc please let me know.

---

<div class="post-metadata">

**Author:** ![jordansissel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jordansissel/32/44957_2.png) [@jordansissel](https://discuss.elastic.co/u/jordansissel)\
**Post date:** [June 4, 2015, 6:37pm UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/2 "2015-06-04T18:37:34Z")

</div>

You can use the file input (set `start_position => beginning` for this!) to read all the old files. The grok filter will help you parse the fields in each event. The date filter will help you tell logstash (and Elasticsearch) the correct time an event occurred.

As for viewing "visits per hour" etc, you can use Kibana for this. Kibana's dashboarding system lets you build a view on your data, and you can share that view with others 😄

---

<div class="post-metadata">

**Author:** ![bdunbar](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@bdunbar](https://discuss.elastic.co/u/bdunbar)\
**Post date:** [June 4, 2015, 9:51pm UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/3 "2015-06-04T21:51:48Z")

</div>

Awesome, Jordan. That is a huge help.

---

<div class="post-metadata">

**Author:** ![bdunbar](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@bdunbar](https://discuss.elastic.co/u/bdunbar)\
**Post date:** [June 9, 2015, 4:37am UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/4 "2015-06-09T04:37:21Z")

</div>

I've got this

#fileimport.conf  
input {  
file {  
path =\> "/var/log/test/app01/doman.com-access\_log-20141201"  
type =\> "apache"  
start\_position =\> "beginning"  
}  
}

filter {  
stuff  
}

output {  
elasticsearch {  
cluster =\> "elasticsearch.local"  
host =\> "127.0.0.1"  
protocol =\> http  
index =\> "muo-logs"  
index\_type =\> "apache"  
}  
}

I launched it like so from shell

$ /opt/logstash/bin/logstash -f fileimport.conf

However, since 'file are followed in a manner similar to 'tail -0f'" it executes but never returns control back to shell. Or at least that is what i surmise: index 'muo-logs' was built, and populated with a few hundred MB of data. Hours later I terminated the process.

Is the correct method to terminate that with '&' so it backgrounds? Or am I using the tool incorrectly?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 9, 2015, 5:25am UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/5 "2015-06-09T05:25:33Z")

</div>

No, you're on the right track. Logstash's file input will never consider a file "done" because it has no way of knowing when no more writes will take place. If you want Logstash to halt when a file has been processed you should feed the input via the [stdin input](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-stdin.html) instead.

---

<div class="post-metadata">

**Author:** ![bdunbar](https://avatars.discourse-cdn.com/v4/letter/b/a3d4f5/32.png) [@bdunbar](https://discuss.elastic.co/u/bdunbar)\
**Post date:** [June 9, 2015, 12:31pm UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/6 "2015-06-09T12:31:51Z")

</div>

Just to make sure I've got my ducks in a row pre-coffee ..

Change 'input' to this

input {  
stdin {  
type =\> "apache"  
}  
}

And execute like so

/opt/logstash/bin/logstash -f fileimport.conf \< /var/log/test/serverlogfile

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 9, 2015, 12:53pm UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/7 "2015-06-09T12:53:23Z")

</div>

Yeah, that looks good.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:38am UTC](https://discuss.elastic.co/t/reports-old-data-oh-my/1947/8 "2017-07-06T05:38:02Z")

</div>


