# Load old data to ElasticSearch

**URL:** <https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557>\
**Category:** Logstash\
**Created:** [March 12, 2018, 1:35pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557 "2018-03-12T13:35:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Codrin\_Costea](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@Codrin\_Costea](https://discuss.elastic.co/u/Codrin_Costea)\
**Post date:** [March 12, 2018, 1:35pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/1 "2018-03-12T13:35:36Z")

</div>

Hello !

I am new to Elastic Search and I would need some help in one problem. I try to load multiple log files from my proxy to Elastic Search, through logstast. The steps that I am following are:

1. Create the .conf file
2. Run it with console
3. Create an index in Kibana

Sometimes it works and loads the logs into Kibana, but the problem is that I don't have all the logs. I am trying to import thousands of logs because I have to analyze some times in my proxy. This is my input field. What commands should I add in order to import all the logs in kibana ?

input  
{  
file {  
path =\> " ..... "  
start\_position=\> beginning"  
}  
}

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [March 12, 2018, 1:40pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/2 "2018-03-12T13:40:16Z")

</div>

Could you share your actual config? also when you paste it here can you use the \</\> tag to format it please?

---

<div class="post-metadata">

**Author:** ![Codrin\_Costea](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@Codrin\_Costea](https://discuss.elastic.co/u/Codrin_Costea)\
**Post date:** [March 12, 2018, 1:42pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/3 "2018-03-12T13:42:43Z")

</div>

```
   input
    {
    	file{
    			path=> "D:/Codrin/access+errors/*"
    			path=> "D:/Codrin/access+errors/*/*"
    			start_position=>"beginning"
    		}
    }
    filter 
    {
    	if [path]=~"error"{
    		grok{ match=> { "message"=>'%{DATE:timestamp} %{TIME:time} %{DATA:type} %{DATA:code} %{DATA:noneed1} %{DATA:noneed2} %{DATA:info} %{DATA:noneed3} %{DATA:ip} %{DATA:noneed4} %{DATA:server} %{DATA:noneed5} %{DATA:request} %{DATA:reques_url} %{DATA:httpver} %{DATA:noneed6} %{DATA:host} %{DATA:noneed7} "%{DATA:referrer}"'}}
    		} else if [path]=~"access"{
    					grok{ match=> { "message"=>'%{IPORHOST:remote_addr} %{USER:user} %{USER:user-name} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:body_bytes_sent} %{NUMBER:NotSureAboutTheFild} %{NUMBER:ResponseTime} %{QS:notsure} %{QS:agent}'}}
    					}
    }
    output
    {
    	stdout{codec => dots}
    	elasticsearch
    		{
    			hosts=>["localhost:9200"]
    			index=>"testerrorandmultiplefiles"
    		}
    }
```

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [March 12, 2018, 1:46pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/4 "2018-03-12T13:46:05Z")

</div>

Thanks,

I think you need to double escape the paths ie: D://Codrin//access+errors//\* but I never ever done anything with windows so I could be wrong here 🙂

---

<div class="post-metadata">

**Author:** ![Codrin\_Costea](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@Codrin\_Costea](https://discuss.elastic.co/u/Codrin_Costea)\
**Post date:** [March 12, 2018, 1:49pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/5 "2018-03-12T13:49:33Z")

</div>

Not sure if it is with //. I think this is not the problem because it reads some logs but others not. That's what I am trying to find.

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [March 12, 2018, 1:55pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/6 "2018-03-12T13:55:27Z")

</div>

I am not sure but this [thread](https://discuss.elastic.co/t/want-to-process-multiple-files-that-exist-in-one-folder/28384/8) might help?

---

<div class="post-metadata">

**Author:** ![Codrin\_Costea](https://avatars.discourse-cdn.com/v4/letter/c/e68b1a/32.png) [@Codrin\_Costea](https://discuss.elastic.co/u/Codrin_Costea)\
**Post date:** [March 12, 2018, 2:47pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/7 "2018-03-12T14:47:29Z")

</div>

Yes indeed ! Apparently this was the problem.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2018, 2:47pm UTC](https://discuss.elastic.co/t/load-old-data-to-elasticsearch/123557/8 "2018-04-09T14:47:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.