Load old data to ElasticSearch

Elastic Stack Logstash
1

Hello !

I am new to Elastic Search and I would need some help in one problem. I try to load multiple log files from my proxy to Elastic Search, through logstast. The steps that I am following are:

  1. Create the .conf file
  2. Run it with console
  3. Create an index in Kibana

Sometimes it works and loads the logs into Kibana, but the problem is that I don't have all the logs. I am trying to import thousands of logs because I have to analyze some times in my proxy. This is my input field. What commands should I add in order to import all the logs in kibana ?

input
{
file {
path => " ..... "
start_position=> beginning"
}
}

2

Could you share your actual config? also when you paste it here can you use the </> tag to format it please?

3 
   input
    {
    	file{
    			path=> "D:/Codrin/access+errors/*"
    			path=> "D:/Codrin/access+errors/*/*"
    			start_position=>"beginning"
    		}
    }
    filter 
    {
    	if [path]=~"error"{
    		grok{ match=> { "message"=>'%{DATE:timestamp} %{TIME:time} %{DATA:type} %{DATA:code} %{DATA:noneed1} %{DATA:noneed2} %{DATA:info} %{DATA:noneed3} %{DATA:ip} %{DATA:noneed4} %{DATA:server} %{DATA:noneed5} %{DATA:request} %{DATA:reques_url} %{DATA:httpver} %{DATA:noneed6} %{DATA:host} %{DATA:noneed7} "%{DATA:referrer}"'}}
    		} else if [path]=~"access"{
    					grok{ match=> { "message"=>'%{IPORHOST:remote_addr} %{USER:user} %{USER:user-name} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:body_bytes_sent} %{NUMBER:NotSureAboutTheFild} %{NUMBER:ResponseTime} %{QS:notsure} %{QS:agent}'}}
    					}
    }
    output
    {
    	stdout{codec => dots}
    	elasticsearch
    		{
    			hosts=>["localhost:9200"]
    			index=>"testerrorandmultiplefiles"
    		}
    }
4

Thanks,

I think you need to double escape the paths ie: D://Codrin//access+errors//* but I never ever done anything with windows so I could be wrong here :slight_smile:

5

Not sure if it is with //. I think this is not the problem because it reads some logs but others not. That's what I am trying to find.

6

I am not sure but this thread might help?

7

Yes indeed ! Apparently this was the problem.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.