Loading logs, located in one machine into elasticsearch, setup in another machine, using logstash


(Gautam) #1

Hi,

I have my logs and logstash running on the one EC2 machine (M1), so I read my logs placed on my local machine with this config:

input { 
    file{
        path => "/path/to/logs/in/M1"
        start_position => "beginning"
    }
}

Now, we have elasticsearch running on a different EC2 machine (M2) and I need to transfer the logs from M1 to elasticsearch in M2 using logstash. I used the following output config:

output {
    stdout { codec => rubydebug }

    elasticsearch {
            hosts => "http://<M2 ip address>:9200"
        index => "logstash-%{+YYYY.MM.dd}"
    }
}

When I run the config file, I get the following error:

04:18:57.640 [[main]>worker0] WARN  logstash.outputs.elasticsearch - UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
04:18:57.646 [[main]>worker0] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>2}
04:18:59.682 [[main]>worker0] WARN  logstash.outputs.elasticsearch - UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
04:18:59.686 [[main]>worker0] ERROR logstash.outputs.elasticsearch - Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
04:19:01.109 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:188] WARN  logstash.outputs.elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x1d08c988 URL:http://10.60.40.120:9200>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://10.60.40.120:9200][Manticore::ConnectTimeout] connect timed out"}
04:19:02.111 [Ruby-0-Thread-17: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-5.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:188] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:url=>#<URI::HTTP:0x55444fcf URL:http://10.60.40.120:9200>, :healthcheck_path=>"/"}

I am new to logstash. Any help is appreciated.


(Mark Walkom) #2

Can you curl <M2 ip address>:9200 and telnet <M2 ip address>:9200?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.