How setting about logstash with proxy connect to es cloud?

Elastic Stack Logstash
1

how setting about logstash with proxy connect to es cloud?
packetbeat can seding logs to es cloud ?
but logstash can't seding system_log in to es cloud.
also use proxy can to es cloud.

logstash logs.

[2017-12-19T12:30:19,138][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/, :path=>"/"}
[2017-12-19T12:30:36,459][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/][Manticore::SocketException] Network is unreachable (connect failed) {:url=>https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/, :error_message=>"Elasticsearch Unreachable: [https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/][Manticore::SocketException] Network is unreachable (connect failed)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2017-12-19T12:30:36,461][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/][Manticore::SocketException] Network is unreachable (connect failed)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2017-12-19T12:30:36,463][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/][Manticore::SocketException] Network is unreachable (connect failed) {:url=>https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/, :error_message=>"Elasticsearch Unreachable: [https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/][Manticore::SocketException] Network is unreachable (connect failed)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
[2017-12-19T12:30:36,466][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch' but Elasticsearch appears to be unreachable or down! {:error_message=>"Elasticsearch Unreachable: [https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/][Manticore::SocketException] Network is unreachable (connect failed)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :will_retry_in_seconds=>2}
[2017-12-19T12:30:38,466][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-12-19T12:30:38,467][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
[2017-12-19T12:30:38,469][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-12-19T12:30:38,470][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
[2017-12-19T12:30:39,169][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/][Manticore::SocketException] Network is unreachable (connect failed)"}
[2017-12-19T12:30:40,171][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic1:xxxxxx@1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/, :path=>"/"}
[2017-12-19T12:30:42,471][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-12-19T12:30:42,472][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-12-19T12:30:42,472][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>8}
[2017-12-19T12:30:42,472][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>8}

packetbeat logs

2017-12-19T12:22:31+08:00 INFO Connected to Elasticsearch version 5.6.5
2017-12-19T12:22:31+08:00 INFO Trying to load template for client: https://1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243
2017-12-19T12:22:32+08:00 INFO Template already exists and will not be overwritten.
2017-12-19T12:22:42+08:00 INFO Non-zero metrics in the last 30s: libbeat.es.call_count.PublishEvents=3 libbeat.es.publish.read_bytes=6065 libbeat.es.publish.write_bytes=7935 libbeat.es.published_and_acked_events=8 libbeat.publisher.messages_in_worker_queues=5 libbeat.publisher.published_events=8
2

i can't see how slove it..

3

is support about logstash connect proxy to es cloud?

4

I don't have an answer myself. I will try to get another Logstash engineer to look at this question.

Patience is necessary, its the holiday season and quite a few people have taken time off.

5

in the mean time, please post your configs, both Logstash and Packetbeat, here.

6

logstash configure

input {  
  stdin { } 
}


filter {
  grok {
    match => {
      "message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} H
TTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) %{QS:referrer} %{QS:agent}'
    }
  }

  date {
    match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale => en
  }

  geoip {
    source => "clientip"
  }

  useragent {
    source => "agent"
    target => "useragent"
  }
}

output {
  stdout {
    codec => dots {}
  }

  elasticsearch {
    hosts => "https://1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/"
    user => "xxx" 
    password => "xxx" 
    index => "apache_elastic_example"
    template => "./apache_template.json"
    template_name => "apache_elastic_example"
    template_overwrite => true
  }
}

packetbeat configure

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "XXX"
  password: "XXX"
7

slove it
logstash output add:
proxy => "xxx.xxx.xxx.xx"
output {
stdout {
codec => dots {}
}

  elasticsearch {
    hosts => "https://1d482615da63165c58e0b465b58398d3.ap-southeast-1.aws.found.io:9243/"
    user => "xxx" 
    password => "xxx" 
    proxy => "xxx.xxx.xxx.xx"
    index => "apache_elastic_example"
    template => "./apache_template.json"
    template_name => "apache_elastic_example"
    template_overwrite => true
  }
1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.