Unable to retreive data from index using elasticsearch input plugin in logstash

    input {
  elasticsearch {
    hosts => [""]
    index => ["heartbeat-*"]
   # query => '{ "query": { "match_all": {} } }'
   # query => '{ "query": { "bool": {"must" : { "term" : { "monitor.status" : "down" } } } } }'
   # query => '{ "query": { "query_string": { "query": "*" } } }'
   # size => 500
   # scroll => "5m"
    #docinfo => true
output {
   # if "_grokparsefailure" in [tags] {
  stdout  { codec => "rubydebug" }
 # }
    elasticsearch {
    hosts => [""]
    manage_template => false
    index => "tp"
   # document_type => "new-type"

This is my logstash configuration

Im getting following error:

[2019-05-16T16:15:59,334][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Elasticsearch index=>"heartbeat-*", hosts=>[""], id=>"49529869826fbe6db5a4ea023f4022b6ebb73e17e4ff1830d2ffb45b9b292152", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_ef14a6ee-dfb5-4070-9a52-faa253c049c8", enable_metric=>true, charset=>"UTF-8">, query=>"{ \"sort\": [ \"_doc\" ] }", size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
  Error: [411] <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The requested URL could not be retrieved</TITLE>
<STYLE type="text/css"><!--BODY{background-color:#ffffff;font-family:verdana,sans-serif}PRE{font-family:sans-serif}--></STYLE>
<H2>The requested URL could not be retrieved</H2>
<HR noshade size="1px">
While trying to process the request:
GET /heartbeat-%2A/_search?scroll=1m&amp;size=1000 HTTP/1.1
Content-Type: application/json
User-Agent: Faraday v0.9.2
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close
Content-Length: 22

The following error was encountered:
Invalid Request

Some aspect of the HTTP Request is invalid.  Possible problems:
<LI>Missing or unknown request method
<LI>Missing URL
<LI>Missing HTTP Identifier (HTTP/1.0)
<LI>Request is too large
<LI>Content-Length missing for POST or PUT requests
<LI>Illegal character in hostname; underscores are not allowed
<P>Your cache administrator is <A HREF="mailto:root">root</A>. 

<BR clear="all">
<HR noshade size="1px">
Generated Thu, 16 May 2019 09:00:09 GMT by TGPROXY-Backup (squid/2.6.STABLE6)

  Exception: Elasticsearch::Transport::Transport::Errors::LengthRequired
  Stack: /home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:202:in `__raise_transport_error'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:319:in `perform_request'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:200:in `do_run'
/home/abc/elk/logstash-6.5.3/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:188:in `run'
/home/abc/elk/logstash-6.5.3/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
/home/abc/elk/logstash-6.5.3/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'

There is a squid proxy between you and Elasticsearch. Is there any indication in the Elasticsearch logs that the request is reaching it?

No there is no indication in elasticsearch logs

That makes me think the problem is in the proxy. Can you not bypass the proxy?

Thanking for you reply but i didnt understand what bypassing proxy mean

You are connecting to and your post suggests that is a Squid proxy, not an elasticsearch instance. I am suggesting you should connect directly to elasticsearch, not to squid.

Ok understood thank you so much

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.