Logstash querying elasticsearch timeout error

[willsy]

Hello,

I have the following error; just seeing if anyone knows where i am setting this? i originally put the timeout setting in the testpipeline.conf for logstash. Any help is greatly appreciated

[2023-07-24T11:59:41,350][WARN ][logstash.inputs.elasticsearch][main][3b44aa36a60d114757681112ebc1cdb657d6c631c725cd9f44a1ad88081ecd1a] Attempt to run query but failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Read timed out"}
[2023-07-24T11:59:41,371][ERROR][logstash.inputs.elasticsearch][main][3b44aa36a60d114757681112ebc1cdb657d6c631c725cd9f44a1ad88081ecd1a] Tried run query unsuccessfully {:message=>"Read timed out", :cause=>#<Java::JavaNet::SocketTimeoutException: Read timed out>}

[carly.richmond]

Hi @willsy,

@willsy

Tried run query unsuccessfully {:message=>"Read timed out", :cause=>#<Java::JavaNet::SocketTimeoutException: Read timed out>}

It looks like Logstash is unable to read from the configured source. Can you share your config and where you are reading from?

[willsy]

Hey @carly.richmond certainly will do. here it is. i am literally just trying to get anything at the moment

input {
 elasticsearch {
 hosts => "localhost:9200"
 ssl_enabled => true
 ssl_verification_mode => none
 api_key => "xxxxxxxxxxxxxxxxxxx"
 index => "*logs*"
 query => '{ "query": { "query_string": { "query": "*" } } }'
 size => 5
 scroll => "5m"
 docinfo => true
 docinfo_target => "[@metadata][doc]"
}
}

output {
  tcp {
host => "A.B.C.D"
port => ABCD
  }
}

[widhalmt (Thomas Widhalm)]

Can you try using curl to query this Elasticsearch instance from the Logstash host? Maybe there's a simple network connection problem.

[willsy]

Morning,

Yea we could, curling was fine. Everything that i was testing was working as expected. I had to create a netcat listener on my tower and send the logs that way to determine the formatting of the logs.

My issue was actually the json codex, for some reason it didnt like the default so i had to change it to json_lines which seemed to sort it out.

I can only assume, (assumptions make an ass out of you and me) that the buffer on my external side couldnt handle the logs as it was coming in in batches and not smaller individual lines. When it came in in individual lines it seemed to work.

so my fix was to add codex => json_lines