Watcher: Error 500 Internal Server Error: An internal server error occurred

Hi there, I have just installed the 6.0 update and when navigating to my watchers I am receiving an internal server error, does anybody know a fix for this? Or any possible reasons why I might be receiving this error?

Thank you in advance,

Jason

Hey,

Are there any logs that go with the 500 error? Was this an upgrade to 6.0 or a fresh install?

Hi Jon,

This is an upgrade to 6.0. I know that there are breaking changes for the watchers after this update but I can't access them to make the updates.

Some logs:

[2017-11-30T09:49:14,164][ERROR][org.elasticsearch.xpack.watcher.input.search.ExecutableSearchInput] failed to execute [search] input for watch [Compromised_Host_Detected], reason [all shards failed]

[org.elasticsearch.xpack.monitoring.exporter.http.VersionHttpResource] version [5.6.2] < [6.0.0-alpha1] and NOT supported for [xpack.monitoring.exporters.found-user-defined]

Could this error be caused by version mismatches?

I know that Elasticsearch is v6.0. Kibana is v6.0 (managed in the cloud). I have run a sudo yum update on my ec2 instance that is running Logstash and it is fully updated.

Jason

Monitoring creates watches and it sounds like the monitoring cluster is on 5.6 still, possibly causing UI errors. You mentioned managed in the cloud, are you using Elastic Cloud? If so can you open a support ticket at support.elastic.co? They'll be able to take a look at the versions upgraded.

Hi @jbudz

I took this up with the elastic cloud team and we have managed to get all of my versions upgraded to 6.0 but this still hasn't fixed the internal server error that I am facing.

Do you know if there is any other reasons why I could be getting this error?

Jason

I also have this exact issue.

New GCP cluster on 6.0 (now on 6.0.1). Trying to access the watches in management gives me the same 500 error code.

Watches still fire, and I can create/modify existing watches through the console without issue (which is what I have been doing).

Thanks,
James

Hi @jamesspi,

It's been reported that if the watches have an email action, but the action does not contain a body property then it will cause this sort of error.

Or if you haven't specified your 'watch' logging action text.

This is the feedback I got from the cloud team.

Jason

Hey @Jasonespo

I'm actually using slack notifications, not e-mail, so not sure this is valid in my case.

Thanks though!

James

@Jasonespo - actually, thanks! This led me to check my slack settings - I didn't have the message text set, just the attachment text. Added this to all my watches and it works now!

@jamesspi

No worries :smile:

I have over 150 watchers to look through.. I'm not sure the exact config of each one..

Is there an easy way to view them? I have them saved in GitHub but I would have to pull them in and manually run them one by one to update the watchers in management.

Jason

@Jasonespo

Thankfully, I only had a handful. I run:

GET .watches/_search/ - this will list your watches and their metadata. You can grab the watch configs out of their, add the missing info and resubmit them. This is what I did.

Might be a pain with 150 though!

James

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.