Not seeing errors is a good thing...
The detection engine really tries to bubble up all and every error it encounters.
I am wondering if your rule is not catching things within the last 5 minutes every time it is run? You said you test ran it within timeline but timeline will look by default through all the indexes that are configured in your advanced settings and it has defaults for those:
Kibana → Management → Advanced Settings →
As a trouble shooting step I would create a test space (or use your default space if it does not impact others using the SIEM) and change your advanced settings to just use that particular index exactly as you have it defined in your custom signal. Test some queries within timeline that are very broad such as
host: *, and then see if you can create a signal that operates with that broad query first.Regardless of it if it works or not, shut it down once done as you don't want broad wild card queries operating against your data sets.
That at least will narrow it down a bit to determine where the culprit might be. If for some reason you misnamed an index when creating a rule (for example) it will not show errors for that particular case. It will just see something does not exist and tell you, yeah there are no signals here.