Hey @stephenb, yes it seems you have a pretty good understanding of the situation.
The alerts created through the metric explorer were all working as intended. However, we create our rules through the kibana API due to an integration with our react app. Looking at the rule from the UI, it looked alright since it had been showing the filterQueryText. Though underneath, it was missing a filterQuery value so in reality it wasn't filtering at all. I would think that the filterQuery would be derived from the filterQueryText since the Kibana didn't complain the missing field but that doesn't seem to be the case
As always, thank you so much for taking the time to look at this issue with me