Hey David, this is awesome! I don't work directly on Observability, so I think someone from that team will be able to provide better feedback on most of your items. However, one item stood out to me:
Let me define a "drop" processor
Similar to above, but instead of grok, we'd just be telling the system that any logs matching our criteria can be dropped.
In Stack Management there's an Ingest Node Pipelines UI which allows you to define and edit pipelines for processing ingested documents. There's a neat debug feature so you can test it out with sample docs and ensure it does what you want. One of the processors you can choose is the drop processor which seems to behave the way you describe. Does this give you what you're looking for? If not can you help me understand what you need and what's missing? Thanks again!