Can Kibana mutate documents?

Elastic Stack Kibana
1

I am building a dashboard to track and correlate exceptions and events in a desktop application. Previously, I was using Visual Studio App Center, but since they enforce limits on the number of events per day, I switched to ELK.

However, one feature I find lacking is the ability to modify or mutate documents in Kibana, for instance to mark an exception, and all matching documents, as "Fixed" so they no longer show up in my dashboard. Are there any open-source solutions that do this, or would I have to build my own frontend?

I am aware that APM is a thing, but I am unable to use the APM exporter, so it's off the table.

2

Logstash can do what your after.

3

I don't think Logstash can do what Trym is asking? They're looking for a way to update docs which are already in Elasticsearch and that they're viewing in Kibana.

As far as I know, there's nothing in Kibana to support this case.

4

Hello Trym,

Can you explain your usecase a bit more? You have a desktop application which runs on many computers and in the case of an error you write the exception and other events to ElasticSearch, correct?

What do you do to fix them? Do you have your project in a version control where you check in your changes(e.g. git)?

My best guess would be to hook into your development process. As an example for GIT:
You could structure your commit message and add the ID of the error to it and add a commit hook to git (e.g. using GitHub - typicode/husky: Git hooks made easy 🐶 woof!). This hook could call the ElasticSearch API to update all documents and add a "fixed" flag to all documents with this error ID.
If you are using a source control with web api(Gitlab, Github, ...) you could also create a LogStash pipeline polling the API and updating all documents which contain an error ID which is mentioned in closed issues.

Best regards
Wolfram

5

Hello Wolfram,

Yes, that is correct. I use GitHub for version control, so this is a great suggestion!

My previous solution created a new issue on GitHub for each unique exception, and marked the related documents as closed once the issue was closed.

To replicate this in Elasticsearch, I could create a watcher based on a webhook, which integrates with Logstash and Elasticsearch.

Thanks!

6

Thanks for confirming. Out of curiosity, is this something that could be implemented in a future release? For instance, to add a field to every document which matches a row in a data table or a query in the Discover page?

7

Damn. Just saw that Watcher requires a gold tier subscription or better, sadly outside of my budget.

I'm kind of scared of updating documents in my index based on commit messages alone. I accidentally enter a wildcard and the entire thing blows up :stuck_out_tongue:

I could probably replicate the watcher behavior by creating a proxy daemon. I dunno if this violates the Elastic license though.

Edit:
Looks like this already exists,. GitHub - morningconsult/go-elasticsearch-alerts: Elasticsearch Alerting Daemon. Will create a daemon which takes the output from go-elasticsearch-alerts, and creates an issue on GitHub.

If an issue on GitHub, with a matching key from the DB, is closed, modify Logstash pipeline and add a field to every document marking them as ignored/closed.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.