Creating and testing a logstash pipeline in Kibana?

Elastic Stack Kibana
1

I have an Elastic Cloud instance running Elasticsearch/Kibana 6.5 and I'm trying to use the new Management > Logstash > Pipelines panel to define a pipeline that runs on a schedule to update a field on each document in one of my indexes.

Now, I'm struggling to understand how to write this pipeline to begin with as I can't find any useful examples to build off of, but right now I can't even tell if this pipeline is running or having errors, etc. I can see that it was created but that's it. I can't figure out how to test it or run it or see what might be wrong.

Simply put, I don't really know where to start with using logstash pipelines. Could anyone possibly point me to a good resource for learning to write pipelines and ideally how to interact with them via Kibana?

Sorry if this isn't the best place to ask this -- I was on the fence about whether to ask here or in the logstash forum.

2

Hi Jeff,

I'd say the first thing you should do is know what specifically you need Logstash to do and how you'll do it. Get your pipeline working locally first, as it's easier to debug your pipeline and LS config. The Pipelines Management UI is great for administering centrally-managed pipelines and making small changes, but it's not great for development.

Check out the Logstash docs to plan what you'll want your pipeline to do. Configure it locally and get it working. After that you can configure Logstash to use central management. You can post back here again if you have further questions.

One more thing to consider - it's possible that Logstash isn't the most relevant tool for your use case. If you want to schedule a task, it may be more useful to use Watcher with the Reindex API.