Creating a pipeline which runs with particular interval on same index

Elastic Stack Kibana
1

Can anyone please tell me, if this is possible ?

I am trying to create a pipeline but I want it to run in particular intervals on the same index

For example :

{
  "description":"Find the Progress",
  "processors":[
    {
      "set":{
        "field":"_source.now",
        "value":"{{_ingest.timestamp}}"
      }
    },

    {
      "script":{
        "lang":"painless",
        "source": """
        ctx['SubmitDate']=ctx['SubmitDate'].substring(0,10)+'T'+ctx['SubmitDate'].substring(11,16)+':00Z';
        if(ctx.containsKey("now") && ctx.containsKey("SubmitDate"))
          {
            ctx['daysleft']=ChronoUnit.DAYS.between(ZonedDateTime.parse(ctx['now']),ZonedDateTime.parse(ctx['SubmitDate']));
        if(ctx['daysleft']>0)
          {
            ctx['Progress']="In Progress";
          }
        if(ctx['daysleft']<=0)
          {
            ctx['Progress']="Completed";
          }
        }
        """
      }
    }
  ]
}

In the above example I want to refresh the Progress field for every day using some calculation.

I am not able find about this any where, I am just curious if that's possible using pipeline and also , I don't know if this is the right way to refresh the fields with some calculation.

Please suggest any methods to perform such calculation and refresh.

2

Hey,

a pipeline is only executed, before a document is indexed, so you would need to run an update by query and reindex all documents. I don't think that is the best approach, so maybe, if you elaborate (without talking about any Elasticsearch/Kibana features), why you need this and what sense this makes for you, maybe we can come up with another solution.

Thanks!

--Alex

3

Hi Alex,

I have a field called "SubmitDate" and I want to find the difference between the submit date and present date which should be stored in a new variable "daysleft".

Now, I want to see the latest updated value of daysleft whenever I want.

Thanks..

4

Hey,

how about a scripted field or a runtime field? See Retrieve selected fields from a search | Elasticsearch Guide [7.13] | Elastic or more recent Runtime fields | Elasticsearch Guide [7.13] | Elastic (which can also be configured in Kibana in the most recent release) and used for searches.

--Alex

5

Hi Alex,

Thanks for guiding me through this, What should I be using in the case of Index pattern ?

I think Runtime fields would be perfect, does that work with index pattern ?

Thanks

6

This should help: Manage index pattern data fields | Kibana Guide [7.13] | Elastic

7

Hi Alex,

Thank you so much for helping me.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.