How to update all docs with query that looks for a regex and update it incase found

Elastic Stack Elasticsearch
1

Hi,

I'm using 7.16.2.

I have a lot of records looks like this

{
        "_index" : "testcases",
        "_type" : "_doc",
        "_id" : "Cv6z8HoBF-BJEaLQYGtU",
        "_score" : 1.0,
        "_source" : {
          "buildVersion" : "1.16.3-alpha+9950",
          "testcase" : {
            "$" : {
              "name" : "/opt/fireglass/1.16.3-alpha+9950/regression_tests/out/cases/activity_logs/basic_activity_logs_search.js - Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation",
              "time" : "14.583",
              "className" : "Contains a log with events 'Network Request' and 'Forward To Isolation"
            }
          },
          "testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation" (18),
          "testcaseStatus" : "Passed",
          "timestamp" : "2021-07-29T05:18:45.071Z"
        }
      }

In the testcaseName field there is an index in brackets, for example "(18)".

I want to run on all documents in the DB on testcaseName field and search for testcaseName that contain this index and remove it.

For example:
"testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation" (18)
Will become:
"testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation"

Just remove the "(18)"

I did it in my code to prevent it happen again, but I need to fix the DB as well.
javascript code:

let newTestName = testName;
    // Find number in parenthesis with space (if exist) before as example: " (33)"
    const match = testName.match(/\s?\([0-9]+\)/gm);

    if (match !== null) {
        // Get only the last finding
        const matchPattern = match[match.length - 1];
        const endIndex = testName.lastIndexOf(matchPattern);
        newTestName = testName.substring(0, endIndex);
    }

How to do the same in Elastic? A query in Dev Tools or should I write code in order to that?
If there is a way to do it from Dev tools let me know how?

Thanks,
Shay

2

The doc recognized by ES is JSON，but this is not a json，How do you save it into es？

"testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation" (18),
3

I build a json in the code and save it to ES

4

So (18) should be in front of " ？

5

I've fixed the code part, so this won't be a problem in the future, my question is how to fix the existing data in Elastic

6

so you want to update data in es
from:

"testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation (18)"

to:

"testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation"

do you ? In es, (18) must be in front of "

7

Yes, the result you show this is my main goal

8

use script pipeline to update your data in es，as below, You can write your own script to suit your specific situation

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """
            if(ctx.testcaseName.endsWith(" (18)")){
              ctx.testcaseName=ctx.testcaseName.substring(0,ctx.testcaseName.lastIndexOf(" (18)"));
            }
          """
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation (18)"
      }
    },
    {
      "_source": {
        "testcaseName" : "Basic activity log search Activity log search after browsing Contains a log with events 'Network Request' and 'Forward To Isolation xxxx"
      }
    }
  ]
}
9

First of all, thank you very much for your response.
I see you wrote _simulate, so this script just do a simulation? If I really want to change the data I use _update instead?

10

you can follow this DOC, use update API with pipeline or script

11

How do I run this script on index? for example my index is: testindexshay.. ?

I've tried:

PUT _ingest/pipeline/testindexshay
{
  "pipeline": {
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": """
            if(ctx.testcaseName.endsWith(" (22)")){
              ctx.testcaseName=ctx.testcaseName.substring(0,ctx.testcaseName.lastIndexOf(" (22)"));
            }
          """
        }
      }
    ]
  }
}

and got this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "[processors] required property is missing",
        "property_name" : "processors"
      }
    ],
    "type" : "parse_exception",
    "reason" : "[processors] required property is missing",
    "property_name" : "processors"
  },
  "status" : 400
}

How do I run on all docs in this index?

12

use update_by_query API with pipeline:

13

I have used this script:

POST /testindexshay/_update_by_query
{
         "script": {
          "lang": "painless",
          "source": """
            if(ctx.testcaseName.endsWith(" (22)")){
              ctx.testcaseName=ctx.testcaseName.substring(0,ctx.testcaseName.lastIndexOf(" (22)"));
            }
          """
}

and got this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : [
          "if(ctx.testcaseName.endsWith(\" (22)\")){\n              ",
          "                   ^---- HERE"
        ],
        "script" : " ...",
        "lang" : "painless",
        "position" : {
          "offset" : 32,
          "start" : 13,
          "end" : 67
        }
      }
    ],
    "type" : "script_exception",
    "reason" : "runtime error",
    "script_stack" : [
      "if(ctx.testcaseName.endsWith(\" (22)\")){\n              ",
      "                   ^---- HERE"
    ],
    "script" : " ...",
    "lang" : "painless",
    "position" : {
      "offset" : 32,
      "start" : 13,
      "end" : 67
    },
    "caused_by" : {
      "type" : "null_pointer_exception",
      "reason" : "cannot access method/field [endsWith] from a null def reference"
    }
  },
  "status" : 400
}
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.