Pipeline question

Elastic Stack Elasticsearch
1

Hi all,

I have a field in my index that is named "Severity". The values can be either 2, 1 or 0. I want to rename those values to Critical, Warning or Informational.

I looked into pipelines and also read the documentation on that, but I don't really know what I should do or use. Any ideas on this?

2

Welcome!

You need to reindex everything in a new index.
How did you index your data the first time?

3

Hi, thanks!

I am very very new to Elasticsearch, so I hope I explain myself right. I upload the data with a CSV file the first time, and now I also created a PowerShell script that uploads the same data. I previously made some pipelines to lowercase some fields and transform some date field, and I indeed reindexed after that.

The data is being indexed by a index template I made, and the Severity field is mapped as a keyword.

I hope I answered your question by this, otherwise please let me know.

4

It would be easier if you can modify the CSV file or your PowerShell script.

But you can also add a script processor which transforms the values you have to a text. Have a look at Script processor | Elasticsearch Guide [7.15] | Elastic

5

Thanks, I did realise that I can adjust my script, and that's an option. But I am also trying to learn, so I am very curious to know how I could to this in Elasticsearch if I couldn't control the datasource, which I'm sure will be the case in the future with other data.

I looked at your link, and find it very hard to follow. But I will see if I can find some further information about that. Thanks so far.

6

Yes. You can create an ingest pipeline and make it the default pipeline for a given index.

Agreed. Writing a Script with painless is not as straightforward as the other processors. :slight_smile:

7

Just a Drive By Thought.... :slight_smile:

Or you can just add a runtime field and skip the whole reindex process.

Just add a runtime field with an if / else block based on the code and emit the value you want :slight_smile:

Your script would look something like (I did not check for syntax)

PUT my-index-000001/
{
  "mappings": {
    "runtime": {
      "serverity_code": {
        "type": "keyword",
        "script": {
          "source": "if (doc['severity'].value.equals('2') {emit('Critical');}
        else if (doc['severity'].value.equals('1') {emit('Warning');}
        else {emit('Informational');}"
        }
      }
    }
  }
}
2 Likes
8

I just forgot about this very nice feature :grin:

9

Will this also work for a lence dashboard? For example a pie chart? Now I see 0/1/2 in the pie and I would like to have the names there.

10

It gives me this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "Failed to parse content to map"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Failed to parse content to map",
    "caused_by" : {
      "type" : "json_parse_exception",
      "reason" : "Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value\n at [Source: (ByteArrayInputStream); line: 7, column: 79]"
    }
  },
  "status" : 400
}

I tried to resolve it myself by searching this forum and Google, it seems that is has something to do with line breaks but so far I was not able to resolve it. I will continue trying, but if you have any idea in the meantime, please let me know!

11

Yes.

12

What exactly did you run?

13 
PUT alert_data/
{
  "mappings": {
    "runtime": {
      "severity_code": {
        "type": "keyword",
        "script": {
          "source": "if (doc['severity'].value.equals('2') {emit('Critical');
          }
          else if (doc['severity'].value.equals('1') {emit('Warning');
          }
          else {emit('Informational');
          }
        }
      }
    }
  }
}
14

Try this:

DELETE alert_data
PUT alert_data/
{
  "mappings": {
    "properties": {
      "severity": {
        "type": "keyword"
      }
    }, 
    "runtime": {
      "severity_code": {
        "type": "keyword",
        "script": {
          "source": """if (doc['severity'].value.equals('2')) { emit('Critical'); }
          else if (doc['severity'].value.equals('1')) { emit('Warning'); }
          else { emit('Informational'); }"""
        }
      }
    }
  }
}
POST alert_data/_doc
{
  "severity": "2"
}
POST alert_data/_doc
{
  "severity": "2"
}
POST alert_data/_doc
{
  "severity": "1"
}
POST alert_data/_doc
{
  "severity": "0"
}
GET alert_data/_search
{
  "size": 0,
  "aggs": {
    "sev": {
      "terms": {
        "field": "severity_code"
      }
    }
  }
}

It gives:

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "sev" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "Critical",
          "doc_count" : 2
        },
        {
          "key" : "Informational",
          "doc_count" : 1
        },
        {
          "key" : "Warning",
          "doc_count" : 1
        }
      ]
    }
  }
}
1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.