Access Field in Ingest-Pipeline

Elastic Stack Elasticsearch
1

Hello Everyone,

I'm currently trying out Ingest-Pipelines, but I'm stuck because of some "field access" issues. My Pipeline is the following.

POST _ingest/pipeline/_simulate
{
"pipeline" : {
"description" : "pipeline for splitting hashes stored in the event.Hash field",
"processors" : [
{
"kv" : {
"field": "event_data.Hash",
"field_split": ",",
"value_split": "=",
"target_field": "event_data",
"ignore_missing": true
},
"set": {
"if": "event_data.SHA256 == 'A993F8C574E0FEA8C1CDCBCD9408D9E2E107EE6E4D120EDCFA11DECD53FA0CAE'",
"field": "suspicious",
"value": "true"
},
"remove": {
"field": "event_data.Hash",
"ignore_missing": true
}
}
]
},
"docs": [
{
"_source" : {
"event_data" : {
"Hash" : "SHA1=9FA11A63B43F83980E0B48DC9BA2CB59D545A4E8,MD5=D7B20F933BE6CDAE41EFBE75548EBA5F,SHA256=A993F8C574E0FEA8C1CDCBCD9408D9E2E107EE6E4D120EDCFA11DECD53FA0CAE,IMPHASH=D989D7ADF6957F1A88BB1332E40317E6"
}
}
}
]
}

Unfortunatly my document isn't marked as "suspicious". I tried using the set-processor "inline" like mentioned here and putting it into a different pipeline. Both didn't work.

2

Any ideas, why the pipeline isn't working as intended?

3

I just tested this on 7.0 and if I start the if part in the set processor with "ctx.event_data.SHA256 ... it works for me[tm].

What version are you on?

--Alex

4

Thanks for the response. I'm using Elasticsearch 7.0.0 and unfortunatly the document isn't marked as "supicious". This is my response if I run the pipeline:

{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"event_data" : {
"SHA256" : "A993F8C574E0FEA8C1CDCBCD9408D9E2E107EE6E4D120EDCFA11DECD53FA0CAE",
"SHA1" : "9FA11A63B43F83980E0B48DC9BA2CB59D545A4E8",
"MD5" : "D7B20F933BE6CDAE41EFBE75548EBA5F",
"IMPHASH" : "D989D7ADF6957F1A88BB1332E40317E6"
}
},
"_ingest" : {
"timestamp" : "2019-05-06T08:09:44.740576Z"
}
}
}
]
}

5

Please take your time to properly format and indent your messages. Because this is really the issue here. The set processor is within the script processor.

Check this example

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "pipeline for splitting hashes stored in the event.Hash field",
    "processors": [
      {
        "kv": {
          "field": "hash",
          "field_split": ",",
          "value_split": "=",
          "target_field": "output",
          "tag": "kv"
        }
      },
      {
        "set": {
          "if": "ctx.output.SHA256 == '1'",
          "field": "suspicious",
          "value": "true",
          "tag": "set"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "hash": "SHA1=2,MD5=3,SHA256=1,IMPHASH=4"
      }
    }
  ]
}

The set processor now is its own element with in the processors array.

Nontheless, an error message might have been useful here. I will open an issue.

6

Ahhh I see the problem now. I just overlooked the fact that I had to put each processor in it's own curly brackets, even tho it is clearly stated in the documentation.

Thanks for your help.