How to create a new field combining values of other field values

Hi All,

I am using elk version 7.2

I have two keywords

1- result
2 - service

using the values of these keywords i want to create another field which will met the condition and keep the data according to the condition in the new field.

My logstash condition for this is

  if [result] == "Accepted" and "Rejected" and [service] == "merged" 
{
  mutate {
   add_field => [ "result1" ]
   }
}

but i am getting this errors while starting the logstash after writing this condition.

[2020-06-22T14:27:19,361][ERROR][logstash.filters.mutate  ] Invalid setting for mutate filter plugin:

  filter {
    mutate {
      # This setting must be a hash
      # This field must contain an even number of items, got 1
      add_field => ["result1"]
      ...
    }
  }

Can anyone please help me out to resolve these issues

Thanks in advance

To add a field you have to configure the key and the value. Therefore it has to be a hash. An array with one value is not enough.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-add_field

Hi Jenni,

Thanks for the response , I have read this link when googled the error but not able to understand correctly.

Can you do the changes in my code to give me the example

The link shows you the exact structure of that setting. But to say it even more clearly:

mutate {
  add_field => { "This is your key/field name" => "This is your value/data/content of the field" }
}

It is unclear what you want to test here, but I very much doubt that is testing what you want.

I have two values "Accepted" and "Rejected" in the [result] keyword.

And one value "merged" in the [service] keyword.

Adding these 3 values from these 2 keywords , I want to create a new field along with the data those values hold

Hi,

I understood the field name part,now my doubt is in content of the field

after assigning the field name i thought the values from the keywords will also get copied to the newly created field

which isn't happening here

Logstash can't read your mind and you never told it to copy any values from one field to another. Do you want to build an array with the content ["Accepted", "Rejected", "merged"] ? I still don't understand what your goal is. Maybe you could post the original event data and what you expect the output to look like.

if [result] == "Accepted" and "Rejected" and [service] == "merged"

This tests whether three things are true

  1. [result] == "Accepted" -- This is a string equality test of the value of the result field
  2. "Rejected" -- I believe this will always be true
  3. [service] == "merged" -- This is a string equality test of the value of the service field

If you want to test whether [result] is either "Accepted" or "Rejected" then you can use an array membership test

if [result] in [ "Accepted", "Rejected" ] and [service] == "merged"

Here as you can see under the result.keyword ,I have

identity accepted and identity rejected

whether in service.keyword i have identity merged

But as per my requirement i need all 3 values within a single keyword

@Jenni are you able to figure out my requirement from the last post?

If the value is actually "Identity Accepted" your conditions wouldn't work anyway because you are querying an exact match with the string "Accepted". You'd need a regex like /Accepted$/ (and are there even any other values than these two for the result?)
You could just post the original event and the result you'd wish to achieve in json format. Then everything would be clear. Will that new field have the values ["Identity Accepted", "Identity merged"] or ["Identity Rejected", "Identity merged"] or did you want to concenate the strings? Either way the "3 values" will always only be two of them (the two combinations I mentioned earlier), right?

well the new field should contain all the 3 strings

"Identity Accepted " , "Identity merged" and "Identity Rejected".

apologies for my previous query where i mentioned the query with "OR" operation which is actually "AND" operator

Any help for the above issue?

You should follow advice from the last response fron Jenni and post the exact input event(s) and the output you expect. Redact whatever you must to, but keep the original structure for result and service and your expected output field.

I presume it will be the only unambiguous way to know, both for the input and output, whether you work with concatenated strings, objects, arrays, etc.

if I'm not wrong your result field contains one of "Accepted" or "Rejected"
if it is the case then your condition should be like this

if [result] == "Identity Accepted" or [result] == "Identity Rejected" and [service] == "Identity merged" {
                mutate {
                        add_field => {"result1" => "%{result}, %{service}"}
                }
}

and if you want the values in array format then you can use mutate-split filter

mutate {
split => {"result1" => ", "}
}

Hi @chitreshg thanks a lot for the reply but the filter wasn't working as expected

It only picks the "if [result] == "Identity Accepted" and the correspondent value's service keyword in the newly created field "result1".

this is how the result1 field looks in kibana visulization

result1.keyword: Descending Count
Identity Accepted, FIRST_REQUEST

where first request is the service keyword value for the identity accepted value

@chitreshg if you found any solution to this issue please update

thanks in advance

You would be much more likely to get a solution if you would show us what your data looks like (either JSON or rubydebug), and what you want the result to look like.

Hi @Badger here is the json data of each value --- Identity Accepted , Identity merged and Identity Rejected

{

"_index": "icp2-2020.06.03",
"_type": "_doc",
"_id": "acbb85b3b45400961be54c2b6362b59844351f3b4dcfed95a038cb2d5979a176",
"_version": 1,
"_score": 4.018989,
"_source": {
"ENTITLEMENT_TYPE": "value_100",
"AT_VAL6": "",
"AT_VAL4": "",
"internalId": "1806",
"AT_CUSTO_S5": "",
"result": "Identity Accepted",
"AT_VAL2": "",
"service": "FIRST_REQUEST",
"host": "",
"AT_CUSTO_S1": "",
"USER_TYPE": "value_90",
"LOG_QUALIFIER": "USER_TASK_RESULT",
"@timestamp": "2020-06-03T11:45:01.893Z",
"AT_EXTENDEDINFO": "",
"logCategory": "BUSINESS",
"owner": "anonymous",
"REQUEST_TYPE": "value_60",
"@version": "1",
"headers": {
"accept_encoding": "gzip,deflate",
"request_method": "PUT",
"http_accept": null,
"connection": "Keep-Alive",
"http_host": ":9090",
"request_path": "/",
"content_length": "1041",
"http_user_agent": "Apache-HttpClient/4.5.7 (Java/1.8.0_102)",
"content_type": "application/json;charset=UTF-8",
"http_version": "HTTP/1.1"
},
"AT_CUSTO_S3": "309",
"AT_VAL3": "37",
"originLogId": "afad5535-a58f-11ea-b158-d94038694644",
"AT_CUSTO_S6": "",
"duration": 150,
"timeStamp": "2015-01-01T01:02:03.904Z",
"origin": "MorphoIA",
"AT_OUTSTANDINGVAL": "",
"businessId": "",
"ER_STATE": "value_40",
"ER_ID": "value_70",
"AT_CUSTO_S9": "",
"AT_CUSTO_S10": "",
"ER_STATUS": "value_50",
"AT_VAL1": "SERVICE_TASK_RESULT",
"AT_CUSTO_S7": "",
"AT_VAL5": "TECHNICAL",
"station": "origin",
"AT_VAL10": "",
"AR_STATUS": "value_30",
"AT_CUSTO_S2": ""
},
"fields": {
"timeStamp": [
"2015-01-01T01:02:03.904Z"
],
"@timestamp": [
"2020-06-03T11:45:01.893Z"
]
},
"highlight": {
"result": [
"@kibana-highlighted-field@Identity@/kibana-highlighted-field@ @kibana-highlighted-field@Accepted@/kibana-highlighted-field@"
],
"result.keyword": [
"@kibana-highlighted-field@Identity Accepted@/kibana-highlighted-field@"
]
}
}

{
"_index": "icp2-2020.06.03",
"_type": "_doc",
"_id": "10f76a8d5b991f7523c55c43bb8e330116b07ad99716610aa70cb427a008761d",
"_version": 1,
"_score": 3.9113598,
"_source": {
"ENTITLEMENT_TYPE": "value_100",
"AT_VAL6": "",
"AT_VAL4": "",
"internalId": "1810",
"AT_CUSTO_S5": "",
"result": "QUEUED",
"AT_VAL2": "",
"service": "Identity merged",
"host": "",
"AT_CUSTO_S1": "",
"USER_TYPE": "value_90",
"LOG_QUALIFIER": "USER_TASK_RESULT",
"@timestamp": "2020-06-03T11:45:03.588Z",
"AT_EXTENDEDINFO": "",
"AT_VAL7": "STARTED",
"activity": "CreateApplicantRequest",
"REQUEST_TYPE": "value_60",
"@version": "1",
"headers": {
"accept_encoding": "gzip,deflate",
"request_method": "PUT",
"http_accept": null,
"connection": "Keep-Alive",
"http_host": "",
"request_path": "/",
"content_length": "1012",
"http_user_agent": "Apache-HttpClient/4.5.7 (Java/1.8.0_102)",
"content_type": "application/json;charset=UTF-8",
"http_version": "HTTP/1.1"
},
"AT_CUSTO_S3": "309",
"AT_VAL3": "37",
"originLogId": "b0af0dc9-a58f-11ea-b158-d94038694644",
"AT_CUSTO_S6": "",
"duration": 150,
"timeStamp": "2015-01-01T01:02:03.904Z",
"origin": "MorphoIA",
"AT_OUTSTANDINGVAL": "",
"businessId": "",
"ER_STATE": "value_40",
"ER_ID": "value_70",
"AT_CUSTO_S9": "",
"AT_CUSTO_S7": "",

},
"fields": {
"timeStamp": [
"2015-01-01T01:02:03.904Z"
],
"@timestamp": [
"2020-06-03T11:45:03.588Z"
]
},
"highlight": {
"service": [
"@kibana-highlighted-field@Identity@/kibana-highlighted-field@ @kibana-highlighted-field@merged@/kibana-highlighted-field@"
],
"service.keyword": [
"@kibana-highlighted-field@Identity merged@/kibana-highlighted-field@"
]
}
}

{
"_index": "icp2-2020.06.01",
"_type": "_doc",
"_id": "a87b87052896c5c6a123186d6da9575f97391a7d3fa9b767eec2c251f6f80d60",
"_version": 1,
"_score": 3.585218,
"_source": {
"ENTITLEMENT_TYPE": "value_100",
"AT_VAL6": "",
"AT_VAL4": "",
"internalId": "1801",
"AT_CUSTO_S5": "",
"result": "Identity Rejected",
"AT_VAL2": "Male",
"service": "FIRST_REQUEST",
"host": "50.50.1.13",
"AT_CUSTO_S1": "",
"USER_TYPE": "value_90",
"LOG_QUALIFIER": "USER_TASK_RESULT",
"@timestamp": "2020-06-01T11:23:43.735Z",
"AT_EXTENDEDINFO": "",
"logCategory": "BUSINESS",
"@version": "1",
"headers": {
"accept_encoding": "gzip,deflate",
"request_method": "PUT",
"http_accept": null,
"connection": "Keep-Alive",
"http_host": "",
"request_path": "/",
"content_length": "1038",
"http_user_agent": "Apache-HttpClient/4.5.7 (Java/1.8.0_102)",
"content_type": "application/json;charset=UTF-8",
"http_version": "HTTP/1.1"
},
"originLogId": "60505416-a3fa-11ea-b158-d94038694644",
"AT_CUSTO_S6": "",
"duration": 150,
"timeStamp": "2015-01-01T01:02:03.904Z",
"origin": "MorphoIA",
"AT_OUTSTANDINGVAL": "",
"businessId": "",
"ER_STATE": "value_40",
"ER_ID": "value_70",
"AT_CUSTO_S9": "",
"AT_CUSTO_S10": "",

},
"fields": {
"timeStamp": [
"2015-01-01T01:02:03.904Z"
],
"@timestamp": [
"2020-06-01T11:23:43.735Z"
]
},
"highlight": {
"result": [
"@kibana-highlighted-field@Identity@/kibana-highlighted-field@ @kibana-highlighted-field@Rejected@/kibana-highlighted-field@"
],
"result.keyword": [
"@kibana-highlighted-field@Identity Rejected@/kibana-highlighted-field@"
]
}
}