How to create a new field combining values of other field values

Matish_Bhuyan · June 22, 2020, 12:46pm

Hi All,

I am using elk version 7.2

I have two keywords

1- result
2 - service

using the values of these keywords i want to create another field which will met the condition and keep the data according to the condition in the new field.

My logstash condition for this is

  if [result] == "Accepted" and "Rejected" and [service] == "merged" 
{
  mutate {
   add_field => [ "result1" ]
   }
}

but i am getting this errors while starting the logstash after writing this condition.

[2020-06-22T14:27:19,361][ERROR][logstash.filters.mutate  ] Invalid setting for mutate filter plugin:

  filter {
    mutate {
      # This setting must be a hash
      # This field must contain an even number of items, got 1
      add_field => ["result1"]
      ...
    }
  }

Can anyone please help me out to resolve these issues

Thanks in advance

Jenni · June 22, 2020, 1:04pm

To add a field you have to configure the key and the value. Therefore it has to be a hash. An array with one value is not enough.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-add_field

Matish_Bhuyan · June 22, 2020, 1:29pm

Hi Jenni,

Thanks for the response , I have read this link when googled the error but not able to understand correctly.

Can you do the changes in my code to give me the example

Jenni · June 22, 2020, 1:33pm

The link shows you the exact structure of that setting. But to say it even more clearly:

mutate {
  add_field => { "This is your key/field name" => "This is your value/data/content of the field" }
}

Badger · June 22, 2020, 1:37pm

It is unclear what you want to test here, but I very much doubt that is testing what you want.

Matish_Bhuyan · June 22, 2020, 2:18pm

I have two values "Accepted" and "Rejected" in the [result] keyword.

And one value "merged" in the [service] keyword.

Adding these 3 values from these 2 keywords , I want to create a new field along with the data those values hold

Matish_Bhuyan · June 22, 2020, 2:34pm

Hi,

I understood the field name part,now my doubt is in content of the field

after assigning the field name i thought the values from the keywords will also get copied to the newly created field

which isn't happening here

Jenni · June 22, 2020, 2:48pm

Logstash can't read your mind and you never told it to copy any values from one field to another. Do you want to build an array with the content ["Accepted", "Rejected", "merged"] ? I still don't understand what your goal is. Maybe you could post the original event data and what you expect the output to look like.

Badger · June 22, 2020, 3:04pm

if [result] == "Accepted" and "Rejected" and [service] == "merged"

This tests whether three things are true

[result] == "Accepted" -- This is a string equality test of the value of the result field
"Rejected" -- I believe this will always be true
[service] == "merged" -- This is a string equality test of the value of the service field

If you want to test whether [result] is either "Accepted" or "Rejected" then you can use an array membership test

if [result] in [ "Accepted", "Rejected" ] and [service] == "merged"

Matish_Bhuyan · June 22, 2020, 3:15pm

Here as you can see under the result.keyword ,I have

identity accepted and identity rejected

whether in service.keyword i have identity merged

But as per my requirement i need all 3 values within a single keyword

Matish_Bhuyan · June 22, 2020, 5:12pm

@Jenni are you able to figure out my requirement from the last post?

Jenni · June 23, 2020, 7:35am

If the value is actually "Identity Accepted" your conditions wouldn't work anyway because you are querying an exact match with the string "Accepted". You'd need a regex like /Accepted$/ (and are there even any other values than these two for the result?)
You could just post the original event and the result you'd wish to achieve in json format. Then everything would be clear. Will that new field have the values ["Identity Accepted", "Identity merged"] or ["Identity Rejected", "Identity merged"] or did you want to concenate the strings? Either way the "3 values" will always only be two of them (the two combinations I mentioned earlier), right?

Matish_Bhuyan · June 23, 2020, 12:20pm

well the new field should contain all the 3 strings

"Identity Accepted " , "Identity merged" and "Identity Rejected".

apologies for my previous query where i mentioned the query with "OR" operation which is actually "AND" operator

Matish_Bhuyan · June 23, 2020, 6:07pm

Any help for the above issue?

andres-perez · June 27, 2020, 12:01am

You should follow advice from the last response fron Jenni and post the exact input event(s) and the output you expect. Redact whatever you must to, but keep the original structure for result and service and your expected output field.

I presume it will be the only unambiguous way to know, both for the input and output, whether you work with concatenated strings, objects, arrays, etc.

chitreshg · June 29, 2020, 12:21pm

if I'm not wrong your result field contains one of "Accepted" or "Rejected"
if it is the case then your condition should be like this

if [result] == "Identity Accepted" or [result] == "Identity Rejected" and [service] == "Identity merged" {
                mutate {
                        add_field => {"result1" => "%{result}, %{service}"}
                }
}

and if you want the values in array format then you can use mutate-split filter

mutate {
split => {"result1" => ", "}
}

Matish_Bhuyan · June 30, 2020, 11:26am

Hi @chitreshg thanks a lot for the reply but the filter wasn't working as expected

It only picks the "if [result] == "Identity Accepted" and the correspondent value's service keyword in the newly created field "result1".

this is how the result1 field looks in kibana visulization

result1.keyword: Descending	Count
Identity Accepted, FIRST_REQUEST

where first request is the service keyword value for the identity accepted value

Matish_Bhuyan · June 30, 2020, 8:17pm

@chitreshg if you found any solution to this issue please update

thanks in advance

Badger · June 30, 2020, 8:44pm

You would be much more likely to get a solution if you would show us what your data looks like (either JSON or rubydebug), and what you want the result to look like.

Matish_Bhuyan · June 30, 2020, 9:14pm

Hi @Badger here is the json data of each value --- Identity Accepted , Identity merged and Identity Rejected

{
"_index": "icp2-2020.06.03",
"_type": "_doc",
"_id": "acbb85b3b45400961be54c2b6362b59844351f3b4dcfed95a038cb2d5979a176",
"_version": 1,
"_score": 4.018989,
"_source": {
"ENTITLEMENT_TYPE": "value_100",
"AT_VAL6": "",
"AT_VAL4": "",
"internalId": "1806",
"AT_CUSTO_S5": "",
"result": "Identity Accepted",
"AT_VAL2": "",
"service": "FIRST_REQUEST",
"host": "",
"AT_CUSTO_S1": "",
"USER_TYPE": "value_90",
"LOG_QUALIFIER": "USER_TASK_RESULT",
"@timestamp": "2020-06-03T11:45:01.893Z",
"AT_EXTENDEDINFO": "",
"logCategory": "BUSINESS",
"owner": "anonymous",
"REQUEST_TYPE": "value_60",
"@version": "1",
"headers": {
"accept_encoding": "gzip,deflate",
"request_method": "PUT",
"http_accept": null,
"connection": "Keep-Alive",
"http_host": ":9090",
"request_path": "/",
"content_length": "1041",
"http_user_agent": "Apache-HttpClient/4.5.7 (Java/1.8.0_102)",
"content_type": "application/json;charset=UTF-8",
"http_version": "HTTP/1.1"
},
"AT_CUSTO_S3": "309",
"AT_VAL3": "37",
"originLogId": "afad5535-a58f-11ea-b158-d94038694644",
"AT_CUSTO_S6": "",
"duration": 150,
"timeStamp": "2015-01-01T01:02:03.904Z",
"origin": "MorphoIA",
"AT_OUTSTANDINGVAL": "",
"businessId": "",
"ER_STATE": "value_40",
"ER_ID": "value_70",
"AT_CUSTO_S9": "",
"AT_CUSTO_S10": "",
"ER_STATUS": "value_50",
"AT_VAL1": "SERVICE_TASK_RESULT",
"AT_CUSTO_S7": "",
"AT_VAL5": "TECHNICAL",
"station": "origin",
"AT_VAL10": "",
"AR_STATUS": "value_30",
"AT_CUSTO_S2": ""
},
"fields": {
"timeStamp": [
"2015-01-01T01:02:03.904Z"
],
"@timestamp": [
"2020-06-03T11:45:01.893Z"
]
},
"highlight": {
"result": [
"@kibana-highlighted-field@Identity@/kibana-highlighted-field@ @kibana-highlighted-field@Accepted@/kibana-highlighted-field@"
],
"result.keyword": [
"@kibana-highlighted-field@Identity Accepted@/kibana-highlighted-field@"
]
}
}

{
"_index": "icp2-2020.06.03",
"_type": "_doc",
"_id": "10f76a8d5b991f7523c55c43bb8e330116b07ad99716610aa70cb427a008761d",
"_version": 1,
"_score": 3.9113598,
"_source": {
"ENTITLEMENT_TYPE": "value_100",
"AT_VAL6": "",
"AT_VAL4": "",
"internalId": "1810",
"AT_CUSTO_S5": "",
"result": "QUEUED",
"AT_VAL2": "",
"service": "Identity merged",
"host": "",
"AT_CUSTO_S1": "",
"USER_TYPE": "value_90",
"LOG_QUALIFIER": "USER_TASK_RESULT",
"@timestamp": "2020-06-03T11:45:03.588Z",
"AT_EXTENDEDINFO": "",
"AT_VAL7": "STARTED",
"activity": "CreateApplicantRequest",
"REQUEST_TYPE": "value_60",
"@version": "1",
"headers": {
"accept_encoding": "gzip,deflate",
"request_method": "PUT",
"http_accept": null,
"connection": "Keep-Alive",
"http_host": "",
"request_path": "/",
"content_length": "1012",
"http_user_agent": "Apache-HttpClient/4.5.7 (Java/1.8.0_102)",
"content_type": "application/json;charset=UTF-8",
"http_version": "HTTP/1.1"
},
"AT_CUSTO_S3": "309",
"AT_VAL3": "37",
"originLogId": "b0af0dc9-a58f-11ea-b158-d94038694644",
"AT_CUSTO_S6": "",
"duration": 150,
"timeStamp": "2015-01-01T01:02:03.904Z",
"origin": "MorphoIA",
"AT_OUTSTANDINGVAL": "",
"businessId": "",
"ER_STATE": "value_40",
"ER_ID": "value_70",
"AT_CUSTO_S9": "",
"AT_CUSTO_S7": "",

},
"fields": {
"timeStamp": [
"2015-01-01T01:02:03.904Z"
],
"@timestamp": [
"2020-06-03T11:45:03.588Z"
]
},
"highlight": {
"service": [
"@kibana-highlighted-field@Identity@/kibana-highlighted-field@ @kibana-highlighted-field@merged@/kibana-highlighted-field@"
],
"service.keyword": [
"@kibana-highlighted-field@Identity merged@/kibana-highlighted-field@"
]
}
}

{
"_index": "icp2-2020.06.01",
"_type": "_doc",
"_id": "a87b87052896c5c6a123186d6da9575f97391a7d3fa9b767eec2c251f6f80d60",
"_version": 1,
"_score": 3.585218,
"_source": {
"ENTITLEMENT_TYPE": "value_100",
"AT_VAL6": "",
"AT_VAL4": "",
"internalId": "1801",
"AT_CUSTO_S5": "",
"result": "Identity Rejected",
"AT_VAL2": "Male",
"service": "FIRST_REQUEST",
"host": "50.50.1.13",
"AT_CUSTO_S1": "",
"USER_TYPE": "value_90",
"LOG_QUALIFIER": "USER_TASK_RESULT",
"@timestamp": "2020-06-01T11:23:43.735Z",
"AT_EXTENDEDINFO": "",
"logCategory": "BUSINESS",
"@version": "1",
"headers": {
"accept_encoding": "gzip,deflate",
"request_method": "PUT",
"http_accept": null,
"connection": "Keep-Alive",
"http_host": "",
"request_path": "/",
"content_length": "1038",
"http_user_agent": "Apache-HttpClient/4.5.7 (Java/1.8.0_102)",
"content_type": "application/json;charset=UTF-8",
"http_version": "HTTP/1.1"
},
"originLogId": "60505416-a3fa-11ea-b158-d94038694644",
"AT_CUSTO_S6": "",
"duration": 150,
"timeStamp": "2015-01-01T01:02:03.904Z",
"origin": "MorphoIA",
"AT_OUTSTANDINGVAL": "",
"businessId": "",
"ER_STATE": "value_40",
"ER_ID": "value_70",
"AT_CUSTO_S9": "",
"AT_CUSTO_S10": "",

},
"fields": {
"timeStamp": [
"2015-01-01T01:02:03.904Z"
],
"@timestamp": [
"2020-06-01T11:23:43.735Z"
]
},
"highlight": {
"result": [
"@kibana-highlighted-field@Identity@/kibana-highlighted-field@ @kibana-highlighted-field@Rejected@/kibana-highlighted-field@"
],
"result.keyword": [
"@kibana-highlighted-field@Identity Rejected@/kibana-highlighted-field@"
]
}
}