# Copy contents of one field to same field of another row if certain condition matches **URL:** \ **Category:** Logstash\ **Created:** [June 28, 2022, 9:32am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339 "2022-06-28T09:32:53Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [June 28, 2022, 9:32am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/1 "2022-06-28T09:32:53Z")
I need to copy contents of “Service” field from No. 15256 into the :path field of No.15257 if “Stream\_Identifier” of both lines in 15256 and 15257 are matching. Kindly suggest how to get this done in the conf file of logstash or any other way to achieve the same Please refer to the image/snippet attached for the above query. ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/e/4e7660ee356998d22820b53298c2e580b8af0386.png) Also given below is my logstash conf file for ref input { file { path =\> "/home/student/test-csv/test-http2-1.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," skip\_header =\> "true" columns =\> ["Time","No.","Source","Destination","Protocol","Length","HTTP\_Header\_Status","Stream\_Identifier","method",":path","String value","Key","Info"] } date { match =\> ["Time", "yyyy-MM-dd HH:mm:ss.SSSSSS"] target =\> " **@timestamp**" timezone =\> "UTC" } ```auto mutate { ``` ```auto ``` ```auto copy => { ":path" => "Service_Request" } ``` ```auto } ``` ```auto ``` ```auto mutate { ``` ```auto split => { ":path" => "/" } ``` ```auto add_field => { "Service" => "%{[:path][1]}" } ``` ```auto remove_field => ["message","host","path","@version"] ``` ```auto } ``` ```auto ``` ```auto aggregate { ``` ```auto task_id => "%{Stream_Identifier}" ``` ```auto code => ' ``` ```auto p = event.get("Service") ``` ```auto if p ``` ```auto map["Service"] = p ``` ```auto elsif map["Service"] ``` ```auto event.set("Service", map["Service"]) ``` ```auto end ``` ```auto ' ``` ```auto } ``` ```auto mutate { ``` ```auto remove_field => [":path"] ``` ```auto } ``` } output { Elasticsearch { hosts =\> "[http://localhost:9200](http://localhost:9200/)" index =\> "demo-http-csv-split-2" } stdout { codec =\> rubydebug } } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 28, 2022, 4:16pm UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/2 "2022-06-28T16:16:00Z")
You need to make adding the "Service" field conditional on the field [:path][1] existing. Take the add\_field out of the mutate+split filter. ``` if [:path][1] { mutate { add_field => { "Service" => "%{[:path][1]}" } } } ``` ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [June 29, 2022, 6:39am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/4 "2022-06-29T06:39:07Z")
Hi Badger - Thanks for your response. Here is sharing the updated logstash conf file as per your suggestion. input { file { path =\> "/home/student/test-csv/test-http2-1.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," skip\_header =\> "true" columns =\> ["Time","No.","Source","Destination","Protocol","Length","HTTP\_Header\_Status","Stream\_Identifier","method",":path","String value","Key","Info"] } date { match =\> ["Time", "yyyy-MM-dd HH:mm:ss.SSSSSS"] target =\> "@timestamp" timezone =\> "UTC" } ``` mutate { copy => { ":path" => "Service_Request" } } mutate { split => { ":path" => "/" } } if [:path][1] { mutate { add_field => { "Service" => "%{[:path][1]}" } } } mutate { remove_field => ["message","host","path","@version"] } aggregate { task_id => "%{Stream_Identifier}" code => ' p = event.get("%{Service}") if p map["Service"] = p elsif map["Service"] event.set("Service", map["Service"]) end ' } mutate { remove_field => [":path"] } ``` } output { ``` elasticsearch { hosts => "http://localhost:9200" index => "demo-http-csv-split" } file { path => "/tmp/my_output_text_file" codec => rubydebug } stdout { codec => rubydebug } ``` } However still the Service field value from 15256 is not getting populated in 15257. Snippet given below for reference. Any modification required in code or aggregation logic ? ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/1/01fa439c180e59a876a4dd9eb71a13615b89229f.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 29, 2022, 4:19pm UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/5 "2022-06-29T16:19:19Z")
Does your configuration comply with the limits on pipeline.workers and pipeline.ordered that are documented for the aggregate filter? ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [June 30, 2022, 6:25am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/6 "2022-06-30T06:25:12Z")
Snippet of the pipeline.ordered and pipeline.worker setting in my logstash.yml file. ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fb1495bbd67f57d73c49cbc70c63468cda65014c.png) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [June 30, 2022, 4:51pm UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/7 "2022-06-30T16:51:10Z")
:path is set (mutate+copy) to the value of Service\_Request. Since that is empty [:path] will be empty. ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [July 1, 2022, 5:43am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/8 "2022-07-01T05:43:06Z")
Yes, but the requirement is to copy the contents of Service\_Request from the row which has an identical Stream\_Identifier. i.e. to copy Service\_Request contents from Row no. 15256 in Row No. 15257 as their Stream\_Identifier matches. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 1, 2022, 3:20pm UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/9 "2022-07-01T15:20:36Z")
Then change your aggregate filter to process [Service\_Request] rather than [Service], and move the code that extracts [Service] from [Service\_Request] to execute after the aggregate. ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [July 4, 2022, 2:43am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/10 "2022-07-04T02:43:46Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/0/3/03a38b2dc0c9bf3f761f4470c2592d8bef0a93a8.png) I have carried out suggested changes but still no joy. I have shared the updated conf file below. Let me know if the same is ok or any changes required. input { file { path =\> "/home/student/test-csv/test-http2-1.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," skip\_header =\> "true" columns =\> ["Time","No.","Source","Destination","Protocol","Length","HTTP\_Header\_Status","Stream\_Identifier","method",":path","String value","Key","Info"] } date { match =\> ["Time", "yyyy-MM-dd HH:mm:ss.SSSSSS"] target =\> "@timestamp" timezone =\> "UTC" } ``` mutate { copy => { ":path" => "Service_Request" } } mutate { split => { ":path" => "/" } } mutate { remove_field => ["message","host","path","@version"] } aggregate { task_id => "%{Stream_Identifier}" code => ' p = event.get("%{Service_Request}") if p map["Service_Request"] = p elsif map["Service_Request"] event.set("Service_Request", map["Service_Request"]) end ' } if [:path][1] { mutate { add_field => { "Service" => "%{[:path][1]}" } } } mutate { remove_field => [":path"] } ``` } output { ``` elasticsearch { hosts => "http://localhost:9200" index => "pcap-csv" } file { path => "/tmp/my_output_text_file" codec => rubydebug } stdout { codec => rubydebug } ``` } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 4, 2022, 2:55am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/11 "2022-07-04T02:55:52Z")
Remove all the remove\_field options, so that you keep all the fields, then post the two events from "/tmp/my\_output\_text\_file" that you want to aggregate (redacted as necessary, but keep [:path], [Service\_Request], and [Service]). ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [July 4, 2022, 5:33am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/12 "2022-07-04T05:33:04Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fac1b9a045d5dbe262811c94bfdb0a9eba9bafe7.png) Please find the sample two events from the /tmp/my\_output\_text\_file. For the highlighted Stream\_Identifier I want Service\_Request from No. =\> 274931 entry to be copied to Service\_Request field of entry in No. =\> 274932 ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [July 4, 2022, 5:33am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/13 "2022-07-04T05:33:49Z")
Adding the text o/p of the two events of the snippet... { "message" =\> ""2022-02-07 19:50:41.590957","274931","10.242.14.148","10.242.22.77","HTTP2","106","","1296227","","/nsmsf-sms/v2/ue-contexts/imsi-525058131708832","","","[TCP Spurious Retransmission] , HEADERS[1296227]"\r", "Protocol" =\> "HTTP2", "Length" =\> "106", "host" =\> "es7", "path" =\> "/home/student/test-csv/test-http2-1.csv", "Stream\_Identifier" =\> "1296227", ":path" =\> [ [0] "", [1] "nsmsf-sms", [2] "v2", [3] "ue-contexts", [4] "imsi-525058131708832" ], "Service\_Request" =\> "/nsmsf-sms/v2/ue-contexts/imsi-525058131708832", "Info" =\> "[TCP Spurious Retransmission] , HEADERS[1296227]", "Service" =\> "nsmsf-sms", "Destination" =\> "10.242.22.77", "@version" =\> "1", "@timestamp" =\> 2022-02-07T19:50:41.590Z, "method" =\> "", "Key" =\> "", "Time" =\> "2022-02-07 19:50:41.590957", "String value" =\> "", "Source" =\> "10.242.14.148", "No." =\> "274931", "HTTP\_Header\_Status" =\> "" } { "message" =\> ""2022-02-07 19:50:41.628290","274932","10.242.22.77","10.242.14.148","HTTP2","71","204","1296227","","","","","[TCP ACKed unseen segment] , HEADERS[1296227]: 204 No Content"\r", "Protocol" =\> "HTTP2", "Length" =\> "71", "host" =\> "es7", "path" =\> "/home/student/test-csv/test-http2-1.csv", "Stream\_Identifier" =\> "1296227", ":path" =\> , "Service\_Request" =\> "", "Info" =\> "[TCP ACKed unseen segment] , HEADERS[1296227]: 204 No Content", "Destination" =\> "10.242.14.148", "@version" =\> "1", "@timestamp" =\> 2022-02-07T19:50:41.628Z, "method" =\> "", "Key" =\> "", "Time" =\> "2022-02-07 19:50:41.628290", "String value" =\> "", "Source" =\> "10.242.22.77", "No." =\> "274932", "HTTP\_Header\_Status" =\> "204" } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 4, 2022, 5:22pm UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/14 "2022-07-04T17:22:08Z")
> [@Ashutosh\_Vaidya](#): > > `p = event.get("%{Service_Request}")` You do not want a sprintf reference there. Try ``` csv { skip_header => true columns => ["Time","No.","Source","Destination","Protocol","Length","HTTP_Header_Status","Stream_Identifier","method",":path","String value","Key","Info"] } date { match => ["Time", "yyyy-MM-dd HH:mm:ss.SSSSSS"] target => "@timestamp" timezone => "UTC" } if [:path] != "" { mutate { copy => { ":path" => "Service_Request" } } } aggregate { task_id => "%{Stream_Identifier}" code => ' p = event.get("Service_Request") if p map["Service_Request"] = p elsif map["Service_Request"] event.set("Service_Request", map["Service_Request"]) end ' } if [Service_Request] { mutate { copy => { "Service_Request" => "[@metadata][path]" } } mutate { split => { "[@metadata][path]" => "/" } } if [@metadata][path][1] { mutate { add_field => { "Service" => "%{[@metadata][path][1]}" } } } } ``` ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [July 5, 2022, 4:58am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/15 "2022-07-05T04:58:39Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fa11ac2c235bd0f4f031340403fca1e5006f31c4.png) I think the below code is working on a row basis. Ideally the o/p of below code should have Service\_Request field populated in the entry on No. 15257 from No.15256. But the same doesn't seem to happen due to which we still have the Service\_Request field empty in No.15257 ``` aggregate { task_id => "%{Stream_Identifier}" code => ' p = event.get("Service_Request") if p map["Service_Request"] = p elsif map["Service_Request"] event.set("Service_Request", map["Service_Request"]) end ``` ' Sample o/p after having the above suggestion of logstash conf file { "path" =\> "/home/student/test-csv/test-http2-1.csv", "Service\_Request" =\> [ [0] "", [1] "nsmsf-sms", [2] "v2", [3] "ue-contexts", [4] "imsi-525058131708832" ], "Length" =\> "106", "HTTP\_Header\_Status" =\> "", "message" =\> ""2022-02-07 19:50:41.590957","274931","10.242.14.148","10.242.22.77","HTTP2","106","","1296227","","/nsmsf-sms/v2/ue-contexts/imsi-525058131708832","","","[TCP Spurious Retransmission] , HEADERS[1296227]"\r", "Info" =\> "[TCP Spurious Retransmission] , HEADERS[1296227]", "Destination" =\> "10.242.22.77", "host" =\> "es7", "Stream\_Identifier" =\> "1296227", "No." =\> "274931", ":path" =\> [ [0] "", [1] "nsmsf-sms", [2] "v2", [3] "ue-contexts", [4] "imsi-525058131708832" ], "method" =\> "", "@timestamp" =\> 2022-02-07T19:50:41.590Z, "Source" =\> "10.242.14.148", "String value" =\> "", "Time" =\> "2022-02-07 19:50:41.590957", "Service" =\> "nsmsf-sms", "Protocol" =\> "HTTP2", "Key" =\> "", "@version" =\> "1" } { "path" =\> "/home/student/test-csv/test-http2-1.csv", "Service\_Request" =\> , ===\> No contents in this field "Length" =\> "71", "HTTP\_Header\_Status" =\> "204", "message" =\> ""2022-02-07 19:50:41.628290","274932","10.242.22.77","10.242.14.148","HTTP2","71","204","1296227","","","","","[TCP ACKed unseen segment] , HEADERS[1296227]: 204 No Content"\r", "Info" =\> "[TCP ACKed unseen segment] , HEADERS[1296227]: 204 No Content", "Destination" =\> "10.242.14.148", "host" =\> "es7", "Stream\_Identifier" =\> "1296227", "No." =\> "274932", ":path" =\> , ====\> No contents in this field "method" =\> "", "@timestamp" =\> 2022-02-07T19:50:41.628Z, "Source" =\> "10.242.22.77", "String value" =\> "", "Time" =\> "2022-02-07 19:50:41.628290", "Protocol" =\> "HTTP2", "Key" =\> "", "@version" =\> "1" } ---
**Author:** ![Ashutosh\_Vaidya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashutosh_vaidya/32/102494_2.png) [@Ashutosh\_Vaidya](https://discuss.elastic.co/u/Ashutosh_Vaidya)\ **Post date:** [July 20, 2022, 5:20am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/16 "2022-07-20T05:20:07Z")
Hi - Any insights or suggestions ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 17, 2022, 5:20am UTC](https://discuss.elastic.co/t/copy-contents-of-one-field-to-same-field-of-another-row-if-certain-condition-matches/308339/17 "2022-08-17T05:20:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.