# How to extract and transform unstructured data into fields

**URL:** <https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916>\
**Category:** Logstash\
**Created:** [July 5, 2017, 1:51pm UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916 "2017-07-05T13:51:19Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sentinel](https://avatars.discourse-cdn.com/v4/letter/s/fbc32d/32.png) [@sentinel](https://discuss.elastic.co/u/sentinel)\
**Post date:** [July 5, 2017, 1:51pm UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916/1 "2017-07-05T13:51:19Z")

</div>

My message index field has field1=A big brown wolf, field2=Yet another furry animal, field3=No guarantee how many fields we will see

This is what is indexed:  
field1=A  
field2=Yet  
field3=No

I'm missing the rest of the words in between the = and the ',' This ends up in the message field  
What would be the appropriate kv filter to use so I can have all the words up to the comma','  
like so:

field1=A big brown wolf  
field2=Yet another furry animal

etc.

thanks

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [July 5, 2017, 2:36pm UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916/2 "2017-07-05T14:36:12Z")

</div>

I do believe this is a limitation of the KV filter (if someone knows a workaround please mention it).  
That said, you can practically achieve the same result with some custom Ruby code.

```auto
filter {
    ruby {
        code => "event.get('message').split(',').each { |v| field, value = v.strip.split('=') ; event.set(field, value) }"
    }
}
```

1. It first splits the message into chunks separated by commas,
2. Then it splits each chuck into key-value fields preserving the included spaces (strip just removes leading/trailing ones from the initial split)

Of course for the above to work, the 'message' field should have the exact structure you posted originally, else code adaptations would be needed.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 5, 2017, 3:14pm UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916/3 "2017-07-05T15:14:38Z")

</div>

Can't you just set `field_split => ","`?

---

<div class="post-metadata">

**Author:** ![sentinel](https://avatars.discourse-cdn.com/v4/letter/s/fbc32d/32.png) [@sentinel](https://discuss.elastic.co/u/sentinel)\
**Post date:** [July 6, 2017, 11:13am UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916/4 "2017-07-06T11:13:19Z")

</div>

Actually it's not so simple .. I have fields with monetary values like this  
Amt=100,087,001.098214 so I get Amt=100 because it's taking me up to the comma. How do I further cater for that?

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [July 6, 2017, 11:33am UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916/5 "2017-07-06T11:33:26Z")

</div>

It would help if you posted an actual log line (preferably containing all corner cases).

---

<div class="post-metadata">

**Author:** ![sentinel](https://avatars.discourse-cdn.com/v4/letter/s/fbc32d/32.png) [@sentinel](https://discuss.elastic.co/u/sentinel)\
**Post date:** [July 6, 2017, 11:37am UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916/6 "2017-07-06T11:37:43Z")

</div>

ClientMapping [counterPartyWID=NOT FOUND, counterParty=NOT FOUND, customerWID=NOT FOUND, customer=NOT FOUND, isCLS=false, isCustomer=true, riskCountry=NOT FOUND], collateralised=false |

So with =\> ',' I'm getting

counterPartyWID=NOT  
customerWID=NOT  
customer=NOT

etc ... I am missing the second word which is the 'FOUND'

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 3, 2017, 11:43am UTC](https://discuss.elastic.co/t/how-to-extract-and-transform-unstructured-data-into-fields/91916/7 "2017-08-03T11:43:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.