# Field spilt **URL:** \ **Category:** Logstash\ **Created:** [July 10, 2019, 8:40am UTC](https://discuss.elastic.co/t/field-spilt/189694 "2019-07-10T08:40:14Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![galtryn](https://avatars.discourse-cdn.com/v4/letter/g/e480ec/32.png) [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Post date:** [July 10, 2019, 8:40am UTC](https://discuss.elastic.co/t/field-spilt/189694/1 "2019-07-10T08:40:14Z")
Hi Guys, I am ingesting some IoT sensor data into Elasticsearch via logstash, the payload of the message is a HEX string which currently looks something like this; ``` 129, 91, 0, 0, 28, 28, 29, 0, 4, 0, 0, 0, 11, 38, 0, 238 ``` This is currently all in one field when it arrives in Elasticsearch, I would like to this field on the , into separate fields. Does anyone out there have experience of doing this? Thanks in advance Nick ---
**Author:** ![sjabiulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjabiulla/32/48429_2.png) [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Post date:** [July 10, 2019, 11:20am UTC](https://discuss.elastic.co/t/field-spilt/189694/2 "2019-07-10T11:20:10Z")
You can parse this using [csv](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html) filter ``` csv{ columns => [field1,field2,field3.............] } ``` You can mention some meaningful fieldname's as you like. ---
**Author:** ![galtryn](https://avatars.discourse-cdn.com/v4/letter/g/e480ec/32.png) [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Post date:** [July 10, 2019, 12:50pm UTC](https://discuss.elastic.co/t/field-spilt/189694/3 "2019-07-10T12:50:51Z")
Hi sjabiulla, I have tried your suggestion of using the CSV filter but I get the below error, ``` Error parsing csv {:field=>"hex", :source=>[129, 75, 10, 0, 24, 24, 24, 0, 4, 0, 0, 0, 60, 54, 0, 214], :exception=>#} ``` Any ideas? ---
**Author:** ![sjabiulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjabiulla/32/48429_2.png) [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Post date:** [July 10, 2019, 12:56pm UTC](https://discuss.elastic.co/t/field-spilt/189694/4 "2019-07-10T12:56:40Z")
can you please provide your config and a sample input ---
**Author:** ![galtryn](https://avatars.discourse-cdn.com/v4/letter/g/e480ec/32.png) [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Post date:** [July 10, 2019, 1:01pm UTC](https://discuss.elastic.co/t/field-spilt/189694/5 "2019-07-10T13:01:11Z")
My current config is ``` filter { csv { source => "hex" columns => [field1,field2] } } ``` the input would be ``` 129, 75, 10, 0, 24, 24, 24, 0, 4, 0, 0, 0, 60, 54, 0, 214 ``` ---
**Author:** ![sjabiulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjabiulla/32/48429_2.png) [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Post date:** [July 10, 2019, 1:08pm UTC](https://discuss.elastic.co/t/field-spilt/189694/6 "2019-07-10T13:08:34Z")
remove **source** from csv filter. ``` filter { csv { columns => [field1,field2] } } ``` I have tested the below config using the sample input you provided and got all fields splitted properly using csv filter. ``` input { stdin { } } filter { csv { columns => [field1,field2] } } output { stdout { codec => rubydebug {} } } ``` **Output:** ``` { "field1" => "129", "column12" => " 0", "host" => "01hw1344293", "@version" => "1", "column8" => " 0", "column16" => " 214", "column5" => " 24", "column11" => " 0", "column14" => " 54", "@timestamp" => 2019-07-10T13:06:16.496Z, "column4" => " 0", "column10" => " 0", "field2" => " 75", "column3" => " 10", "column13" => " 60", "column15" => " 0", "column9" => " 4", "message" => "129, 75, 10, 0, 24, 24, 24, 0, 4, 0, 0, 0, 60, 54, 0, 214\r", "column6" => " 24", "column7" => " 24" } ``` ---
**Author:** ![galtryn](https://avatars.discourse-cdn.com/v4/letter/g/e480ec/32.png) [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Post date:** [July 10, 2019, 1:11pm UTC](https://discuss.elastic.co/t/field-spilt/189694/7 "2019-07-10T13:11:04Z")
I really need to the CSV data to come from the hex field as that is the field it arrives in Elastic search in, is it not possible to do it that way? ---
**Author:** ![sjabiulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjabiulla/32/48429_2.png) [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\ **Post date:** [July 10, 2019, 1:14pm UTC](https://discuss.elastic.co/t/field-spilt/189694/8 "2019-07-10T13:14:09Z")
That's the reason I have asked you for the config and sample input. If you provide me the wrong config and wrong sample input then of-course my answers won't help you. ---
**Author:** ![galtryn](https://avatars.discourse-cdn.com/v4/letter/g/e480ec/32.png) [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Post date:** [July 10, 2019, 1:32pm UTC](https://discuss.elastic.co/t/field-spilt/189694/9 "2019-07-10T13:32:47Z")
Here is the full pipeline ``` input { http { host => "0.0.0.0" port => "8080" } } filter { json { source => "message" target => "json_log" } if [message] =~ /^\s*$/ { drop { } } } filter { ruby { init => "require 'base64'" code => "event.set( '[payload]', Base64.decode64(event.get('[json_log][data]')).unpack('C*'))" } } filter { mutate { copy => { "payload" => "hex" } } } filter { csv { source => "hex" columns => [field1,field2] } } output ``` The hex values come in encrypted thats why have to decrypt and end up with the hex values ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 10, 2019, 1:49pm UTC](https://discuss.elastic.co/t/field-spilt/189694/10 "2019-07-10T13:49:47Z")
You can convert the array of integers to a string that csv will parse using ``` ruby { code => 'event.set("payload", event.get("payload").to_s[1..-2])' } ``` The [1..-2] is to strip off the [and] that to\_s will add. ---
**Author:** ![galtryn](https://avatars.discourse-cdn.com/v4/letter/g/e480ec/32.png) [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Post date:** [July 11, 2019, 8:27am UTC](https://discuss.elastic.co/t/field-spilt/189694/11 "2019-07-11T08:27:31Z")
Thank you badger I shall try it this morning ---
**Author:** ![galtryn](https://avatars.discourse-cdn.com/v4/letter/g/e480ec/32.png) [@galtryn](https://discuss.elastic.co/u/galtryn)\ **Post date:** [July 11, 2019, 8:43am UTC](https://discuss.elastic.co/t/field-spilt/189694/12 "2019-07-11T08:43:33Z")
I added the ruby code to the pipeline and got the following error Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"############-2019.07.11", :\_type=\>"\_doc", :routing=\>nil}, #LogStash::Event:0x1f4d7ec7], :response=\>{"index"=\>{"\_index"=\>"##########-2019.07.11", "\_type"=\>"\_doc", "\_id"=\>"qbIq4GsB\_o88aajI8mVC", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse field [payload] of type [long] in document with id 'qbIq4GsB\_o88aajI8mVC'", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"For input string: "129, 99, 15, 0, 21, 20, 21, 0, 4, 0, 0, 0, 12, 59, 0, 202""}}}}} Not sure where to go from here ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 11, 2019, 1:24pm UTC](https://discuss.elastic.co/t/field-spilt/189694/13 "2019-07-11T13:24:10Z")
I think that is telling you that payload is a string, but elasticsearch has a [mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html) that tells it to expect a long in that field. Renaming the field with mutate might fix it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 8, 2019, 1:25pm UTC](https://discuss.elastic.co/t/field-spilt/189694/14 "2019-08-08T13:25:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.