# How & When to process/convert data in ELK

**URL:** <https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816>\
**Category:** Elasticsearch\
**Created:** [September 16, 2020, 10:25am UTC](https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816 "2020-09-16T10:25:31Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![aspat](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@aspat](https://discuss.elastic.co/u/aspat)\
**Post date:** [September 16, 2020, 10:25am UTC](https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816/1 "2020-09-16T10:25:31Z")

</div>

Hello,

I am planning to use ELK  
My current SQL table is like below

Time, Field ID, Filed\_name, Field\_Sub\_ID, value, extra value, type

To view this data we have provided excel add-in.  
User can provide basic filter to this addin (filter like date,time from & to. Filed\_id, sub id)

**Value filed contain encoded data (HEX )**  
14 C9 80 00 04 00 00 40 00 C4 04 61 39 E0 00 01 00 00 61 02 08 07 6C 00 00 40 00 14 00 00 00 36 C0 19 80 00 3B 62 00 00  
86 00  
**Actual data we see in excel is like**  
WellID = 1 DrillState = 0 MudAvailable = 0 LastLevel = 150 PowerOnContext = xyzz  
Type = Bore Operation = Semi OperatorMonitoring = sd209 .......... this is very big data more than 30 fields

Which gets converted by excel addin.  
**For conversion addin uses xml file** , which tells which bit contain what data.  
**Example** :-

```
<?xml version="1.0"?>
<!-- Drill DataMessage (id=456) -->
<message class="Message">
    <record class="Record" id="WellID" offset="0">
      <record class="Record" id="Info" offset="0">
        <field class="int" id="WellID" size="1"/>
        <field class="bool" id="DrillState" size="1"/>
        <field class="bool" id="MudAvailable" size="1"/>
        <field class="int" id="LastLevel" size="1"/>
        <field class="int" id="PowerOnContext" size="1"/>
        <field class="int" id="Type" size="1"/>
        <field class="string" id="Operation" size="1"/>
        <field class="int" id="OperatorMonitoring" size="2"/>        
      </record>
    </record>
</message>

```

We have different XML schema files, based on type of filed.

**If I am using ELK,**  
**My quires-**

1. When we should do such configuration,  
Before inserting data (at logstach ) or while fetching or viewing the data(in elasticearch or kibana side)
2. How this is possible, I can write own script or can I use our own CPP exe to do this operation.

Can someone please help on this.  
Thanks in advance 😊

Regards,  
ASH

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 16, 2020, 9:30pm UTC](https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816/2 "2020-09-16T21:30:41Z")

</div>

Please format your code/logs/config using the `</>` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂

Processing of data like this should happen before it goes into Elasticsearch.  
If the content is XML, you can use the XML filter in Logstash.

---

<div class="post-metadata">

**Author:** ![aspat](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@aspat](https://discuss.elastic.co/u/aspat)\
**Post date:** [September 17, 2020, 8:09am UTC](https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816/3 "2020-09-17T08:09:36Z")

</div>

Hello Mark,

Thanks for reply , I have made changes the you said 🙂  
actually XML is just to tell how to deode hex data. it just tell which bit having what value.

Can I create custom filters .

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 17, 2020, 9:56pm UTC](https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816/4 "2020-09-17T21:56:03Z")

</div>

Ok, to be clear do you want to decode that hex?

---

<div class="post-metadata">

**Author:** ![aspat](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@aspat](https://discuss.elastic.co/u/aspat)\
**Post date:** [September 18, 2020, 5:22am UTC](https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816/5 "2020-09-18T05:22:37Z")

</div>

Yes Mark,  
and how to decode is mentioned in xml format.

we have different type of data, so there are around 80 -90 XMLs.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 16, 2020, 5:22am UTC](https://discuss.elastic.co/t/how-when-to-process-convert-data-in-elk/248816/6 "2020-10-16T05:22:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.