# Is it possible to change output structure

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759>\
**Category:** Logstash\
**Created:** [October 24, 2018, 9:16am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759 "2018-10-24T09:16:52Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 24, 2018, 9:16am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/1 "2018-10-24T09:16:52Z")

</div>

Hello,  
I got an output structure like that:

```auto
{
              "path" => "/home/data/test.xml",
          "@version" => "1",
        "@timestamp" => 2018-10-24T08:49:15.480Z,
    "Properties" => {
               "tag" => [
            [0] {
                   "pluginname" => "LastUnauthenticatedResults",
                      "content" => "1539"
            },
            [1] {
                   "pluginname" => "Scan",
                      "content" => "false"
            }
        ],
        "ReportItem" => [
            [0] {
                          "port" => "0",
                      "severity" => "0",
            },
            [1] {
                        "port" => "22"
                    "severity" => "3"
            },
            [2] {
                        "port" => "80"
                    "severity" => "5"
           }
        ]
    },
                "ip" => "11.111.11.1"
}

```

I am not able to index this structure.  
Is it possible to get output like that

```auto
{
          
        "ReportItem" => [
            [0] {
                          "port" => "0",
                      "severity" => "0",
                    "pluginname" => "LastUnauthenticatedResults",
                       "content" => "1539"
                            "ip" => "11.111.11.1"
            },
            [1] {
                        "port" => "22"
                    "severity" => "3"
                  "pluginname" => "LastUnauthenticatedResults",
                     "content" => "1539"
                          "ip" => "11.111.11.1"
            },

            [2] {
                        "port" => "80"
                    "severity" => "5"
                  "pluginname" => "Scan",
                     "content" => "false"
                          "ip" => "11.111.11.1"
            }
        ]
   }

```

What i need to use if that's possible?

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 24, 2018, 9:50am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/2 "2018-10-24T09:50:12Z")

</div>

Yup, it is possible by writing the ruby code, but if you could provide the input code, it is easy to provide the solution.

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 24, 2018, 9:53am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/3 "2018-10-24T09:53:26Z")

</div>

Sure, my input code is this:

```auto
input {
  file {
    path => "/home/data/test.xml"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "</system>"
      what => "next"
      negate => true
      max_lines => 333333
    }
  }
}

filter {
  xml {
    source => "message"
    store_xml => false
    xpath => ["/system/Report/ReportHost","ReportHost"]    
  }
  mutate {
    remove_field => ["message","@version"]
  }
  split {
    field => "ReportHost"
  }
  xml {
    source => "ReportHost"
    target => "[xml_content]"
    force_array => false
  }
  ruby {
    code => '
      event.get("[xml_content]").each do |key, value|
        event.set(key, value)
      end
    '
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

```

Do you also need my input data structure?

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 24, 2018, 9:55am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/4 "2018-10-24T09:55:48Z")

</div>

Also provide the sample xml file also

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 24, 2018, 10:33am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/5 "2018-10-24T10:33:50Z")

</div>

my xml sample

```auto
<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
<tag pluginnname="Scan">false</tag>
</HostProperties>
<ReportItem port="0" severity="0">
</ReportItem>
<ReportItem port="22" severity="3">
</ReportItem>
<ReportItem port="80" severity="5">
</ReportItem>
</ReportHost>
</Report>
</system>

```

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 24, 2018, 12:38pm UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/6 "2018-10-24T12:38:31Z")

</div>

The xml file which you have posted has an error, didnot close the tag  
\<tag pluginname="LastUnauthenticatedResults"\>1539\</tag\>

The input code is as follows as per your requirement,

input {  
file {  
path =\> "D:/xxxxx/ELKStack/sample.xml"  
start\_position =\> "beginning"  
sincedb\_path =\> "NUL"  
codec =\> multiline {  
pattern =\> ""  
negate =\> "true"  
what =\> "previous"  
auto\_flush\_interval =\> 1  
max\_lines =\> 333333  
}  
}  
}

filter {  
xml {  
source =\> "message"  
target =\> "parsed"  
store\_xml =\> "false"  
xpath =\> [  
"/system/Report/ReportHost/@ip","ip",  
"/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname",  
"/system/Report/ReportHost/HostProperties/tag/text()","content",  
"/system/Report/ReportHost/ReportItem/@port","portname",  
"/system/Report/ReportHost/ReportItem/@severity","severity"   
]  
}

ruby {  
code =\> "   
i = event.get('ip')  
n = event.get('pluginname')  
p = event.get('portname')  
s = event.get('severity')  
carr =   
s.each\_index { |k|  
h = { 'portname' =\> p[k] , 'severity' =\> s[k] , 'ip' =\> i[0], 'pluginname' =\> n[0] }  
carr \<\< h  
}  
event.set('reportitem', carr) "  
}  
mutate {  
remove\_field =\> ["message","@version"]  
}

}

output {  
stdout {  
codec =\> rubydebug  
}  
}

the output is as shown in the image below

![Capture](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a28319cf5fa406cd54d726cf9833766411d328ec.png)

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 24, 2018, 1:26pm UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/7 "2018-10-24T13:26:52Z")

</div>

Nice, that works.

How am I able to split the array of reportitem, so i can get each array([0],[1],[2]) an own event, because if i index now, i just get one event.  
The events should looks like this:

event[0]  
Severity: 0  
ip: 11.111.11  
port: 0  
HistPropertiesName: LastUnauthenticatedResults

event[1]  
Severity: 3  
ip: 11.111.11  
port: 22  
HistPropertiesName: LastUnauthenticatedResults

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 24, 2018, 1:37pm UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/8 "2018-10-24T13:37:45Z")

</div>

Have you tried using the [split filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-split.html) on the `reportitem` field?

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 24, 2018, 1:43pm UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/9 "2018-10-24T13:43:34Z")

</div>

Now I tried and it's perfect. Thx a lot guys 🙂

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 25, 2018, 7:33am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/10 "2018-10-25T07:33:54Z")

</div>

> [@balumurari1](#):
>
> ruby {  
> code =\> "  
> i = event.get('ip')  
> n = event.get('pluginname')  
> p = event.get('portname')  
> s = event.get('severity')  
> carr =   
> s.each\_index { |k|  
> h = { 'portname' =\> p[k] , 'severity' =\> s[k] , 'ip' =\> i[0], 'pluginname' =\> n[0] }  
> carr \<\< h  
> }

Is there a reason why you pick **s.** each\_index?

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 25, 2018, 7:37am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/11 "2018-10-25T07:37:57Z")

</div>

the reason why i choose **s** because it contains 3 elements and i want to iterate the loop for 3 times.  
You need to mention the name of the array, where you can iterate with the size of the elements

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 25, 2018, 10:05am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/12 "2018-10-25T10:05:13Z")

</div>

I have some more questions.

When i have a log, where are elements with the same name but different values, how can i get them indexed in the same index?  
I.E.

```auto
<?xml version="1.0" ?>
<system>
<Report name="Scan">
<ReportHost ip="11.111.11.1"><HostProperties>
<tag pluginname="LastUnauthenticatedResults">1539<tag>
</HostProperties>
<ReportItem port="80" severity="5">
**<bid>1111</bid>**
 **<bid>2222</bid>**
</ReportItem>
</ReportHost>
</Report>
</system>

```

I guess i need smth like

```auto
s.each_index { |k,j|
h = { 'portname' => p[k] , 'severity' => s[k] , 'ip' => i[0], 'pluginname' => n[0], 'bid'[j]=>[k] }
carr << h
}

```

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 25, 2018, 10:14am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/13 "2018-10-25T10:14:50Z")

</div>

yes you need to iterate again, give me the sample output required for you

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 25, 2018, 10:15am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/14 "2018-10-25T10:15:56Z")

</div>

> [@humalog](#):
>
> event[0]  
> Severity: 0  
> ip: 11.111.11  
> port: 0  
> HistPropertiesName: LastUnauthenticatedResults  
> bid: 1111  
> bid: 2222

this

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 25, 2018, 10:18am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/15 "2018-10-25T10:18:56Z")

</div>

For this, again you get "bid" like array of objects and you need to iterate(bid array) within the loop( reportitem array)

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 25, 2018, 10:38am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/16 "2018-10-25T10:38:51Z")

</div>

I work with ruby since yesterday. Have to search how it works.

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 25, 2018, 10:39am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/17 "2018-10-25T10:39:40Z")

</div>

The input code as per your requirement is as follows,

input {  
file {  
path =\> "D:/xxxxx/ELKStack/sample.xml"  
start\_position =\> "beginning"  
sincedb\_path =\> "NUL"  
codec =\> multiline {  
pattern =\> ""  
negate =\> "true"  
what =\> "previous"  
auto\_flush\_interval =\> 1  
max\_lines =\> 333333  
}  
}  
}

filter {  
xml {  
source =\> "message"  
target =\> "parsed"  
store\_xml =\> "false"  
xpath =\> [  
"/system/Report/ReportHost/@ip","ip",  
"/system/Report/ReportHost/HostProperties/tag/@pluginname","pluginname",  
"/system/Report/ReportHost/HostProperties/tag/text()","content",  
"/system/Report/ReportHost/ReportItem/@port","portname",  
"/system/Report/ReportHost/ReportItem/@severity","severity",  
"/system/Report/ReportHost/ReportItem/bid/text()","bidvalue"  
]  
}

ruby {  
code =\> "   
i = event.get('ip')  
n = event.get('pluginname')  
p = event.get('portname')  
s = event.get('severity')  
b = event.get('bidvalue')  
carr =   
s.each\_index { |k|  
h = { 'portname' =\> p[k] , 'severity' =\> s[k] , 'ip' =\> i[0], 'pluginname' =\> n[0],'bid1' =\> b[0],'bid2' =\> b[1] }  
carr \<\< h  
}  
event.set('reportitem', carr) "  
}  
mutate {  
remove\_field =\> ["message","@version"]  
}

}

output {  
stdout {  
codec =\> rubydebug  
}  
}

The output is as shown in below image,

![Capture](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d2d81e61fd416d19ceb24c5e308d031822feb5c.png)

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 25, 2018, 10:40am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/18 "2018-10-25T10:40:47Z")

</div>

hope above solution help you, 🙂

---

<div class="post-metadata">

**Author:** ![humalog](https://avatars.discourse-cdn.com/v4/letter/h/cc9497/32.png) [@humalog](https://discuss.elastic.co/u/humalog)\
**Post date:** [October 25, 2018, 11:05am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/19 "2018-10-25T11:05:45Z")

</div>

the problem is, in my log the elementnames are exactly the same. they both called and i cant say  
bid1=\>b[0]  
bid2=\>b[1]

---

<div class="post-metadata">

**Author:** ![balumurari1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/balumurari1/32/39203_2.png) [@balumurari1](https://discuss.elastic.co/u/balumurari1)\
**Post date:** [October 25, 2018, 11:11am UTC](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759/20 "2018-10-25T11:11:11Z")

</div>

Based on the xml file given by you, i have tested and it worked fine. Please check the output image attached above.

Please let me know if you have an unformatted xml file, if so, post them.

[Next page](https://discuss.elastic.co/t/is-it-possible-to-change-output-structure/153759.md?page=2)