# Xml filter plugin - creating nested field out of null object: Can't get text on a END\_OBJECT

**URL:** https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928
**Category:** Logstash
**Created:** [August 1, 2016, 8:40pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928 "2016-08-01T20:40:50Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![whyapenny](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@whyapenny](https://discuss.elastic.co/u/whyapenny)
#### Post date: [August 1, 2016, 8:40pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/1 "2016-08-01T20:40:50Z")

</div>

FYI, Here is the error seen:

```auto
"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [entry.AppId.raw]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a END_OBJECT at 1:1416"}}}}, :level=>:warn}
```

I have been tracking this issue down for a while and I believe i have finally come to the root of the issue. I have an xml document that has fields sometimes null, and sometimes not. When the field is populated, xml filter parses it correctly into its own field, for example:

calling xml filter:

`
    xml {
       target => entry
       source => message
       force_array => false
    } `

example of xml:

```auto
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<AppId>ConsumeAll</AppId>
```

will result in the following:

```auto
"entry": {
  "AppId": "ConsumeAll"
}
```

This is exactly how it should be, which matches my mapping set for the index, and if all records were like this, i assume i would not have any issues. HOWEVER, if the xml field is null, then we hit issues. Instead of being a null string/object, it gets created as the parent of a nested field such as:

example of xml:

```auto
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<AppId/>
```

will result in the following:

```auto
"entry": {
  "AppId": {
  
  }
}
```

At this point the entry fails because elasticsearch cannot match that to the template. I have tried removing the field with:

```auto
mutate{
  remove_field => ["[entry][AppID]" ]
}
```

but the filter does not remove the field, i assume because it is a nested field even though it has nothing nested. Any help is greatly appreciated as i have been beating my head against the wall on this one.

---

<div class="post-metadata">

### Author: ![whyapenny](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@whyapenny](https://discuss.elastic.co/u/whyapenny)
#### Post date: [August 3, 2016, 2:58pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/2 "2016-08-03T14:58:05Z")

</div>

I have tried deleted every field that gets pulled in blank, such as:

```auto
"entry": {
  "AppId": {
  }
}
```

by doing the following:

```auto
mutate{
  remove_field => ["[entry][AppId]" ]
}
```

yet it does not get removed. Perhaps this is because AppId is itself now the parent of a nested field, and thus not getting removed? Any ideas how i could remove that? Also, it would be nice if i could find a way to remove it ONLY if it is an empty nested field, and leave it when it is in fact populated.

Are there any conditions i could use to determine when entry.AppId is an empty nested field and is there a mutate function that will delete it?

---

<div class="post-metadata">

### Author: ![gmoskovicz](https://avatars.discourse-cdn.com/v4/letter/g/ecae2f/32.png) [@gmoskovicz](https://discuss.elastic.co/u/gmoskovicz)
#### Post date: [August 3, 2016, 4:40pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/3 "2016-08-03T16:40:30Z")

</div>

> [@whyapenny](#):
>
> \<?xml version="1.0" encoding="UTF-8"?\>\n\<AppId/\>

Hi !

What exact Logstash version are you using? In addition, can you please share the complete LS configuration?

Thanks!

--Gabriel

---

<div class="post-metadata">

### Author: ![whyapenny](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@whyapenny](https://discuss.elastic.co/u/whyapenny)
#### Post date: [August 3, 2016, 5:41pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/4 "2016-08-03T17:41:28Z")

</div>

we are using logstash-2.3.2. The configs are really long, and a compilation of over a dozen files. the config pertaining to this call is isolated to the xml piece. ie:

```auto
if "app" in [tags]{
    xml {
       target => entry
       source => message
       force_array => false
    }
    date{
       match => ["[entry][Timestamp]", "YYYY-MM-dd HH:mm:ss.SSS Z", "ISO8601" ]
    }
}
```

---

<div class="post-metadata">

### Author: ![whyapenny](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@whyapenny](https://discuss.elastic.co/u/whyapenny)
#### Post date: [August 3, 2016, 5:44pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/5 "2016-08-03T17:44:54Z")

</div>

also, to expand and add the input and output piece, the result config should be as simple as:

```auto
input{
 beats{
   port => 5044
 }
}
filter{
if "app" in [tags]{
    xml {
       target => entry
       source => message
       force_array => false
    }
    date{
       match => ["[entry][Timestamp]", "YYYY-MM-dd HH:mm:ss.SSS Z", "ISO8601" ]
    }
}
}
if "app" in [tags]{
   elasticsearch {
   hosts => ["es01", "es02"]
   workers => 16
   index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
 }
```

---

<div class="post-metadata">

### Author: ![gmoskovicz](https://avatars.discourse-cdn.com/v4/letter/g/ecae2f/32.png) [@gmoskovicz](https://discuss.elastic.co/u/gmoskovicz)
#### Post date: [August 3, 2016, 5:58pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/6 "2016-08-03T17:58:34Z")

</div>

> [@whyapenny](#):
>
> \<?xml version="1.0" encoding="UTF-8"?\>\n\<AppId/\>

Hi!

So i think that i might be missing something. I am testing the following which should be a similar test than what you are doing:

```
input{

	stdin{}
}
filter{
   
   xml {
       target => entry
       source => message
       force_array => false
    }
    date{
       match => ["[entry][Timestamp]", "YYYY-MM-dd HH:mm:ss.SSS Z", "ISO8601" ]
    }
}

output{
	stdout{codec=>rubydebug}
}

```

The result of this is the following:

```
<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<AppId>ConsumeAll</AppId>

{
       "message" => "<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<AppId>ConsumeAll</AppId>",
      "@version" => "1",
    "@timestamp" => "2016-08-03T17:44:28.578Z",
          "host" => "Gabriels-MacBook-Pro.local",
         "entry" => "ConsumeAll"
}

<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<AppId/>

{
       "message" => "<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<AppId/>",
      "@version" => "1",
    "@timestamp" => "2016-08-03T17:44:35.114Z",
          "host" => "Gabriels-MacBook-Pro.local",
         "entry" => {}
}

```

---

<div class="post-metadata">

### Author: ![gmoskovicz](https://avatars.discourse-cdn.com/v4/letter/g/ecae2f/32.png) [@gmoskovicz](https://discuss.elastic.co/u/gmoskovicz)
#### Post date: [August 3, 2016, 6:02pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/7 "2016-08-03T18:02:09Z")

</div>

> [@whyapenny](#):
>
> remove\_field =\> ["[entry][AppID]" ]

By the way, if i do the following, then the field inside entry is removed:

```
{
       "message" => "<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<AppId/>",
      "@version" => "1",
    "@timestamp" => "2016-08-03T18:00:05.140Z",
          "host" => "Gabriels-MacBook-Pro.local",
         "entry" => {}
}

```

I just tried to force my document to contain that field with the following:

```
   xml {
       target => entry
       source => message
       force_array => false
    }

    mutate {
    	rename => ['entry', '[entry][AppID]']
    }

	mutate { 
		remove_field => ["[entry][AppID]" ] 
	}

```

So it's working on my side, however i am sure that we are missing a single thing that is causing this.

Thanks!

--Gabriel

---

<div class="post-metadata">

### Author: ![whyapenny](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@whyapenny](https://discuss.elastic.co/u/whyapenny)
#### Post date: [August 3, 2016, 6:15pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/8 "2016-08-03T18:15:59Z")

</div>

I did not provide the full xml since i was hopeful it would be a known issue and i was just missing something. try this xml pulled down from filebeat output:

```auto
"@timestamp": "2016-08-03T18:02:11.219Z",
  "beat": {
    "hostname": "app01",
    "name": "app01"
  },
  "count": 1,
  "input_type": "log",
  "message": "2016-08-03 13:54:32,084 INFO app.Test.Logstash [Job0] - \u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cns0:AppLogTest xmlns:ns0=\"http://log.internal.pri/log/Namespaces/Interface.xsd\"\u003e\u003cns0:Name\u003eProc/Comm/framework.proc\u003c/ns0:Name\u003e\u003cns0:ApplId/\u003e\u003cns0:Origin/\u003e\u003cns0:Subject\u003eStart Test Logging\u003c/ns0:Subject\u003e\u003cns0:conId/\u003e\u003cns0:OriginName/\u003e\u003cns0:Class\u003elogManagement\u003c/ns0:Class\u003cns0:Timestamp\u003e2016-08-03T13:54:32.083-04:00\u003c/ns0:Timestamp\u003e\u003c/ns0:AppLogTest\u003e \t ",
  "offset": 236901,
  "source": "/app/log/test.log",
  "tags": [
    "app",
    "test",
    "log",
    "app01"
  ],
  "type": "logtest"
```

For CSV filter i see the option "skip\_empty\_columns". I was hoping there was something similar in xml filter but I have not found anything to that effect.

---

<div class="post-metadata">

### Author: ![gmoskovicz](https://avatars.discourse-cdn.com/v4/letter/g/ecae2f/32.png) [@gmoskovicz](https://discuss.elastic.co/u/gmoskovicz)
#### Post date: [August 3, 2016, 6:40pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/9 "2016-08-03T18:40:00Z")

</div>

So it looks like the xml is the following:

```
<?xml version="1.0" encoding="UTF-8"?>
<ns0:AppLogTest xmlns:ns0=xxxxxxxxxx>
   <ns0:Name>xxxxxxxxxx</ns0:Name>
   <ns0:ApplId />
   <ns0:Origin />
   <ns0:Subject>Start Test Logging</ns0:Subject>
   <ns0:conId />
   <ns0:OriginName />
   <ns0:Class>logManagement</ns0:Class>
   <ns0:Timestamp>2016-08-03T13:54:32.083-04:00</ns0:Timestamp>
</ns0:AppLogTest>

```

That will generate a json with a field name `ApplId`. It needs to have the complete field name and lower case `d`. The following shuold work in this case:

```
	mutate { 
		remove_field => ["[entry][ApplId]" ] 
	}

```

If you add the correct field name , then the document generated is the following:

```
{
       "message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?><ns0:AppLogTest xmlns:ns0=\"http://log.internal.pri/log/Namespaces/Interface.xsd\"><ns0:Name>Proc/Comm/framework.proc</ns0:Name><ns0:ApplId/><ns0:Origin/><ns0:Subject>Start Test Logging</ns0:Subject><ns0:conId/><ns0:OriginName/><ns0:Class>logManagement</ns0:Class><ns0:Timestamp>2016-08-03T13:54:32.083-04:00</ns0:Timestamp></ns0:AppLogTest>",
      "@version" => "1",
    "@timestamp" => "2016-08-03T18:36:19.066Z",
          "host" => "Gabriels-MacBook-Pro.local",
         "entry" => {
         "xmlns:ns0" => "http://log.internal.pri/log/Namespaces/Interface.xsd",
              "Name" => "Proc/Comm/framework.proc",
            "Origin" => {},
           "Subject" => "Start Test Logging",
             "conId" => {},
        "OriginName" => {},
             "Class" => "logManagement",
         "Timestamp" => "2016-08-03T13:54:32.083-04:00"
    }
}

```

Please let me know if this is the actual issue.

Thanks!

--Gabriel

---

<div class="post-metadata">

### Author: ![whyapenny](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@whyapenny](https://discuss.elastic.co/u/whyapenny)
#### Post date: [August 4, 2016, 3:40pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/10 "2016-08-04T15:40:32Z")

</div>

Ok, so looks like I can remove completely and it resolves the issue. I guess im still curious how/why it gets pulled in as a nested field instead of empty string.

Finally, my last question is how would i remove that field ONLY when it is empty? Since it isn't an empty string field and instead an empty nested field the following hasn't worked:  
this one fails to fulfill condition:

```auto
if "" in [entry][ApplId]{
	mutate { 
		remove_field => ["[entry][ApplId]" ] 
	}
}
```

this one actuates EVERY time:

```auto
if [entry][ApplId][]{
    mutate { 
        remove_field => ["[entry][ApplId]" ] 
    }
}
```

so is there a way i can write a condition which will remove the field only when it is empty, and not all or nothing? When there is information presented in ApplId i would prefer to keep that field.

---

<div class="post-metadata">

### Author: ![gmoskovicz](https://avatars.discourse-cdn.com/v4/letter/g/ecae2f/32.png) [@gmoskovicz](https://discuss.elastic.co/u/gmoskovicz)
#### Post date: [August 4, 2016, 4:08pm UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/11 "2016-08-04T16:08:03Z")

</div>

Does this works for you?

```
if !([entry][ApplId] =~ /.+/) {
	mutate { 
		remove_field => ["[entry][ApplId]" ] 
	}
}

```

This will check if the field has any value. If doesn't , it'll prune that field.

Please try out this and let us know if it works.

Thanks!

--Gabriel

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:44am UTC](https://discuss.elastic.co/t/xml-filter-plugin-creating-nested-field-out-of-null-object-cant-get-text-on-a-end-object/56928/12 "2017-07-06T04:44:51Z")

</div>