# Indexing data in elasticsearch through logstash using xml files

**URL:** <https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242>\
**Category:** Logstash\
**Created:** [November 26, 2018, 8:52pm UTC](https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242 "2018-11-26T20:52:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrey\_Querejeta](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@Andrey\_Querejeta](https://discuss.elastic.co/u/Andrey_Querejeta)\
**Post date:** [November 26, 2018, 8:52pm UTC](https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242/1 "2018-11-26T20:52:05Z")

</div>

Hello, everybody:  
I am new using ELK, although I have managed to index data into elasticsearch through logstash, using the CSV filter.  
For several days I have tried to use the XML filter without results. I have read several posts but I have not found the solution to the problem.  
The index is not created in elasticsearch, does'nt show anything in the log.  
I would appreciate the help, because I don't know what else to do, I've really tried everything, but I haven't been able to index anything to ES using this filter.

XML file

* * *
 1 1 25 2012 1 2012/01/20 15:37:35 2012/01/01 00:00:00 1 TICK 2012/01/20 15:37:35 3 26 1000 2012 1 2012/02/07 15:19:34 2012/01/01 00:00:00 3 TICK 2012/02/07 15:19:34 

logstash.conf

* * *

input {  
file {  
path =\> "C:/Bitnami/elk/logstash/data/tabla.xml"   
start\_position =\> "beginning"  
sincedb\_path =\> "C:/Bitnami/elk/logstash/data/dev/nullo"  
codec =\> multiline  
{  
pattern =\> "^\<?main .\*\>"  
negate =\> true  
what =\> "previous"  
}   
}  
}

filter {  
xml {  
source =\> "message"  
force\_array =\> false  
remove\_namespaces =\>true  
store\_xml =\> true  
target =\> "doc"  
xpath =\> [  
"//main/DATA\_RECORD/IDRANGODOCTRAZA/text()", "myid",  
"//main/DATA\_RECORD/CODIGOTIPODOC/text()", "codigo"  
]  
}  
mutate {  
remove\_field =\> ["message", "@metadata"]  
}   
}

output {  
elasticsearch {  
index =\> "table"  
hosts =\> ["127.0.0.1:9200"]  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![Andrey\_Querejeta](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@Andrey\_Querejeta](https://discuss.elastic.co/u/Andrey_Querejeta)\
**Post date:** [November 26, 2018, 9:07pm UTC](https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242/2 "2018-11-26T21:07:23Z")

</div>

here is my xml... again  
\<main\>  
\<DATA\_RECORD\>  
\<IDRANGODOCTRAZA\>1\</IDRANGODOCTRAZA\>  
\<DESDE\>1\</DESDE\>  
\<HASTA\>25\</HASTA\>  
\<ANNO\>2012\</ANNO\>  
\<IDTIPOOPERMOVDOC\>1\</IDTIPOOPERMOVDOC\>  
\<FECHAMOV\>2012/01/20 15:37:35\</FECHAMOV\>  
\<FECHAENTREGA\>2012/01/01 00:00:00\</FECHAENTREGA\>  
\<ID\_USUA\>1\</ID\_USUA\>  
\<CODIGOTIPODOC\>TICK\</CODIGOTIPODOC\>  
\<FECHACREACION\>2012/01/20 15:37:35\</FECHACREACION\>  
\</DATA\_RECORD\>  
\<DATA\_RECORD\>  
\<IDRANGODOCTRAZA\>3\</IDRANGODOCTRAZA\>  
\<DESDE\>26\</DESDE\>  
\<HASTA\>1000\</HASTA\>  
\<ANNO\>2012\</ANNO\>  
\<IDTIPOOPERMOVDOC\>1\</IDTIPOOPERMOVDOC\>  
\<FECHAMOV\>2012/02/07 15:19:34\</FECHAMOV\>  
\<FECHAENTREGA\>2012/01/01 00:00:00\</FECHAENTREGA\>  
\<ID\_USUA\>3\</ID\_USUA\>  
\<CODIGOTIPODOC\>TICK\</CODIGOTIPODOC\>  
\<FECHACREACION\>2012/02/07 15:19:34\</FECHACREACION\>  
\</DATA\_RECORD\>  
\<DATA\_RECORD\>  
\<IDRANGODOCTRAZA\>4\</IDRANGODOCTRAZA\>  
\<DESDE\>26\</DESDE\>  
\<HASTA\>50\</HASTA\>  
\<ANNO\>2012\</ANNO\>  
\<IDTIPOOPERMOVDOC\>3\</IDTIPOOPERMOVDOC\>  
\<FECHAMOV\>2012/02/07 15:27:29\</FECHAMOV\>  
\<FECHAENTREGA\>2012/01/01 00:00:00\</FECHAENTREGA\>  
\<ID\_VEND\>8\</ID\_VEND\>  
\<ID\_USUA\>3\</ID\_USUA\>  
\<CODIGOTIPODOC\>TICK\</CODIGOTIPODOC\>  
\<FECHACREACION\>2012/02/07 15:27:29\</FECHACREACION\>  
\</DATA\_RECORD\>  
\<DATA\_RECORD\>  
\<IDRANGODOCTRAZA\>5\</IDRANGODOCTRAZA\>  
\<DESDE\>51\</DESDE\>  
\<HASTA\>1000\</HASTA\>  
\<ANNO\>2012\</ANNO\>  
\<IDTIPOOPERMOVDOC\>3\</IDTIPOOPERMOVDOC\>  
\<FECHAMOV\>2012/03/23 10:51:02\</FECHAMOV\>  
\<FECHAENTREGA\>2012/01/01 00:00:00\</FECHAENTREGA\>  
\<ID\_VEND\>3\</ID\_VEND\>  
\<ID\_USUA\>5\</ID\_USUA\>  
\<CODIGOTIPODOC\>TICK\</CODIGOTIPODOC\>  
\<FECHACREACION\>2012/03/23 10:51:02\</FECHACREACION\>  
\</DATA\_RECORD\>  
\<DATA\_RECORD\>  
\<IDRANGODOCTRAZA\>6\</IDRANGODOCTRAZA\>  
\<DESDE\>51\</DESDE\>  
\<HASTA\>1000\</HASTA\>  
\<ANNO\>2012\</ANNO\>  
\<IDTIPOOPERMOVDOC\>4\</IDTIPOOPERMOVDOC\>  
\<FECHAMOV\>2012/03/23 11:03:19\</FECHAMOV\>  
\<FECHAENTREGA\>2012/01/01 00:00:00\</FECHAENTREGA\>  
\<ID\_VEND\>3\</ID\_VEND\>  
\<ID\_USUA\>5\</ID\_USUA\>  
\<CODIGOTIPODOC\>TICK\</CODIGOTIPODOC\>  
\<FECHACREACION\>2012/03/23 11:03:19\</FECHACREACION\>  
\</DATA\_RECORD\>  
\</main\>

---

<div class="post-metadata">

**Author:** ![Andrey\_Querejeta](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@Andrey\_Querejeta](https://discuss.elastic.co/u/Andrey_Querejeta)\
**Post date:** [November 27, 2018, 1:54pm UTC](https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242/4 "2018-11-27T13:54:02Z")

</div>

Thank you for answering. I tried your suggestion, but it didn't work either. I think there's something wrong with my ELK, but I don't know what it can be. Here is the logstash:  
[2018-11-27T08:35:56,309][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2018-11-27T08:35:56,400][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.4.2"}  
[2018-11-27T08:35:57,935][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>8, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}  
[2018-11-27T08:35:58,105][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://127.0.0.1:9200/](http://127.0.0.1:9200/)]}}  
[2018-11-27T08:35:58,106][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://127.0.0.1:9200/](http://127.0.0.1:9200/), :path=\>"/"}  
[2018-11-27T08:35:58,177][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://127.0.0.1:9200/](http://127.0.0.1:9200/)"}  
[2018-11-27T08:35:58,214][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6}  
[2018-11-27T08:35:58,214][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[2018-11-27T08:35:58,216][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//127.0.0.1:9200](https://127.0.0.1:9200)"]}  
[2018-11-27T08:35:58,217][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}  
[2018-11-27T08:35:58,221][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://127.0.0.1:9200/](http://127.0.0.1:9200/)]}}  
[2018-11-27T08:35:58,222][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://127.0.0.1:9200/](http://127.0.0.1:9200/), :path=\>"/"}  
[2018-11-27T08:35:58,222][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[2018-11-27T08:35:58,228][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://127.0.0.1:9200/](http://127.0.0.1:9200/)"}  
[2018-11-27T08:35:58,245][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6}  
[2018-11-27T08:35:58,245][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[2018-11-27T08:35:58,247][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//127.0.0.1:9200](https://127.0.0.1:9200)"]}  
[2018-11-27T08:35:58,247][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil}  
[2018-11-27T08:35:58,249][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}}  
[2018-11-27T08:35:59,759][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
[2018-11-27T08:35:59,759][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections  
[2018-11-27T08:35:59,760][INFO][logstash.pipeline] Pipeline started successfully {:pipeline\_id=\>"main", :thread=\>"#\<Thread:0x546e06b8@C:/Bitnami/elk/logstash/logstash-core/lib/logstash/pipeline.rb:157 sleep\>"}  
[2018-11-27T08:35:59,774][INFO][logstash.agent] Pipelines running {:count=\>1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>}  
[2018-11-27T08:35:59,869][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 27, 2018, 1:55pm UTC](https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242/5 "2018-11-27T13:55:40Z")

</div>

Sorry, I just realised my mistake, your original config was correct. There is no error in your log however and is starting correctly?

---

<div class="post-metadata">

**Author:** ![Andrey\_Querejeta](https://avatars.discourse-cdn.com/v4/letter/a/8edcca/32.png) [@Andrey\_Querejeta](https://discuss.elastic.co/u/Andrey_Querejeta)\
**Post date:** [November 27, 2018, 2:07pm UTC](https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242/6 "2018-11-27T14:07:17Z")

</div>

Apparently ELK is working, in fact, I test others functionalities in Kibana with other indexs and everything works well, but the problem is that it does not index my xml data. Can you give me an example of a logstash.conf and an xml file that works for you, for me to test it on another computer? I don't know what else I can do. Yesterday I posted this topic in this forum because I had already tried several examples of this filter without results. Thanks in advance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2018, 2:07pm UTC](https://discuss.elastic.co/t/indexing-data-in-elasticsearch-through-logstash-using-xml-files/158242/7 "2018-12-25T14:07:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.