How can i put an xml file into elasticsearch by spring data elasticseach

Elastic Stack Elasticsearch
1

hi,
i'm new in elasticsearch, i tried to load xml file into elasticsearch by logstash, but sometimes it loads and sometimes it doesnt load, that's why decide to use filebeat to load it, it loads the file but the logstash filter doesnt work

finally i decide to use spring data elasticsearch but i dont have any idea how to use it

anybody can help me???

2

It would be better to explain what were the problems you had when using filebeat / logstash for example.

About Spring Data, that's a too vague question and I have no idea what to answer.

You can also look at FSCrawler which supports xml files. https://fscrawler.readthedocs.io/en/latest/admin/fs/local-fs.html#indexing-xml-docs

3

when i use filebeat, it load all the document file and put it into one field, the filter doesnt work i dont know why

4

about the spring i want to load a simple file into elasticsearch by spring-data-elastic , i want that the elasticsearch understand all fields, save every tag as a column of the data base and the data into the tag like an enregistrement

5

I can't tell without knowing what you did.

6

filebeats.yml
filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /var/log/test.xml

#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["logstash:5044"]

loadxml_filebeat.conf

input {
beats{
port => 5044
}
}

filter {
xml {
store_xml => false
source => "message"
xpath =>
[
"/xmldata/head1/id/text()", "id",
"/xmldata/head1/date/text()", "date",
"/xmldata/head1/key1/text()", "key1"
]
}
}

output {
elasticsearch {
index => "logstash_xml_filebeat"
hosts => "elasticsearch:9200"
document_id =>"%{id}"
}

}

test.xml

Value1 Value2 0001 01-01-2016 09:00:00 Value3
7

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

Also tell what is the problem with this? What went wrong, error logs...

8 
filebeat.inputs:
- type: log
 enabled: true
paths:
 - /var/log/test.xml


#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["logstash:5044"]

logstash config file

input {
beats{
port => 5044
}
}

filter {
xml {
store_xml => false
source => "message"
xpath =>
[
"/xmldata/head1/id/text()", "id",
"/xmldata/head1/date/text()", "date",
"/xmldata/head1/key1/text()", "key1"
]
}
}

output {
elasticsearch {
    index => "logstash_xml_filebeat"
    hosts => "elasticsearch:9200"
    document_id =>"%{id}"
}

}
9 
<xmldata>
 <head1>
  <key1>Value1</key1>
  <key2>Value2</key2>
  <id>0001</id>
  <date>01-01-2016 09:00:00</date>
 </head1>
 <head2>
  <key3>Value3</key3>
 </head2>
</xmldata>
10

@dadoonet??

11

As I wrote:

Also tell what is the problem with this? What went wrong, error logs...

That could help to give you an answer.
Also I recommend using the logstash stdout plugin to print how logstash is parsing your data.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.