Retrieve metadata of document in response to queries

Elastic Stack Elasticsearch
1

I am indexing the text content of a file by using
res = es.index(index=myIndex,body=jsonDict), where jsonDict is just the text of a pdf file in a json dictionary.
I am able to query the content and retrieve the relevant text.
My question is : how do I index the file contents so I could also retrieve file info such as :-
"file" : {
"extension" : "pdf",
"content_type" : "application/pdf",
"created" : "2019-10-01T15:28:31.000+0000",
"last_modified" : "2019-10-01T15:28:31.000+0000",
"last_accessed" : "2019-10-01T15:38:50.000+0000",
"indexing_date" : "2019-10-01T15:39:08.055+0000",
"filesize" : 877861,

and
"path" : {
"root" : "4a997482a3826d51751b8e7c01e476c",
"virtual" : "/P_GB27980_20120213.pdf",
"real" : "/Users/madabhuc/Documents/IR/presto/eval/P_GB27980_20120213.pdf"

Do I need to explicitly provide this info when I submit the files to ElasticSearch for indexing ?
Thanks.

2

You need to provide what you want to index. The full document that is.

3

The background is that I am trying to index pages of a pdf document which is currently not supported by fscrawler. So I am trying to interface with ElasticSearch by myself.
It is not clear to me how you are providing the file to ES : are you writing the tika output to a file and then sending it to ES ?

Thanks for your help.

4

From which language? Java?

5

python

6

In jsonDict you should provide the full JSON content, including the metadata and the extracted text.

7

including filesize ? How do I know path.root as in the example ?

" path" : {
"root" : "4a997482a3826d51751b8e7c01e476c",
"virtual" : "/P_GB27980_20120213.pdf",
"real" : "/Users/madabhuc/Documents/IR/presto/eval/P_GB27980_20120213.pdf"
....
}

8

You send whatever you want to index. If this data is not needed, don't send it.
I have hard time to understand what your actual problem is.

9

When I look at the output in response to a query, I see not just the content, but also data such as "path" as seen above. I understand that I have to pass in whatever data that I want to be seen in a query result. It is not clear to me how i would know the value of "root" as an example. or can i assume that such data is generated automatically by ES.

10

No data is generated automatically.

11

id is, that's why i wanted to clarify.
So how do you know the value of "root" ?

Many Thanks.

12

I don't know.

I can tell you how FSCrawler computes all that but that's another story. It's all FSCrawler internals.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.