# How to index text files (pdf, doc, txt...) in Java?

**URL:** <https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678>\
**Category:** Elasticsearch\
**Created:** [December 20, 2022, 6:28pm UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678 "2022-12-20T18:28:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![gagidza](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@gagidza](https://discuss.elastic.co/u/gagidza)\
**Post date:** [December 20, 2022, 6:28pm UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678/1 "2022-12-20T18:28:28Z")

</div>

Hello everyone, some time ago I started to deal with Elasticsearch.  
Now I am faced with the task of indexing various text files from a certain directory for subsequent search by their contents. Since Elasticsearch uses Apache Tika to parse various files, I think this is what I need. At this stage, it will be enough to save files by 4 parameters: name, file type, absolute path and file content.  
I have already read similar posts ([How to store .pdf or .txt file and search in it using Java](https://discuss.elastic.co/t/how-to-store-pdf-or-txt-file-and-search-in-it-using-java/304025); [How to index and store pdf file in elastic search using spring boot?](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289?page=3)) and the article [Attachment processor | Elasticsearch Guide [8.5] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/attachment.html#attachment). But so far it's not entirely clear to me how to write a similar request for indexing a file using Java.  
If someone met with a similar problem and has experience in solving it, I will be glad of any help. Thank you in advance.

---

<div class="post-metadata">

**Author:** ![thn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thn/32/8061_2.png) [@thn](https://discuss.elastic.co/u/thn)\
**Post date:** [December 20, 2022, 6:52pm UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678/2 "2022-12-20T18:52:12Z")

</div>

> **[Indexing single documents | Elasticsearch Java API Client \[8.5\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/8.5/indexing.html)**

for a given directory,

1. Get a list of files
2. For each file,  
2.1 get the metadata such as file name, file path, file type  
2.2 get file content (for PDF, extract the text)  
2.3 index the data

or  
-\> create a JSON doc with four fields: file\_name, file\_path, file\_type, file\_content  
-\> then index this JSON doc [(using raw JSON data)](https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/8.5/indexing.html#_using_raw_json_data)

---

<div class="post-metadata">

**Author:** ![gagidza](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@gagidza](https://discuss.elastic.co/u/gagidza)\
**Post date:** [December 20, 2022, 7:08pm UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678/3 "2022-12-20T19:08:51Z")

</div>

Thanks, I chose the same steps to complete.  
I have a solution for this problem for pdf files. I am using pdfbox to get the content. But I want an elegant solution for all the large number of file types. And I understand that Elasticsearch is able to parse various documents itself. Or am I wrong?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 20, 2022, 7:41pm UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678/4 "2022-12-20T19:41:01Z")

</div>

Not sure it helps but you can also give a look at fscrawler project. 😊

---

<div class="post-metadata">

**Author:** ![gagidza](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@gagidza](https://discuss.elastic.co/u/gagidza)\
**Post date:** [December 21, 2022, 7:42am UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678/5 "2022-12-21T07:42:19Z")

</div>

I want to share the results, I think it will be useful to someone:

```auto
public Document parseFile(String filePath) throws IOException, TikaException, SAXException {
        Map<String, String> metaMap = new HashMap<>();
        String content = "";

        File file = new File(filePath);
        InputStream input = new ByteArrayInputStream(FileUtils.readFileToByteArray(file));
        BodyContentHandler handler = new BodyContentHandler(-1);
        Metadata metadata = new Metadata();
        Parser parser = new AutoDetectParser();
        parser.parse(input, handler, metadata, new ParseContext());

        content = handler.toString();
        Arrays.stream(metadata.names()).forEach(n -> metaMap.put(n, metadata.get(n)));
        
        Document document = new Document();
        document.setName(file.getName());
        document.setPath(file.getAbsolutePath());
        document.setText(content);
        
        return document;
    }

```

```auto
public String indexDocument(String filePath) throws IOException, TikaException, SAXException {
        Document document = documentService.parseFile(filePath);

        ElasticsearchClient client = esRestClient.getElasticSearchClient();

        IndexResponse response = client.index(i -> i
                .index(DOCUMENT_INDEX)
                .document(document));

        logger.info("File {} located in the directory: {} was successfully indexed.", document.getName(), document.getPath());

        return response.result().toString();
    }

```

```auto
@GetMapping("/parse")
    public void indexDocuments() {
        List<String> documents = Arrays.asList(PDF_TEXT, PDF_IMAGE, DOC, DOCX, TXT);
        for (String doc : documents) {
            service.indexDocument(doc);
        }
    }

```

This is an intermediate option, I am not using metadata information yet and the document has only 3 fields so far (name, absolute path and content).

Summing up, with the help of Apache Teak it was possible to get all the necessary information from various types of text files. It was not possible to get the contents from the pdf file created from the pictures. I will continue to look for a more beautiful and flexible solution through the attachment processor.

Also thanks for mentioning the fscrawler project. Very interesting project, should be studied.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 21, 2022, 9:48am UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678/6 "2022-12-21T09:48:53Z")

</div>

> [@gagidza](#):
>
> Also thanks for mentioning the fscrawler project. Very interesting project, should be studied.

Yeah. It does basically what you wrote... 😉

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 18, 2023, 9:48am UTC](https://discuss.elastic.co/t/how-to-index-text-files-pdf-doc-txt-in-java/321678/7 "2023-01-18T09:48:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.