# How to index and store pdf file in elastic search using spring boot? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 21, 2020, 4:04am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289 "2020-02-21T04:04:02Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 10, 2020, 8:36am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/23 "2020-03-10T08:36:55Z")
Yes I have gone through that documentation. In that I came across the creation of pipeline using PUT \_ingest/pipeline/attachment. But whatever we are creating or inserting is done via kibana or postman.But I would like to create pipeline via my java code. ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 10, 2020, 8:42am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/24 "2020-03-10T08:42:24Z")
I also have one more doubt.As u mentioned ingest plugin is used to extract the info from pdf, but here my main concern is to store the files into elastic search ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 10, 2020, 8:59am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/25 "2020-03-10T08:59:48Z")
> [@samanvi\_sweety](#): > > ut I would like to create pipeline via my java code. The documentation is here: [Create or update pipeline API | Java REST Client [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-ingest-put-pipeline.html) > [@samanvi\_sweety](#): > > As u mentioned ingest plugin is used to extract the info from pdf, but here my main concern is to store the files into Elasticsearch Well. I'm always hesitating doing that. Elasticsearch is not ideal to store big binary blobs. So I'd say that it can be ok if the size of your documents is limited to say 10kb. If you are planning to store mega bytes of documents, that's another story. It comes with a lot of costs. Like the network bandwidth when you are going to fetch the 10 top relevant results. 1mb \* 10 equals 10mb for each search response... You need to think about it... I prefer storing the binary in another datastore or on a file system and just index in Elasticsearch the extracted content and the metadata. ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 10, 2020, 9:12am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/26 "2020-03-10T09:12:02Z")
> [@dadoonet](#): > > Well. I'm always hesitating doing that. Elasticsearch is not ideal to store big binary blobs. So I'd say that it can be ok if the size of your documents is limited to say 10kb. > If you are planning to store mega bytes of documents, that's another story. > > It comes with a lot of costs. Like the network bandwidth when you are going to fetch the 10 top relevant results. 1mb \* 10 equals 10mb for each search response... > You need to think about it... > > I prefer storing the binary in another datastore or on a file system and just index in Elasticsearch the extracted content and the metadata. Okay thanku so much for ur time ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 10, 2020, 10:14am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/27 "2020-03-10T10:14:15Z")
Our file size would be max 130kb. So can we store them by using ingest plugin? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 10, 2020, 11:06am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/28 "2020-03-10T11:06:31Z")
Sounds like possible to me. but note that there is no relationship between "storing" the binary in elasticsearch and indexing its content in elasticsearch. Yon can do: - index content (with the ingest attachment plugin) - store the file - index content (with the plugin) and store the content So it depends on the use case I'd say. ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 11, 2020, 5:38am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/29 "2020-03-11T05:38:16Z")
I dont want to index the content. I would like to index the file. Can we store direct file into ES? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 5:54am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/30 "2020-03-11T05:54:33Z")
What does mean "index the file"? Indexing and storing are different things. ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 11, 2020, 5:57am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/31 "2020-03-11T05:57:24Z")
Giving some index to the file so that it can be fetched by using that index. and later storing it. If its possible ,Can we store file directly without any indexing.I just want to store the file into ES. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 5:58am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/32 "2020-03-11T05:58:54Z")
Yes you can. Just use the binary data type. ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 11, 2020, 6:09am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/33 "2020-03-11T06:09:04Z")
Where do I need to mention as binary/ For example I am creating pipeline by using the code below.Where exactly should i make it as binary ```auto String source = "{\"description\":\"my set of processors\"," + "\"processors\":[{\"set\":{\"field\":\"data\",\"value\":\"encodedfile\"}}]}"; PutPipelineRequest request = new PutPipelineRequest( "ourpipeline", new BytesArray(source.getBytes(StandardCharsets.UTF_8)), XContentType.JSON ); AcknowledgedResponse response = client().ingest().putPipeline(request, RequestOptions.DEFAULT); ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 6:26am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/34 "2020-03-11T06:26:40Z")
In the mapping. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/binary.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/binary.html) If you just want to store the binary file as BASE64, you don't need a pipeline. So you don't need the ingest attachment plugin. ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 11, 2020, 6:39am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/35 "2020-03-11T06:39:54Z")
Okay but all those put requests are performed via postman or kibana to store the data.What if we want to use the above approach of binary data type through java code? ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 11, 2020, 9:19am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/36 "2020-03-11T09:19:53Z")
And also I have gone through the binary data type.It is clearly mentioned that it cannot be searched or stored by default.So what if I want to fetch my file which i stored earlier?What should I do at that time? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 10:27am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/37 "2020-03-11T10:27:54Z")
> It is clearly mentioned that it cannot be searched or stored by default. It can not be searched as it is not indexed. You told me that you don't want to index it but just store it. So we are good. The field is not stored by default. It is a good default. As the binary is also stored within the `_source` field. > [@samanvi\_sweety](#): > > So what if I want to fetch my file which i stored earlier?What should I do at that time? ```auto PUT my_index { "mappings": { "properties": { "name": { "type": "text" }, "blob": { "type": "binary" } } } } PUT my_index/_doc/1 { "name": "Some binary blob", "blob": "U29tZSBiaW5hcnkgYmxvYg==" } GET my_index/_doc/1 ``` This will give you back something like: ```auto { // ... "_source": { "name": "Some binary blob", "blob": "U29tZSBiaW5hcnkgYmxvYg==" } } ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 11, 2020, 10:33am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/38 "2020-03-11T10:33:19Z")
> What if we want to use the above approach of binary data type through java code? You need to create the mapping accordingly. See [Update mapping API | Java REST Client [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-put-mapping.html) or [Create Index API | Java REST Client [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-create-index.html) Then you need to use the Index API as you already did: ```auto @PostMapping("/upload") public String upload() { String filePath = "C://x.pdf"; String encodedfile = null; RestHighLevelClient restHighLevelClient = null; File file = new File(filePath); try { FileInputStream fileInputStreamReader = new FileInputStream(file); byte[] bytes = new byte[(int) file.length()]; fileInputStreamReader.read(bytes); encodedfile = new String(Base64.getEncoder().encodeToString(bytes)); } catch (IOException e) { } try { restHighLevelClient = new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http"))); } catch (Exception e) { } Map jsonMap = new HashMap<>(); jsonMap.put("Name", "samanvi"); jsonMap.put("postDate", new Date()); jsonMap.put("hra", encodedfile); IndexRequest request = new IndexRequest("index","_doc","56") .index("index") .source("field",jsonMap); try { IndexResponse response = restHighLevelClient.index(request, RequestOptions.DEFAULT); } catch(ElasticsearchException | IOException e) { } return "uploaded"; } ``` ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 11, 2020, 10:46am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/39 "2020-03-11T10:46:17Z")
Thanks for ur time.I'll give it a try and get back ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 13, 2020, 10:01am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/41 "2020-03-13T10:01:49Z")
Hi David, I have tried implementing the above approach.Index is succesfully getting created but i am unable to pass the encoded string via put mapping ```auto @GetMapping("/final") public String trail() throws IOException { String filePath="C:\\Users\\xyz.pdf"; String encodedfile = null; File file = new File(filePath); try { FileInputStream fileInputStreamReader = new FileInputStream(file); byte[] bytes = new byte[(int) file.length()]; fileInputStreamReader.read(bytes); encodedfile = new String(Base64.getEncoder().encodeToString(bytes)); } catch (IOException e) { e.printStackTrace(); } IndexRequest request1 = new IndexRequest("twitter","_doc","56"); request1.source( "{\n" + " \"properties\": {\n" + " \"message\": {\n" + " \"type\": \"binary\"\n" + " }\n" + " }\n" + "}", XContentType.JSON); IndexResponse createIndexResponse = client().index(request1, RequestOptions.DEFAULT); System.out.print(createIndexResponse); PutMappingRequest request3 = new PutMappingRequest("twitter"); request3.source("{\n" + " \"message\": encodedfile " + "}", XContentType.JSON); AcknowledgedResponse putMappingResponse1 = client().indices().putMapping(request3, RequestOptions.DEFAULT); return "done"; } ``` I am getting the below error: ```auto 2020-03-13 15:21:02.627 ERROR 1660 --- [nio-8085-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=parse_exception, reason=Failed to parse content to map]]; nested: ElasticsearchException[Elasticsearch exception [type=json_parse_exception, reason=Unrecognized token 'encodedfile': was expecting ('true', 'false' or 'null') at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@21a5b7d2; line: 2, column: 25]]];] with root cause ``` Can you please help ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 13, 2020, 10:53am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/42 "2020-03-13T10:53:59Z")
You are mixing the index document request and the put mapping request. This should be in the put mapping request: ``` "{\n" + " \"properties\": {\n" + " \"message\": {\n" + " \"type\": \"binary\"\n" + " }\n" + " }\n" + "}", ``` And this should be the index request: ``` "{\n" + " \"message\": encodedfile " + "}" ``` Note that you first create the index, the mapping and then you can index documents. ---
**Author:** ![samanvi\_sweety](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samanvi_sweety/32/56190_2.png) [@samanvi\_sweety](https://discuss.elastic.co/u/samanvi_sweety)\ **Post date:** [March 16, 2020, 5:53am UTC](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289/44 "2020-03-16T05:53:14Z")
Thanku so much. It is working after changing as per ur instructions. Here is what I did ```auto @GetMapping("/final") public String trail() throws IOException { String filePath="C:\\Users\\w8ben.pdf"; String encodedfile = null; File file = new File(filePath); try { FileInputStream fileInputStreamReader = new FileInputStream(file); byte[] bytes = new byte[(int) file.length()]; fileInputStreamReader.read(bytes); encodedfile = new String(Base64.getEncoder().encodeToString(bytes)); } catch (IOException e) { e.printStackTrace(); } System.out.println(encodedfile); //creating index CreateIndexRequest request = new CreateIndexRequest("twitter"); CreateIndexResponse createIndexResponse = client().indices().create(request, RequestOptions.DEFAULT); System.out.print(createIndexResponse); //mapping properties PutMappingRequest request2 = new PutMappingRequest("twitter"); request2.source( "{\n" + " \"properties\": {\n" + " \"message\": {\n" + " \"type\": \"binary\"\n" + " }\n" + " }\n" + "}", XContentType.JSON); AcknowledgedResponse putMappingResponse = client().indices().putMapping(request2, RequestOptions.DEFAULT); IndexRequest request3 = new IndexRequest("twitter","_doc","56"); request3.source("{\n" + " \"message\": " +"\""+ encodedfile +"\""+ "}", XContentType.JSON); IndexResponse createIndexResponse1= client().index(request3, RequestOptions.DEFAULT); return "done"; } ``` [Previous page](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289.md?page=1) [Next page](https://discuss.elastic.co/t/how-to-index-and-store-pdf-file-in-elastic-search-using-spring-boot/220289.md?page=3)