# Searching through PDF attachments and other documents in ElasticSearch with one query

**URL:** <https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519>\
**Category:** Elasticsearch\
**Created:** [September 30, 2020, 2:04pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519 "2020-09-30T14:04:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bertrand\_Nau](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand_nau/32/45778_2.png) [@Bertrand\_Nau](https://discuss.elastic.co/u/Bertrand_Nau)\
**Post date:** [September 30, 2020, 2:04pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519/1 "2020-09-30T14:04:43Z")

</div>

Hello everybody,

I'm using the ingest-attachment plugin to parse PDF files in an ElasticSearch 7 cluster. Each PDF file gives additional informations to an already existing document.

I try to create a query which retrieves all the documents which contains a given text, either in their properties, either in their corresponding PDF file.

Ideally, I would like to store the PDF file content as a field of the already existing document, but I can't find a way to do it with the ingest-attachment plugin.

As a workaround, I thought of making a kind of one-to-one join query, but some sources say that it should be avoided if possible.

Is there a proper solution for this use case?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 30, 2020, 4:25pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519/2 "2020-09-30T16:25:51Z")

</div>

Welcome!

OOTB the ingest attachment plugin also stores the binary content. I'm not sure I understood your question or problem. Could you explain?

That said, I'm not a fan of storing the binary content in Elasticsearch. Instead, I'd recommend storing it in a 3rd party service (like an http service) and just store the link inside the final document (in addition to the extracted text).

I'd do something like:

```auto
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    },{
      "remove": {
        "field": "data"
      }
    }
  ]
}
PUT my-index-00001/_doc/my_id?pipeline=attachment
{
  "url": "http://server/path/to/file.txt",
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
GET my-index-00001/_doc/my_id

```

This should give:

```auto
{
  "found": true,
  "_index": "my-index-00001",
  "_type": "_doc",
  "_id": "my_id",
  "_version": 1,
  "_seq_no": 22,
  "_primary_term": 1,
  "_source": {
    "url": "http://server/path/to/file.txt",
    "attachment": {
      "content_type": "application/rtf",
      "language": "ro",
      "content": "Lorem ipsum dolor sit amet",
      "content_length": 28
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Bertrand\_Nau](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand_nau/32/45778_2.png) [@Bertrand\_Nau](https://discuss.elastic.co/u/Bertrand_Nau)\
**Post date:** [September 30, 2020, 5:20pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519/3 "2020-09-30T17:20:51Z")

</div>

Hi @dadoonet, thank you for your answer.

Indeed, I have created the same pipeline in order to keep the text content of the PDF without the binary content.

I have an index containing people with some properties (firstname, lastname...). For each people, I can have a resume, and I want to find the people by firstname, lastname, or resume content.

Currently, I have two different indexes, one for the people, and one for the resumes and I don't know how to make the query. I suppose that the modelling of the data should be improved. For instance, people metadata and resume content could be in the same document, but I don't know if it's possible with the ingest-attachment plugin.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 30, 2020, 7:03pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519/4 "2020-09-30T19:03:18Z")

</div>

yeah. I'd most likely do something like:

```auto
PUT my-index-00001/_doc/my_id?pipeline=attachment
{
  "name": "David P",
  "birthdate": "XX/XX/XXXX",
  "country": "France",
  "resume": {
    "url": "http://server/path/to/file.txt",
    "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  }
}

```

Another idea is to use the enrich processor to do the lookup at index time. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/match-enrich-policy-type.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/match-enrich-policy-type.html)

---

<div class="post-metadata">

**Author:** ![Bertrand\_Nau](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bertrand_nau/32/45778_2.png) [@Bertrand\_Nau](https://discuss.elastic.co/u/Bertrand_Nau)\
**Post date:** [October 1, 2020, 1:25pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519/5 "2020-10-01T13:25:23Z")

</div>

Hi @dadoonet, thanks for your answer. I've tested and it works perfectly!!!

I have one last question regarding this topic. Is it possible to put the attachment without resending the people properties (name, birthdate...)?

Currently, I need to do a GET query before every PUT query in order to retrieve the people properties.

I have checked the Update API, and it's not able to call a pipeline. I have also checked the script processor, but I feel it can't solve it neither.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 1, 2020, 1:46pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519/6 "2020-10-01T13:46:57Z")

</div>

No. You need to send everything again.

The `GET` you are doing could be done though by the enrich processor. It would do the lookup at index time. See

> **[Example: Enrich your data based on exact values | Elasticsearch Reference...](https://www.elastic.co/guide/en/elasticsearch/reference/current/match-enrich-policy-type.html)**

In which case, you would only send something like:

```auto
PUT my-index-00001/_doc/my_id?pipeline=attachment
{
  "username": "dadoonet",
  "resume": {
    "url": "http://server/path/to/file.txt",
    "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0=",
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 29, 2020, 1:47pm UTC](https://discuss.elastic.co/t/searching-through-pdf-attachments-and-other-documents-in-elasticsearch-with-one-query/250519/7 "2020-10-29T13:47:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.