# Request to extract metadata fields in the self-hosted content extraction service

**URL:** <https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131>\
**Category:** Elasticsearch\
**Tags:** docker, connectors\
**Created:** [February 26, 2024, 3:07pm UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131 "2024-02-26T15:07:38Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![jan.stap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan.stap/32/97404_2.png) [@jan.stap](https://discuss.elastic.co/u/jan.stap)\
**Post date:** [February 26, 2024, 3:07pm UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/1 "2024-02-26T15:07:38Z")

</div>

Hi,

I'm using the [self-hosted content extraction service](https://www.elastic.co/guide/en/enterprise-search/8.12/connectors-content-extraction.html#connectors-content-extraction-data-extraction-service) to be able to ingest office documents larger than 10 MBytes via the [Elastic S3 connector](https://www.elastic.co/guide/en/enterprise-search/8.12/connectors-s3.html), which we need for a client project.

I notice that only the document content is extracted into event field "body" and not metadata fields like "title", "description", "author", etc. When using the Elasticsearch Attachment processor instead, these field are extracted, but that method is limited to \<=10 Mbytes documents.

Looking into the data-extraction-service docker container and into the [connector code](https://github.com/elastic/connectors/tree/8.12/connectors), I find that infact metadata _is_ extracted (I tested that by running a local Apache Tika of the same 2.8.0 version), but the script `/app/lua/tika-response-body.lua` passes only the `X-TIKA:content` field to the client, where it is assigned to the "body" field by [source.py](https://github.com/elastic/connectors/blob/8.12/connectors/source.py), line 814.

It looks straightforward to include these metadata fields, like is done for the Elasticsearch/Attachment processor case. Is this planned within short?

Thanks,

Jan Stap

---

<div class="post-metadata">

**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\
**Post date:** [February 29, 2024, 10:51am UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/2 "2024-02-29T10:51:20Z")

</div>

Hi @jan.stap

Thanks for your question!  
The Data Extraction Service is still in beta, because we're still trying to gather feedback on how it _should_ behave. On one hand, we want it to be feature rich, on another hand, we don't want to absorb the support burden of 100% of the Tika Server APIs and features. If you're a customer, I'd definitely ask you to work with Support to file an Enhancement Request so that we can capture your use case and plan for it. Alternatively, if you're interested, we can try to connect you directly with one of our product managers (in a less public thread, of course).

---

<div class="post-metadata">

**Author:** ![jan.stap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan.stap/32/97404_2.png) [@jan.stap](https://discuss.elastic.co/u/jan.stap)\
**Post date:** [March 4, 2024, 9:20am UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/3 "2024-03-04T09:20:36Z")

</div>

Hi Sean,

Thanks! For now we will use Apache Tika directly to ingest office documents into the Elastic Stack, but I'll file the enhancement request with Support, to help you guys plan on further dev on the Data Extraction Service.

Kind regards,  
Jan

---

<div class="post-metadata">

**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\
**Post date:** [March 4, 2024, 1:57pm UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/4 "2024-03-04T13:57:58Z")

</div>

Do note, if your office documents are under 100MB, you can use the [Attachment Processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/attachment.html) in an Elasticsearch Ingest Pipeline. It is able to return a lot more (if not all) metadata fields.

---

<div class="post-metadata">

**Author:** ![jan.stap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan.stap/32/97404_2.png) [@jan.stap](https://discuss.elastic.co/u/jan.stap)\
**Post date:** [March 21, 2024, 3:07pm UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/5 "2024-03-21T15:07:15Z")

</div>

Hi Sean,

Aha, thanks for the info! So you are saying (as I understand) that the 10MB limit is specific to the Elastic Connector?

We do have some documents over 100 MB though. For the \<=10MB case we already created an @custom pipeline on the search index, fetching the metadata fields from the Attachment processor output. The default pipeline only passes the .content field of the result.

Thanks,  
Jan

---

<div class="post-metadata">

**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\
**Post date:** [March 21, 2024, 7:56pm UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/6 "2024-03-21T19:56:39Z")

</div>

@jan.stap yes, the 10mb is/was a limit that's hardcoded in the connector codebase. In its most recent branches, [this has been made configurable](https://github.com/elastic/connectors/pull/2080). The 10mb limit was a legacy holdover that didn't really have a valid justification anymore. If you're on an older branch, feel free to fork the repo and just modify this constant to increase the limit up to ~100MB: [connectors/connectors/source.py at 8.12 · elastic/connectors · GitHub](https://github.com/elastic/connectors/blob/8.12/connectors/source.py#L41)

> We do have some documents over 100 MB though

The big fish always do. 🙂 This is where Tika or the Data Extraction Service shine, because the 100MB limit comes from Elasticsearch's `http.max_content_length` default value, which we strongly recommend you not change. This limits payload size in Elasticsearch requests, so the Attachment Processor can't receive documents larger than this. So you'd need to choose to either drop such files (often, files this large only negatively impact search relevance anyway), or use a tool that lets you extract content on the edge before sending it to Elasticsearch.

> The default pipeline only passes the .content field of the result.

Yeah, this was an intentional tradeoff. We figured it was better to not index more fields (increasing customer costs) by default, and let that be somthing you opt into. In the `@custom` workaround, hopefully you saw that the metadata fields are available to you to move/copy/process since they're generated before your `@custom` pipeline and removed _after_ your `@custom` pipeline.

---

<div class="post-metadata">

**Author:** ![Sean\_Story](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sean_story/32/69987_2.png) [@Sean\_Story](https://discuss.elastic.co/u/Sean_Story)\
**Post date:** [April 3, 2024, 4:17pm UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/7 "2024-04-03T16:17:58Z")

</div>

@jan.stap I wanted to let you know that we just made the repo for the Data Extraction Service public.

> **[GitHub - elastic/data-extraction-service](https://github.com/elastic/data-extraction-service)**
>
> Contribute to elastic/data-extraction-service development by creating an account on GitHub.

Since you'd invested the effort to reverse-engineer the project, I figured you might be excited to know that you now have easier access to the code, and to create Github issues/feature requests.

---

<div class="post-metadata">

**Author:** ![jan.stap](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jan.stap/32/97404_2.png) [@jan.stap](https://discuss.elastic.co/u/jan.stap)\
**Post date:** [April 8, 2024, 9:29am UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/8 "2024-04-08T09:29:24Z")

</div>

Hi Sean,

Thanks! Sorry for the late reply, I had an Easter holiday.

Cheers, Jan

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 6, 2024, 9:29am UTC](https://discuss.elastic.co/t/request-to-extract-metadata-fields-in-the-self-hosted-content-extraction-service/354131/9 "2024-05-06T09:29:51Z")

</div>

Closed on May 6, 2024, 9:29am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.