# Ingest attachmnet increase file content size to index

**URL:** <https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624>\
**Category:** Elasticsearch\
**Created:** [April 11, 2018, 12:19pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624 "2018-04-11T12:19:05Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 11, 2018, 12:19pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/1 "2018-04-11T12:19:06Z")

</div>

Hello. I am using Elasticsearch 5.4 and ingest-attachment plugin. It works fine with index, search, analyzing with file content up to 32kb, but I have a requirement for indexing and searching big files.  
I found the solution here in max\_content\_length: [https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-http.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-http.html)  
But my problem that I am using TransportClient and when I try to index big file I get the following exception:

java.lang.IllegalArgumentException: Document contains at least one immense term in field="attachment.content" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[80, 114, 101, 112, 97, 114, 101, 100, 32, 101, 120, 99, 108, 117, 115, 105, 118, 101, 108, 121, 32, 102, 111, 114, 32, 86, 106, 97, 99, 104]...', original message: bytes can be at most 32766 in length; got 395087

Could you explain me how to increase the file content size for TransportClient?

---

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 17, 2018, 2:13pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/2 "2018-04-17T14:13:42Z")

</div>

Hello. Could you answer my above question? I have to solve this problem in nearly future.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 17, 2018, 2:58pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/3 "2018-04-17T14:58:23Z")

</div>

I think you are wrongly using the ingest attachment plugin.

Could you tell what is your mapping, what is the ingest pipeline and how you index a document?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [April 18, 2018, 6:59am UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/4 "2018-04-18T06:59:09Z")

</div>

I think the error message is saying, that there is a **single** crazy big term in the document. It seems that the splitting in many terms was not successful or possible.

Is it possible that this document contains such a term? Or that tika was not able to create multiple terms out of this document...

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 18, 2018, 7:29am UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/5 "2018-04-18T07:29:47Z")

</div>

My theory is that he is indexing the BASE64 attachment has not been launched on this field.

---

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 18, 2018, 10:58am UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/6 "2018-04-18T10:58:40Z")

</div>

Thank you for replying.

**localhost:9200/\_ingest/pipeline/attachment**

```
{
"attachment": {
    "description": "Extract attachment information",
    "processors": [
        {
            "attachment": {
                "field": "payload",
                "indexed_chars": "-1",
                "properties": [
                    "content",
                    "content_type",
                    "content_length",
                    "title",
                    "language"
                ]
            },
            "remove": {
                "field": "payload"
            }
        }
    ]
} }

```

**localhost:9200/payload\_index/my\_type/\_mapping**

```
{
"payload_index": {
	"mappings": {
		"my_type": {
			"properties": {
				"attachment": {
					"properties": {
						"content": {
							"type": "text",
							"fields": {
								"_lowercase": {
									"type": "text",
									"analyzer": "_lowercase"
								}
							},
							"analyzer": "english"
						},
						"content_length": {
							"type": "long"
						},
						"content_type": {
							"type": "text",
							"fields": {
								"keyword": {
									"type": "keyword",
									"ignore_above": 256
								}
							}
						},
						"language": {
							"type": "text",
							"fields": {
								"keyword": {
									"type": "keyword",
									"ignore_above": 256
								}
							}
						},
						"title": {
							"type": "text",
							"fields": {
								"keyword": {
									"type": "keyword",
									"ignore_above": 256
								}
							}
						}
					}
				},
				"payload": {
					"type": "text",
					"fields": {
						"_lowercase": {
							"type": "text",
							"analyzer": "_lowercase"
						}
					},
					"analyzer": "english"
				}
			}
		}
	}
} }

```

There is a peace of code how I index a document:

```
TransportClient transportClient = new PreBuiltTransportClient( Settings.builder()
          .put( "cluster.name", "my_cluster" )
          .put( "node.name", "my_node" ).build() )
          .addTransportAddresses( new InetSocketTransportAddress(
            InetAddress.getByName( "127.0.0.1" ),
            elasticConf().getPortNumber( 9300) ) );

```

> XContentBuilder xContentBuilder = jsonBuilder().startObject();  
> xContentBuilder.field( "payload", bytes );  
> IndexRequestBuilder requestBuilder = transportClient.prepareIndex( "payload\_index", "my\_type", id );  
> requestBuilder.setPipeline( "attachment" );  
> requestBuilder.setSource( xContentBuilder.endObject() ).execute().actionGet();

**localhost:9200/payload\_index/my\_type/\_search**

```
{
"took": 1,
"timed_out": false,
"_shards": {
	"total": 5,
	"successful": 5,
	"failed": 0
},
"hits": {
	"total": 1,
	"max_score": 1,
	"hits": [
		{
			"_index": "payload_index",
			"_type": "my_type",
			"_id": "1",
			"_score": 1,
			"_source": {
				"attachment": {
					"content_type": "text/plain; charset=ISO-8859-1",
					"language": "en",
					"content": "Test attachment content.",
					"content_length": 24
				}
			}
		}
	]
} }

```

Really I don't need BASE64 attachment.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 18, 2018, 2:13pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/7 "2018-04-18T14:13:11Z")

</div>

Could you run the `_ingest/pipeline/attachment/_simulate` API with your BASE64 content that is failing?

And paste here the result or upload to [gist.github.com](http://gist.github.com)?

---

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 18, 2018, 4:21pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/8 "2018-04-18T16:21:41Z")

</div>

I put the whole BASE64 content file but it is too large for output all 'payload' here, so i cut it here for display. "content\_length": 402701

**localhost:9200/\_ingest/pipeline/attachment/\_simulate**

```
{ "docs" : [
      { "_index": "payload_index",
          "_type": "my_type",
          "_id": "5",
          "_source": {
          "payload": "VW5kZXIgQ29uc3RydWN0aW9uOiAgVGhlIGJvb2sgeW914oCZcmUgcmVhZGluZyBpcyBzdGlsbCB1bmRlcmRldmVsb3BtZW50LiBBcyBwYXJ0IG9mIG91ciBCZXRhIGJvb2sgcHJvZ3JhbSwgd2XigJlyZSByZWxlYXNpbmd0aGlzIGNvcHkgd2VsbCBiZWZvcmUgYSBub3JtYWwgYm9vayB3b3VsZCBiZSByZWxlYXNlZC4gVGhhdHdheSB5b3XigJlyZSBhYmxlIHRvIGdldCB0aGlzIGNvbnRlbnQgYSBjb3VwbGUgb2YgbW9udGhzIGJlZm9yZWl04oCZcyBhdmFpbGFibGUgaW4gZmluaXNoZWQgZm9ybS"
          }
       } ] } 

```

**Response:** Status: 200 OK, Time: 1539 ms, Size: 409.06 KB

```
{
"docs": [
    {
        "doc": {
            "_type": "my_type",
            "_id": "5",
            "_index": "payload_index",
            "_source": {
                "attachment": {
                    "content_type": "text/plain; charset=UTF-8",
                    "language": "en",
                    "content": "Under Construction: The book you’re reading is still underdevelopment. As part of our Beta book program, we’re releasingthis copy well before a normal book would be released. Thatway you’re able to get this content a couple of months beforeit’s available in finished form",
                    "content_length": 402701
                }
            },
            "_ingest": {
                "timestamp": "2018-04-18T15:56:43.736Z"
            }
        }
    }
] }

```

But there are about 39000 highlighted blue symbols in 'content' and others are coloured by black in it. Is it possible to index the whole text?

---

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 20, 2018, 8:11am UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/9 "2018-04-20T08:11:15Z")

</div>

Is my mapping wrong? And how to index the big files in my situation when I don't need BASE64?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 20, 2018, 12:43pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/10 "2018-04-20T12:43:43Z")

</div>

Everything looks good. Are you sure you defined the pipeline when indexing ?

Sounds like you did but I want to double check as I don't get the full picture here.

---

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 20, 2018, 12:57pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/11 "2018-04-20T12:57:03Z")

</div>

I do it when I build request for indexing.

> [@Vikentyi](#):
>
> XContentBuilder xContentBuilder = jsonBuilder().startObject();
> 
> xContentBuilder.field( "payload", bytes );
> 
> IndexRequestBuilder requestBuilder = transportClient.prepareIndex( "payload\_index", "my\_type", id );
> 
> requestBuilder.setPipeline( "attachment" );
> 
> requestBuilder.setSource( xContentBuilder.endObject() ).execute().actionGet();

Should I put the encoded Base64 bytes for index like in simulate? And what is the maximum file size for indexing?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 20, 2018, 1:12pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/12 "2018-04-20T13:12:29Z")

</div>

Yes. What did you put in there?

---

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 20, 2018, 1:14pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/13 "2018-04-20T13:14:48Z")

</div>

> [@Vikentyi](#):
>
> xContentBuilder.field( "payload", bytes );

I put there bytes as it is.

But when i run simulate i put base64 encoded.

I have set http.max\_content\_length: 500 mb

But there are about 39000 highlighted blue symbols in 'content' and others are coloured by black in it. Is it possible to index the whole text? And what is the maximum file size for indexing?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 20, 2018, 1:34pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/14 "2018-04-20T13:34:00Z")

</div>

> But there are about 39000 highlighted blue symbols in 'content' and others are coloured by black in it.

What does it mean?

---

<div class="post-metadata">

**Author:** ![Vikentyi](https://avatars.discourse-cdn.com/v4/letter/v/41988e/32.png) [@Vikentyi](https://discuss.elastic.co/u/Vikentyi)\
**Post date:** [April 20, 2018, 3:46pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/15 "2018-04-20T15:46:15Z")

</div>

When i run simulate command for big file with content-length 402701 I got that.

I don't have any problem with small files but big files raise the described above exception (the first record in this discussion).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 20, 2018, 4:59pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/16 "2018-04-20T16:59:37Z")

</div>

> I got that.

What do you get? Can you do a screenshot?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2018, 4:59pm UTC](https://discuss.elastic.co/t/ingest-attachmnet-increase-file-content-size-to-index/127624/17 "2018-05-18T16:59:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.