Hello,
we have around 20000 products in 25 different languages. Each language has an own index (products_en, products_fr, ...)
when 1000 product data (code, name, description) per language is gathered, we send for every language a bulk request.
this works. but once we also index a pdf (200-400KB), then elasticsearch returns after some time "(429) Too Many Requests"
but why? the number of requests did not change, only the size of the bulk json and only one bulk request is send at a time
this is performed on a Windows Server 2012 (73GB memory, 8 core 2.60 GHz) which is for development
index config:
{
"settings": {
"index": {
"number_of_shards": 1,
"number_of_replicas": 1
}
}
}
ingest attachment config:
{
"description": "Extract attachment information",
"processors": [
{
"foreach": {
"field": "attachments",
"processor": {
"attachment": {
"field": "_ingest._value.data",
"target_field": "_ingest._value.attachment",
"properties": [ "content" ]
}
}
}
}, {
"foreach": {
"field": "attachments",
"processor" : {
"remove" : { "field" : "_ingest._value.data" }
}
}
}
]
}
Regards