What will be the mapping and DSL for creating a organisation employee profile user with maximum upload of 3 photos for storing and searching his photos please of image with timestamp

[rajends1 (Agassis)]

A.What will be the mapping and DSL query structure example for creating a organisation employee profile user

1. with maximum upload of n photo limit with
2. image size nMb and
3. x height , y width

for storing and searching his photos please of image with timestamp please.

B. is it good fast perfomance and easier to store profile image
1.Directly in ES for search or
2. store in say other Aws S3 and access meta data image catalogue in ES for search ..
.which option is best pls. ?

[warkolm (Mark Walkom)]

Welcome to our community!

Elasticsearch isn't a binary store, so I would use it for the metadata search and then store the images elsewhere.

[rajends1 (Agassis)]

Hi Warkolm ,

Thanks for taking time and answering the question , really helpful.

I have below futher questions on your comment.

As per below discussion ,it was said that images can be stored in binary image type in below 2 links in AWS elasticsearch uses AWS EBS storage . please suggest

1. Can Elastic hold/store images?

Can Elastic hold/store images?

Is it possible for Elastic to store images? If so, are there only specific data types? Would they be processed or stored the same way all other data is stored i.e. index, type, docs, etc.

dadoonetDavid PilatoDeveloper | Evangelist

Jul 2019

You can store images using the binary type. But I'd not store big images IMHO.
See Binary field type | Elasticsearch Guide [8.11] | Elastic

2. Binary field type | Elasticsearch Guide [8.11] | Elastic

[dadoonet (David Pilato)]

You mentioned some megabytes for images. This is not "small" IMO.

Something like few bytes or kilobytes would be probably ok.

If you want to do it, I'd probably store images in another dedicated index and index the metadata in the main index.

[rajends1 (Agassis)]

Thanks Dadoonet , currently the File size limit for user profile photos is : 400 kb. we planned to use AWS elasticsearch with default EBS storage as one of option . yes elasticsearch image index and another elasticsearch metadata index is wonderful idea .thanks again .

[dadoonet (David Pilato)]

We won't be able to support you as AWS elasticsearch is not running our official code and service.

Did you look at Cloud by Elastic, also available if needed from AWS Marketplace ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next ...

[rajends1 (Agassis)]

Thanks for the inforamtion

[rajends1 (Agassis)]

Hi Dadoonet,
With respect to having 2 indexes do you prefer parent-child or nested objects type please ?
for performance and below requirements
of adding say 3 images to a customer profile ..

eg., Data struture is

cutomer details (index 1)
->
image details (index2 )
(customer can add 3 profile images) with timestamp ,latest display as main image and search photo functionality ..

[dadoonet (David Pilato)]

None of them.

I'd just put the id of the images or the url in the main index. And I'd "do the join" on the client side. Which means multiple calls if you want to display the results.

[rajends1 (Agassis)]

Thanks , Join seems good idea.

I have a query on the same as we use stream technology of current data like customer in one db and stream to ES .

so will it be the "join" in mongo/dynamodb stream alone or both ES and mangodb/dynamodb . as we use stream of dynamodb to ES of customer details .

ie., we store the customer details in seperate db like mongodb /dynamodb and stream details to elasticsearch ,ES is used for search faster query .

in this case, can we still keep customer details in mongo/dynamo and add their photo details alone directly in elastic search by updating the mapping/index in elastic search directly ( directly means without stream of photo details) so do the join of customer table stream in mango /dynamodb and the customer id and images details directly in ES ?

storing binaary images base64 in dynamo or mango ie., having joins in mangodb/dynamo with images and then streaming in ES may be bad idea as the images size is large and has constrain in dynamodb like 400kb per item

please advise ?

[dadoonet (David Pilato)]

I don't know DynamoDB or MongoDB so I can't comment.

But if the binary images are already there, I'd keep them in that place instead of copying into elasticsearch the images.

A GET by id in MongoDB is normally super fast I believe.

[rajends1 (Agassis)]

Thanks Again. The binary images are never implemented in dynamodb as this adding photos to customer profile is new requirement. so will it be ok to directlyadd images in ES while the customer data is still in dynamo and also streamed into ES

so the ES will have part of customer data streamed from dynamo and other join part is images directly (not streamed from dynamo) added into ES are associated to customer data streamed from dynamo .

so before adding new photos we need to get customer id created by streaming from dynamo in ES and then add customer images to it for searching in dynamo latter.