Seeking a developer to help extend Elasticsearch to connect with Web3 API

Elastic Stack Elasticsearch
1

Hi,

I am looking for a developer who is able to help integrate Web3 into Elasticsearch.

I am one of the developers working on the Web3 interface for SIA decentralised storage.

I will be mainly using Elasticsearch with AI LLM models. I am looking to store the Elasticsearch index on the decentralised storage. The plan is that this will be open-source so that anyone can use.

Do you have relevant experience and can you help me?

Best,
Jules Lai
(CTO of Fabstir)

2

It looks like there's a S3-compatible interface for Sia already, so it should already work as a backend for snapshots and searchable snapshots.

Apart from that, Elasticsearch doesn't really work well with high-write-latency storage today. That's one of the key areas of innovation for the stateless architecture on which we're currently working. Again, AWS S3 is one of the possible backends.

3

Sorry for my delay to reply.

Thanks for your answer. The S3 interface is for Amazon's storage, not Web3.
I'm thinking of Web3 ways of keeping the backups censorship resistant and for it to be private and under ownership of users rather than a centralised platform or authority.

How easy would it be to code it so that snapshots get stored directly to decentralised storage. Is there an API that I only need to implement for it to integrate into Elasticsearch?

4

You can look at this plugin to get some inspiration:

5

Sorry if I'm misunderstanding something, but https://filebase.com/ claims to be a S3-compatible interface to Web3 decentralised storage. It sounds like exactly what you want.

6

Thank you for the code reference.
I'm getting a picture for what needs to be done. I will put together a plan/proposal.

What I want to do is put out an open-source framework that will allow AI models that can use Elasticsearch to store its snapshots to decentralised storage. From there to open up Web3 use cases.

7

It could be but S5 does use a CDN for better latency etc., but after talking to the developer, the s3 compatible interface has not been built yet.

My plan is to use the s5 web3 client to store the snapshot serialised data to s5 SIA more directly and have better performance benefits.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.