ELK on cloud

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hello guys, I've a question about the cloud service. Me and my company thinking about using ElasticCloud for our product but our usage is pretty particular.
We'd like to use an ELK stack for logging purpose but we want also to use Logstash and Elastic search to gathering and indexing some other datas for our search feature on our app.
My question is when we subscribe to the Cloud do we have access to the full Elastic stack (especially, Beats, Logstash, Elasticsearch and Kibana) ?

2

We only offer Elasticsearch and Kibana at this stage - https://www.elastic.co/cloud/as-a-service

We're working towards Logstash, but running Beats (eg filebeat) doesn't make a lot of sense as they usually run to read local sources :slight_smile:

3

Make sense. thanks for the quick answer. An other question, we'd like to synchronize a part of our Neo4J graph oriented database into elasticsearch indexes. I found that page https://neo4j.com/developer/elastic-search/ but no one of these solutions are viable for our use case. Is somebody think about an other solution for sync Neo4J and Elastic index ?

4

I’m sure @ikwattro has ideas!

1 Like
5

Thanks @dadoonet for pinging me.

I can answer only for one the two solutions on the neo4j developer page. The GraphAware neo4j-to-elasticsearch plugin. It has a very flexible json replication definition and is used for example at Airbnb for the search part of their Data Portal (https://www.youtube.com/watch?v=gayXC2FDSiA)

Knowing that this plugin cannot cover all use cases, I would need to understand "why it is not viable" in order to
propose you an adequate alternative

6

Thanks for your response @ikwattro.
I've took a look at it but it seems really hard to define our own index mapping with it. The configuration file addons are not really well documented perhaps in order to sell some consulting service.

7

We discourage completely to define your ES index mapping via this plugin. The role of the plugin is to replicate data, not to manage your ES cluster.

8

So as I said this solution may not be viable for our use case.
We need to index some nodes of our Neo4J database in order to search on it using the powerfull search API of Elasticsearch. If the plugin only replicate datas without indexing them the way we need for the search it's not matching our uses, Are you agree ?

9

I do not agree, you need to define your ES index mapping before replicating data. In that way, the data being replicated to ES will be indexed in the way defined in your ES mapping.

1 Like
10

I tried to inject my datas in an existing index via this conf line:

#optional, Elasticsearch index name, default is neo4j-index
com.graphaware.module.ES.index=my_index_name

But it create new indexes with prefix of this index name like:
my_index_name-node
my_index_name-relationships

Maybe I messed up somewhere else in the conf file, maybe in this line:

#optional, specify which nodes to index in Elasticsearch, defaults to all nodes
com.graphaware.module.ES.node=hasLabel('my_targeted_label')

Again I'd like to have a better documentation about these conf lines because what we have here it's really not enought.

11

I generally suggest to use the json definition documented here :

also if there are questions best is to use StackOverflow or Github Issues on the repository

12

I think that's super nice to keep this thread here. It could help other people to find this thread who are asking the same.

Or at least, please add the link to SOF here as a follow up.

Thanks Christophe anyway for the help here!

13

@ikwattro Thanks a lot for the insight I'll dig a bit more on this documentation, I'll get back to you to tell if I'm able to do what I want with it.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.