Different ways to add documents?

Elastic Search
1

Heya

I've been playing around with App Search the last couple of days and so far I am really enjoying it!
I would like to move the search functionality of my PHP-based forum to AS and have played around with adding some documents.

Right now I have a PHP script that gets the data from MySQL, does a bit of manipulation (convert timestamps to ISO etc) and then ship 100 documents at a time to AS/

Now I am wondering, how to do it better / faster / easier using - I don't know - JDBC? Different shipping methods from MySQL to AS?
But I am unsure of what would be the "right" way to do it. As far as I have found out, there is no Logstash in AS? Or is there?

Any tips will be much appreciated! Seeing that this is quite a new (and awesome) product, I could not find any specific App Search related answers or tutorials

Thanks!

2

Welcome!

I shared most of my thoughts there: http://david.pilato.fr/blog/2015/05/09/advanced-search-for-your-legacy-application/ although it applies to Elasticsearch but I believe the same philosophy applies. :slight_smile:

Basically, I'd recommend modifying the application layer if possible and send data to elasticsearch in the same "transaction" as you are sending your data to the database.

3

Hi!

I've read your article a couple of days ago and like the idea - but I am not so sure it is the easiest way with my existing forum solution: I don't want to alter the base code and have not found the right hooks for CRUD operations that the forum uses

Even if I were to go your proposed route: the question remains, how to add the initial data of round about 1m documents

4

I'd probably use Logstash. It now has an App Search connector. See https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elastic_app_search.html

So I'd read from the DB with a jdbc input plugin and write to App search.

Note that in the past (LS was not really able to do that), I used the exact same application layer I built to do it.
So my app was able to send the current activity to Elasticsearch but also to read the whole database again and reindex everything.

But for this you need to "hack" your app.

5

awesome!

Does AS come with LS?
Or asked differently: I need to setup jdbc with logstash and then, as an output use the plugin to send it to my AS server?

6

You have to install the AS output plugin in LS if that was your question.

7

does AS come with LS? Where would I need to install it?
I thought AS uses Beats?

8

I don't understand.

does AS come with LS?

I guess no.

Where would I need to install it?

Install what? The plugin? In logstash.

I thought AS uses Beats?

May be but I'm really unsure about that. And I'm pretty sure it's not related anyway.

9

hm ok. let me try to then explain better:
for App Search I needed to install Elastic Search 7.x and have it running, then, download the zip and run it from within the bin/app-search.
The systemd installation required a worker for filebeat and app search itself, but I could not find any information regarding Logstash

So if I were to use the App Search output plugin for Logstash, I would need to actually have Logstash installed somewhere (if it does not come with App Search).

If I do need to set it up separately, then I am asking for the best way to do this.
Same server as App Search? Different server? Do I only need Logstash (as it will then speak to App Search directly)?

Hope this is more clear, if not, let me know :slight_smile:

10

Hope this is more clear

Yes. Thanks.

Yes.

I'd probably make it run on the same machine where App Search is running.

In that case I think you'd need:

  • an Elasticsearch cluster
  • AppSearch server
  • Logstash and the app search output plugin
11

perfect! I'll look into it :slight_smile:
thank you for your help!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.