Get added documents immedately

Elastic Stack Elasticsearch
1

Hi everyone.
I have an application that stores documents to the elasticsearch.
Also, I have another web application and I want to show added documents to the elasticsearch immediately in a page (similar to a socket between my dashboard and elasticsearch).
Can anyone offer me a solution?

2

Hi, Ghasem.

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-get.html
Enjoy.

3

There's no "change API" available. So you most likely need to send may be the documents to a 3rd party system like Kafka and monitor the queue for both indexation in elasticsearch and present the results you need through the websocket.

I don't think there's a way to do that only with elasticsearch until the changes api is actually implemented. See

4

Hi Mikhail
Thank you for your reply.

But this is only a simple Http GET request and is not appropriate for live log trail.
Because with the GET request, I have to send a http request periodically to query|search my won desired indices .
But I'm looking for a solution like as a socket connection between the logstash/elasticsearch and my own application.

I think I have to do the following steps:

  1. Add my own application in logstash output (in addition to elasticsearch output in logstash), so that logstash send the processed documents directly to my own app too.
  2. listen to the specified port of logstash output, in my own app.
  3. Now I can receive the documents in my own app.

Is there a better solution than the above?

5

Hi @dadoonet
Thank you for your suggestion.
I'm going to add the Kafka to my architecture as soon as possible. So it seems a good solution.
But now what do you think about the below solutions :

  1. Direct relation between logstash -> my app?
  2. This library : https://github.com/sivasamyk/logtrail
6

I was not aware that the data is coming from Logstash.

So may be you can just put 2 outputs:

Would that work?

7

@dadoonet
Yes it works.
But do you think calling a Http endpoint repeatedly (hundreds of times per second) is a good solution?
In my case, whenever a user logs in to the dashboard, I have to show him/her logs that occurs live (real-time).
I think I need a way to notify the logstash when user is logged-in, and therefore logstash must keeps alive the connection and call the desired Http endpoint, and vice versa whenever user is not logged-in, tell to logstash close the connection and don't send logs anymore.

How can I do it?

8

Do you know that we have a "similar" feature in Kibana in the Logs UI.
It basically tails the logs in the UI.

9

Yes, But I'm not going to use kibana for end users. We have our own dashboard and want to allow the user to see only their own logs.
Therefore, I'm looking for the best solution to relate logstash to my own app, preferably by a bidirectional way between them:

step(1). app notifies logstash to start send logs
step(2). logstash ---> sends logs
step(3). app notifies logstash to stop sending logs
10

I just meant that if Kibana can do it, you can also probably mimic a similar behavior in your app?

11

So, At first I have to examine kibana source code to see how it is doing it.
But I think the above three steps is not provided by the kibana.

12

If you have a reliable time stamp you can periodically query for new documents based on this. You could add such a time stamp at the time of indexing through an ingest node pipeline if you do not have one.

13

@Christian_Dahlqvist
I have timestamp for every log message. my scenario is as the following:

  1. Java app -> saves logs to some files (by logback.xml and RollingFileAdapter).
  2. Filebeat monitors all log files and send them to logstash.
  3. logstash do more process on the logs and stores them as elasticsearch documents.
  4. I can query|see all of these stored documents in my own dashboard now by calling a Http endpoint and do search on elasticsearch.

But the last above step is not working automatically, because it needs page refreshing|calling Http endpoint.

I have a web socket between dashboard and back-end and I'm looking for what is the best way to involve the elasticsearch/logstash into this architecture, so that whenever a log occurs (filebeat->logstash->app or filebeat->logstash->elasticsearch->app), it will be sent to dashboard automatically.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.