Elasticsearch to SQL

Elastic Stack Logstash
1

Hi,

Im new in Elasticsearch.
I have a project to collect syslogs from some servers using custom grok filter.
I can write different logs to different indexes and works fine.
Kibana reporting is a great tool, but my goal is to use SQL for PowerBI as reporting.
My main issue is that using REST API only get 10000 rows.
Is that possible to use another DB engine with Lucene like MariaDB and export some data from ELK to MariaDB in the same server?
How i do that? Off course without the limit of rows, because at this time i have some issues with limit rows, i want all data from one day.

Some help.

Regards,
Marcos

2

I have not tested it but I believe this could be done. Use an elasticsearch input. Then either use a third-party jdbc output or use a jdbc_streaming filter to execute INSERTs (in which case you do not need an output).

3

Hi Badger,

Thanks for your reply.
At this moment i use logstash as output.
What i need is that the output become logstash and MariaDB Database.
I try to use jdbc output without success, as you said use the jdbc streaming to execute inserts into MariaDB? Is that correct?

4

As I said, I have not tried it, but I think that would work.

5

Ok.
I will try this.
Thank you.

6

Hi Badger,

It works with jdbc output plugin.
Another question the records appears in Kibana second by second, is it possible to change de save time between 5 by 5 minutes?
Because if i saved all records second by second de database increases very shortly.
Is that a definition in elasticsearch?

Thank you.

7

I cannot think of a way to do that.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.