Getting dynamic data from mysql db

Elastic Stack Elasticsearch
#1

Hi all. Today I have tried to get dynamic data from mysql db via logstash. I have installed elastic search and have run, than have installed logstash and configured db.config , which in I have
parameters => {"user_name" => "Jhon"}
stament => "SELECT * FROM user where name= :user_name "

I need that the user_name to be dynamicly. I am using Elastic Search PHP Library and I want to send this parameters via PHP. Please help. what I need to do ?

#2

logstash can't have a "dynamic" config field while you could have a environment variable or use event fields to help you but that will not help you in the way your asking but your asking 2 separate questions

  1. I don' believe you will find logstash to work like the way you want. The idea for Logstash would be to pull all user data and populate the Elasticsearch Index

  2. Then your query from PHP against Elastic search could then "filter" on a userid and return only their data.

#3

Dear Ed. Thank you for replying. Can you suggest me a way to solve this problem. ?

#4

yah as I mentioned in answer 1 pull all your user data in to elasticsearch not just one user
"SELECT * FROM user"

then from php just add your user to what ever your existing PHP query is

#5

Thanks. But I want to use large sql with more than 4 joins and mysql functions like match(abc) against('abc' IN BOOLEAN MODE) . Is there a way to use function like this in ELastic ?

#6

no elasticsearch does not join documents, but you can issue your sql statement to pull all the joins of all users. if I get you right you want all this users data in Elasticsearch.

from the little that I know about what your doing,

is that you have a Mysql Database with all your user data, and you want to load that data in to elasticsearch. Which is totally possible,

However Logstash is only a simple reader of data with some data manipulations capacity, doing complex data or dynamic data is not really its strong suite. As an example but there are plenty of other answers to accomplish this

you could write a php command (or lang of choice) to query and write the data to a file in say JSON , then either have that same PHP insert it to Elastic search or have Logstash do it.

Heck you probably could set up a DB Trigger to post the data directly to Logstash via HTTP every-time the user updates his data (a quick example) don't know if it is good for you.
http://www.mooreds.com/wordpress/archives/1497

#7

Ok. So I should create query with joins and set it in logstash. Than using PHP I need to filter all getting data ? I am correct ?

#8

yes correct. that is probably the easiest way.

#9

Thanks !!!!!

#10

yah the " selcet * from user join blah blah blah

will create an event in one document, then when you do an elastic query
for "john doe" via php it will return you a document that has all of the
data you wanted in it.

#11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.