How to set the configuration in logstash to run dynamically

Elastic Stack Elasticsearch
1

I am pulling the data from pgadmin and pushing to elastic search through logstash..I am using JDBC driver in Logstash but how can i update dynamically like postgres updates ..can any one suggest..E.s &L.s ,kibana Version is 6.1.2...Thankz

2

With LS or any other ETL you can only schedule an execution every x seconds/minutes/xxx

You can't do that in real time if this is what you meant.

3

Thankz a lot for your valuable information..actually i am working on report part i had choose the kibana to generate reports i am new to 'ELK'... as per my understanding after reading your reply there is no option to update dynamically in real time..is m correct ?? ..i am facing this thing If i need last day information (data) from 'pgadmin' i need to run logstash ,this is very useless .so every morning i am keep doing this thing to get the refresh data..can u suggest me if any other options have..thankz

4

I wrote some of my thoughts there: https://david.pilato.fr/blog/2015-05-09-advanced-search-for-your-legacy-application/

May be it can help.

as per my understanding after reading your reply there is no option to update dynamically in real time..is m correct ??

No. If you have in your SQL table a date field like lastModified, then you can probably SELECT all records that have been modified since the last run of LS.
But that's more something to solve on the SQL database end IMO.

5

Thanks for such prompt reply ..i will follow your steps.

6

facing issue at logstash..just i need the updated data i dont know how to use schedule in my config file for a day..can i discuss here or should i start new thread..??this is my config file

stdin: db.conf

input {
jdbc {

	jdbc_connection_string=>"jdbc:postgresql://localhost/joy"
	
	jdbc_user=>"yyy"
	
	jdbc_password=>"yyy"
	
	jdbc_driver_library=>"Desktop/Drivers/postgresql-9.1-901.jdbc4.jar"
	#the name of the driver class for postgres
	jdbc_driver_class=>"org.postgresql.Driver"
          
	#our query
	statement=>"SELECT * from arron;"
}

}
output {
stdout{codec=>json_lines
}
elasticsearch {
#protocol=>http
index=>"joy"
document_type=>"tem"
hosts => ["local"]
user => "alexjon"
password => "thomas"

}

}

Just suggest me how to use per a day "schedule => "* * * * *"" not for a second/minute..
In my config file i am not using any filters or plugins like grok..so plz suggest me..thankz

7

Please format your code using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

Please edit your post.

8

Sry to say I am very new to elastic search..plz help me to understand ,should i post by putting my index name in dev tools??like in this format GET my index/_search
{
"query": {
"match": {
"foo": "bar"
}
}
}..
is i am on wrong track??

9

i am facing issue with logstash.

  1. i just stucked with this issue When i try to run the logstash duplicate data generated.
    so plz suggest me how to solve this issue..thankz
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.