I have parse data through logstash i have 3 records in sql server the data is parse it shows
if you look the attached image it shows 24 records and logstash shutdown after parsing data from database is there any way to logstash not shutdown and i insert record in sql and new record is showing in kibana?? and why it shows 24 records instead of 3 records
syad, more information is required.
I think you trying to ask two different question here correct?
you said you have 3 record in sql database?
but you see 24 record? if that is the case then you have run your logstash 8 time and everytime it duplicated the records 8x3 = 24
you have to use schecule => in your input section and then it will not shutdown your logstash and will run it on x interval.
put your config file example, put your data input example and we should be able to tell you more.
this is my config file
input {
jdbc {
# SqlServer jdbc connection string to our database, employeedb
# "jdbc:sqlserver://HostName\instanceName;database=DBName;user=UserName;password=Password"
jdbc_connection_string => "jdbc:sqlserver://localhost:1433;database=i3insight;user=;password="
# The user we want to execute our statement as
#sincedb_path => "NUL"
jdbc_user => nil
# The path to our downloaded jdbc driver
jdbc_driver_library => "C:/Program Files/sqljdbc_6.2/enu/jre8/sqljdbc4-2.0.jar"
# The name of the driver class for SqlServer
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
# Query for testing purpose
statement => "SELECT * from [i3insight].[dbo].[mcb]"
}
}
filter {
date {
match => ["date", "yyyy-MM-dd HH:mm:ss.SSS a"]
target => "date"
#}
#}
output {
#stdout { codec => json_lines }
elasticsearch {
hosts => ["localhost:9200"]
index => "connect"
}
stdout { codec => json_lines }
}
now the problem is that when the data is parse in logstash through sql server when 2 records update logstash shutdown automatically and in logstash 2 records is parsed but when i open elasticsearch it shows 4 records or 8 records everytime i change the index name and run it fisrt time but still it shows 4 or 8 records
what version you are using?
Go to discover and find out how many events you have there.
i am using 7.3.2 version
can you run a query in sqlserver and see how many records are there? is it 2 or 4?
elasticsearch will create duplicate record everytime you run your query.
for example you have two record
custname: foo
clientname: bar
custname:abc
clientname:def
elasticsearch will enter this record with some doument_id
when you run it again. it will create new id for both record and enter it
when i run query in sql server it shows 2 records when logstash parse the data it show 2 records parse when i see in kibana it shows 4 records
issue resolved for duplication. can someone tell me when parsing data from logstash is there way around that logstash service will not shutdown
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.