Logstash Data Parsing Problem

Elastic Stack Kibana
1

I have parse data through logstash i have 3 records in sql server the data is parse it shows

image
image1366×768 88 KB
if you look the attached image it shows 24 records and logstash shutdown after parsing data from database is there any way to logstash not shutdown and i insert record in sql and new record is showing in kibana?? and why it shows 24 records instead of 3 records

2

syad, more information is required.

I think you trying to ask two different question here correct?

you said you have 3 record in sql database?
but you see 24 record? if that is the case then you have run your logstash 8 time and everytime it duplicated the records 8x3 = 24

you have to use schecule => in your input section and then it will not shutdown your logstash and will run it on x interval.

put your config file example, put your data input example and we should be able to tell you more.

3

this is my config file
input {
jdbc {
# SqlServer jdbc connection string to our database, employeedb
# "jdbc:sqlserver://HostName\instanceName;database=DBName;user=UserName;password=Password"
jdbc_connection_string => "jdbc:sqlserver://localhost:1433;database=i3insight;user=;password="
# The user we want to execute our statement as
#sincedb_path => "NUL"
jdbc_user => nil
# The path to our downloaded jdbc driver
jdbc_driver_library => "C:/Program Files/sqljdbc_6.2/enu/jre8/sqljdbc4-2.0.jar"
# The name of the driver class for SqlServer
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
# Query for testing purpose
statement => "SELECT * from [i3insight].[dbo].[mcb]"
}
}

filter {

date {

match => ["date", "yyyy-MM-dd HH:mm:ss.SSS a"]

target => "date"

#}
#}
output {
#stdout { codec => json_lines }
elasticsearch {
hosts => ["localhost:9200"]
index => "connect"
}
stdout { codec => json_lines }
}
now the problem is that when the data is parse in logstash through sql server when 2 records update logstash shutdown automatically and in logstash 2 records is parsed but when i open elasticsearch it shows 4 records or 8 records everytime i change the index name and run it fisrt time but still it shows 4 or 8 records

4

what version you are using?
Go to discover and find out how many events you have there.

5

i am using 7.3.2 version

6

can you run a query in sqlserver and see how many records are there? is it 2 or 4?

elasticsearch will create duplicate record everytime you run your query.

for example you have two record

custname: foo
clientname: bar

custname:abc
clientname:def

elasticsearch will enter this record with some doument_id

when you run it again. it will create new id for both record and enter it

7

when i run query in sql server it shows 2 records when logstash parse the data it show 2 records parse when i see in kibana it shows 4 records

8

issue resolved for duplication. can someone tell me when parsing data from logstash is there way around that logstash service will not shutdown

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.