# Inconsistent data **URL:** \ **Category:** Kibana\ **Tags:** lens, visualisation\ **Created:** [September 16, 2024, 7:29am UTC](https://discuss.elastic.co/t/inconsistent-data/366608 "2024-09-16T07:29:39Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![SamehSaeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samehsaeed/32/145262_2.png) [@SamehSaeed](https://discuss.elastic.co/u/SamehSaeed)\ **Post date:** [September 16, 2024, 7:29am UTC](https://discuss.elastic.co/t/inconsistent-data/366608/1 "2024-09-16T07:29:40Z")
Hello, I'm getting different count of records when switching between lens and discover, the only explanation I could think of is due to duplicates but after extracting a CSV file, i did not find any duplicates. Lens count (unique): 25139 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/e/ee94c8f0bd0d7a6a5473a005a6e55a28cd5b33a7.png) Discover count : 25078 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/3/636eba53684d5234957a906aa47accc929ca704a.png) This is really confusing because the unique count (in lens) should be less if there are any duplicates, yet today the unique count has more records, couple of days ago for the same period (00 to 10 AM) it was the opposite. ---
**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\ **Post date:** [September 16, 2024, 8:30am UTC](https://discuss.elastic.co/t/inconsistent-data/366608/2 "2024-09-16T08:30:47Z")
Hi @SamehSaeed, In Discover have you applied any filter ? Not sure about `call_history` in the image? Make sure you're applying same filter and same query both of the places. ---
**Author:** ![SamehSaeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samehsaeed/32/145262_2.png) [@SamehSaeed](https://discuss.elastic.co/u/SamehSaeed)\ **Post date:** [September 16, 2024, 8:40am UTC](https://discuss.elastic.co/t/inconsistent-data/366608/3 "2024-09-16T08:40:07Z")
Hi @ashishtiwari1993 , I have applied the same filter in both, actually I can see both results in lens if I use "unique count" and normal count (same as in discover) , still not sure why unique count gets more records : Unique count : 25,531 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/a/ba512732d35c01150469ae03fd1e9e864886d9a6.png) Normal count : 25,382 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5f6b067f67f7c89f6e38e8d55677953ae96b7d08.png) Discover (since you cant apply unique filter) : 25,382 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/5/b55698adbffe70340f92f973c3a488b1425d7e10.png) ---
**Author:** ![SamehSaeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samehsaeed/32/145262_2.png) [@SamehSaeed](https://discuss.elastic.co/u/SamehSaeed)\ **Post date:** [September 16, 2024, 9:04am UTC](https://discuss.elastic.co/t/inconsistent-data/366608/4 "2024-09-16T09:04:56Z")
I found some duplicate values at the time of creating the new index "Asia/Riyadh" is UTC+3 so around 2:45 AM. I don't know how I can remove these duplicates however, selecting unique count should get unique records even if I have duplicates in the index correct? ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/0/60c2ee2578427c48969e48a46d90df56f012bdcf.png) ---
**Author:** ![jughosta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jughosta/32/107160_2.png) [@jughosta](https://discuss.elastic.co/u/jughosta)\ **Post date:** [September 16, 2024, 9:53am UTC](https://discuss.elastic.co/t/inconsistent-data/366608/5 "2024-09-16T09:53:06Z")
Added #visualisation ---
**Author:** ![Marco\_Liberati](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marco_liberati/32/82953_2.png) [@Marco\_Liberati](https://discuss.elastic.co/u/Marco_Liberati)\ **Post date:** [September 19, 2024, 3:38pm UTC](https://discuss.elastic.co/t/inconsistent-data/366608/6 "2024-09-19T15:38:50Z")
Added #lens ---
**Author:** ![Marco\_Liberati](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marco_liberati/32/82953_2.png) [@Marco\_Liberati](https://discuss.elastic.co/u/Marco_Liberati)\ **Post date:** [September 19, 2024, 3:39pm UTC](https://discuss.elastic.co/t/inconsistent-data/366608/7 "2024-09-19T15:39:52Z")
Is there any record with multiple values for the field `seqId`? ---
**Author:** ![SamehSaeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samehsaeed/32/145262_2.png) [@SamehSaeed](https://discuss.elastic.co/u/SamehSaeed)\ **Post date:** [September 24, 2024, 8:33am UTC](https://discuss.elastic.co/t/inconsistent-data/366608/8 "2024-09-24T08:33:43Z")
No, each record has only one `seqid` value . logstash configuration : ```auto input { jdbc { # jdbc_driver_library => " **** \ojdbc11.jar" jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver" jdbc_connection_string => " ****" jdbc_user => " ****" jdbc_password => " ****" jdbc_validate_connection => true jdbc_default_timezone => "Asia/Riyadh" jdbc_validation_timeout => 120 schedule => "*/10 * * * *" connection_retry_attempts => 3 last_run_metadata_path => " **** /config/metadata/callhistory-lastrun" tracking_column => "seqid" tracking_column_type => "numeric" use_column_value => "true" statement => " SELECT /*+ PARALLEL8 */ CALLER_ID, SERVED_NUMBER, ISD_SKILL_GROUP, DNIS, MASTER_DNIS, AGENT_NODE, AGENT_ID, ANI_CUST_TYPE, ANI_CUST_SUB_TYPE, CALL_LANG, CUSTOMER_PRIORITY, CUSTOMER_TYPE, END_REASON_CODE, SERVICE_TYPE, SOURCE_IP, VXML_GATEWAY_IP, SESSION_ID, CALL_START_TIME, CALL_END_TIME, IDB_DATE_TIME, CISCO_IP, SEQID, CALL_HISTORY, CHANNEL, PHONE_TYPE FROM ( SELECT CALLER_ID, SERVED_NMBR AS SERVED_NUMBER, ISD_SKILL_GROUP, DNIS, MASTER_DNIS, AGENT_NODE, AGENT_ID, ANI_CUST_TYPE, ANI_CUST_SUB_TYPE, CALL_LANG, CUSTOMER_PRIORITY, CUSTOMER_TYPE, END_REASON_CODE, SERVICE_TYPE, SOURCE_IP, VXML_GATEWAY_IP, SESSION_ID, CALL_START_DATE_TIME AS CALL_START_TIME, CALL_END_DATE_TIME AS CALL_END_TIME, IDB_DATE_TIME, CISCO_IP, SEQID, CALL_HISTORY, CHANNEL, CASE WHEN (INSTR(CALL_HISTORY, 'Check GSM Customer Service/prepaid', -1) > 0 OR INSTR(CALL_HISTORY, 'Check GSM Customer Type/prepaid', -1) > 0 OR INSTR(CALL_HISTORY, 'Check Disconnect Customer Type/prepaid', -1) > 0) THEN 'PREPAID' WHEN INSTR(CALL_HISTORY, 'Check GSM Customer Type For IE Customers/prepaid', -1) > 0 THEN 'IE PREPAID' WHEN (INSTR(CALL_HISTORY, 'Check GSM Customer Service/postpaid', -1) > 0 OR INSTR(CALL_HISTORY, 'Check GSM Customer Type/postpaid', -1) > 0 OR INSTR(CALL_HISTORY, 'Check Disconnect Customer Type/postpaid', -1) > 0) THEN 'POSTPAID' WHEN INSTR(CALL_HISTORY, 'Check GSM Customer Type For IE Customers/postpaid', -1) > 0 THEN 'IE POSTPAID' WHEN (INSTR(CALL_HISTORY, 'Check Phone Type/landline', -1) > 0 OR INSTR(CALL_HISTORY, 'Check Disconnect Customer Type/landline', -1) > 0) THEN 'LANDLINE' WHEN INSTR(CALL_HISTORY, 'Check Phone Type For IE Customers/landline', -1) > 0 THEN 'IE LANDLINE' END AS PHONE_TYPE FROM CALL_HISTORY_DETAILS_ETL_TEST WHERE SEQID > :sql_last_value ) " tags => ["callhistorytest"] } } filter { } output{ elasticsearch { hosts => ["https://localhost:9200/"] index => "bkgroupindex_callhistory-%{+YYYY.MM}" document_id => "%{seqid}" user => " ****" password => " ****" ssl => true ssl_certificate_verification => false } } ```