How to avoid duplicate values in ealstic search 5.6.4

[narasiman1986 (narasiman)]

i want to avoid duplicated values using elastic search 5.6.4

document_id => "%{[@metadata][_id]}"

i written the above code it is correct?

output {

elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "test" }
stdout { codec => rubydebug }

document_id => "%{[@metadata][_mrd]}"

}

[narasiman1986 (narasiman)]

want to delete the duplicates the below code is correct

the below code is written in the logstash file under config file.
file type is conf file.

output {

elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "test" }
stdout { codec => rubydebug }

document_id => "%{[@metadata][_mrdno]}"
}

want to remove the duplicates of mrdno column.

please let me know

[narasiman1986 (narasiman)]

i am displaying the records from mysql database to elasticsearch through logstash.

when i execute the query in mysql

i am importing the csv file to mysql database.

select casesummary from clinicalsample where mrdno= '1145740'

it returns the output three records.

but in kibana discover for that mrdno records displaying duplicates records

in my sql shows three records only but in kibana discover records shows

Medical Record No. : 1145740Patient Name : MR. MANEESH RAMKI C B Age : 19 Year(s)Sex : maleAddress : NO:4, ASTALAKSHMI FLATS, HINDU COLONY, 2ND CROSS STREET, ULLAGARAM, ---,TAMIL NADU, INDIA - 600091.Report Date : 2015-04-01 Optometrist : JAYASHREE A Start Time : 2015-04-01 16:47 Consultant : Purpose of Visit:Others--OU: HVF 24-2 SS Done, OU: Normal Fields 13407External File Uploaded - Patient Info1145740 OD.pdf - fileExternal File Uploaded - Patient Info1145740 OS.pdf - file

Medical Record No. : 1145740Patient Name : MR. MANEESH RAMKI C B Age : 19 Year(s)Sex : maleAddress : NO:4, ASTALAKSHMI FLATS, HINDU COLONY, 2ND CROSS STREET, ULLAGARAM, ---,TAMIL NADU, INDIA - 600091.Report Date : 2015-04-01 Optometrist : ASHIM DEY Start Time : 2015-04-01 10:01 Consultant : Dr.MONA KHURANA Purpose of Visit:Regular checkup--------Main ComplaintsEye Symptom Description Duration Unit Onset Progression Remarks Associated Symptoms OU.NIL.OU: no complain regarding vision /any other ocular symptoms.Past Ocular HistoryEye Disease Duration Unit Past Surgery Surgery Date Previous Medication Past Investigation Other Trtmt OU.GLAUCOMA SUSPECT - 365.00b.6.Year(s).Past Medical HistoryDescription Duration Unit Past Surgery Surgery Date Previous Medication Past Investigation Prev Hospitalization Nil .Nil.NoFamilyHistoryDisease Relation Age Duration Unit Treatment Status GLAUCOMA - SN001,GrandfatherGLAUCOMA - SN001,Father Current Medication : Nil History entered by Optometrist was reviewed and Authorized by Dr.MONA KHURANA Visual Acuity/Refraction Prescription Date Eye Spherical Cylindrical Axis Add Prism Base Direction Lens type Glass status REMARKS --/--/nil pgOD------------------VISUAL ACUITYWithout GlassWith GlassContact LensWith PHOD6/5 N6 30 -- -- -- -- -- -- -- OS6/5 N6 30 -- -- -- -- -- -- -- TYPE OF CHARTSnellen NEAR VISION CHARTN Chart REFRACTIONRetinoScopyQuality ofCycloplegicRefractionEyeDSphDCylAxisReflexTypeDSphDCylAxisOD+0.00 -- -- Clear -- -- -- -- OS+0.00 -0.25 180 Clear -- -- -- -- ACCEPTANCEOD+0.00 -- -- 6/5 OD-- N6 30 Prefers plano OS+0.00 -- -- 6/5 OS-- N6 30 Prefers plano GLASS PRESCRIPTIONDistance VisionAddAcceptanceEyeDSphDCylAxisBCVAEyeDSphBCVAcmPreferenceOD+0.00 -- -- 6/5 OD-- N6 30 OS+0.00 -- -- 6/5 OS-- N6 30 Proposed PrismREFINING PROCEDUREPROPOSED PRISMEyeDuochromeJCCFoggingEyePDBasePDBaseODDuochrome balanced -- Yes OD-- -- -- -- OSDuochrome balanced -- Yes OS-- -- -- -- Cover Test / EOMCover Test / EOM CoverTestOrtho OcularMovement OcularMotility OD:FullOcularMotility OS:Full PupilPUPILODOSSizeNormal Normal ShapeNormal Normal Reaction to LightNormal Normal Reaction to NearNormal Normal Pupil Morphology Face Examination EXTERNAL EXAMPostureNormal FaceNormal Lash and ConjunctivaLIDSODOSUpper Lid MarginNormal Normal Lower Lid MarginNormal Normal Upper LashNormal Normal Lower LashNormal Normal Upper PunctumNormal Normal Lower PunctumNormal Normal CONJUCTIVAODOSUpper PalpebralPapillae Papillae Lower PalpebralNormal Normal Bulbar NasalNormal Normal Bulbar TemporalNormal Normal LimbusNormal Normal FornixNormal Normal Cornea/ Anterior ChamberCORNEA/ANTERIOR CHAMBERODOSScleraNormal Normal CorneaNormal Normal Anterior Chamber Central DepthNormal Normal Anterior Chamber Peripheral DepthDeep Deep Von Herick's GradingIV IV Iris/ Lens OD OS IrisNormal Normal LensClear Clear IOP IOPIOP METHODActual Time OD mm of Hg OS mm of Hg USER NAME Applanation10:111616ASHIM DEYGonioscopyGonioscopyODOSDim Illumination-ODDim Illumination-OSScleral spurScleral spurScleral spurScleral spurScleral spurScleral spurScleral spurScleral spurPAS: NOTMPig: 1+Remarks:OU: Flat iris configurationPAS: NOTMPig: 1+Remarks:Compression-ODCompression-OS Vitals signs PULSE : -76per min Blood Pressure:- 120/80mm of Hg Temperature : -Afebrile Respiratory Rate : - 20per mm Dilatation InstructionsDILATATIONODOSOUDilatationRequired Required Drug Used Number of times Which Eye Tropicamide Plus.3.OU.Fundus DrawingDiagnosisDiagnosisEYEDESCRIPTION OUEMMETROPIA - 367.91a OUGLAUCOMA SUSPECT Nutrional ScreeningNo special dietary requirements advicedFollowUpFOLLOW-UPReview After Instructions Action Planfamily h/o glaucoma, cdr asymmetry.. adv: HVF, GOCT and Disc phot Ou and rev back..wnl.. rtc 1 yearExamined by Consultant: Dr.MONA KHURANA Time: 01-04-2015 17:14:15

Medical Record No. : 1145740Patient Name : MR. MANEESH RAMKI C B Age : 19 Year(s)Sex : maleAddress : NO:4, ASTALAKSHMI FLATS, HINDU COLONY, 2ND CROSS STREET, ULLAGARAM, ---,TAMIL NADU, INDIA - 600091.Report Date : 2015-04-01 Optometrist : DEBAYAN MAHESH Start Time : 2015-04-01 15:01 Consultant : Diagnostic/Therapeutic ProceduresOptical Coherence TomographyCorrelate Clinically.REMARKS:RNFL: OU: NORMAL RNFL THICKNSS NOTED IN ALL QUADRANTS, AVERAGE THICKNESS OD: 96 OS: 92 MICRONS. ASSYMETRIC DISC SIZE NOTED, GCC: OU: NORMAL GCC THICKNESS NOTED IN ALL ALL SECTORS, AVERAGE THICKNESS OD: 82 OS: 82 MICRONS.External File Uploaded - Patient InfoInvestigation Report - GOCT_1145740 OU GCC.pdf - fileExternal File Uploaded - Patient InfoInvestigation Report - GOCT_1145740 OU RNFL.pdf - file

In kibana display records as follows

Medical Record No. : 1145740Patient Name : MR. MANEESH RAMKI C B Age : 19 Year(s)Sex : maleAddress : NO:4, ASTALAKSHMI FLATS, HINDU COLONY, 2ND CROSS STREET, ULLAGARAM, ---,TAMIL NADU, INDIA - 600091.Report Date : 2015-04-01 Optometrist : JAYASHREE A Start Time : 2015-04-01 16:47 Consultant : Purpose of Visit:Others--OU: HVF 24-2 SS Done, OU: Normal Fields 13407External File Uploaded - Patient Info1145740 OD.pdf - fileExternal File Uploaded - Patient Info1145740 OS.pdf - file

Medical Record No. : 1145740Patient Name : MR. MANEESH RAMKI C B  Age : 19 Year(s)Sex : maleAddress : NO:4, ASTALAKSHMI FLATS, HINDU COLONY, 2ND CROSS STREET, ULLAGARAM, ---,TAMIL NADU, INDIA - 600091.Report Date : 2015-04-01 Optometrist : DEBAYAN MAHESH Start Time : 2015-04-01 15:01 Consultant :  Diagnostic/Therapeutic ProceduresOptical Coherence TomographyCorrelate Clinically.REMARKS:RNFL: OU: NORMAL RNFL THICKNSS NOTED IN ALL  QUADRANTS, AVERAGE THICKNESS   OD: 96   OS:  92 MICRONS. ASSYMETRIC DISC SIZE NOTED,   GCC: OU: NORMAL GCC THICKNESS NOTED IN ALL   ALL SECTORS,  AVERAGE THICKNESS OD: 82  OS:  82 MICRONS.External File Uploaded - Patient InfoInvestigation Report - GOCT_1145740 OU GCC.pdf - fileExternal File Uploaded - Patient InfoInvestigation Report - GOCT_1145740 OU RNFL.pdf - file

[dadoonet (David Pilato)]

Please stop cross-posting. You already asked there How to remove duplicate values in ealstic search about the same topic.

Although it's in a bit different way this time.