Logstash unable to process more no of documents

[PRASHANT_MEHTA]

Hello All,
I'm tyring to process 40000 documents by running perl script.The issue faced is that in stack management I can see the index being get cereated but the documents are not inserting the index.
For a seperate usecase I was able to get arountd 17000 documents wherein the script was executing every 30 min.Going by this logic I increased the time to 1 hour to get 40000 documents in my index,but unable to understand why its not inserting.
I'm testing it in test instance with 4 core cpu and JVM heap of 2 GB.

Logstash JVM config:
-Xms1g
-Xmx2g

In the below config file I tried running script every 1 hour and aternatively also tried to run specific time once per day.Still the documents aren't ingested in index.In logs also I dont see any error or exceptions but can see the events.
Could anyone suggest what might be the reason or how to troubleshoot?

input {
   exec {
      command => '/k/app/MIS/logstash/logstash-7.9.1/scripts/tcgeometry/run_tcs_geometry.sh'
      schedule => "0 58 9 * * *"
   }
}
filter {
   if [message] =~ "^\{.*\}[\s\S]*$" {
      json {
         source => "message"
         target => "parsed_json"
		 remove_field => "message"
      }
	  
      split {
         field => "[parsed_json][MSLgeometryMonitorResponse]"
         target => "geometry"
         remove_field => [ "parsed_json" ]
      }	  
   }
   else {
     drop { }
   }
}
output {
   elasticsearch {
      hosts => "http://abc:9200"
	  ilm_pattern => "{now/d}-000001"
      ilm_rollover_alias => "tis-monitor-geometry"
	  ilm_policy => "tis-monitor-geometry-policy"
	  doc_as_upsert => true
	  document_id => "%{[geometry][uniqueId]}" 
   } 
}

• pipeline.id: tcgeometry
  queue.type: persisted
  path.config: "/k/app/MIS/logstash/logstash-7.9.1/scripts/tcgeometry/tc_geometrycfg"

Thanks

[leandrojmp (Leandro Pereira)]

@PRASHANT_MEHTA

document_id => "%{[geometry][uniqueId]}"

Since you are using a custom value for the _id field, are you sure that you have 40000 unique ids? This is the first thing you need to check

You could add the following output and see if you really have 40000 unique values.

output {
 file {
   path => "/path/to/an/output/file.log"
   codec => line { format => "%{[geometry][uniqueId]}"}
 }
}

[PRASHANT_MEHTA]

Hi @leandrojmp ,

Thanx for your time to look into this, as u mentioned correctly, the unique ids were not getting generated.Backend logic were not creating all the ids.Now the ids are generated and all are present in index.

Thanks