Single logstash config file should send data to mutiple indices

Hello All,

Kindly suggest if something wrong I'm doing here.

Logstash: 8.8.2

I have two indices: mis-monitoring-webserver and mis-monitoring-webui.

  1. I want some additional fileds to be created based on some condition satisfield and go to mis-monitoring-webui index.(For this if :else used in filter)
    2)Now in output filter data never goes to mis-monitoring-webui index.
    3)Now if IF-ELSE (mis-monitoring-webui) is removed for output filter the data goes to first index i.e mis-monitoring-webserve
    4)I want the entire data to go to first index (WEBSERVER) and on condition meet on applied filter- new fields to be created and only that should go to NEW INDEX-(WEBUI)

I tried debugging, stuck now.
Note in second index: webui only the required fields should come and not whole object data(i.e only required fields)

input {
   exec {
      command => '$ABCD_APP/mis/monitoring/scripts/monitoring_core.ksh -c $ABCD_CONFIG_DIR/mis/globalconfiguration.properties -e ac_tomcat_monitoring.pl'
      schedule => "*/10 * * * * *"
   }
}

filter {
   split {}
   
   if [message] =~ "^\{.*\}[\s\S]*$" {
      json {
         source => "message"
         target => "parsed_json"
         remove_field => "message"
      }
      
      split {
         field => "[parsed_json][mis]"
         target => "tomcat"
         remove_field => [ "parsed_json" ]
      }

      # Check for critical conditions and add fields accordingly
      if [tomcat][memory_status] == "Critical" or [tomcat][error_status] == "Critical" or [tomcat][request_status] == "Critical" or [tomcat][server_status] == "DOWN" {
         mutate {
            add_field => {
               "UsecaseStatus" => "Critical"
               "UsecaseCategory" => "Tomcat"
            }
         }
      } else {
         # Set default values if no critical conditions
         mutate {
            add_field => {
               "UsecaseStatus" => "Normal"
               "UsecaseCategory" => "Tomcat"
            }
         }
      }
   }
   else {
      drop { }
   }
}

output {
   elasticsearch {
      hosts => "https://abc:443"
      ilm_pattern => "{now/d}-000001"
      ilm_rollover_alias => "mis-monitoring-webserver"
      ilm_policy => "mis-monitoring-common-policy"
      api_key => ""
      ssl_enabled => true
      ssl_certificate_authorities => ""
      http_compression => true
      data_stream => false
   }

   # Conditional block to redirect events based on UsecaseStatus field
   if [UsecaseStatus] == "Critical" {
      elasticsearch {
         hosts => "https://abc:443"
         ilm_pattern => "{now/d}-000001"
         ilm_rollover_alias => "mis-monitoring-webui"
         ilm_policy => "mis-monitoring-common-policy"
         api_key => ""
         ssl_enabled => true
         ssl_certificate_authorities => ""
         http_compression => true
         data_stream => false
         doc => {
            "UsecaseStatus" => "%{UsecaseStatus}"
            "UsecaseCategory" => "%{UsecaseCategory}"
         }
      }
   } else {
      elasticsearch {
         hosts => "https://abc:443"
         ilm_pattern => "{now/d}-000001"
         ilm_rollover_alias => "mis-monitoring-webui"
         ilm_policy => "mis-monitoring-common-policy"
         api_key => ""
         ssl_enabled => true
         ssl_certificate_authorities => ""
         http_compression => true
         data_stream => false
         doc => {
            "UsecaseStatus" => "Normal"
            "UsecaseCategory" => "Tomcat"
         }
      }
   }
}

Couuld someone please suggest here something?

Conditions look OK, however it's not clear where do you want to keep data and how you will present as with the same data view or separated.
Does mis-monitoring-common-policy and template define in which index data go?
I don't se indices names except common mis-monitoring-common-policy ILM policy.
According to your code, data should ends up in the logstash index, something like this: ecs-logstash-2024.02.06-000001

If you want separated indices, then use separated indices:
index => "mis-monitoring-webserver"
index => "mis-monitoring-webui index"
These indicesare without ILM. For ILM, you have to create the template with pattern, create the policy and add the policy to the template.

According to the documentation, there is no "doc" parameter, only document_id

Hello @Rios ,

Thanx for your time to look into this!
I will keep it simple to help understand better.

I have mutiple usecases which have there own indices managed by there own index templates and have common policy governing them.
Now for example, I have below usecases which have this common fields in them and I want only these fields should come to this Index : mis-monitoring-webui.
usecaseStatus:"Normal" or "Critical",usecaseCategory:"Tomact" are the fields present in all indices and I want this only to come in MIS-MONITORING WEBUI

Tomact Index: "usecaseStatus":"Normal","usecaseCategory":"Tomact"
Oracle Index: "usecaseStatus":"Critical","usecaseCategory":"Oracle"
Webserver Index: "usecaseStatus":"Critical", "usecaseCategory":"Webserver"

Now only the above fields i.e usecaseStatus and usecaseCategory only should come in mis-monitoring-webui index and REST EVENTS OF THIS INDIVIDUAL USECASES SHOULD NOT COME

Now again read my config, you might understand now what I am trying to achieve.What is done in above config, will be done for all usecases to generate this addition fields: usecasestatus,usecasecategory

Basically SQL JOIN, which can't be done in elasticsearch, so this is my implementation .But somehow data dont come in mis-monitoring-webui index,but when removed this index from output filter then data comes to mis-monitoring-webserver.

mis-monitoring-webserver has lot of fields but I just need this to send usecaseCtaegory and usecaseStatus field only to MIS-MONITORING-WEBUI index.

FINAL GOAL IS BELOW IMAGE:

image

Thanx

Have you used ruby debug in output? It might be something related to data and the UsecaseStatus field hasn't been added, not sure.
If there is too much data, you can also write the output in a file or simply select 10-20 records and try to achieve your condition.

I would also temporary remove ILM and write directly to indices to set the conditions correctly. When data arrived on good location, switch to ILM.

What do you want to achieve with?

 doc => {
            "UsecaseStatus" => "%{UsecaseStatus}"
            "UsecaseCategory" => "%{UsecaseCategory}"
         }

Hello

With below fields in doc, I'm just trying to send only those two fields from webserver index to webui index, webserver has many fields and from there only this two fields should be sent to webui index.

doc => {
"UsecaseStatus" => "%{UsecaseStatus}"
"UsecaseCategory" => "%{UsecaseCategory}"
}

You can use the prune white list in order to select which fields are allowed.