Conditional index in output

Elastic Stack Logstash
1

I was doing research and some testing in my environment. I am using Filebeat to read 2 kinds of logs, that go into 2 different index patterns. I looked up conditionals in output, and found some articles, but one had a solution that worked.

I realized that it was simple. I added a field that allowed me to set the index name in the filter section, and then access it in the output section of the logstash config.

    filter {
    	if [log][file][path] =~ /\/data\/iboss\/.+/ {
    	csv {
    		columns => ["thedate","thetime",.... ""]
    		}
    	#first record will have the field names, drop it
    	if [thedate] == "date" { drop { } }
    	    #combines the date and time fields, so we can use them to create the timestamp
            mutate { add_field => { "log_timestamp" => "%{thedate} %{thetime}" } }
            date { match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss" ] }
    		#this section removes the filebeat default field info that is not useful for this index about the host and the agent
            mutate  { remove_field => ['host.architecture','host.containerized','host.hostname','host.id','host.ip','host.mac','host.name','host.os.codename','host.os.family','host.os.kernel','host.os.name','host.os.platform','host.os.version','message','agent.ephemeral_id','agent.hostname','agent.id','agent.type','agent.version'] }
    	    mutate { add_field => { theindex => "iboss" } }
    	
    	}
    	if [log][file][path] =~ /\/data\/test\/.+/ {
          		csv {
            	columns => [ "area","area_title","area_type","naics","naics_title","i_group","own_code","occ_code","occ_title","o_group","tot_emp","emp_prse","jobs_1000","loc_quotient","pct_total","h_mean","a_mean","mean_prse","h_pct10","h_pct25","h_median","h_pct75","h_pct90","a_pct10","a_pct25","a_median","a_pct75","a_pct90","annual","hourly" ]
         		}
    	mutate { add_field => { theindex => "national_salary" } }
    	}
        }

    output {
      elasticsearch { 
      hosts => ["localhost:9200"] 
      index => "%{[theindex]}" 
     }
    }

Setting the "theindex" variable in each section, then using it in the index assignment in the elasticsearch plugin allowed us to specify the index based on conditions. This worked for me, and I hope someone else finds it helpful.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.