# How to configure different indexes in logstash

**URL:** <https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/248752>\
**Category:** Logstash\
**Created:** [September 16, 2020, 12:56am UTC](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/248752 "2020-09-16T00:56:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![pheathers](https://avatars.discourse-cdn.com/v4/letter/p/67e7ee/32.png) [@pheathers](https://discuss.elastic.co/u/pheathers)\
**Post date:** [September 16, 2020, 12:56am UTC](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/248752/1 "2020-09-16T00:56:52Z")

</div>

Hi,

I'm trying to ingest multiple (two right now) different formatted logs via Filebeat into Logstash and have them sent to different indexes based on some criteria. The first thought was to use the **source** (path) to identify which log is being ingested and then use the **source** to determine which index the logs should be index into.

One of the sets of logs is a custom app log and the other is IIS logs. The intent is to ingest the logs into an index where I can use the Filebeat IIS module and Kibana Dashboards [link](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-iis.html).

Here is my logstash.config ...

```auto
input {
    beats {
		port => 5044
    }
}
  
filter {
    mutate {
        gsub => ["message", "^.{1,37}(.*)$","\1"]
    }
    json {
        source => "message"      
    }   
	
    grok {
        match => ["source", "%{GREEDYDATA}\\W3SVC1\\%{DATA:iisLogSource}.log"]
    } 
	
	grok { 
		match => { "Timestamp" => "%{TIMESTAMP_ISO8601:logdate}" } 
	}
	
	date {
		match => ["logdate", "ISO8601"]
	}	
	if ("" in [TimeSpan]) {
		grok {
			match => { "TimeSpan" => "%{INT:hours}:%{INT:minutes}:%{INT:seconds}.%{INT:subsecond}" }
			
		}
		ruby {
			code => '
				subsecond = event.get("subsecond")
				if subsecond
					subsecond = subsecond.to_f / (10 ** subsecond.length)
					event.set("elapsed", 3600 * event.get("hours").to_f + 60 * event.get("minutes").to_f + event.get("seconds").to_f + subsecond)
				end
			'
			remove_field => ["hours", "minutes", "seconds", "subsecond"]
		}		
	}	
}
  
output {
	if [iisLogSource] <> "" {
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "filebeat-iis-%{+YYYY.MM.dd}"
		}		
	} else {
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "dclogstash-%{+YYYY.MM.dd}"
		}
	}
}

```

I was trying to use iisLogSource as a generic variable allowing me to determine if the log came from a particular path (C:\SomePath\W3SVC1\filename.log).

Thanks for your help in advance.

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [September 16, 2020, 1:08am UTC](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/248752/2 "2020-09-16T01:08:24Z")

</div>

If the 2 different sources go to 2 different indexes you can use [tags](https://www.elastic.co/guide/en/beats/filebeat/7.9/add-tags.html) on the filebeat side.

---

<div class="post-metadata">

**Author:** ![pheathers](https://avatars.discourse-cdn.com/v4/letter/p/67e7ee/32.png) [@pheathers](https://discuss.elastic.co/u/pheathers)\
**Post date:** [September 16, 2020, 5:11pm UTC](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/248752/3 "2020-09-16T17:11:01Z")

</div>

Thanks @aaron-nimocks.

I was able to find a solution which worked for me and I believe it was basically what you were referring to.

First I added field associated with the log in my filebeat.yml ...

```auto
filebeat.inputs:

- type: log
  enabled: true
  paths: 
    - C:\PerfElastic\Logs\*.json
  fields: 
    log_type: diagnostics    

- type: log
  enabled: true
  paths: 
    - C:\PerfElastic\Logs\testiis\*.log
  fields: 
    log_type: iis  

```

... and then in the logstash.conf, checked for the field in my output ...

```auto
output {
	if ([fields][log_type] == "iis"){
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "filebeat-iis-%{+YYYY.MM.dd}"
		}		
	} else {
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "dclogstash-%{+YYYY.MM.dd}"
		}
	}
}

```

Additional notes...

- This post ([link](https://stackoverflow.com/questions/38830663/how-to-define-seperated-indexes-for-different-logs-in-filebeat-elk)) was helpful
- YAMLLINT was very helpful [YAMLLINT](http://www.yamllint.com/) when checking to make sure your syntax is correctly formatted

Hopefully this will help someone in the future.

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [September 16, 2020, 5:16pm UTC](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/248752/4 "2020-09-16T17:16:59Z")

</div>

@pheathers yes, that's what I was suggesting and glad you got it all worked out! 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 14, 2020, 5:17pm UTC](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/248752/5 "2020-10-14T17:17:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.