# How to tweak logstash -\> elasticsearch indexing process

**URL:** <https://discuss.elastic.co/t/how-to-tweak-logstash-elasticsearch-indexing-process/83401>\
**Category:** Logstash\
**Created:** [April 24, 2017, 10:28am UTC](https://discuss.elastic.co/t/how-to-tweak-logstash-elasticsearch-indexing-process/83401 "2017-04-24T10:28:24Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 24, 2017, 10:28am UTC](https://discuss.elastic.co/t/how-to-tweak-logstash-elasticsearch-indexing-process/83401/1 "2017-04-24T10:28:24Z")

</div>

Hello, my logstash configurations create a separate index for each log file, I want to add them all together in a single elasticsearch index, how can I do this?

Here's how my logstash.conf looks like:

```
input {
tcp {
    port => 5000
    codec => multiline {
        pattern => "^%{TIMESTAMP_ISO8601} "
        negate => true
        what => previous
    }
    }
}

filter {
	## Finished merchants

## Stock status (in, out)
grok{
		match => ["message", "'in_stock_items_count': %{NUMBER:instock_items:int}"]
}
grok{
		match => ["message", "'out_stock_items_count': %{NUMBER:outofstock_items:int}"]
}

## Scraped items, invalid items
grok{
		match => ["message", "'item_scraped_count': %{NUMBER:scraped_items:int}"]
}
grok{
		match => ["message", "'invalid_items_count': %{NUMBER:invalid_items:int}"]
}

## Zero Priced
grok{
		match => ["message", "'zero_price_items_count': %{NUMBER:zero_priced_items:int}"]
}

## Item Duration
grok{
		match => ["message", "'iteration_duration': %{NUMBER:iteration_duration:float}"]
}

## timestamp
grok{
		match => ["message", "%{DATE_EU:timestamp}"]
}
date{
	    match => ["timestamp", "yy-MM-dd"]
	    target => "@timestamp"
	}
}

output {
   	if "_grokparsefailure" not in [tags]{
	
		elasticsearch {
			hosts => "elasticsearch:9200"
		}
	    }
    }
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 24, 2017, 10:40am UTC](https://discuss.elastic.co/t/how-to-tweak-logstash-elasticsearch-indexing-process/83401/2 "2017-04-24T10:40:11Z")

</div>

Based on that config it should do that.  
Are you seeing something else?

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [April 24, 2017, 10:49am UTC](https://discuss.elastic.co/t/how-to-tweak-logstash-elasticsearch-indexing-process/83401/3 "2017-04-24T10:49:14Z")

</div>

I added three log files named (01012017.log, 02012017.log, 03012017.log) and I ran  
`curl http://localhost:9200/_aliases?pretty=1`  
the output was

```
{
  "logstash-2017.01.03" : {
    "aliases" : { }
  },
  ".kibana" : {
    "aliases" : { }
  },
  "logstash-2017.01.01" : {
    "aliases" : { }
  },
  "logstash-2017.01.02" : {
    "aliases" : { }
  }
}

```

3 different indices, right?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 24, 2017, 10:52am UTC](https://discuss.elastic.co/t/how-to-tweak-logstash-elasticsearch-indexing-process/83401/4 "2017-04-24T10:52:24Z")

</div>

Yes, one per day. In your case you happen to have one logfile per day, giving you the impression that it's one index per logfile.

Change the elasticsearch output's index option. If you inspect its default value you'll understand why you're getting one index per day and changing it to e.g. one index per month should be obvious. Make sure you understand why time-based indexes are pretty convenient before changing to one index for _all_ events.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2017, 10:55am UTC](https://discuss.elastic.co/t/how-to-tweak-logstash-elasticsearch-indexing-process/83401/5 "2017-05-22T10:55:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.