How to tweak logstash -> elasticsearch indexing process

Elastic Stack Logstash
1

Hello, my logstash configurations create a separate index for each log file, I want to add them all together in a single elasticsearch index, how can I do this?

Here's how my logstash.conf looks like:

input {
tcp {
    port => 5000
    codec => multiline {
        pattern => "^%{TIMESTAMP_ISO8601} "
        negate => true
        what => previous
    }
    }
}

filter {
	## Finished merchants

## Stock status (in, out)
grok{
		match => [ "message", "'in_stock_items_count': %{NUMBER:instock_items:int}" ]
}
grok{
		match => [ "message", "'out_stock_items_count': %{NUMBER:outofstock_items:int}" ]
}

## Scraped items, invalid items
grok{
		match => [ "message", "'item_scraped_count': %{NUMBER:scraped_items:int}" ]
}
grok{
		match => [ "message", "'invalid_items_count': %{NUMBER:invalid_items:int}" ]
}

## Zero Priced
grok{
		match => [ "message", "'zero_price_items_count': %{NUMBER:zero_priced_items:int}" ]
}

## Item Duration
grok{
		match => [ "message", "'iteration_duration': %{NUMBER:iteration_duration:float}" ]
}

## timestamp
grok{
		match => [ "message", "%{DATE_EU:timestamp}" ]
}
date{
	    match => [ "timestamp", "yy-MM-dd" ]
	    target => "@timestamp"
	}
}

output {
   	if "_grokparsefailure" not in [tags]{
	
		elasticsearch {
			hosts => "elasticsearch:9200"
		}
	    }
    }
2

Based on that config it should do that.
Are you seeing something else?

3

I added three log files named (01012017.log, 02012017.log, 03012017.log) and I ran
curl http://localhost:9200/_aliases?pretty=1
the output was

{
  "logstash-2017.01.03" : {
    "aliases" : { }
  },
  ".kibana" : {
    "aliases" : { }
  },
  "logstash-2017.01.01" : {
    "aliases" : { }
  },
  "logstash-2017.01.02" : {
    "aliases" : { }
  }
}

3 different indices, right?

4

Yes, one per day. In your case you happen to have one logfile per day, giving you the impression that it's one index per logfile.

Change the elasticsearch output's index option. If you inspect its default value you'll understand why you're getting one index per day and changing it to e.g. one index per month should be obvious. Make sure you understand why time-based indexes are pretty convenient before changing to one index for all events.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.