Elasticsearch query in Logstash config file for aggregation and re-indexing not working

Elastic Stack Logstash
1

I have data that is already indexed and now I want to perform some querying and aggregating to re-index the data into smaller specific indices. However I have trouble writing the Elasticsearch query where I just want to retrieve all values of the IP field, get the count of how many times each IP showed up in my data and thereafter store them in a new index. When I view the new index in Kibana, all fields are still showing up which I do not want.

Here is my Logstash config file code:

input {
  elasticsearch {
    hosts => "localhost:9200"
    index => "Old_Index"	
    query => '{
	"_source": "ip.keyword",
	"size": 0,
	"query": {
		"bool": {
			"must": [{
				"match_all": {}
			}]
		}
	},
	"aggs": {
		"by ip": {
			"terms": {
				"field": "ip.keyword"
			},
			"aggs": {
				"ip_count": {
					"value_count": {
						"field": "ip.keyword"
					}
				}
			}
		}
	}
	}'
	}
}					

output { 
  elasticsearch { 
    hosts => "localhost:9200"
    index => "NewIndex" 
  }
}

Still very new to this, any help is appreciated.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.