solved it ...
needed to use UNIX not UNIX_MS
the indexing looks a bit slow ...
still 4cpu 8gig of ram :\ ~ 2000log seconds... a bit low :\
Regarding indexing rate. I just did a perf test using Logstash to receive netflow data:
- 1xCPU 6 physical cores or 12 logical cores, 16GB RAM.
- The server is dedicated for netflow generator and Logstash only, ES is on another server.
- 1Gbps network between 2 servers.
- 20K flow/sec generated
Flows sent from Logstash/received by Elasticsearch
No filter at all: 15K flows/sec
GeoIP filter for 2 IP fields: 7K flows/sec
GeoIP filter + logstash-cidr-filter to tag network based on IP + some if statements: 2.5K flows/sec
I guess playing with Filter Worker may help https://www.elastic.co/guide/en/logstash/2.2/breaking-changes.html#_filter_worker_default_change
I have geofilter on two field
and its a VM on ESX server the ressources are not dedicated. but allowed.
so I assume, according to your stats.. its decent then
thank, im already running 20workers.