Hi,
I use the combination of rsyslog and elasticsearch which seems to be pretty
common.
My requirement is something in the lines of this:
- I will have as many as 16 processes running parallely and each process
has 23 threads sending syslog information. (using syslog( ) system call)
The message itself will look something like this "ip=x.x.x.x,
name=abcd, loglevel=3, this is a test log" - My current architecture for logs is file based where I separate each
process log to a different file. - I tried to log the above message with standalone mysql /sqlite (use SQL
insert statements) where I had a single database and multiple tables for
each process, however the performance was poor. ( I had high cpu usage on a
4 core x86_64 server having 32gb memory) - I start all processes one by one they run for a few hours and stop them
all at once and collect/examine the logs. - I wont need to store the logs permanently, so I can go ahead and delete
them once the analysis is over.
Is it possible in rsyslog to
- create 1 index and multiple types for each of the process through
rsyslog (or outside of it first using curl) based on some name that
identifies the process - Query indexes separately for values (e.g. 1 process may have logs from
ip addresses 1.1.1.1 to 1.1.1.10 and other process may have logs from
1.1.1.11 to 1.1.1.20 etc)
So my index should look something like this curl -XGET
localhost:9200/index//_search?q=ip:"x.x.x.x"
"process1"
"ip" : "x.x.x.x"
"name": "abcd"
"log": "test log"
"process2"
"ip" : "y.y.y.y"
"name": "wxyz"
"log": "test log"
thanks a lot in advance
regards
Mahesh
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.