Here is a brainier. I am trying to track the number of the requests between server and user ( I need to know the elapsed time per each request), and I am trying to catch the number and the elapsed time of the small API requests withing the big request. I grokked a few lines: the one where the request begin and the one where it ends - that is how I define my Big request. Everything that is inside those two lines contains some unnecessary info and the info about inside requests and the times it took them to be completed. So, I wrote a third line to grok it. I thought I link all processes by using tracking ID of the event. Then my big requests would be easily correlated with small API requests. I also used aggregate filter to define my request range. However, my config does not work the way I need it to. It runs OK but it does not allow me to visualize the data the way I need it (please see the example one). Then I decided to remove aggregate filter and just find my big requests by a tag and maybe later play with Kibana visualization.It works, however, I can combine those terms when I need to build a table were I would see the big request elapsed time because it is related only to the end request part and it does fail to help me to see api request within each big request. Any suggestions? (please see the second example.
example 1:
input {
file {
path => "C:/ELK/WITSML/Rigcloud_witsml_store_all_log.log"
start_position => "beginning"
}
}
filter {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - %{GREEDYDATA:logger}. URI: %{NOTSPACE:log_message}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:WITSML_request_bytes}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - %{GREEDYDATA:logger}. URI: %{NOTSPACE:log_message}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:request_bytes}, Elapsed Time: %{NUMBER:WITSML_elapsed_time}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} -%{GREEDYDATA:logger} %{NOTSPACE:log_message}, Username: %{USERNAME}, StatusCode: %{WORD}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
if [logger] == "Begin Request" {
aggregate {
task_id => "%{tracking_ID}"
code => "map['WITSML_message_add'] = event.get('message')"
map_action => "create"
}
}
if [logger] == "apiURI" {
aggregate {
task_id => "%{tracking_ID}"
code => "map['WITSML_message_add']+= event.get('message')"
map_action => "update"
}
}
if [logger] == "End Request" {
aggregate {
task_id => "%{tracking_ID}"
code => "event.set('message', map['WITSML_message_add']+= event.get('message'))"
map_action => "update"
end_of_task => true
timeout => 120
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "rigcloud_witsml_store_all_logg"
template => "C:/ELK/mytemplate.json"
}}
Example 2:
input {
file {
path => "C:/ELK/WITSML/Rigcloud_witsml.log"
start_position => "beginning"
}
}
filter {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - %{GREEDYDATA:witsml_begin_request}. URI: %{NOTSPACE:log_message}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:WITSML_request_bytes}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} - %{GREEDYDATA:witsml_end_request}. URI: %{NOTSPACE:log_message}, Referrer: , UserHostAddress: %{IP:user_host_ip}, UserHostName: %{IP:user_host_name}, TotalBytes: %{NUMBER:request_bytes}, Elapsed Time: %{NUMBER:WITSML_elapsed_time}"}
}
grok {
match => {"message" => "%{TIMESTAMP_ISO8601} %{NOTSPACE:tracking_ID} %{NOTSPACE:thread_ID} %{WORD:log_level} %{NOTSPACE:method} -%{GREEDYDATA:api_request} %{NOTSPACE:log_message}, Username: %{USERNAME}, StatusCode: %{WORD}, Elapsed Time: %{NUMBER:api_elapsed_time}"}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "rigcloud_witsml"
template => "C:/ELK/mytemplate.json"
}
}