How to combine Web Proxy logs into one message

(Александр Ли) #1

Please excuse me for my English.

We successfully doing many things through logstash. (parse with csv filter around 25 fields, decode URL more amazing things).
But thousands referer web request can't allow to analyze log.
For one click by user Proxy generate many string of logs and many categories.
Of course we tried use aggregate filter. It's worked if referrer url in one deep and user do new attempt to site.

for example

  1. Url_request :
  2. Url_request : referrer :

We simple add new field with this condition if referer not exist request is main. if referer exist - main and next do aggregate filter.

But what we do if request isn't "new"?

For example

User come from lunch and his browser stay in (of course all aggregtion timeouts are lost). And he clicks to link in

  1. Url_request referer:

So. second example isn't big problem...

The big problem is
How to aggregate web request that have more 1 deep?

For example

1)Url_request :
2)Url_request : referrer :
3)Url_request : referer :

Good example is YouTube...

Thanks for any advice

(Александр Ли) #2

Okay. These code allow to have shared field "Proxy_pvc" for future aggregate. Soon I will try to build time management.

if ("cloned_event" in [tags]){
	aggregate {
		push_map_as_event_on_timeout => true
		map_action => "create_or_update"
		inactivity_timeout => 9999
		task_id => "%{Proxy_username}%{Proxy_client_ip}%{Proxy_useragent}"
		code => "
		pvc_temp = event.get('Proxy_pvc')
		url = event.get('Proxy_url_full')
		referer = event.get('Proxy_referer_url')
		current_time =
		if map['database'] == nil
			map['database'] = {}

		if referer == nil
			event.set('Proxy_pvc', url)
			if map['database'][url] == nil
				map['database'][url] ||= []
			map['database'].each do |temp_key, temp_value|
				if referer == temp_key
					map['database'][temp_key] << url
					event.set('Proxy_pvc', referer)
					map['database'].each do |temp_key2, temp_value2|
						if temp_value2.include?(referer)
							temp_pvc = map['database'].key(temp_value2)
							map['database'][temp_pvc] << url
							event.set('Proxy_pvc', temp_pvc)

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.