How to combine Web Proxy logs into one message

Please excuse me for my English.

We successfully doing many things through logstash. (parse with csv filter around 25 fields, decode URL more amazing things).
But thousands referer web request can't allow to analyze log.
For one click by user Proxy generate many string of logs and many categories.
Of course we tried use aggregate filter. It's worked if referrer url in one deep and user do new attempt to site.

for example

  1. Url_request : example.com/
  2. Url_request : blablabla.com/12321.txt referrer : example.com

We simple add new field with this condition if referer not exist request is main. if referer exist example.com - main and next do aggregate filter.

But what we do if request isn't "new"?

For example

User come from lunch and his browser stay in example.com (of course all aggregtion timeouts are lost). And he clicks to link in example.com.

  1. Url_request example.com/news1 referer: example.com

So. second example isn't big problem...

The big problem is
How to aggregate web request that have more 1 deep?

For example

1)Url_request : example.com/
2)Url_request : blablabla.com/12321.js referrer : example.com
3)Url_request : zxc.com referer : blablabla.com/12321.js

Good example is YouTube...

Thanks for any advice

Okay. These code allow to have shared field "Proxy_pvc" for future aggregate. Soon I will try to build time management.

if ("cloned_event" in [tags]){
	aggregate {
		push_map_as_event_on_timeout => true
		map_action => "create_or_update"
		inactivity_timeout => 9999
		task_id => "%{Proxy_username}%{Proxy_client_ip}%{Proxy_useragent}"
		code => "
		pvc_temp = event.get('Proxy_pvc')
		url = event.get('Proxy_url_full')
		referer = event.get('Proxy_referer_url')
		current_time = Time.now
		if map['database'] == nil
			map['database'] = {}
		end

		if referer == nil
			event.set('Proxy_pvc', url)
			if map['database'][url] == nil
				map['database'][url] ||= []
			end
		else
			map['database'].each do |temp_key, temp_value|
				if referer == temp_key
					map['database'][temp_key] << url
					event.set('Proxy_pvc', referer)
				else
					map['database'].each do |temp_key2, temp_value2|
						if temp_value2.include?(referer)
							temp_pvc = map['database'].key(temp_value2)
							map['database'][temp_pvc] << url
							event.set('Proxy_pvc', temp_pvc)
						end
					end
				end
			end
		end

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.