# How to combine Web Proxy logs into one message

**URL:** <https://discuss.elastic.co/t/how-to-combine-web-proxy-logs-into-one-message/120484>\
**Category:** Logstash\
**Created:** [February 19, 2018, 3:50pm UTC](https://discuss.elastic.co/t/how-to-combine-web-proxy-logs-into-one-message/120484 "2018-02-19T15:50:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![khabme](https://avatars.discourse-cdn.com/v4/letter/k/4491bb/32.png) [@khabme](https://discuss.elastic.co/u/khabme)\
**Post date:** [February 19, 2018, 3:50pm UTC](https://discuss.elastic.co/t/how-to-combine-web-proxy-logs-into-one-message/120484/1 "2018-02-19T15:50:27Z")

</div>

Please excuse me for my English.

We successfully doing many things through logstash. (parse with csv filter around 25 fields, decode URL more amazing things).  
But thousands referer web request can't allow to analyze log.  
For one click by user Proxy generate many string of logs and many categories.  
Of course we tried use aggregate filter. It's worked if referrer url in one deep and user do new attempt to site.

for example

1. Url\_request : [example.com/](http://example.com/)
2. Url\_request : [blablabla.com/12321.txt](http://blablabla.com/12321.txt) referrer : [example.com](http://example.com)

We simple add new field with this condition if referer not exist request is main. if referer exist [example.com](http://example.com) - main and next do aggregate filter.

But what we do if request isn't "new"?

For example

User come from lunch and his browser stay in [example.com](http://example.com) (of course all aggregtion timeouts are lost). And he clicks to link in [example.com](http://example.com).

1. Url\_request [example.com/news1](http://example.com/news1) referer: [example.com](http://example.com)

So. second example isn't big problem...

The big problem is  
How to aggregate web request that have more 1 deep?

For example

1)Url\_request : [example.com/](http://example.com/)  
2)Url\_request : [blablabla.com/12321.js](http://blablabla.com/12321.js) referrer : [example.com](http://example.com)  
3)Url\_request : [zxc.com](http://zxc.com) referer : [blablabla.com/12321.js](http://blablabla.com/12321.js)

Good example is YouTube...

Thanks for any advice

---

<div class="post-metadata">

**Author:** ![khabme](https://avatars.discourse-cdn.com/v4/letter/k/4491bb/32.png) [@khabme](https://discuss.elastic.co/u/khabme)\
**Post date:** [March 4, 2018, 5:17pm UTC](https://discuss.elastic.co/t/how-to-combine-web-proxy-logs-into-one-message/120484/2 "2018-03-04T17:17:00Z")

</div>

Okay. These code allow to have shared field "Proxy\_pvc" for future aggregate. Soon I will try to build time management.

```
if ("cloned_event" in [tags]){
	aggregate {
		push_map_as_event_on_timeout => true
		map_action => "create_or_update"
		inactivity_timeout => 9999
		task_id => "%{Proxy_username}%{Proxy_client_ip}%{Proxy_useragent}"
		code => "
		pvc_temp = event.get('Proxy_pvc')
		url = event.get('Proxy_url_full')
		referer = event.get('Proxy_referer_url')
		current_time = Time.now
		if map['database'] == nil
			map['database'] = {}
		end

		if referer == nil
			event.set('Proxy_pvc', url)
			if map['database'][url] == nil
				map['database'][url] ||= []
			end
		else
			map['database'].each do |temp_key, temp_value|
				if referer == temp_key
					map['database'][temp_key] << url
					event.set('Proxy_pvc', referer)
				else
					map['database'].each do |temp_key2, temp_value2|
						if temp_value2.include?(referer)
							temp_pvc = map['database'].key(temp_value2)
							map['database'][temp_pvc] << url
							event.set('Proxy_pvc', temp_pvc)
						end
					end
				end
			end
		end
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 1, 2018, 5:17pm UTC](https://discuss.elastic.co/t/how-to-combine-web-proxy-logs-into-one-message/120484/3 "2018-04-01T17:17:08Z")

</div>

Closed on April 1, 2018, 5:17pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.