# Bluecoat proxySG log parsing
**URL:** https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634
**Category:** Logstash
**Created:** [July 15, 2016, 3:26pm UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634 "2016-07-15T15:26:30Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [July 15, 2016, 3:26pm UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/1 "2016-07-15T15:26:30Z")
Hi, I am facing a problem related to bluecoat proxySG logs;
I have my bluecoat log files stored in a server running an instance of logstash, let's call it server 1. The input section of the configuration file is also on server 1. I set up a second server that runs a second instance of logstash, along with with elasticsearch. Let's call it server 2.
I set up a csv filter to parse my logs, and placed the whole configuration on server 2, but KIBANA doesn't seem to take filter section into account. I am pretty sure my configuration is Okay, because it somehow works when I place it on server 1, but have no idea why it is ignored on server 2. I can see raw logs though, so the problem is not coming from an empty imput.
Maybe someone went through the same situation and could help me to shed light on why the parsing is not working?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 17, 2016, 8:46am UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/2 "2016-07-17T08:46:43Z")
Providing your config would be helpful.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [July 18, 2016, 8:32am UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/3 "2016-07-18T08:32:51Z")
All right, this is the input configuration, placed on server 1 (running redis and logstash):
```
input {
file {
path => "/path/to/logs/*.log"
type => "logstash"
tags => ["bluecoat"]
start_position => "beginning"
}
}
```
Here is the configuration on server 2 (running elasticserach and logstash):
```
input {
redis {
host => "server1"
data_type => "list"
type => "redis-input"
key => "logstash"
threads => 4 # number (optional), default: 1
}
}
filter {
if [type] == "logstash" {
if [path] == "/path/to/log/file-name.log" {
# drop comment lines
if ([message] =~ /^#/) {
drop{}
}
csv {
columns => ["localtime", "time_taken", "c_ip", "cs_x_forwarded_for", "sc_status", "s_action", "sc_bytes", "cs_bytes", "cs_method", "cs_uri_scheme", "cs_host", "cs_uri_port", "cs_uri_path", "cs_uri_query", "cs_username", "cs_auth_group", "s_hierarchy", "s_supplier_name", "rs_content_type", "cs_referer", "cs_user_agent", "sc_filter_result", "cs_categories", "x_virus_id", "s_ip"]
separator => " "
source => message
remove_field => ["message","host","path","@version","@timestamp"]
}
if [localtime] {
date {
match => ["localtime", "[dd/MMM/YYYY:HH:mm:ss Z]"]
}
}
#enrich log entry with destination geolocation info
if ([c_ip] and [c_ip] != "-" ){
mutate {
add_field => {"cs_host_ip" => "%{c_ip}"}
}
dns {
resolve => ["cs_host_ip"]
action => "replace"
}
geoip {
source => "cs_host_ip"
}
}
# parse User-Agent header
if ([cs_user_agent] and [cs_user_agent] != "" and [cs_user_agent] != "-") {
useragent {
source => "cs_user_agent"
prefix => "user_agent_"
}
}
# split Blue web site categories into an array
if ([cs_categories] and [cs_categories] != "" and [cs_categories] != "-") {
mutate {
split => { "cs_categories" => ";" }
}
}
# type convert number fields
mutate {
convert => ["sc_bytes", "integer",
"time_taken", "integer",
"r_port", "integer",
"s_port", "integer",
"cs_bytes", "integer",
"duration", "integer" ]
}
}
}
}
```
I have to mention that bluecoat logs are not the only ones treated by Logstash. I have other logs coming via Syslog, and their parsing works fine.
And here is what a single log line looks like:
```
"[21/May/2014:17:12:44 +0100]" 18 10.20.2.33 - 200 TCP_HIT 5689 169 GET http www.google.fr 80 / - NavInternet - - 211.85.89.15 pub/sky - - OBSERVED "url_allowed;Search Engines/Portals" - 163.10.165.24
```
Thanks in advance.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 18, 2016, 10:11am UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/4 "2016-07-18T10:11:04Z")
Given
```
type => "redis-input"
```
in your redis input configuration, is the
```
if [type] == "logstash" {
```
conditional really correct? Use a `stdout { codec => rubydebug }` to inspect exactly what your events look like when Logstash is done with them.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [July 18, 2016, 3:31pm UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/5 "2016-07-18T15:31:26Z")
Yes, in fact there is the "type" parameter that may be wrong in the elasticsearch configuration (not in the redis one as you mentioned), but it doesn't seem to affect the other parsings.
I'll run a debug and give you a feed back as soon as possible.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [July 19, 2016, 4:17pm UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/6 "2016-07-19T16:17:46Z")
I changed the configuration. I am actually looking inside the tags array ( the one defined in redis input) for the element "bluecoat" and passing the event to the csv filter if the element is found, like this:
```
filter {
if "bluecoat" in [tags] {
# drop comment lines
if ([message] =~ /^#/) {
drop{}
}
...
```
Nothing's changed, I still get no results.
I even tried with Grok:
```
filter {
if "bluecoat" in [tags] {
# drop comment lines
if ([message] =~ /^#/) {
drop{}
}
grok {
match => ["message","\"\[%{DAY}/%{MONTH}/%{YEAR}:%{TIME} %{ISO8601_TIMEZONE}\]\" %{NUMBER:time_taken} %{IP:c_ip} %{IP:cs_x_forwarded_for} %{NUMBER:status} %{GREEDYDATA:s_action} %{NUMBER:sc_bytes} %{NUMBER:cs_bytes} %{WORD:cs_method} %{URIPROTO:cs_uri_scheme} %{IPORHOST:cs_host} %{NUMBER:cs_uri_port} %{URIPATH:cs_uri_path} %{URI:cs_uri_query} %{USERNAME:cs_username} %{GREEDYDATA:cs_auth_group} %{GREEDYDATA:s_hierarchy} %{GREEDYDATA:s_supplier_name} %{GREEDYDATA:rs_content_type} %{GREEDYDATA:cs_referer} %{GREEDYDATA:cs_user_agent} %{GREEDYDATA:sc_filter_result} %{GREEDYDATA:cs_categories} %{GREEDYDATA:x_virus_id} %{IP:s_ip}"]
add_tag => ["grokked_bluecoat"]
tag_on_failure => ["ERROR : grok matching failed"]
}
```
Same thing. I tried to run a debug, I get a java buffer over flow exception...
Here is the command I typed:
```
/path/to/logstash --debug -l /path/to/log_file -f /path/to/config/
```
Here is the exception I get:
```
HeapByteBuffer.java:183:in `put': java.nio.BufferOverflowException
```
The whole configuration above, from the beggining, is on server 2
Does it ring any bell?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [July 20, 2016, 11:25am UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/7 "2016-07-20T11:25:31Z")
> HeapByteBuffer.java:183:in `put': java.nio.BufferOverflowException
Nothing I've seen before. Do you have the full stack trace?
Again, getting the result of a `stdout { codec => rubydebug }` output for a message that isn't correctly filtered would be helpful.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [July 20, 2016, 1:48pm UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/8 "2016-07-20T13:48:46Z")
> [@magnusbaeck](#):
>
> HeapByteBuffer.java:183:in `put': java.nio.BufferOverflowException
>
> Nothing I've seen before. Do you have the full stack trace?
The stack is too big and can't fit in here, here is the first part of it:
```
Exception in thread "[main]>worker2" Exception in thread "[main]>worker1" java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at org.jruby.util.io.ChannelStream.bufferedWrite(ChannelStream.java:1100)
at org.jruby.util.io.ChannelStream.fwrite(ChannelStream.java:1277)
at org.jruby.RubyIO.fwrite(RubyIO.java:1541)
at org.jruby.RubyIO.write(RubyIO.java:1412)
at org.jruby.RubyIO$INVOKER$i$1$0$write.call(RubyIO$INVOKER$i$1$0$write.gen)
at org.jruby.RubyClass.finvoke(RubyClass.java:743)
at org.jruby.runtime.Helpers.invoke(Helpers.java:505)
at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:364)
at org.jruby.RubyIO.write(RubyIO.java:2490)
at org.jruby.RubyIO.putsSingle(RubyIO.java:2478)
at org.jruby.RubyIO.puts1(RubyIO.java:2407)
at org.jruby.RubyIO.puts(RubyIO.java:2380)
at org.jruby.RubyIO$INVOKER$i$puts.call(RubyIO$INVOKER$i$puts.gen)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at rubyjit.Cabin::Outputs::IO$$\=\^\^_a759d2d46347217e5f0d618914b90c91265cea161954102985.block_0$RUBY$ __file__ (/path/to/logstash/vendor/local_gems/54c50e3f/cabin-0.8.1/lib/cabin/outputs/io.rb:52)
at rubyjit$Cabin::Outputs::IO$$\=\^\^_a759d2d46347217e5f0d618914b90c91265cea161954102985$block_0$RUBY$ __file__.call(rubyjit$Cabin::Outputs::IO$$\=\^\^_a759d2d46347217e5f0d618914b90c91265cea161954102985$block_0$RUBY$ __file__ )
at org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)
at org.jruby.runtime.Block.yield(Block.java:142)
at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:149)
at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
at rubyjit.Cabin::Outputs::IO$$\=\^\^_a759d2d46347217e5f0d618914b90c91265cea161954102985. __file__ (/path/to/logstash/vendor/local_gems/54c50e3f/cabin-0.8.1/lib/cabin/outputs/io.rb:50)
at rubyjit.Cabin::Outputs::IO$$\=\^\^_a759d2d46347217e5f0d618914b90c91265cea161954102985. __file__ (/path/to/logstash/vendor/local_gems/54c50e3f/cabin-0.8.1/lib/cabin/outputs/io.rb)
at org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at org.jruby.runtime.callsite.ShiftLeftCallSite.call(ShiftLeftCallSite.java:24)
at rubyjit.Cabin::Subscriber$$\=\^\^_10bd20586d047ca25f2f05ebf603ebf4af74e6dd1954102985. __file__ (/path/to/logstash/vendor/local_gems/54c50e3f/cabin-0.8.1/lib/cabin/subscriber.rb:10)
at rubyjit.Cabin::Subscriber$$\=\^\^_10bd20586d047ca25f2f05ebf603ebf4af74e6dd1954102985. __file__ (/path/to/logstash/vendor/local_gems/54c50e3f/cabin-0.8.1/lib/cabin/subscriber.rb)
at org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)
at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
at org.jruby.runtime.callsite.ShiftLeftCallSite.call(ShiftLeftCallSite.java:24)
at rubyjit.Cabin::Channel$$publish_2fedc2a9a4a36f3ad01bf44594d627e37af6b87f1954102985.block_2$RUBY$ __file__ (/path/to/logstash/vendor/local_gems/54c50e3f/cabin-0.8.1/lib/cabin/channel.rb:196)
```
The parsing works fine now. I have my csv bluecoat fields on KIBANA thanks to my latest configuration. ALthough the fields are not recongnized on dashboard visualizations, is it normal? They are tagged with a question mark on the discover KIBANA page.
---
### Author:  [@jpry](https://discuss.elastic.co/u/jpry)
#### Post date: [July 20, 2016, 1:59pm UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/9 "2016-07-20T13:59:28Z")
Problem solved after reloading the parameters, in the settings section. Thank you all for your help.
If you find something on the java stack error, this would help, thanks again.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:47am UTC](https://discuss.elastic.co/t/bluecoat-proxysg-log-parsing/55634/10 "2017-07-06T04:47:18Z")