# Bluecoat Grok filter **URL:** \ **Category:** Logstash\ **Created:** [August 31, 2017, 12:15pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997 "2017-08-31T12:15:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Maekee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maekee/32/21708_2.png) [@Maekee](https://discuss.elastic.co/u/Maekee)\ **Post date:** [August 31, 2017, 12:15pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/1 "2017-08-31T12:15:27Z")
Hello, I am running the ELK-stack on a server and want to parse the Bluecoat Proxy logs. I have created a grok filter and it works when i test it in the Grok Debugger ([http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/)) but when i add the filter in the logstash.json and run it it doesnt create the fields in kibana for me. The full message just says on the message field. This is my logstash.json file input { beats { port =\> 5044 type =\> "log" } } filter { if ([message] =~ /^#/) { drop { } } if [type] == "bluecoat" { grok =\> { match =\> { "message" =\> "^%{TIMESTAMP\_ISO8601:date} %{NUMBER:time\_taken} %{IP:c\_ip} %{USER:cs\_username} %{NOTSPACE:cs\_auth\_group} %{NOTSPACE:s\_supplier\_name} %{NOTSPACE:s\_supplier\_ip} %{NOTSPACE:s\_supplier\_country} %{NOTSPACE:s\_supplier\_failures} %{NOTSPACE:x\_exception\_id} %{NOTSPACE:sc\_filter\_result} %{QUOTEDSTRING:cs\_categories} %{NOTSPACE:cs\_Referer} %{NOTSPACE:cs\_status} %{NOTSPACE:s\_action} %{NOTSPACE:cs\_method} %{NOTSPACE:rs\_Content-Type} %{NOTSPACE:cs\_uri\_scheme} %{NOTSPACE:cs\_host} %{NOTSPACE:cs\_uri\_port} %{NOTSPACE:cs\_uri\_path} %{NOTSPACE:cs\_uri\_query} %{NOTSPACE:cs\_uri\_extension} %{QUOTEDSTRING:cs\_User\_Agent} %{IP:s\_ip} %{NUMBER:sc\_bytes} %{NUMBER:cs\_bytes} %{NOTSPACE:x\_virus\_id} %{QUOTEDSTRING:x\_bluecoat\_application\_name} %{QUOTEDSTRING:x\_bluecoat\_application\_operation} %{NUMBER:cs\_threat\_risk} %{NOTSPACE:x\_bluecoat\_transaction\_uuid} %{NOTSPACE:x\_icap\_reqmod\_header\_XICAPMetadata} %{NOTSPACE:x\_icap\_respmod\_header\_XICAPMetadata} %{NOTSPACE:cs\_auth\_type} %{NOTSPACE:x\_auth\_credential\_type}" } } } } output { elasticsearch { hosts =\> "localhost:9200" manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } stdout {} } and this is the filebeat config related to this file: - input\_type: log # Paths that should be crawled and fetched. Glob based paths. paths: - D:\LOGS\BC\*.log.gz document\_type: bluecoat encoding: utf-8 scan\_frequency: 5s Dont understand why it ignores the new fields.. is this kibana config? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 31, 2017, 12:20pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/2 "2017-08-31T12:20:34Z")
I suspect `type => "log"` overrides the type assignment in the Filebeat configuration. ---
**Author:** ![Maekee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maekee/32/21708_2.png) [@Maekee](https://discuss.elastic.co/u/Maekee)\ **Post date:** [August 31, 2017, 12:35pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/3 "2017-08-31T12:35:25Z")
Thanks (tack), i removed the line from the logstash config file and restarted logstash, created a new log and its still no new fields available ☹ ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 31, 2017, 12:42pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/4 "2017-08-31T12:42:11Z")
What does an event look like then? Copy/paste from Kibana's JSON tab so we can see the raw JSON. ---
**Author:** ![Maekee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maekee/32/21708_2.png) [@Maekee](https://discuss.elastic.co/u/Maekee)\ **Post date:** [August 31, 2017, 12:50pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/5 "2017-08-31T12:50:43Z")
I took one event: { "\_index": "filebeat-2017.08.31", "\_type": "doc", "\_id": "AV44TaLGzwkFvvGLDiXa", "\_version": 1, "\_score": null, "\_source": { "@timestamp": "2017-08-31T12:39:23.355Z", "beat": { "hostname": "SERVER01", "name": "SERVER01", "version": "5.5.2" }, "input\_type": "log", "message": "2017-08-30 08:25:01 78 192.168.10.1 user123 DOMAIN\InternetUsers [sub.domain.com](http://sub.domain.com) 192.168.11.1 Sweden - - OBSERVED "Web Ads/Analytics" - 200 TCP\_NC\_MISS GET image/gif http [sub.domain.com](http://sub.domain.com) 80 /src/media/cookie.aspx ?xid=JUNga1BqQ7j9twX7R\_y1vJUi aspx "Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko" 192.168.12.11 383 761 - "none" "none" 2 80c18e0c98d2dd54-000000007012b0b5-0000000059a6765c - "{ %22expect\_sandbox%22: false }" Digest Kerberos", "offset": 4278, "source": "D:\LOGS\BC\SG\_main\_\_7.log.gz", "type": "bluecoat" }, "fields": { "@timestamp": [ 1504183163355 ] }, "sort": [ 1504183163355 ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 31, 2017, 3:08pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/6 "2017-08-31T15:08:43Z")
I don't know what's going on here. Have you actually configured Filebeat to send to Logstash rather than directly to Elasticsearch? ---
**Author:** ![Maekee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maekee/32/21708_2.png) [@Maekee](https://discuss.elastic.co/u/Maekee)\ **Post date:** [August 31, 2017, 3:36pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/7 "2017-08-31T15:36:30Z")
Hi, No, doesnt seem like it. This was the config i had in filebeat.yml output.elasticsearch: # Array of hosts to connect to. hosts: ["localhost:9200"] #output.logstash: # The Logstash hosts #hosts: ["localhost:5044"] I switched that around, so the output.logstash is now Active (and not the output.elasticsearch) but now new files doesnt arrive into kibana. Do i need to configure any special input in the logstash config? Looks like this at the moment: input { beats { port =\> 5044 type =\> "log" } } But, the information in the output: index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" Seems to have been applied when i look at events in Kibana, \_Index have the value "filebeat-2017.08.31" for example. What am i missing? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 31, 2017, 4:40pm UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/8 "2017-08-31T16:40:15Z")
Logstash has no idea of whether the files Filebeat should read are new or old, so if old data arrives to Logstash and Elasticsearch as expected but new files aren't picked up then you have a Filebeat problem. ---
**Author:** ![Maekee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maekee/32/21708_2.png) [@Maekee](https://discuss.elastic.co/u/Maekee)\ **Post date:** [September 1, 2017, 5:56am UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/9 "2017-09-01T05:56:42Z")
I solved this, updated the beat agent config, and also uploaded the beat template to elasticsearch which i missed. I am new to this but trying to convince my workplace not to buy splunk if we can solve our needs with the ELK-stack. Alot of help in the Community and i appreciate that i get assistance when i get stuck. Stort tack Magnus! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 29, 2017, 5:57am UTC](https://discuss.elastic.co/t/bluecoat-grok-filter/98997/10 "2017-09-29T05:57:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.