# Begginers help grok and check if data is making it into elastic search **URL:** \ **Category:** Logstash\ **Created:** [November 1, 2017, 9:06pm UTC](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099 "2017-11-01T21:06:00Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![burntfaceman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/burntfaceman/32/23671_2.png) [@burntfaceman](https://discuss.elastic.co/u/burntfaceman)\ **Post date:** [November 1, 2017, 9:06pm UTC](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099/1 "2017-11-01T21:06:00Z")
Hi, I have just started to play with elasticstack for the first time. My objective to create a simple to use system for firewall rule log aggregation/searching. I have followed the basic guide to setup logstash, Elasticsearch (with Xpack) , kibana ( with Xpack) all on the same host. I have been using [http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/) to develop my grok paser/filter whatever you call it. but I am failing to make the system work end to end and I cant even verify if my logstash configuration is working correctly. This is currently what my only logstash config file looks like > input{ > tcp { > port =\> "55514" > type =\> "syslog-F5" > } > udp { > port =\> "55514" > type =\> "syslog-F5" > } > } > filter { > if [type] =~ "syslog-F5" { > grok { > match =\> { "message" =\> "^%{TIMESTAMP\_ISO8601:SYSLOG\_TIME} %{IPV4:DEVICEIP} %{HOSTNAME:HOSTNAME}|%{UNIXPATH:CONTEXT\_NAME}|(?\[a-zA-Z\_]\*)|%{POSINT:RD}|%{UNIXPATH:ACL\_NAME}|%{WORD:INUSE}|%{UNIXPATH:RULENAME}|%{WORD:ACTION}|((?\[a-zA-Z\_]+))?|%{IPV4:SRC\_IP}|%{NUMBER:SRC\_PORT}|%{IPV4:DST\_IP}|%{NUMBER:DST\_PORT}|%{WORD:PROTOCOL}|%{UNIXPATH:VLANID}|%{GREEDYDATA:RAW\_DATE}" } > } > date { > match =\> ["RAW\_DATE" , "MMM dd yyyy HH:mm:ss"] > } > } > } > output { > elasticsearch { > hosts =\> localhost > user =\> elastic > password =\> changeme > } > file{ > codec =\> plain > path =\> "/mnt/data/logstash/test.log" > } > } Here is an example log taken from the output log file > 2017-11-01T14:57:00.536Z 10.107.20.10 test2.mydomain.com.location|/Common/N82-Prod-Prod|Route Domain|1|/Comm-Prod|Enforced|/Common/Prod-Prod-Outside:Users\_to\_Prod\_VRF-ip|Accept||10.200.5.5|16121|10.101.40.2|1967|UDP|/Common/VL\_Prod-Prod-Outside\_F5|Nov 01 2017 14:56:28 So that log is decoded fine by my grok on the site [http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/) but after that this is where my confusion begins. 1. with my output setting a file should I not be seeing the key:pair values in it? Currently I only see un processed messages as they look like coming from syslog. If that's correct how do I check if my grok script is working? 2. how do I check if data is making it from logstash into Elasticsearch? I am not seeing any data in Kibana and get the cant find index pattern logstash-\* error. my logstash log looks like > [2017-11-02T01:54:09,473][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://elastic:xxxxxx@localhost:9200/](http://elastic:xxxxxx@localhost:9200/)]}} > [2017-11-02T01:54:09,474][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://elastic:xxxxxx@localhost:9200/](http://elastic:xxxxxx@localhost:9200/), :path=\>"/"} > [2017-11-02T01:54:09,652][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://elastic:xxxxxx@localhost:9200/](http://elastic:xxxxxx@localhost:9200/)"} > [2017-11-02T01:54:09,675][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} > [2017-11-02T01:54:09,680][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::Elasticsearch", :hosts=\>["[//localhost](https://localhost)"]} > [2017-11-02T01:54:09,708][INFO][logstash.pipeline] Starting pipeline {"id"=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>5, "pipeline.max\_inflight"=\>500} > [2017-11-02T01:55:05,938][INFO][logstash.pipeline] Pipeline main started > [2017-11-02T01:55:05,939][INFO][logstash.inputs.udp] Starting UDP listener {:address=\>"0.0.0.0:55514"} > [2017-11-02T01:55:05,947][INFO][logstash.inputs.udp] UDP listener started {:address=\>"0.0.0.0:55514", :receive\_buffer\_bytes=\>"62464", :queue\_size=\>"2000"} > [2017-11-02T01:55:05,957][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} > [2017-11-02T01:55:06,183][INFO][logstash.outputs.file] Opening file {:path=\>"/mnt/data/logstash/test.json"} and my Elasticsearch log looks like > [2017-11-02T00:47:35,820][INFO][o.e.c.m.MetaDataMappingService] [SSnuhdW] [logstash-2017.11.01/i8SUUAl3TDOKGJCtYBZN0w] create\_mapping [syslog-F5] > [2017-11-02T01:12:38,416][INFO][o.e.c.m.MetaDataMappingService] [SSnuhdW] [logstash-2017.11.01/i8SUUAl3TDOKGJCtYBZN0w] update\_mapping [syslog-F5] > [2017-11-02T01:30:00,001][INFO][o.e.x.m.MlDailyMaintenanceService] triggering scheduled [ML] maintenance tasks > [2017-11-02T01:30:00,002][INFO][o.e.x.m.a.DeleteExpiredDataAction$TransportAction] [SSnuhdW] Deleting expired data > [2017-11-02T11:00:00,262][INFO][o.e.c.m.MetaDataCreateIndexService] [SSnuhdW] [logstash-2017.11.02] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_] > [2017-11-02T11:00:00,310][INFO][o.e.c.m.MetaDataMappingService] [SSnuhdW] [logstash-2017.11.02/OV1XNcv4TH2AARaFN3lZFw] create\_mapping [syslog-F5] > [2017-11-02T11:00:05,587][INFO][o.e.c.m.MetaDataCreateIndexService] [SSnuhdW] [.monitoring-es-6-2017.11.02] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc] > [2017-11-02T11:00:08,758][INFO][o.e.c.m.MetaDataCreateIndexService] [SSnuhdW] [.monitoring-kibana-6-2017.11.02] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[1], mappings [doc] > [2017-11-02T11:00:46,128][INFO][o.e.c.m.MetaDataCreateIndexService] [SSnuhdW] [.watcher-history-6-2017.11.02] creating index, cause [auto(bulk api)], templates [.watch-history-6], shards [1]/[1], mappings [doc] > [2017-11-02T11:00:46,156][INFO][o.e.c.m.MetaDataMappingService] [SSnuhdW] [.watcher-history-6-2017.11.02/sWXegmLzTh2o-mPXpck2mA] update\_mapping [doc] > [2017-11-02T11:00:46,199][INFO][o.e.c.m.MetaDataMappingService] [SSnuhdW] [.watcher-history-6-2017.11.02/sWXegmLzTh2o-mPXpck2mA] update\_mapping [doc] I would appreciate any help 🙂 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 2, 2017, 8:32am UTC](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099/2 "2017-11-02T08:32:09Z")
> with my output setting a file should I not be seeing the key:pair values in it? Not with the plain codec that you've configured for your file output. To debug inputs and filters I always recommend using a rubydebug codec. > how do I check if data is making it from logstash into Elasticsearch? I am not seeing any data in Kibana and get the cant find index pattern logstash-\* error. Your ES logs prove that you're getting data. Perhaps it's a permissions issue, i.e. the user you're logged into Kibana as doesn't have permission to read the logstash-2017.11.02 index? ---
**Author:** ![burntfaceman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/burntfaceman/32/23671_2.png) [@burntfaceman](https://discuss.elastic.co/u/burntfaceman)\ **Post date:** [November 2, 2017, 9:25pm UTC](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099/3 "2017-11-02T21:25:12Z")
Thax for getting back to me, On the first point my build suffers a failure in Jruby when I set rubydebug codec so I cant use it. Also interestingly the data displayed in that log has additional data prepended compared to the actual payload data. So I was getting Grok failures so I just had to remove the first two pattern matches. On the Second point you are 100% this was my issue, I was logged into kibana as the kibana user, I had to log in as the elastic user. issuing curl -GET [http://elastic:changeme@localhost:9200/\_cat/indices](http://elastic:changeme@localhost:9200/_cat/indices)? | grep log showed that the data was getting to elastic stack and the counts where growing. I was able to customize the template for elastic search easy enough following : > **[​Little Logstash Lessons: Using Logstash to help create an Elasticsearch mapping...](https://www.elastic.co/blog/logstash_lesson_elasticsearch_mapping)** > > How to use Logstash together with Elasticsearch to create custom mapping templates. Improve your Elasticsearch storage and performance! [https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) So its all looking pretty good, I know this next question is a little off topic but I haven't been able to find a current answer. Do CIDR based lookups work in Kibana? I have only be able to use a range style command so far. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 3, 2017, 6:28am UTC](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099/4 "2017-11-03T06:28:03Z")
> On the first point my build suffers a failure in Jruby when I set rubydebug codec so I cant use it. I've never heard about that before. > Do CIDR based lookups work in Kibana? I don't believe so, no. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 1, 2017, 6:28am UTC](https://discuss.elastic.co/t/begginers-help-grok-and-check-if-data-is-making-it-into-elastic-search/106099/5 "2017-12-01T06:28:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.