# Elasticsearch - Logstash communication problem **URL:** \ **Category:** Logstash\ **Created:** [November 18, 2017, 1:11am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217 "2017-11-18T01:11:57Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![homardboy](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@homardboy](https://discuss.elastic.co/u/homardboy)\ **Post date:** [November 18, 2017, 1:11am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/1 "2017-11-18T01:11:57Z")
**Hi,** **I discovered Logstash, elasticsearch and kibana few days ago, and i'm now trying to have a kibana Dashboard of my Squid's log from Pfsense, but i got some issues ...** **The logs from Squid are from the web trafic of my LAN.** **A default log entry look like this :** Nov 17 21:01:10 192.168.1.1 (squid-1): 1510952470.370 233176 192.168.1.103 TCP\_TUNNEL/200 5931 CONNECT [etherpad.fr:443](http://etherpad.fr:443) - ORIGINAL\_DST/195.154.57.241 - **The Squid's log are send to a NAS server with syslog-ng on /var/log/pfsense/pfSense.long.** **On this remote server, i installed logstash, elasticsearch and kibana.** **My grok for the logs is the following, and is working fine on an online grok debugger :** \_%{WORD:month} %{NUMBER:date} %{TIME:timestamp} %{IP:proxy} (%{WORD:squid\_version}-%{INT:http\_status\_code}): %{NUMBER:header1} %{NUMBER:header2} %{IP:ip\_source} %{WORD:protocole}/%{INT:code} %{NUMBER:header3} %{WORD:status} %{USER:fqdn\_destination}:%{NUMBER:port\_destination} - %{WORD:label\_destination}/%{IP:ip\_destination} \_- **I made a pfsense\_log.conf on the logstash directory, with the following options :** input { file { path =\> "/var/log/pfsense/pfSense.log" start\_position =\> beginning sincedb\_path =\> "/dev/null" } } filter { grok { match =\> { "message" =\> "%{WORD:month} %{NUMBER:date} %{TIME:timestamp} %{IP:proxy} (%{WORD:squid\_version}-%{INT:http\_status\_code}): %{NUMBER:header1} %{NUMBER:header2} %{IP:ip\_source} %{WORD:protocole}/%{INT:code} %{NUMBER:header3} %{WORD:status} %{USER:fqdn\_destination}:%{NUMBER:port\_destination} - %{WORD:label\_destination}/%{IP:ip\_destination} -" } } } output { stdout { codec =\> plain { charset =\> "ISO-8859-1" } } elasticsearch { hosts =\> ["127.0.0.1:9200"] index =\> "pfsense\_log" template =\> "/home/user/ELK/logstash-2.2.2/bin/pfsense\_template.json" template\_name =\> "pfsense\_log" template\_overwrite =\> true } } **And a pfsense\_template.json :** { "template": "pfsense\_log", "settings": { "index.refresh\_interval": "5s" }, "mappings": { "_default_": { "dynamic\_templates": [ { "message\_field": { "mapping": { "index": "analyzed", "omit\_norms": true, "type": "string" }, "match\_mapping\_type": "string", "match": "message" } }, { "string\_fields": { "mapping": { "index": "analyzed", "omit\_norms": true, "type": "string", "fields": { "raw": { "index": "not\_analyzed", "ignore\_above": 256, "type": "string" } } }, "match\_mapping\_type": "string", "match": "\*" } } ], "properties": { "date": { "type": "date", "format": "date\_time\_no\_millis" }, "proxy": { "type": "string", "index": "not\_analyzed" }, "ip\_source": { "type": "string", "index": "not\_analyzed" }, "header1": { "type": "integer", "index": "not\_analyzed" }, "protocole": { "type": "string", "index": "not\_analyzed" }, "http\_status\_code": { "type": "integer", "index": "not\_analyzed" }, "fqdn\_destination": { "type": "string", "index": "not\_analyzed" }, "status": { "type": "string", "index": "not\_analyzed" }, "label\_destination": { "type": "string", "index": "not\_analyzed" }, "squid": { "type": "string", "index": "not\_analyzed" }, "ip\_destination": { "type": "string", "index": "not\_analyzed" }, "\_all": { "enabled": true } } } } } **Wen i run logstash (logstash -f pfsense\_log.conf), i see that the data from my log file are loaded :** "path"=\>"/var/log/pfsense/pfSense.log", "host"=\>"user-virtual-machine", "month"=\>"Nov", "date"=\>"17", "timestamp"=\>"22:12:41", "proxy"=\>"192.168.1.1", "squid\_version"=\>"squid", "http\_status\_code"=\>"1", "header1"=\>"1510956761.029", "header2"=\>"181845", "ip\_source"=\>"192.168.1.103", "protocole"=\>"TCP\_TUNNEL", "code"=\>"200", "header3"=\>"16464", "status"=\>"CONNECT", "fqdn\_destination"=\>"[etherpad.fr](http://etherpad.fr)", "port\_destination"=\>"443", "label\_destination"=\>"ORIGINAL\_DST", "ip\_destination"=\>"195.154.57.241"}, @lut={"path"=\>[{"message"=\>"Nov 17 22:12:41 192.168.1.1 (squid-1): 1510956761.029 181845 192.168.1.103 TCP\_TUNNEL/200 16464 CONNECT [etherpad.fr:443](http://etherpad.fr:443) - ORIGINAL\_DST/195.154.57.241 -", "@version"=\>"1", "@timestamp"=\>"2017-11-18T00:42:38.562Z" **So i know that logstash take my log file and apply filter on it. But when i start elasticsearch, and when i try to reach 127.0.0.1:9200/pfsense\_log (the name of my index), elasticsearch can't find the file, and i got an "Index\_not\_found\_exeption" error ... I'm not really sure if i need to manually create the "pfsense\_log" file, and where to put it 😕 (elasticsearch/conf ? ) Or maybe i miss Something else ? Any idee ?** **Thanks 😄** ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 18, 2017, 9:18am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/2 "2017-11-18T09:18:11Z")
Which version are you using? Is there anything in the Elasticsearch and/or Logstash logs? ---
**Author:** ![homardboy](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@homardboy](https://discuss.elastic.co/u/homardboy)\ **Post date:** [November 18, 2017, 1:49pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/3 "2017-11-18T13:49:28Z")
Hi, I am using Logstash version 2.2.2, elasticsearch 2.2.1 and kibana 4.4.2. And yes, i have some very strange things in my logs from logstash and elasticsearch : ![logstashlog1](https://us1.discourse-cdn.com/elastic/original/3X/6/8/6838734c5e47c774c458e1ba297967315c240fe9.PNG) And for elasticsearch : ![elasticsearchlog2](https://us1.discourse-cdn.com/elastic/original/3X/d/2/d2990d8806c7b36c4173e2eb54d18702c62efb61.PNG) ![elasticsearchlog3](https://us1.discourse-cdn.com/elastic/original/3X/2/c/2ced9d07ab9f2fa12538b4380982f221888c30e9.PNG) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 18, 2017, 1:54pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/4 "2017-11-18T13:54:11Z")
If you are just starting to use the stack, why are you not using the latest versions?? ---
**Author:** ![homardboy](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@homardboy](https://discuss.elastic.co/u/homardboy)\ **Post date:** [November 18, 2017, 1:59pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/5 "2017-11-18T13:59:56Z")
That's because i read a lot of post about problem related to different version of logstash and elasticsearch trying to communicate together ... So i search which version of the bundle is 100% working fine together and i used it 😊 Let me try again with the latest version 🙂 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 18, 2017, 2:03pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/6 "2017-11-18T14:03:41Z")
It used to be a bit confusing as the components had different versioning schemes. From version 5.0, we however changed to unified releases, where it is expected that all components of the stack are at the same version. This is a lot easier, and also makes the support matrix considerably more compact. 🙂 ---
**Author:** ![homardboy](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@homardboy](https://discuss.elastic.co/u/homardboy)\ **Post date:** [November 18, 2017, 2:34pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/7 "2017-11-18T14:34:43Z")
Oh ok, i see 😄 I just try with the 6.0.0 version of the stack, but i can't run logstash and elasticsearch at the same time, it's about an insufficient java runtime memory allocation ... But my virtual machine as something like 4Go of RAM and 4 processor of 1 core 😅 Should i try with the 5.0.0 version ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 18, 2017, 2:41pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/8 "2017-11-18T14:41:13Z")
4GB of RAM should be fine for version 6.0, as I believe both by default use a 1GB heap. ---
**Author:** ![homardboy](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@homardboy](https://discuss.elastic.co/u/homardboy)\ **Post date:** [November 18, 2017, 3:00pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/9 "2017-11-18T15:00:10Z")
I play a little bit with the jvm.option file and it's all good now ! And i can reach my index : ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bb0a5482ad91702103c93c355218ebc95a1f522f.PNG) Regardeless to my output, i don't know if the grok filter is working, or if the data is parse ... Anyway, thank's for your help 😄 I'm going to search and try if it work ! ---
**Author:** ![homardboy](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@homardboy](https://discuss.elastic.co/u/homardboy)\ **Post date:** [November 18, 2017, 3:55pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/10 "2017-11-18T15:55:46Z")
Well, it's working ! I have my log from pfsense in my kibana Dashboard !! Thank you again, i'm really happy 😄 I just need to find a way to parse my incomming logs with multiple grok filter (i have different synthaxe depending on the log : ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/5/0/5074f39938da753f8f35891430ef80a4fa94a6ea.PNG) ) Problem solve ! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 18, 2017, 10:12pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/11 "2017-11-18T22:12:48Z")
Just a quick tip, it's always good to not post pictures of text. Some people block pics and sometimes they are really hard to see. But in all cases pictures mean we cannot copy/paste your data to try and help 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 16, 2017, 10:12pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-communication-problem/108217/12 "2017-12-16T22:12:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.