# Normalize the logs with ELK **URL:** \ **Category:** Logstash\ **Created:** [April 26, 2019, 3:58pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684 "2019-04-26T15:58:27Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 11:36am UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/21 "2019-05-09T11:36:19Z")
the name of script is : test-switch ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 9, 2019, 11:39am UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/22 "2019-05-09T11:39:18Z")
What I was after is something like this (my script is named logger.sh ``` $ ls -l logger.sh -rwx------ 1 user group 147 Apr 13 2017 logger.sh ``` As you can see the script is executable by user `user` (because `-rwx------`). ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 11:41am UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/23 "2019-05-09T11:41:01Z")
![02](https://us1.discourse-cdn.com/elastic/original/3X/f/a/fa4db03da51f0b57df72ce4ded3b8463534c7369.png) ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 11:41am UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/24 "2019-05-09T11:41:28Z")
![01](https://us1.discourse-cdn.com/elastic/original/3X/d/5/d5286dca4fd0733c44b629c99ec4831f5437d7fa.png) ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 11:59am UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/25 "2019-05-09T11:59:48Z")
![cccccccc](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3ede976d70865d333bbae2ecfcdcfbb28bb127b0.png) ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 12:01pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/26 "2019-05-09T12:01:04Z")
@A_B i must tape chmod +x test-switch.sh ... but when I run the script; connection refused ![re](https://us1.discourse-cdn.com/elastic/original/3X/e/e/ee48250deb555bb3d8dfb26024baa53e094b91b9.png) ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 9, 2019, 12:20pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/27 "2019-05-09T12:20:31Z")
You have to of corse change the hostname and port in this comment ```auto $ nc -v -z -u logstash.example.com 5515 ``` You did change the port but should it not be `1514` ? If the above comment does not succeed then the script will not work. You can see if Logstash is listening on port `1514` with this as well ``` # lsof -n -P -i :1514 ``` That command should return something like ``` COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 26964 logstash 173u IPv4 105197915 0t0 UDP *:1514 ``` ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 12:33pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/28 "2019-05-09T12:33:39Z")
![07](https://us1.discourse-cdn.com/elastic/original/3X/4/e/4effadbc72aa79593881b7e77f18fea5899cf7d0.png) how i make logstash lestenning in port 1514 ( centos7) ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 9, 2019, 1:18pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/29 "2019-05-09T13:18:17Z")
Is Logstash running at all? As I have written already ```auto # cat logstash.conf input { udp { port => 1514 } } filter {} output { stdout { codec => rubydebug } } # /path/to/logstash/bin/logstash -f /path/to/logstash.conf ``` That would start Logstash, listening on UDP port `1514`. But that starts Logstash in the foreground so as soon as you exit Logstash stops again... I do not use Centos so not sure how the Logstash service is managed. ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 1:47pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/30 "2019-05-09T13:47:33Z")
yes i do the same in /etc/logstash/conf.d/switch-cisco.conf ![a](https://us1.discourse-cdn.com/elastic/original/3X/7/6/765eb4e7b34b962aa05a6c5b3d70cfbb55a167f5.png) for the installation of ELK under centos 7 server I followed this link > **[How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on CentOS 7...](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-centos-7)** > > In this tutorial for CentOS 7, you will learn how to install all of the components of the Elastic Stack, a collection of open-source software produced by Elastic which allows you to search, analyze, and visualize logs generated from any source in any ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 9, 2019, 1:50pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/31 "2019-05-09T13:50:34Z")
How do you start Logstash? Is it running as a service? ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 1:54pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/32 "2019-05-09T13:54:01Z")
![service%20logstash](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c9ac37985ed09a00ee506bbf13f2334ce9c1c6da.png) ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 9, 2019, 2:02pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/33 "2019-05-09T14:02:19Z")
What does `cat /etc/logstash/logstash.yml` show? Or whatever your main Logstash config file is called... ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 2:16pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/34 "2019-05-09T14:16:03Z")
the file logstash is a big file, the script does not work ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 9, 2019, 2:20pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/35 "2019-05-09T14:20:27Z")
What I am trying to find out is what your `path.config` is set to. Below an example from one of my machines. ```auto --- path.data: "/var/lib/logstash" path.config: "/etc/logstash/conf.d" path.logs: "/var/log/logstash" log.level: warn dead_letter_queue.enable: true dead_letter_queue.max_bytes: 10240mb path.dead_letter_queue: "/var/dead_letter_queue" pipeline.batch.size: 25 pipeline.batch.delay: 5 ``` As you can see ``` path.config: "/etc/logstash/conf.d" ``` For me that folder looks like ``` # ls -l /etc/logstash/conf.d total 16 -rw-r----- 1 root logstash 6007 Apr 10 08:27 filter -rw-r----- 1 root logstash 2246 Nov 23 11:32 input -rw-r----- 1 root logstash 230 Apr 10 08:27 output ``` I'm trying to help you figure out if Logstash is actually using your config from above because it sure looks like it is not, because Logstash is not listening on port `1514`. ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 9, 2019, 2:33pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/36 "2019-05-09T14:33:02Z")
![conffffff](https://us1.discourse-cdn.com/elastic/original/3X/2/6/2629d6e5fb13741cbd94415f755693497b3546e1.png) ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 9, 2019, 2:54pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/37 "2019-05-09T14:54:22Z")
I do not know if you can have several configuration files for different inputs. So I would suggest to at least during troubleshooting to move all inputs to one file. Try adding the `udp` input to 02-beats-input.conf The resulting file should look something like ``` input { beats { port => 5044 } udp { port => 1514 } } ``` Then restart Logstash and see if it is now listening on port `1514` with `lsof`. ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 10, 2019, 7:01am UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/38 "2019-05-10T07:01:23Z")
@A_B you are right , I want to know if i have several configuration files for different inputs. it causes problems ? ☹ my goal is to collect the Cisco switch logs and Palo alto firewall, for the palo alto, I found this link, I followed it but it's not clear to me, > **[ELK + Palo Alto Networks](https://anderikistan.com/2016/03/26/elk-palo-alto-networks/)** > > Why ELK Stack Chances are that if you’re here you already know what the ELK Stack is and what it is used for. For the uninitiated ELK is actually an acronym (Elasticsearch / Logstash / Kibana). Here’s a brief overview: Logstash – Data collection and... I i don't know for logstash, if I put a single input for several devices, or each device has an input file and filter file and output file, each one apart ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [May 10, 2019, 7:17am UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/39 "2019-05-10T07:17:01Z")
This is how my config files are arranged ``` # ls -l /etc/logstash/conf.d total 16 -rw-r----- 1 root logstash 6007 Apr 10 08:27 filter -rw-r----- 1 root logstash 2246 Nov 23 11:32 input -rw-r----- 1 root logstash 230 Apr 10 08:27 output ``` One file per config file "section". I have probably 10 inputs, many filters and try to use only one output (although I have used several in the past). It is your choice if you use more than one input. Personally, I would probably use one for Cisco devices and another one for the Palo Alto devices. This would let you add tags and fields to the inputs that can be very useful in the filter section. As an example, here is the Logstash input config for the Juniper switch and firewall logs where I work ``` # Juniper log JSON input syslog { port => 5518 grok_pattern => "%{SYSLOG5424PRI}%{NUMBER:syslog_severity} %{SYSLOGLINE}" type => "juniper" add_field => { "[@metadata][index]" => "juniper" "[@metadata][log_prefix]" => "dc" } } ``` First of all, Juniper logs are in `syslog`format, so the input type is `syslog`. This input listens on both TCP and UDP according to the [documentation](https://www.elastic.co/guide/en/logstash/7.0/plugins-inputs-syslog.html). The `port`is self-explanatory. The `syslog` input allows using _grok_ already on the input which is nice. I the logs were in JSON this would not be necessary. The `type` adds a filed which can be used in the filter section but I usually use the added @metadata fields for that. I also use those @metadata fields for setting Elasticsearch index name. Having said all that, I would start as simple as possible and add complexity as you go. ---
**Author:** ![Labidi\_Ayoub](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/labidi_ayoub/32/45116_2.png) [@Labidi\_Ayoub](https://discuss.elastic.co/u/Labidi_Ayoub)\ **Post date:** [May 10, 2019, 2:28pm UTC](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684/40 "2019-05-10T14:28:28Z")
so you advise me to put a single input and filter file and output `udp` input to 02-beats-input.conf ![beats](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a2bf55355528b30564a09411e2902c1c5e301c51.png) ![on](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8c40231dc3385641436ab29ef013a0890bd22abe.png) something was wrong, can be juniper it's not like cisco in the configuration. please there is no guide or tutto for cisco switch log collection with ELK i am stuck ☹ [Previous page](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684.md?page=1) [Next page](https://discuss.elastic.co/t/normalize-the-logs-with-elk/178684.md?page=3)