# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1037 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1038 --- ## [Parsing SQL Error Logs & Best Practices](https://discuss.elastic.co/t/parsing-sql-error-logs-best-practices/74716)
**Author:** [@locolbd](https://discuss.elastic.co/u/locolbd)\ **Replies:** 2\ **Last updated:** [February 13, 2017, 6:24pm UTC](https://discuss.elastic.co/t/parsing-sql-error-logs-best-practices/74716 "2017-02-13T18:24:28Z")
I am in the early stages of collecting SQL logs from the sql errrorlog file. My DBAs are slowing giving me information as to the type of data they would like parsed into fields etc. I would like to parse the domain an… --- ## [Reduce similar log entries](https://discuss.elastic.co/t/reduce-similar-log-entries/74913)
**Author:** [@yami](https://discuss.elastic.co/u/yami)\ **Replies:** 0\ **Last updated:** [February 13, 2017, 5:50pm UTC](https://discuss.elastic.co/t/reduce-similar-log-entries/74913 "2017-02-13T17:50:16Z")
I've read https://discuss.elastic.co/t/reduce-duplicate-or-similar-log-entries/29029 how can i Reduce duplicate or similar log entries with collate? I used it like that: collate { interval =\> "1s" order =\>… --- ## ["Fetched and Invalid Config" Error](https://discuss.elastic.co/t/fetched-and-invalid-config-error/74568)
**Author:** [@jonarmani](https://discuss.elastic.co/u/jonarmani)\ **Replies:** 4\ **Last updated:** [February 13, 2017, 4:16pm UTC](https://discuss.elastic.co/t/fetched-and-invalid-config-error/74568 "2017-02-13T16:16:44Z")
Getting the official Logstash Docker container running on CentOS 7. Strange error after 15 or 30 seconds of launching the instance: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.prope… --- ## [Aggregate all event messages](https://discuss.elastic.co/t/aggregate-all-event-messages/74866)
**Author:** [@ademicky](https://discuss.elastic.co/u/ademicky)\ **Replies:** 1\ **Last updated:** [February 13, 2017, 4:10pm UTC](https://discuss.elastic.co/t/aggregate-all-event-messages/74866 "2017-02-13T16:10:42Z")
Hi Everyone, I am sorry if I'm posting this in the wrong place. I have an issue with logstash aggregate filter which has been on going for 2 weeks now. I am a java developer, so dont understand much of ruby. The challen… --- ## [Indexes failing to auto create from filebeat](https://discuss.elastic.co/t/indexes-failing-to-auto-create-from-filebeat/74785)
**Author:** [@funtimes\_ninja](https://discuss.elastic.co/u/funtimes_ninja)\ **Replies:** 7\ **Last updated:** [February 13, 2017, 3:25pm UTC](https://discuss.elastic.co/t/indexes-failing-to-auto-create-from-filebeat/74785 "2017-02-13T15:25:57Z")
I've have an issue with an indexes not being created anymore on more than one occasion. Previously I thought maybe it was because I upgrade from 4.x to 5.x, and installed x-pack(monitoring). Last night/this morning I sto… --- ## [Filter grok : Few lines differents in log file](https://discuss.elastic.co/t/filter-grok-few-lines-differents-in-log-file/74529)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 4\ **Last updated:** [February 13, 2017, 3:08pm UTC](https://discuss.elastic.co/t/filter-grok-few-lines-differents-in-log-file/74529 "2017-02-13T15:08:34Z")
Hello, I have a little problem about performing of of my "q\_compt" file (file text) 80% of lines have 4 fields 10% have 3 fields 10% have 4 fields but have differents informations in. Here my filter expression : … --- ## [Logstash process config](https://discuss.elastic.co/t/logstash-process-config/74894)
**Author:** [@ggp](https://discuss.elastic.co/u/ggp)\ **Replies:** 0\ **Last updated:** [February 13, 2017, 2:27pm UTC](https://discuss.elastic.co/t/logstash-process-config/74894 "2017-02-13T14:27:37Z")
I have a 9 node cluster with Master Nodes = 3 Data Nodes = 4 Client Nodes = 2 so if i want to run Logstash process should they be communicating with Client Only ? similar to the consumers point to Client Node to pr… --- ## [Logstash Bad url error](https://discuss.elastic.co/t/logstash-bad-url-error/74875)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 0\ **Last updated:** [February 13, 2017, 12:04pm UTC](https://discuss.elastic.co/t/logstash-bad-url-error/74875 "2017-02-13T12:04:23Z")
Hi, Im using logstash new version 5.2.0. Im facing issue while running the bat file in windows. Please find the message below, Command to run: C:\>C:\\logstash-5.2.0\\bin\\logstash.bat --path.settings C:/logstash-5.2.0/c… --- ## [Indexing CSV data into elasticsearch](https://discuss.elastic.co/t/indexing-csv-data-into-elasticsearch/55871)
**Author:** [@Meenakshi07](https://discuss.elastic.co/u/Meenakshi07)\ **Replies:** 7\ **Last updated:** [February 13, 2017, 2:13pm UTC](https://discuss.elastic.co/t/indexing-csv-data-into-elasticsearch/55871 "2017-02-13T14:13:32Z")
I have tried the below code to import data from CSV into elasticsearch using logstash: input { file { path =\> \["D:\\test.csv"\] type =\> "csv" start\_position =\> "beginning" } } filter { c… --- ## [Logstash-codec-nmap install fails 5.2](https://discuss.elastic.co/t/logstash-codec-nmap-install-fails-5-2/74881)
**Author:** [@CaveJohnson](https://discuss.elastic.co/u/CaveJohnson)\ **Replies:** 0\ **Last updated:** [February 13, 2017, 12:52pm UTC](https://discuss.elastic.co/t/logstash-codec-nmap-install-fails-5-2/74881 "2017-02-13T12:52:27Z")
I;m trying to install the nmap codec for logstash however the install fails on the latest version (5.2) with the error logstash-codec-nmap (\>= 0) java depends on logstash-core (\< 3.0.0, \>= 2.0.0) java logstash-core (\>… --- ## [Verification failed for logstash-output-zabbix](https://discuss.elastic.co/t/verification-failed-for-logstash-output-zabbix/29383)
**Author:** [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\ **Replies:** 18\ **Last updated:** [February 13, 2017, 12:01pm UTC](https://discuss.elastic.co/t/verification-failed-for-logstash-output-zabbix/29383 "2017-02-13T12:01:19Z")
Hi, I am trying to install logstash-output-zabbix plugin with below command. $sudo ./plugin install logstash-output-zabbix But it shows below error. " Validating logstash-output-zabbix "Unable to download data … --- ## [Problem getting logstash started](https://discuss.elastic.co/t/problem-getting-logstash-started/74854)
**Author:** [@agrawalsaurabh](https://discuss.elastic.co/u/agrawalsaurabh)\ **Replies:** 2\ **Last updated:** [February 13, 2017, 11:40am UTC](https://discuss.elastic.co/t/problem-getting-logstash-started/74854 "2017-02-13T11:40:16Z")
Hi, I am a newbie and trying to use the stack of Elastic Search, Logstash and Kibana working together to parse the apache log file and visualize it. I started with this URL - https://www.elastic.co/downloads/logstash w… --- ## [Unable to install the elapsed plugin on windows](https://discuss.elastic.co/t/unable-to-install-the-elapsed-plugin-on-windows/74609)
**Author:** [@GautamP](https://discuss.elastic.co/u/GautamP)\ **Replies:** 6\ **Last updated:** [February 13, 2017, 8:46am UTC](https://discuss.elastic.co/t/unable-to-install-the-elapsed-plugin-on-windows/74609 "2017-02-13T08:46:46Z")
Continuing the discussion from Query to access distinct fields from different documents: I'm trying to install the logstash elapsed plugin on windows 7. I ran the code C:\\ELK\\logstash-5.2.0\\bin\>logstash-plugin.bat in… --- ## [Help with a logstash instance processing 4000 messages/s](https://discuss.elastic.co/t/help-with-a-logstash-instance-processing-4000-messages-s/73824)
**Author:** [@Laverio](https://discuss.elastic.co/u/Laverio)\ **Replies:** 12\ **Last updated:** [February 13, 2017, 8:26am UTC](https://discuss.elastic.co/t/help-with-a-logstash-instance-processing-4000-messages-s/73824 "2017-02-13T08:26:58Z")
Hi, I'm suffering a serious tuning issue processing messages from a kafka queue. The architecture is the following: ASA Firewall -\> logstash input -\> kafka queue (32 partitions) -\> logstash output (4 workers) -\> elas… --- ## [Exclude property in File input plugin](https://discuss.elastic.co/t/exclude-property-in-file-input-plugin/74343)
**Author:** [@spravn789](https://discuss.elastic.co/u/spravn789)\ **Replies:** 3\ **Last updated:** [February 13, 2017, 7:25am UTC](https://discuss.elastic.co/t/exclude-property-in-file-input-plugin/74343 "2017-02-13T07:25:10Z")
I have a directory with multiple log file \[\*.log\], among which i need to exclude few logs that starts with CBATest.log,BPLTest.log and ARCTest.log. Below provided is the configuration used. But these files are not exclu… --- ## [Configure multi line in logstash error with multi input file](https://discuss.elastic.co/t/configure-multi-line-in-logstash-error-with-multi-input-file/74840)
**Author:** [@Tr\_ng\_Trang](https://discuss.elastic.co/u/Tr_ng_Trang)\ **Replies:** 0\ **Last updated:** [February 13, 2017, 7:21am UTC](https://discuss.elastic.co/t/configure-multi-line-in-logstash-error-with-multi-input-file/74840 "2017-02-13T07:21:59Z")
i configure file filebeat.yml like: filebeat.prospectors: - paths: - "/var/log/payload" fields: {log\_type: payload} - paths: - "/var/log/ont" fields: {log\_type: ont} # The regexp Pattern that has to … --- ## [BRO IDS integration with ELK Stack](https://discuss.elastic.co/t/bro-ids-integration-with-elk-stack/74839)
**Author:** [@ruby](https://discuss.elastic.co/u/ruby)\ **Replies:** 0\ **Last updated:** [February 13, 2017, 7:14am UTC](https://discuss.elastic.co/t/bro-ids-integration-with-elk-stack/74839 "2017-02-13T07:14:46Z")
I have installed BRO IDS and filebeat shipper on a System and ELK stack on another system Now i want BRO -IDS alerts from one system to another, what need to configure on ELK stack one logstash configuration file BRo i… --- ## [Logstash Update a document in elasticsearch](https://discuss.elastic.co/t/logstash-update-a-document-in-elasticsearch/74039)
**Author:** [@banu](https://discuss.elastic.co/u/banu)\ **Replies:** 10\ **Last updated:** [February 6, 2017, 11:16am UTC](https://discuss.elastic.co/t/logstash-update-a-document-in-elasticsearch/74039 "2017-02-06T11:16:45Z")
Trying to update a specific field in elasticsearch through logstash. Is it possible to update only a set of fields through logstash ? Please find the code below, filter { csv { separator =\> "|… --- ## [Logstash json filter parsed fields cannot be read within logstash](https://discuss.elastic.co/t/logstash-json-filter-parsed-fields-cannot-be-read-within-logstash/74598)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 4\ **Last updated:** [February 13, 2017, 5:37am UTC](https://discuss.elastic.co/t/logstash-json-filter-parsed-fields-cannot-be-read-within-logstash/74598 "2017-02-13T05:37:35Z")
I am parsing a json file with "codec =\> json" in the input and " json { source=\>message }" in the filter. I have also tried alternating the two. The parsed fields cannot be read by logstash using "if \[comment\]". This… --- ## [Access arrays in array](https://discuss.elastic.co/t/access-arrays-in-array/74828)
**Author:** [@yxc](https://discuss.elastic.co/u/yxc)\ **Replies:** 0\ **Last updated:** [February 13, 2017, 2:04am UTC](https://discuss.elastic.co/t/access-arrays-in-array/74828 "2017-02-13T02:04:52Z")
Hi all, I have a nested field with arrays in array in JSON like the following: { "foo": { "bar": \[ \[ "a", "b" \], \[ "c", "d" \] \] } } The following is my config file: input { file … --- ## [Do you even SNMP bro?](https://discuss.elastic.co/t/do-you-even-snmp-bro/74690)
**Author:** [@kopacko](https://discuss.elastic.co/u/kopacko)\ **Replies:** 7\ **Last updated:** [February 13, 2017, 2:02am UTC](https://discuss.elastic.co/t/do-you-even-snmp-bro/74690 "2017-02-13T02:02:51Z")
Does Logstash store any information in SNMP anywhere? If not, why? --- ## [Not able to parse date in format "YYYY-mm-dd:HH:mm:ss" using grok pattern](https://discuss.elastic.co/t/not-able-to-parse-date-in-format-yyyy-mm-dd-hh-mm-ss-using-grok-pattern/74656)
**Author:** [@anisen](https://discuss.elastic.co/u/anisen)\ **Replies:** 1\ **Last updated:** [February 12, 2017, 6:31pm UTC](https://discuss.elastic.co/t/not-able-to-parse-date-in-format-yyyy-mm-dd-hh-mm-ss-using-grok-pattern/74656 "2017-02-12T18:31:44Z")
My Log file looks like this: 2015-06-12:00:08:55 195.160.1.191 POST /abc/xyz - 80 - 192.160.1.129 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) - 401 0 0 37 2015-06-12:00:08:54 195.160.1.191 POST… --- ## [Jdbc update and delete](https://discuss.elastic.co/t/jdbc-update-and-delete/74595)
**Author:** [@Greentea](https://discuss.elastic.co/u/Greentea)\ **Replies:** 1\ **Last updated:** [February 12, 2017, 6:24pm UTC](https://discuss.elastic.co/t/jdbc-update-and-delete/74595 "2017-02-12T18:24:58Z")
Hi I am currently using logstash-plugin-jdbc to pull data from mysql to elasticsearch. I want to know is it possible to update or delete a single document in elasticsearch by modifying mysql? For example, I got 1000 ro… --- ## [Separate shards](https://discuss.elastic.co/t/separate-shards/74781)
**Author:** [@Amin1](https://discuss.elastic.co/u/Amin1)\ **Replies:** 9\ **Last updated:** [February 12, 2017, 10:20am UTC](https://discuss.elastic.co/t/separate-shards/74781 "2017-02-12T10:20:50Z")
Hi all I have design this structure for my enterprise network with more than 280 server : Logstash(1)===\>Docker (Redis)=====\>Logstash(2)=======\>Elasticsearch(Clustre) my config is as follow : logstash1 : i… --- ## [Using logstash-output-gelf can I print my own message that will describe the issue cause](https://discuss.elastic.co/t/using-logstash-output-gelf-can-i-print-my-own-message-that-will-describe-the-issue-cause/74790)
**Author:** [@sivaram](https://discuss.elastic.co/u/sivaram)\ **Replies:** 1\ **Last updated:** [February 12, 2017, 6:53am UTC](https://discuss.elastic.co/t/using-logstash-output-gelf-can-i-print-my-own-message-that-will-describe-the-issue-cause/74790 "2017-02-12T06:53:01Z")
logFile $ sudo vi /var/log/nova/nova-compute.log 2017-01-25 02:05:35.748 20820 CRITICAL nova \[req-a5d425ae-b08e-450b-ac83-142733c35c99 - - - - -\] AccessRefused: (0, 0): (403) ACCESS\_REFUSED - Login was refused using a… --- ## [Suggestion for AWS Cloudtrail Logging!](https://discuss.elastic.co/t/suggestion-for-aws-cloudtrail-logging/74787)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 0\ **Last updated:** [February 12, 2017, 6:29am UTC](https://discuss.elastic.co/t/suggestion-for-aws-cloudtrail-logging/74787 "2017-02-12T06:29:45Z")
Hello, We have multiple cloudtrail accounts in our s3 bucket. eg:- my- bucket --\> AwsLogs --\> 12312312312 --\> team-x 12312312222 --\> team-y … --- ## [Can not start logstash with jdbc to postgresql9.5.5,except an error"role 'greg' does not exist"](https://discuss.elastic.co/t/can-not-start-logstash-with-jdbc-to-postgresql9-5-5-except-an-error-role-greg-does-not-exist/74623)
**Author:** [@Wynter\_Greg](https://discuss.elastic.co/u/Wynter_Greg)\ **Replies:** 1\ **Last updated:** [February 12, 2017, 1:33am UTC](https://discuss.elastic.co/t/can-not-start-logstash-with-jdbc-to-postgresql9-5-5-except-an-error-role-greg-does-not-exist/74623 "2017-02-12T01:33:38Z")
This is the first time I use logstash.I installed logstash5.2.0 by downloading logstash-5.2.0.tar.gz. The input is: input { jdbc { jdbc\_driver\_library =\> "jdbc-drivers/postgresql-9.4.1212.jar" jdbc\_driver\_c… --- ## [Logstash not finding installed gems](https://discuss.elastic.co/t/logstash-not-finding-installed-gems/74776)
**Author:** [@gmcgilli](https://discuss.elastic.co/u/gmcgilli)\ **Replies:** 1\ **Last updated:** [February 11, 2017, 9:17pm UTC](https://discuss.elastic.co/t/logstash-not-finding-installed-gems/74776 "2017-02-11T21:17:29Z")
Hi, I'm new to pretty the entire stack so learning. I'm having trouble getting logstash to recognize an installed gem that I want to use. I don't know if it matters but the gem was installed in the shared folder for so… --- ## [Logstash is failing to read a csv file](https://discuss.elastic.co/t/logstash-is-failing-to-read-a-csv-file/74133)
**Author:** [@danilo\_gimenez\_ramos](https://discuss.elastic.co/u/danilo_gimenez_ramos)\ **Replies:** 0\ **Last updated:** [February 6, 2017, 10:04pm UTC](https://discuss.elastic.co/t/logstash-is-failing-to-read-a-csv-file/74133 "2017-02-06T22:04:56Z")
Hello, I'm using logstash to read some csv files and then send to elastic, but I don't know why, the logstash can't read a file completely without a error. I'm receiving various errors like these: ":timestamp=\>"2017-… --- ## [Parsing and Calculating Navigation TIming](https://discuss.elastic.co/t/parsing-and-calculating-navigation-timing/74761)
**Author:** [@cuenca](https://discuss.elastic.co/u/cuenca)\ **Replies:** 0\ **Last updated:** [February 11, 2017, 2:23pm UTC](https://discuss.elastic.co/t/parsing-and-calculating-navigation-timing/74761 "2017-02-11T14:23:55Z")
Hi Everyone, I am having trouble parsing and calculating performance Navigation Timing data I have in a csv. I was able to parse the fields but not sure how to approach the calculations (below) properly. Some points … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1036) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1038)