# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=778 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 779 --- ## [Netflow default dashboard wrong fields](https://discuss.elastic.co/t/netflow-default-dashboard-wrong-fields/128239)
**Author:** [@Ivan44785372](https://discuss.elastic.co/u/Ivan44785372)\ **Replies:** 1\ **Last updated:** [April 17, 2018, 6:59am UTC](https://discuss.elastic.co/t/netflow-default-dashboard-wrong-fields/128239 "2018-04-17T06:59:31Z")
Hello. I have installed ELK using docker 6.2.3. I can see that standard netflow dashboard/visualisation use netflow.dst\_addr but I don't have such field. I have: netflow.ipv4\_dst\_addr netflow.bytes (dashboard) -\> netf… --- ## [Logstash avro parse error](https://discuss.elastic.co/t/logstash-avro-parse-error/127898)
**Author:** [@questbit](https://discuss.elastic.co/u/questbit)\ **Replies:** 4\ **Last updated:** [April 17, 2018, 5:48am UTC](https://discuss.elastic.co/t/logstash-avro-parse-error/127898 "2018-04-17T05:48:24Z")
Hi I am trying to process an avro file from logstash. version - jdk 1.8 - logstash 6.2.2 - https://github.com/logstash-plugins/logstash-codec-avro 3.2.3 logstash test.conf input { file { path =\> \["test.avro"\] c… --- ## [Not able to start logstash after applying xpack 6.3](https://discuss.elastic.co/t/not-able-to-start-logstash-after-applying-xpack-6-3/127158)
**Author:** [@rg912](https://discuss.elastic.co/u/rg912)\ **Replies:** 1\ **Last updated:** [April 17, 2018, 4:06am UTC](https://discuss.elastic.co/t/not-able-to-start-logstash-after-applying-xpack-6-3/127158 "2018-04-17T04:06:04Z")
Hello All am trying to install logstash but couldnt, after applying xpack 6.3 its failing with below error. have already verified permission on /usr/share/logstash/data/ and it looks good. any one faced below issue ? … --- ## [Docker build - issues starting X-pack environment - pwd](https://discuss.elastic.co/t/docker-build-issues-starting-x-pack-environment-pwd/128223)
**Author:** [@Robert\_Bardo](https://discuss.elastic.co/u/Robert_Bardo)\ **Replies:** 1\ **Last updated:** [April 17, 2018, 2:06am UTC](https://discuss.elastic.co/t/docker-build-issues-starting-x-pack-environment-pwd/128223 "2018-04-17T02:06:09Z")
Hello, I am trying to start my stack and all starts just fine except logstash. I have set my env variable in the compose file to set the ES bootstrap password. I have set the kibana config to use this bootstrap elasti… --- ## [JDBC Connect Error](https://discuss.elastic.co/t/jdbc-connect-error/128139)
**Author:** [@seungtae](https://discuss.elastic.co/u/seungtae)\ **Replies:** 3\ **Last updated:** [April 17, 2018, 1:05am UTC](https://discuss.elastic.co/t/jdbc-connect-error/128139 "2018-04-17T01:05:06Z")
How can I sove this Error..? using password?? Is there an option for using password in the Reference? Error: \[ERROR\]\[logstash.pipeline \] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\ **Author:** [@simmel](https://discuss.elastic.co/u/simmel)\ **Replies:** 1\ **Last updated:** [April 16, 2018, 10:03pm UTC](https://discuss.elastic.co/t/solved-many-metrics-to-graphite-via-include-metrics-but-from-metadata/125730 "2018-04-16T22:03:34Z") I don't want to send our metrics fields which are many (ES complains that we have more than 1000 fields) and unwanted to elasticsearch. Our previous approach was to use: graphite { host =\> "graphite.domain.tld" fie… --- ## [Multiple Syslog Inputs on one port](https://discuss.elastic.co/t/multiple-syslog-inputs-on-one-port/127956)
**Author:** [@Joachim\_Hermans](https://discuss.elastic.co/u/Joachim_Hermans)\ **Replies:** 4\ **Last updated:** [April 16, 2018, 9:40pm UTC](https://discuss.elastic.co/t/multiple-syslog-inputs-on-one-port/127956 "2018-04-16T21:40:36Z")
Hello, I'm a student doing studies around Centralized Logging. I've setup an ELK system, Kibana, Logstash, Elasticsearch, Filebeat, Nginx, Metricbeat and packetbeat. I've been messing around with many different logs. … --- ## [Logstash input elasticsearch - Scheduling](https://discuss.elastic.co/t/logstash-input-elasticsearch-scheduling/128242)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 1\ **Last updated:** [April 16, 2018, 8:05pm UTC](https://discuss.elastic.co/t/logstash-input-elasticsearch-scheduling/128242 "2018-04-16T20:05:10Z")
I'm trying to follow Elasticsearch input plugin | Logstash Reference (Scheduling). Let's say, following is part of my input (elasticsearch): input { elasticsearch { hosts =\> "localhost" query =\> '{ "query": {… --- ## [Logstash JDBC error java.lang.IllegalStateException](https://discuss.elastic.co/t/logstash-jdbc-error-java-lang-illegalstateexception/128096)
**Author:** [@kundavaram](https://discuss.elastic.co/u/kundavaram)\ **Replies:** 4\ **Last updated:** [April 16, 2018, 7:57pm UTC](https://discuss.elastic.co/t/logstash-jdbc-error-java-lang-illegalstateexception/128096 "2018-04-16T19:57:40Z")
Im running on Elastic 6.2.3 Logstash 6.2.3 Kibana 6.2.3 JDBC : sqljdbc\_6.4 My logstash thew below error while reading from SQL server table \[ERROR\] 2018-04-15 21:39:26.275 \[main\] Logstash - java.lang.IllegalStateExce… --- ## [Logstash output csv index name](https://discuss.elastic.co/t/logstash-output-csv-index-name/128231)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 4\ **Last updated:** [April 16, 2018, 7:13pm UTC](https://discuss.elastic.co/t/logstash-output-csv-index-name/128231 "2018-04-16T19:13:19Z")
I'm using Csv output plugin | Logstash Reference to write events to file on disk. snippet from my logstash configuration: output { csv { fields =\> \["@timestamp","XXX","YYY","ZZZ"\] path =\> "/tmp/XXX-%{+YYYY-MM… --- ## [Logstash configuration when ES is down](https://discuss.elastic.co/t/logstash-configuration-when-es-is-down/128229)
**Author:** [@mww1](https://discuss.elastic.co/u/mww1)\ **Replies:** 1\ **Last updated:** [April 16, 2018, 6:53pm UTC](https://discuss.elastic.co/t/logstash-configuration-when-es-is-down/128229 "2018-04-16T18:53:47Z")
Hi, what happens when I want to stream my output data from logstash to ElasticSearch when ES is down ? how does logstash take care? and how can I configure my logstash to ensure my pipeline executed successfully. please… --- ## [Logstash output streaming when ES is down](https://discuss.elastic.co/t/logstash-output-streaming-when-es-is-down/128230)
**Author:** [@mww1](https://discuss.elastic.co/u/mww1)\ **Replies:** 3\ **Last updated:** [April 16, 2018, 5:47pm UTC](https://discuss.elastic.co/t/logstash-output-streaming-when-es-is-down/128230 "2018-04-16T17:47:04Z")
Hi, what happens when I want to stream my output data from logstash to ElasticSearch when ES is down ? how do I configure my logstash? --- ## [Kafka input failover](https://discuss.elastic.co/t/kafka-input-failover/128027)
**Author:** [@jspeer](https://discuss.elastic.co/u/jspeer)\ **Replies:** 3\ **Last updated:** [April 16, 2018, 5:25pm UTC](https://discuss.elastic.co/t/kafka-input-failover/128027 "2018-04-16T17:25:14Z")
If I have a kafka topic replicated across 2 brokers, will logstash failover to the 2nd broker if the first becomes unavailable? It's not working, for me, but I'm not sure that logstash will failover in this instance. Wh… --- ## [Logstash doesn't dump all my events to elasticsearch](https://discuss.elastic.co/t/logstash-doesnt-dump-all-my-events-to-elasticsearch/128056)
**Author:** [@woshitom](https://discuss.elastic.co/u/woshitom)\ **Replies:** 13\ **Last updated:** [April 16, 2018, 4:14pm UTC](https://discuss.elastic.co/t/logstash-doesnt-dump-all-my-events-to-elasticsearch/128056 "2018-04-16T16:14:32Z")
I have a postgresql table "relou" having more than 2.8m records: \> select count(\*) from relou; \> count \> --------- \> 2853566 I'm indexing this table to elasticsearch using logstash: input { jdbc { jdb… --- ## [Data not getting indexed in elastic search](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207)
**Author:** [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\ **Replies:** 7\ **Last updated:** [April 16, 2018, 3:58pm UTC](https://discuss.elastic.co/t/data-not-getting-indexed-in-elastic-search/128207 "2018-04-16T15:58:35Z")
\[root@ukmr0xtmd01 conf.d\]# /usr/share/logstash/bin/logstash -f siae.conf WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify the path using --path.setting… --- ## [Grok filter not extracting fields](https://discuss.elastic.co/t/grok-filter-not-extracting-fields/127960)
**Author:** [@jarnold](https://discuss.elastic.co/u/jarnold)\ **Replies:** 27\ **Last updated:** [April 16, 2018, 3:20pm UTC](https://discuss.elastic.co/t/grok-filter-not-extracting-fields/127960 "2018-04-16T15:20:54Z")
Hi All, I've created the following grok filter; %{TIMESTAMP\_ISO8601:Date} %{SYSLOG5424SD:Client\_IP} %{WORD:Log\_Level} %{JAVACLASS:Java\_Class} %{GREEDYDATA:Freeform\_Text} Which appears to be working according to the gr… --- ## [Why would DeadLetterQueue fail like this?](https://discuss.elastic.co/t/why-would-deadletterqueue-fail-like-this/127664)
**Author:** [@smbullet](https://discuss.elastic.co/u/smbullet)\ **Replies:** 2\ **Last updated:** [April 16, 2018, 3:13pm UTC](https://discuss.elastic.co/t/why-would-deadletterqueue-fail-like-this/127664 "2018-04-16T15:13:09Z")
\[2018-04-11T10:37:31,479\]\[ERROR\]\[logstash.pipeline \] A plugin had an unrecoverable error. Will restart this plugin. Pipeline\_id:main Plugin: \"/home/logstash\_user/logsta… --- ## [Grok out a field withing another field](https://discuss.elastic.co/t/grok-out-a-field-withing-another-field/125673)
**Author:** [@cchooks2](https://discuss.elastic.co/u/cchooks2)\ **Replies:** 4\ **Last updated:** [April 16, 2018, 3:00pm UTC](https://discuss.elastic.co/t/grok-out-a-field-withing-another-field/125673 "2018-04-16T15:00:33Z")
Hello, I have the following message: \[Linux\]\[Baseline\]\[rwbdfrmmop03\]\[Process\]\[srmclient\]\[SiSExclude\] I want to extract the whole message as monitor\_name \[Linux\]\[Baseline\]\[rwbdfrmmop03\]\[Process\]\[srmclient\]\[SiSExclud… --- ## [KV generated field names contain spoaces and capitals](https://discuss.elastic.co/t/kv-generated-field-names-contain-spoaces-and-capitals/127963)
**Author:** [@leoponton](https://discuss.elastic.co/u/leoponton)\ **Replies:** 3\ **Last updated:** [April 16, 2018, 1:59pm UTC](https://discuss.elastic.co/t/kv-generated-field-names-contain-spoaces-and-capitals/127963 "2018-04-16T13:59:00Z")
I'm using the kv plugin to extract data from Windows ForwardedEvents. Typical data to process looks like: Client Version: 1.21.204.0 Client Policy ID: e964d551-3d3c-4a8f-8b9c-a99cce9b7ce5 IP Addresses: 192.168.0.10 Proc… --- ## [Pfsense 2.4 logs to logstash error](https://discuss.elastic.co/t/pfsense-2-4-logs-to-logstash-error/128194)
**Author:** [@francescouk](https://discuss.elastic.co/u/francescouk)\ **Replies:** 0\ **Last updated:** [April 16, 2018, 1:25pm UTC](https://discuss.elastic.co/t/pfsense-2-4-logs-to-logstash-error/128194 "2018-04-16T13:25:40Z")
Hi, I´m trying to send pfsense logs to logstash but keep´s giving me error \_geoip\_lookup\_failure and \_grokparsefailure. Because I´m newbie to this scene, I was using this conf from web and it was working until upgrade … --- ## [Collecting data from SOAP webservices for alarming](https://discuss.elastic.co/t/collecting-data-from-soap-webservices-for-alarming/127795)
**Author:** [@nazeemsoeltan](https://discuss.elastic.co/u/nazeemsoeltan)\ **Replies:** 1\ **Last updated:** [April 16, 2018, 12:21pm UTC](https://discuss.elastic.co/t/collecting-data-from-soap-webservices-for-alarming/127795 "2018-04-16T12:21:23Z")
I'm currently figuring out if we can use Logstash for collecting specific data from external services we're calling with our API. Eventually to be used for alarm triggers. I'll try to explain the case first. Customers … --- ## [Implement concurrency while inserting data to ES using logstash](https://discuss.elastic.co/t/implement-concurrency-while-inserting-data-to-es-using-logstash/128120)
**Author:** [@manjur](https://discuss.elastic.co/u/manjur)\ **Replies:** 3\ **Last updated:** [April 16, 2018, 11:42am UTC](https://discuss.elastic.co/t/implement-concurrency-while-inserting-data-to-es-using-logstash/128120 "2018-04-16T11:42:47Z")
Hi, I am using below config file for logstash. I have used schedule= 1 min as jdbc scheduler which keep on inserting data to ES in every 1 min. But when number of records are more i am not able to identify weather it is… --- ## [Read source filename from beats and redirect output to different Elastic Search index](https://discuss.elastic.co/t/read-source-filename-from-beats-and-redirect-output-to-different-elastic-search-index/127987)
**Author:** [@pavanssetty](https://discuss.elastic.co/u/pavanssetty)\ **Replies:** 3\ **Last updated:** [April 16, 2018, 11:41am UTC](https://discuss.elastic.co/t/read-source-filename-from-beats-and-redirect-output-to-different-elastic-search-index/127987 "2018-04-16T11:41:02Z")
I want to read the source of the csv data and redirect the output to a different ElasticSearch index. I tried as below (if condition with source in output) by following various discussions but it did not work. It is alw… --- ## [Cofig test is not working](https://discuss.elastic.co/t/cofig-test-is-not-working/128149)
**Author:** [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\ **Replies:** 1\ **Last updated:** [April 16, 2018, 11:36am UTC](https://discuss.elastic.co/t/cofig-test-is-not-working/128149 "2018-04-16T11:36:32Z")
\[root@ukmr0xtmd01 opt\]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/siae.conf -configtest WARNING: Could not find logstash.yml which is typically located in $LS\_HOME/config or /etc/logstash. You can specify… --- ## [Fallback http output](https://discuss.elastic.co/t/fallback-http-output/128154)
**Author:** [@luca.cavazzana](https://discuss.elastic.co/u/luca.cavazzana)\ **Replies:** 1\ **Last updated:** [April 16, 2018, 11:35am UTC](https://discuss.elastic.co/t/fallback-http-output/128154 "2018-04-16T11:35:41Z")
Hi, I have a simple configuration with a http output plugin firing to a custom application. What I'd like to obtain is to log on another Elastic Search any http:50x status codes that logstash may get when forwarding th… --- ## [If/Else not working as expected in filter](https://discuss.elastic.co/t/if-else-not-working-as-expected-in-filter/128173)
**Author:** [@va1entin](https://discuss.elastic.co/u/va1entin)\ **Replies:** 2\ **Last updated:** [April 16, 2018, 11:35am UTC](https://discuss.elastic.co/t/if-else-not-working-as-expected-in-filter/128173 "2018-04-16T11:35:19Z")
Hey folks, I am a bit confused regarding use of if else constructs in logstash. I have the following config: filter { if "apache\_access" in \[tags\] { grok { match =\> { "message" =\> "%{HTTPD\_COMMONLOG}" } … --- ## [Logstash: Filter out the service to file](https://discuss.elastic.co/t/logstash-filter-out-the-service-to-file/128136)
**Author:** [@jawad846](https://discuss.elastic.co/u/jawad846)\ **Replies:** 3\ **Last updated:** [April 16, 2018, 11:26am UTC](https://discuss.elastic.co/t/logstash-filter-out-the-service-to-file/128136 "2018-04-16T11:26:02Z")
Hi folks, I just want to filter out only apache and MySQL logs from the logstash output, I had configured the pipeline.conf file its not working. please suggest me to solve this # INPUT HERE input { beats { p… --- ## [Parquet output codec](https://discuss.elastic.co/t/parquet-output-codec/128177)
**Author:** [@hartfordfive](https://discuss.elastic.co/u/hartfordfive)\ **Replies:** 0\ **Last updated:** [April 16, 2018, 11:24am UTC](https://discuss.elastic.co/t/parquet-output-codec/128177 "2018-04-16T11:24:51Z")
I am currently index data into Elasticsearch although I'm looking for a simple method to also store logs on an HDFS cluster in parquet format for longer term storage and to allow for search/map reduce jobs over many year… --- ## [Check the pipeline stats logstash 6.2.X](https://discuss.elastic.co/t/check-the-pipeline-stats-logstash-6-2-x/128174)
**Author:** [@EloaCardoso](https://discuss.elastic.co/u/EloaCardoso)\ **Replies:** 0\ **Last updated:** [April 16, 2018, 11:18am UTC](https://discuss.elastic.co/t/check-the-pipeline-stats-logstash-6-2-x/128174 "2018-04-16T11:18:47Z")
Hi, I was reading the content at the following section: and I got an error when I typed localhost:9600/\_node/stats/pipelines?pretty in the browser address bar. This works for me: localhost:9600/\_node/stats/pipe… --- ## [Aggegation of log lines](https://discuss.elastic.co/t/aggegation-of-log-lines/128166)
**Author:** [@Ganesha\_Venkanna](https://discuss.elastic.co/u/Ganesha_Venkanna)\ **Replies:** 0\ **Last updated:** [April 16, 2018, 10:15am UTC](https://discuss.elastic.co/t/aggegation-of-log-lines/128166 "2018-04-16T10:15:16Z")
I would like to aggregate the below 3 lines (everything same except timestamp) 04/13/2018 19:31:20 \[ACPServerThread\_226\] VERBOSE SERVER - ::ACPSCConnectionResponder::SiteController Connection Accepted for SiteControll… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=777) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=779)