# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=833 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 834 --- ## [Grok filter ignore first line always](https://discuss.elastic.co/t/grok-filter-ignore-first-line-always/117359)
**Author:** [@parthkdas](https://discuss.elastic.co/u/parthkdas)\ **Replies:** 1\ **Last updated:** [January 29, 2018, 8:35am UTC](https://discuss.elastic.co/t/grok-filter-ignore-first-line-always/117359 "2018-01-29T08:35:04Z")
I am trying to read large csv and txt file from different sources. The sources may give us csv or txt file for same data. I want to ignore the first the line of each files when they are first time parsed , as they repr… --- ## [Mutate not converting text to integer](https://discuss.elastic.co/t/mutate-not-converting-text-to-integer/117256)
**Author:** [@mikebw](https://discuss.elastic.co/u/mikebw)\ **Replies:** 6\ **Last updated:** [January 29, 2018, 7:43am UTC](https://discuss.elastic.co/t/mutate-not-converting-text-to-integer/117256 "2018-01-29T07:43:25Z")
I'm trying to grok a field with regexp and convert it to an integer, but it's not converting to an integer. if \[type\] == "server\_log" { grok { match =\> \[ "message", ".\*mediaDurationMs=(?\\\… --- ## [JDBC plugin reloads everything, everytime](https://discuss.elastic.co/t/jdbc-plugin-reloads-everything-everytime/117254)
**Author:** [@TheEvilDonut](https://discuss.elastic.co/u/TheEvilDonut)\ **Replies:** 1\ **Last updated:** [January 29, 2018, 7:21am UTC](https://discuss.elastic.co/t/jdbc-plugin-reloads-everything-everytime/117254 "2018-01-29T07:21:12Z")
Hello All. I am trying to get data from a SQL Server BD with a very simple query. For some reason everytime I restart Logstash the whole data gets loaded into Elasticsearch. The marker file seems OK, the very last ID i… --- ## [Queries and Issues faced with Logstash](https://discuss.elastic.co/t/queries-and-issues-faced-with-logstash/117338)
**Author:** [@SawyerFord](https://discuss.elastic.co/u/SawyerFord)\ **Replies:** 1\ **Last updated:** [January 29, 2018, 7:14am UTC](https://discuss.elastic.co/t/queries-and-issues-faced-with-logstash/117338 "2018-01-29T07:14:51Z")
I have 5 queries and issues faced with Logstash. Whether I really need to configure logstash.conf with input and output plugins. I did not configured anything inside logstash.conf , any how I am getting logs in JSON. … --- ## [Connecting to Pervasive](https://discuss.elastic.co/t/connecting-to-pervasive/117368)
**Author:** [@Amitendra\_Kumar](https://discuss.elastic.co/u/Amitendra_Kumar)\ **Replies:** 1\ **Last updated:** [January 29, 2018, 7:06am UTC](https://discuss.elastic.co/t/connecting-to-pervasive/117368 "2018-01-29T07:06:56Z")
I am trying to connect to Pervasive through Logstash but not able to. Can somebody please help? --- ## [Logstash can not use path option in ruby](https://discuss.elastic.co/t/logstash-can-not-use-path-option-in-ruby/117148)
**Author:** [@Chunbo\_Liao](https://discuss.elastic.co/u/Chunbo_Liao)\ **Replies:** 5\ **Last updated:** [January 29, 2018, 7:03am UTC](https://discuss.elastic.co/t/logstash-can-not-use-path-option-in-ruby/117148 "2018-01-29T07:03:57Z")
I try to use ruby script file in my lostash conf file(filter). but it failed, the error logs: \[2018-01-26T21:12:49,988\]\[ERROR\]\[logstash.filters.ruby \] Unknown setting 'path' for ruby \[2018-01-26T21:12:49,998\]\[ERRO… --- ## [Error in Logstash](https://discuss.elastic.co/t/error-in-logstash/116646)
**Author:** [@Krunal\_kalaria](https://discuss.elastic.co/u/Krunal_kalaria)\ **Replies:** 3\ **Last updated:** [January 29, 2018, 6:56am UTC](https://discuss.elastic.co/t/error-in-logstash/116646 "2018-01-29T06:56:59Z")
Hello Everyone, I am trying to integrate ELK Stack 5.6.4 in RHEL 7 with ArcSight smartconnector with this link: https://www.elastic.co/guide/en/logstash/5.6/arcsight-module.html#arcsight-instructions-smartconnector Af… --- ## [How to convert String to long in logstash](https://discuss.elastic.co/t/how-to-convert-string-to-long-in-logstash/116029)
**Author:** [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)\ **Replies:** 2\ **Last updated:** [January 29, 2018, 5:37am UTC](https://discuss.elastic.co/t/how-to-convert-string-to-long-in-logstash/116029 "2018-01-29T05:37:53Z")
Example from my log i am getting response time as String in ES i want to convert it to long mutate{ convert =\> {"respnseTime" =\> "long"} } it throws me an error --- ## [Splitt Message after Hostname](https://discuss.elastic.co/t/splitt-message-after-hostname/116977)
**Author:** [@A.Klos](https://discuss.elastic.co/u/A.Klos)\ **Replies:** 4\ **Last updated:** [January 29, 2018, 5:16am UTC](https://discuss.elastic.co/t/splitt-message-after-hostname/116977 "2018-01-29T05:16:29Z")
Hi, I have following line: message: \<30\>Jan 24 13:59:45 10.243.x.x named\[10148\]: queries: client 10.243.x.x#47205 (88.255.x.x.in-addr.arpa): query: 88.255.x.x.in-addr.arpa IN PTR + (10.243.x.x) I allready split it and… --- ## [HTTP\_POLLER Plugin for Logstash](https://discuss.elastic.co/t/http-poller-plugin-for-logstash/110498)
**Author:** [@Rym\_Guerbi\_Michaut](https://discuss.elastic.co/u/Rym_Guerbi_Michaut)\ **Replies:** 29\ **Last updated:** [January 28, 2018, 9:38pm UTC](https://discuss.elastic.co/t/http-poller-plugin-for-logstash/110498 "2018-01-28T21:38:36Z")
Hi there, I'm new at using ELK and I have to import data with HTTP\_POLLER (json statement). I don't really understand some stuff about input plugins for Logstash. My question is: Do I have to download the plugin to … --- ## [Will a plugin packaged on windows work on linux?](https://discuss.elastic.co/t/will-a-plugin-packaged-on-windows-work-on-linux/117236)
**Author:** [@makibroshett](https://discuss.elastic.co/u/makibroshett)\ **Replies:** 1\ **Last updated:** [January 28, 2018, 9:34pm UTC](https://discuss.elastic.co/t/will-a-plugin-packaged-on-windows-work-on-linux/117236 "2018-01-28T21:34:53Z")
My target linux machine runs logstash and has no internet access. Only my windows machine has that, so can I build/package a plugin on windows and have that running on Linux ? Thanks for the help! Olivier --- ## [Handling duplicate data in Logstash + Elastic Search](https://discuss.elastic.co/t/handling-duplicate-data-in-logstash-elastic-search/116229)
**Author:** [@Patrik\_Iselind](https://discuss.elastic.co/u/Patrik_Iselind)\ **Replies:** 2\ **Last updated:** [January 28, 2018, 9:28pm UTC](https://discuss.elastic.co/t/handling-duplicate-data-in-logstash-elastic-search/116229 "2018-01-28T21:28:10Z")
Hi, First some background: I have an ELK stack running in an docker compose environment, so far for learning purposes. Logstash gets its input from a AWS S3 bucket and sends its output to the Elastic Search server. The… --- ## [Custom add\_field storing as an array with value duplicated in Elasticsearch](https://discuss.elastic.co/t/custom-add-field-storing-as-an-array-with-value-duplicated-in-elasticsearch/117277)
**Author:** [@nodesocket](https://discuss.elastic.co/u/nodesocket)\ **Replies:** 2\ **Last updated:** [January 28, 2018, 8:40pm UTC](https://discuss.elastic.co/t/custom-add-field-storing-as-an-array-with-value-duplicated-in-elasticsearch/117277 "2018-01-28T20:40:38Z")
Hello. I am trying to add a custom field msgid and while the groc and field is working, the result is an array with the value duplicated twice. Here is an example raw message: Jan 26 22:58:01 ip-172-31-23-201 mailqueue… --- ## [Dissect plugin reads too much into last field](https://discuss.elastic.co/t/dissect-plugin-reads-too-much-into-last-field/117354)
**Author:** [@marekf](https://discuss.elastic.co/u/marekf)\ **Replies:** 0\ **Last updated:** [January 28, 2018, 10:33am UTC](https://discuss.elastic.co/t/dissect-plugin-reads-too-much-into-last-field/117354 "2018-01-28T10:33:41Z")
Hey folks, I am using dissect plugin to slice Webserver log line into JSON document. Logstash version is 6.0.1. Here is NGINX log format: log\_format main '$remote\_addr ($http\_x\_forwarded\_for) - $remote\_user \[$time\_i… --- ## [Grok patterns with ( { \[ . , \\ /](https://discuss.elastic.co/t/grok-patterns-with/117298)
**Author:** [@Farhad\_Yousefi](https://discuss.elastic.co/u/Farhad_Yousefi)\ **Replies:** 1\ **Last updated:** [January 27, 2018, 1:38pm UTC](https://discuss.elastic.co/t/grok-patterns-with/117298 "2018-01-27T13:38:07Z")
Hi . I want to catch my exception with ELK but my exception is full of ( { \[ . , \\ / , " ' character. How can I index them in grok . thank you my log file: Exception in \*\*\* CoreLevel\*\*\* occured. Date&Time: 2018-01-21… --- ## [Logtash dont capture Logger.error logs using logback](https://discuss.elastic.co/t/logtash-dont-capture-logger-error-logs-using-logback/116976)
**Author:** [@SawyerFord](https://discuss.elastic.co/u/SawyerFord)\ **Replies:** 2\ **Last updated:** [January 27, 2018, 9:32pm UTC](https://discuss.elastic.co/t/logtash-dont-capture-logger-error-logs-using-logback/116976 "2018-01-27T21:32:04Z")
Below is the configuration of logback.xml I tried many ways but it does not encode or print Logger.error statements \ \ \ **Author:** [@Xoom](https://discuss.elastic.co/u/Xoom)\ **Replies:** 0\ **Last updated:** [January 27, 2018, 10:34pm UTC](https://discuss.elastic.co/t/dns-filter-in-logstash-netflow-module/117337 "2018-01-27T22:34:16Z") We are ingesting Netflow v9 datagram with Logstash 5.6.4 onto our 5.6.4 elastic cluster. This is working great and kibana dashboards and visualizations are very handy. One thing we are looking to do is to transform IP ad… --- ## [Logtash dont capture Logger.error logs](https://discuss.elastic.co/t/logtash-dont-capture-logger-error-logs/116952)
**Author:** [@SawyerFord](https://discuss.elastic.co/u/SawyerFord)\ **Replies:** 1\ **Last updated:** [January 27, 2018, 9:15pm UTC](https://discuss.elastic.co/t/logtash-dont-capture-logger-error-logs/116952 "2018-01-27T21:15:03Z")
Below is the configuration of logback.xml I tried many ways but it does not encode or print Logger.error statements \ \ \ **Author:** [@some7](https://discuss.elastic.co/u/some7)\ **Replies:** 1\ **Last updated:** [January 27, 2018, 4:34pm UTC](https://discuss.elastic.co/t/how-to-test-udp-input-for-logstash/117309 "2018-01-27T16:34:41Z") I want to send messages into logstash listening on localhost UDP port to test that it works. I'm on linux and would like to use ncat, how can I do it? Maybe it's wrong but echo test | ncat -v -u localhost 5999 has an err… --- ## [\_grokparsefailure on \[fields\]\[document\_type\]](https://discuss.elastic.co/t/grokparsefailure-on-fields-document-type/117312)
**Author:** [@mr.szop](https://discuss.elastic.co/u/mr.szop)\ **Replies:** 2\ **Last updated:** [January 27, 2018, 4:21pm UTC](https://discuss.elastic.co/t/grokparsefailure-on-fields-document-type/117312 "2018-01-27T16:21:33Z")
Hey guys, I'm having a hard time to understand my mistake getting a \_grokparsefailure on a Logstash filter. Im unsing an Apache access log and a NGINX access log which are prospected by Filebeat and then send to Logstas… --- ## [Configuring logs and monitoring different logstash running on same AWS instance](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192)
**Author:** [@kseth](https://discuss.elastic.co/u/kseth)\ **Replies:** 4\ **Last updated:** [January 27, 2018, 1:39am UTC](https://discuss.elastic.co/t/configuring-logs-and-monitoring-different-logstash-running-on-same-aws-instance/115192 "2018-01-27T01:39:03Z")
Hi, We are using systemctl for running 3 logstash processes, one for each environment : dev, qa and prod on AWS centos box. We are unable to figure out how to configure logging for these 3 different logstash processes a… --- ## [Logstash s3 input plugin failed to work with subdirectories](https://discuss.elastic.co/t/logstash-s3-input-plugin-failed-to-work-with-subdirectories/117220)
**Author:** [@Nikolay\_Petrov](https://discuss.elastic.co/u/Nikolay_Petrov)\ **Replies:** 0\ **Last updated:** [January 26, 2018, 3:57pm UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-failed-to-work-with-subdirectories/117220 "2018-01-26T15:57:57Z")
Hi, I'm observing an couple of difficulties, until trying to grab the data from s3, if they stored on that way: trailbucketname/usw1/AWSLogs/361190373704/CloudTrail/us-west-1////361190373704\_CloudTrail\_us-west-1\_\*.jso… --- ## [Empty configuration for pipeline\_id: test](https://discuss.elastic.co/t/empty-configuration-for-pipeline-id-test/116963)
**Author:** [@akiyamayami](https://discuss.elastic.co/u/akiyamayami)\ **Replies:** 1\ **Last updated:** [January 26, 2018, 9:31pm UTC](https://discuss.elastic.co/t/empty-configuration-for-pipeline-id-test/116963 "2018-01-26T21:31:03Z")
Hi, i have problem after i install x-pack for elk, logstash can't read config for pipeline when i enable security in kibana and elastic \[52\] i already try create pipeline in kinana but not working \[58\] But if i disab… --- ## [Centralized pipeline management - pipeline terminating](https://discuss.elastic.co/t/centralized-pipeline-management-pipeline-terminating/116357)
**Author:** [@rpendela](https://discuss.elastic.co/u/rpendela)\ **Replies:** 1\ **Last updated:** [January 26, 2018, 9:28pm UTC](https://discuss.elastic.co/t/centralized-pipeline-management-pipeline-terminating/116357 "2018-01-26T21:28:22Z")
Hello community, I am trying to use the centralize pipeline management feature , I followed the document https://www.elastic.co/guide/en/logstash/6.0/logstash-centralized-pipeline-management.html I have a x-pack licenc… --- ## [Logstash failed to parse timestamp field, Invalid format, malformed (grok, date filter, add\_field)](https://discuss.elastic.co/t/logstash-failed-to-parse-timestamp-field-invalid-format-malformed-grok-date-filter-add-field/117135)
**Author:** [@thirusama](https://discuss.elastic.co/u/thirusama)\ **Replies:** 3\ **Last updated:** [January 26, 2018, 6:10pm UTC](https://discuss.elastic.co/t/logstash-failed-to-parse-timestamp-field-invalid-format-malformed-grok-date-filter-add-field/117135 "2018-01-26T18:10:17Z")
Hello, I am getting the error "failed to parse \[logdate\]","reason"=\>"failed to parse \[logdate\]", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Invalid format: "2018-Jan-16T11:07:42" is malformed at "-Ja… --- ## [Fields in messages](https://discuss.elastic.co/t/fields-in-messages/116785)
**Author:** [@kayaHL](https://discuss.elastic.co/u/kayaHL)\ **Replies:** 3\ **Last updated:** [January 26, 2018, 5:41pm UTC](https://discuss.elastic.co/t/fields-in-messages/116785 "2018-01-26T17:41:43Z")
Hi guys, I am new to Elasticsearch... I am using filebeat to ship the multiple line unstructured messages to logstash and use grok filter to structure the data. However, I noticed in the application logs, between messag… --- ## [Dead letter queue duplicating the stream](https://discuss.elastic.co/t/dead-letter-queue-duplicating-the-stream/114821)
**Author:** [@NGC\_2359](https://discuss.elastic.co/u/NGC_2359)\ **Replies:** 1\ **Last updated:** [January 26, 2018, 5:26pm UTC](https://discuss.elastic.co/t/dead-letter-queue-duplicating-the-stream/114821 "2018-01-26T17:26:30Z")
Hi Elastic folks, I’m using the ElasticStack 6.1.1 and have been trying to configure the dead letter queue feature with the following: pipelines.yml: - pipeline.id: main path.config: ... - pipeline.id: dead path.c… --- ## [Where to put file for ruby filter?](https://discuss.elastic.co/t/where-to-put-file-for-ruby-filter/117228)
**Author:** [@clannadqs](https://discuss.elastic.co/u/clannadqs)\ **Replies:** 1\ **Last updated:** [January 26, 2018, 5:09pm UTC](https://discuss.elastic.co/t/where-to-put-file-for-ruby-filter/117228 "2018-01-26T17:09:17Z")
Hey, I have a pretty simple ruby filter that is using a CSV file as a reference for input while adding fields based on what was provided by the input. Keep in mind the csv file is not coming from input. My problem comes… --- ## [Query error with JDBC with cassandra input](https://discuss.elastic.co/t/query-error-with-jdbc-with-cassandra-input/117222)
**Author:** [@nomedx](https://discuss.elastic.co/u/nomedx)\ **Replies:** 0\ **Last updated:** [January 26, 2018, 3:59pm UTC](https://discuss.elastic.co/t/query-error-with-jdbc-with-cassandra-input/117222 "2018-01-26T15:59:41Z")
Hi, I have a problem with logstash jdbc plugin (I use dbschema.CassandraJdbcDriver) and cassandra. When I start logstash I obtain always this error in logstash logs: \[ERROR\]\[logstash.inputs.jdbc \] Java::ComDatastax… --- ## [Which filters ship with logstash docker image?](https://discuss.elastic.co/t/which-filters-ship-with-logstash-docker-image/117218)
**Author:** [@makibroshett](https://discuss.elastic.co/u/makibroshett)\ **Replies:** 1\ **Last updated:** [January 26, 2018, 4:20pm UTC](https://discuss.elastic.co/t/which-filters-ship-with-logstash-docker-image/117218 "2018-01-26T16:20:53Z")
Hi, Having redployed my existing logstash installation using the official Docker image (6.1). One of my pipeline failed to load because it uses UUID filter, which was not there. I don't recall having that problem with … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=832) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=834)