# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=910 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 911 --- ## [503 error on Logstash plugin install in Docker build](https://discuss.elastic.co/t/503-error-on-logstash-plugin-install-in-docker-build/100240)
**Author:** [@YeeP](https://discuss.elastic.co/u/YeeP)\ **Replies:** 1\ **Last updated:** [September 12, 2017, 6:15pm UTC](https://discuss.elastic.co/t/503-error-on-logstash-plugin-install-in-docker-build/100240 "2017-09-12T18:15:06Z")
I am trying to get the logstash-output-elasticsearch plugin working with Logstash. I am running into an issue with the install: Current Dockerfile: FROM logstash/logstash:5.6.0 ENV HTTPS\_PROXY="http://corporate-proxy.c… --- ## [Logstash is throwing errors](https://discuss.elastic.co/t/logstash-is-throwing-errors/100155)
**Author:** [@Narasimha\_Gajje](https://discuss.elastic.co/u/Narasimha_Gajje)\ **Replies:** 6\ **Last updated:** [September 12, 2017, 6:02pm UTC](https://discuss.elastic.co/t/logstash-is-throwing-errors/100155 "2017-09-12T18:02:51Z")
Hi Team, I have downloaded LogStash and configured to push logs to ElasticSearch. When I start the LogStash with command logstash -f ../conf/logstash.yml but getting below error. Even I tried to start Logstash with co… --- ## [Logstash KV plugin working](https://discuss.elastic.co/t/logstash-kv-plugin-working/99338)
**Author:** [@shashwat](https://discuss.elastic.co/u/shashwat)\ **Replies:** 4\ **Last updated:** [September 12, 2017, 4:55pm UTC](https://discuss.elastic.co/t/logstash-kv-plugin-working/99338 "2017-09-12T16:55:50Z")
Hi, I am trying to use logstash’s KV plugin. I have following log format: time taken for transfer for all files in seconds=23 transfer start time= 201708030959 transfer end time = 201708030959 My .conf file has follow… --- ## [Fingerprint](https://discuss.elastic.co/t/fingerprint/100236)
**Author:** [@paolo1](https://discuss.elastic.co/u/paolo1)\ **Replies:** 0\ **Last updated:** [September 12, 2017, 3:23pm UTC](https://discuss.elastic.co/t/fingerprint/100236 "2017-09-12T15:23:10Z")
Is it possible to instruct Fingerprint to log duplicates (or ignore them entirely), rather than having logstash update a document in ES? The only way I can think of doing this is to restrict the user and role we have as… --- ## [CSV plugin problem](https://discuss.elastic.co/t/csv-plugin-problem/100070)
**Author:** [@avik\_ewu](https://discuss.elastic.co/u/avik_ewu)\ **Replies:** 3\ **Last updated:** [September 12, 2017, 2:33pm UTC](https://discuss.elastic.co/t/csv-plugin-problem/100070 "2017-09-12T14:33:48Z")
\[image\] My sample data is like this and my config file is below. data not converted properly. input { file { path =\> "/home/node/Copy/SDP/SDP-testdata/sdptestdata.csv" start\_position=\> beginning #sincedb\_path =\> "/… --- ## [Log4j input plugin](https://discuss.elastic.co/t/log4j-input-plugin/100104)
**Author:** [@xiaopeng\_zhang](https://discuss.elastic.co/u/xiaopeng_zhang)\ **Replies:** 5\ **Last updated:** [September 12, 2017, 12:55pm UTC](https://discuss.elastic.co/t/log4j-input-plugin/100104 "2017-09-12T12:55:10Z")
I know it's deprecated.i just wang to use it Conveniently. when i use it.it run normally.it doesn/t accept any log. input { log4j { mode =\> "server" port =\> 80 } } but when i use tcp .it get log and there are man… --- ## [Gsub not working](https://discuss.elastic.co/t/gsub-not-working/100176)
**Author:** [@Charlotte\_EGM](https://discuss.elastic.co/u/Charlotte_EGM)\ **Replies:** 10\ **Last updated:** [September 12, 2017, 12:53pm UTC](https://discuss.elastic.co/t/gsub-not-working/100176 "2017-09-12T12:53:43Z")
Hello, I'm trying to remove \[\] from my data as it is not supported in Kibana. Generally, I'm doing it very well with gsub (with http input). Now I'm using rabbitmq input and gsub is not working. I can't figure out why… --- ## [Creating Dummy Entry via ConfigFile](https://discuss.elastic.co/t/creating-dummy-entry-via-configfile/100198)
**Author:** [@Sharivari](https://discuss.elastic.co/u/Sharivari)\ **Replies:** 1\ **Last updated:** [September 12, 2017, 11:46am UTC](https://discuss.elastic.co/t/creating-dummy-entry-via-configfile/100198 "2017-09-12T11:46:13Z")
Hi everyone, I just started the basic dummy entry creation with stdin via: $ sudo docker exec -it \ /bin/bash and # /opt/logstash/bin/logstash --path.data /tmp/logstash/data \\ -e 'input { stdin { }… --- ## [Deserializing an avro file in file input plugin](https://discuss.elastic.co/t/deserializing-an-avro-file-in-file-input-plugin/100195)
**Author:** [@Pratheek](https://discuss.elastic.co/u/Pratheek)\ **Replies:** 2\ **Last updated:** [September 12, 2017, 11:26am UTC](https://discuss.elastic.co/t/deserializing-an-avro-file-in-file-input-plugin/100195 "2017-09-12T11:26:56Z")
Hi, Is it possible to deserialize an avro file using file input plugin ? In the case of kafka input plugin there are settings such as key deserializer class and value deserializer class,is there anything as such for fi… --- ## [Logstash: Can't generate new event in custom filter using ruby](https://discuss.elastic.co/t/logstash-cant-generate-new-event-in-custom-filter-using-ruby/99714)
**Author:** [@Jay\_Dabhi](https://discuss.elastic.co/u/Jay_Dabhi)\ **Replies:** 2\ **Last updated:** [September 12, 2017, 11:02am UTC](https://discuss.elastic.co/t/logstash-cant-generate-new-event-in-custom-filter-using-ruby/99714 "2017-09-12T11:02:43Z")
0 down vote favorite I am designing the my custom log-stash filter using the ruby.This filter generates the event after certain period.I am generating the new event from the thread.Following is the my code of the thre… --- ## [Authenticating clients - beats](https://discuss.elastic.co/t/authenticating-clients-beats/100189)
**Author:** [@Ulrar](https://discuss.elastic.co/u/Ulrar)\ **Replies:** 4\ **Last updated:** [September 12, 2017, 10:43am UTC](https://discuss.elastic.co/t/authenticating-clients-beats/100189 "2017-09-12T10:43:13Z")
Hi, I'm trying to setup ELK at work to centralize our client's logs. It seems to work fine, but I'm a bit concerned that any of our client could basically fiddle with the filebeats (or anything else ..) config, use the … --- ## [Logstah performance issues](https://discuss.elastic.co/t/logstah-performance-issues/99128)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 28\ **Last updated:** [September 12, 2017, 9:05am UTC](https://discuss.elastic.co/t/logstah-performance-issues/99128 "2017-09-12T09:05:48Z")
Hello, I am so tired about Logstash problem. I have total 2 850 000 messages every 15 minutes (3 document type differents), and On one of type, I have 2 hours of delays. 2 850 000 message every 15 minutes = 190 000 me… --- ## [Cannot properly assign timestamp tracking column value (JDBC plugin)](https://discuss.elastic.co/t/cannot-properly-assign-timestamp-tracking-column-value-jdbc-plugin/100177)
**Author:** [@c\_sharper](https://discuss.elastic.co/u/c_sharper)\ **Replies:** 0\ **Last updated:** [September 12, 2017, 8:55am UTC](https://discuss.elastic.co/t/cannot-properly-assign-timestamp-tracking-column-value-jdbc-plugin/100177 "2017-09-12T08:55:13Z")
I have problem with assigning properly last 'sql\_last\_value'. I wanted to assign it with my last timestamp which is stored in database. Here is my config: input { jdbc { jdbc\_driver\_library =\> "/usr/share/logstash/l… --- ## [Winglobeat via Logstash for a new indices](https://discuss.elastic.co/t/winglobeat-via-logstash-for-a-new-indices/99764)
**Author:** [@hispeed](https://discuss.elastic.co/u/hispeed)\ **Replies:** 2\ **Last updated:** [September 12, 2017, 6:58am UTC](https://discuss.elastic.co/t/winglobeat-via-logstash-for-a-new-indices/99764 "2017-09-12T06:58:54Z")
Hi, I have installed the winlogbeat 5.4 on a Windows Server 2016 machine. The servie is started. I have configured the ELK server this server is also running fine. output.logstash: The Logstash hosts hosts: \["192.168… --- ## [How to provide custom 'JAVA\_HOME' path for Logstash?](https://discuss.elastic.co/t/how-to-provide-custom-java-home-path-for-logstash/99814)
**Author:** [@venkaty](https://discuss.elastic.co/u/venkaty)\ **Replies:** 1\ **Last updated:** [September 12, 2017, 6:39am UTC](https://discuss.elastic.co/t/how-to-provide-custom-java-home-path-for-logstash/99814 "2017-09-12T06:39:35Z")
Hi, Currently I am using Logstash 2.1.1 version. I want to use custom 'JAVA\_HOME' path instead of global environment. is it possible ? --- ## [File o/p plugin not working](https://discuss.elastic.co/t/file-o-p-plugin-not-working/100007)
**Author:** [@ashish235](https://discuss.elastic.co/u/ashish235)\ **Replies:** 15\ **Last updated:** [September 12, 2017, 6:21am UTC](https://discuss.elastic.co/t/file-o-p-plugin-not-working/100007 "2017-09-12T06:21:25Z")
Hi Team, Does anyone using file o/p plugin to just ship logs from remote location to a central location? ES needs a lot of storage so that's not a viable option for me, so I need to ship logs and store logs as files on… --- ## [Logstash gzip\_lines , tail latest written file to text file](https://discuss.elastic.co/t/logstash-gzip-lines-tail-latest-written-file-to-text-file/99926)
**Author:** [@shashwat](https://discuss.elastic.co/u/shashwat)\ **Replies:** 2\ **Last updated:** [September 12, 2017, 6:15am UTC](https://discuss.elastic.co/t/logstash-gzip-lines-tail-latest-written-file-to-text-file/99926 "2017-09-12T06:15:33Z")
Hi, I am using logstash 5.5. I am using gzip\_lines plugin to parse the archived files. As , this plugin expects a text file having archived file with absolute path line by line. If this text file is static , i.e. if I w… --- ## [Need to convert string into date\_time format](https://discuss.elastic.co/t/need-to-convert-string-into-date-time-format/100153)
**Author:** [@vipinkumar](https://discuss.elastic.co/u/vipinkumar)\ **Replies:** 4\ **Last updated:** [September 12, 2017, 6:03am UTC](https://discuss.elastic.co/t/need-to-convert-string-into-date-time-format/100153 "2017-09-12T06:03:18Z")
Hi All I am working on csv files and in that file there is a column as DATE/TIME in which the date and time is coming in following format. 2017-09-12-0915 YYYY-mm-dd-HHmm I tried to convert this to DATE/TIME to date\_t… --- ## [Using logstash for a propietary application stack](https://discuss.elastic.co/t/using-logstash-for-a-propietary-application-stack/100128)
**Author:** [@kaiser\_m](https://discuss.elastic.co/u/kaiser_m)\ **Replies:** 1\ **Last updated:** [September 12, 2017, 1:26am UTC](https://discuss.elastic.co/t/using-logstash-for-a-propietary-application-stack/100128 "2017-09-12T01:26:11Z")
hi, We have an application stack where several different components hosted on different servers talk to each other. The components are as below, Apache Http Server - Web Server IBM Websphere Portal - Web Application … --- ## [Logstash parsing string+xml](https://discuss.elastic.co/t/logstash-parsing-string-xml/99907)
**Author:** [@ElkPdxUser](https://discuss.elastic.co/u/ElkPdxUser)\ **Replies:** 1\ **Last updated:** [September 11, 2017, 8:59pm UTC](https://discuss.elastic.co/t/logstash-parsing-string-xml/99907 "2017-09-11T20:59:58Z")
Hello, I need to get the transaction id from the input message. During execution, the config script runs without error, however, it is not able to parse the transaction id from the xml. Any help? Log Input: 2017-09-… --- ## [Contention of logstash in sql server](https://discuss.elastic.co/t/contention-of-logstash-in-sql-server/100108)
**Author:** [@Santos\_Javier\_Salaza](https://discuss.elastic.co/u/Santos_Javier_Salaza)\ **Replies:** 3\ **Last updated:** [September 11, 2017, 8:22pm UTC](https://discuss.elastic.co/t/contention-of-logstash-in-sql-server/100108 "2017-09-11T20:22:53Z")
good morning I have a detail with a connection of database sql server to logstash earlier this one worked correctly but when changing the table for the query this mara the following error: \[2017-09-11T12: 24: 05,936\] \[… --- ## [Can we add field in \_grokparsefailure if block?](https://discuss.elastic.co/t/can-we-add-field-in--grokparsefailure-if-block/100101)
**Author:** [@kant.gaurav](https://discuss.elastic.co/u/kant.gaurav)\ **Replies:** 2\ **Last updated:** [September 11, 2017, 6:53pm UTC](https://discuss.elastic.co/t/can-we-add-field-in--grokparsefailure-if-block/100101 "2017-09-11T18:53:50Z")
is it possible like below? if "\_grokparsefailure" in \[tags\] { add\_field =\> \["exceptionClass", "other"\] drop {} } right now it is throwing error. --- ## [Logstash doesn't index when I restart the services but works when I test it](https://discuss.elastic.co/t/logstash-doesnt-index-when-i-restart-the-services-but-works-when-i-test-it/100095)
**Author:** [@bharathdev](https://discuss.elastic.co/u/bharathdev)\ **Replies:** 1\ **Last updated:** [September 11, 2017, 6:23pm UTC](https://discuss.elastic.co/t/logstash-doesnt-index-when-i-restart-the-services-but-works-when-i-test-it/100095 "2017-09-11T18:23:50Z")
Here is my conf file in /etc/logstash/conf.d/apachelog.conf...I don't see index in kibana But when i enter this command it works /opt/logstash/bin/logstash -f /etc/logstash/conf.d/apachelog.conf and I see the index in… --- ## [Issue of protobuf for logstash codec](https://discuss.elastic.co/t/issue-of-protobuf-for-logstash-codec/100102)
**Author:** [@fluency03](https://discuss.elastic.co/u/fluency03)\ **Replies:** 0\ **Last updated:** [September 11, 2017, 4:58pm UTC](https://discuss.elastic.co/t/issue-of-protobuf-for-logstash-codec/100102 "2017-09-11T16:58:47Z")
I am trying to use protobuf for logstash codec. I am following : https://github.com/codekitchen/ruby-protocol-buffers https://github.com/logstash-plugins/logstash-codec-protobuf http://tech.trivago.com/2016/01/19/logst… --- ## [How to pull specific data from a single log line and index that data as fields available in every document](https://discuss.elastic.co/t/how-to-pull-specific-data-from-a-single-log-line-and-index-that-data-as-fields-available-in-every-document/98919)
**Author:** [@newb](https://discuss.elastic.co/u/newb)\ **Replies:** 3\ **Last updated:** [September 11, 2017, 3:31pm UTC](https://discuss.elastic.co/t/how-to-pull-specific-data-from-a-single-log-line-and-index-that-data-as-fields-available-in-every-document/98919 "2017-09-11T15:31:48Z")
Goal is: for each line in the log, there should be a document in elastic containing the ‘message’ (text after time stamp). Each document should also contain fields for the the project name, plan name, and build #. \<–this… --- ## [Google BigQuery Plugin problem](https://discuss.elastic.co/t/google-bigquery-plugin-problem/100081)
**Author:** [@vkruoso](https://discuss.elastic.co/u/vkruoso)\ **Replies:** 0\ **Last updated:** [September 11, 2017, 3:18pm UTC](https://discuss.elastic.co/t/google-bigquery-plugin-problem/100081 "2017-09-11T15:18:25Z")
I'm using the plugin to get logs from HAProxy into the BigQuery tables. I can parse the lines using Grok and send the parsed data to the plugin. My problem is that the plugin can't seem to handle uploads efficiently. Th… --- ## [Change @timestamp to time from actual log file](https://discuss.elastic.co/t/change-timestamp-to-time-from-actual-log-file/100011)
**Author:** [@Ferrow](https://discuss.elastic.co/u/Ferrow)\ **Replies:** 5\ **Last updated:** [September 11, 2017, 1:55pm UTC](https://discuss.elastic.co/t/change-timestamp-to-time-from-actual-log-file/100011 "2017-09-11T13:55:06Z")
Hello everybody.. Im trying to send log files from an Ubuntu client to my ES server. That seems to work fine although I can not seem to get the time field right. (want the time stamp from the log file, not when it has … --- ## [Logstash issue with json input from beats \[Solved\]](https://discuss.elastic.co/t/logstash-issue-with-json-input-from-beats-solved/100039)
**Author:** [@Rob\_Davidson](https://discuss.elastic.co/u/Rob_Davidson)\ **Replies:** 5\ **Last updated:** [September 11, 2017, 1:54pm UTC](https://discuss.elastic.co/t/logstash-issue-with-json-input-from-beats-solved/100039 "2017-09-11T13:54:06Z")
In short I am getting the following error when I view the console terminal on logstash JSON parse error, original data now in message field {:error=\>#\ **Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 6\ **Last updated:** [September 11, 2017, 12:58pm UTC](https://discuss.elastic.co/t/value-split-is/99123 "2017-09-11T12:58:26Z") Hi, I have kv text with the value split “:”. In the text I also have some “:”. For example : (after the https.) Refferer(URI):https://wifiondemand.xfinity.com/wod/selfservice/?lang=en&ui\_style=light My filter looks … --- ## [Adding field to column-Mysql](https://discuss.elastic.co/t/adding-field-to-column-mysql/99313)
**Author:** [@vineets928](https://discuss.elastic.co/u/vineets928)\ **Replies:** 4\ **Last updated:** [September 11, 2017, 12:14pm UTC](https://discuss.elastic.co/t/adding-field-to-column-mysql/99313 "2017-09-11T12:14:42Z")
Hi, I have a mysql table having column name as name, fathername, dob, age, address, dateofjoining, currentlocation. I want to add a field i.e. employee and filter data as attached. I have some idea about mutate, but h… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=909) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=911)