# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1067 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1068 --- ## [Some problems about logstash5 with non time based jdbc input](https://discuss.elastic.co/t/some-problems-about-logstash5-with-non-time-based-jdbc-input/68111)
**Author:** [@reo7189](https://discuss.elastic.co/u/reo7189)\ **Replies:** 1\ **Last updated:** [December 9, 2016, 6:31am UTC](https://discuss.elastic.co/t/some-problems-about-logstash5-with-non-time-based-jdbc-input/68111 "2016-12-09T06:31:45Z")
1./ With logstash jdbc plugin, I got postgresql db pipeline to Elasticsearch, but I noticed the index size in elasticsearch doubled every time I restart logstash, and kibana showed doubled data counts as well, why? and h… --- ## [Logstash not indexing s3 input](https://discuss.elastic.co/t/logstash-not-indexing-s3-input/68068)
**Author:** [@nixmind](https://discuss.elastic.co/u/nixmind)\ **Replies:** 1\ **Last updated:** [December 7, 2016, 1:36pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-s3-input/68068 "2016-12-07T13:36:04Z")
Hi, I have an issue diffucult to troubleshoot. I had set up an architecture with filebeat as forwarder on my web APIs, a logstash listening to filebeat on port 5044 on a dedicated machine, and indexing log messages on … --- ## [Logstash-outputs-elasticsearch aws endpoint authentication](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-aws-endpoint-authentication/68359)
**Author:** [@api-gateway](https://discuss.elastic.co/u/api-gateway)\ **Replies:** 2\ **Last updated:** [December 9, 2016, 1:23am UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-aws-endpoint-authentication/68359 "2016-12-09T01:23:25Z")
Context: Using an Elastic.co hosted version of ELK (on AWS) Attempting to build the filebeat-\>logstash-\>elasticsearch pipeline Configured the ouput {elasticsearch {...}} to point to remote stack During startup o… --- ## [Response code: 429](https://discuss.elastic.co/t/response-code-429/68470)
**Author:** [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Replies:** 9\ **Last updated:** [December 9, 2016, 12:50am UTC](https://discuss.elastic.co/t/response-code-429/68470 "2016-12-09T00:50:37Z")
Hi All, I am getting below error and I understand from other thread I am overloading elasticsearch. I am looking on what to dail down to get rid of this problem and get the most of performance. I have 9 LS nodes. My l… --- ## [Defaulting pipeline worker threads to 1](https://discuss.elastic.co/t/defaulting-pipeline-worker-threads-to-1/68465)
**Author:** [@Honda\_fred\_elk](https://discuss.elastic.co/u/Honda_fred_elk)\ **Replies:** 1\ **Last updated:** [December 8, 2016, 8:42pm UTC](https://discuss.elastic.co/t/defaulting-pipeline-worker-threads-to-1/68465 "2016-12-08T20:42:57Z")
Hello, I have following config for logstash.. input { file { path =\> "/appl/log/TestLogs/\*.log" start\_position =\> "beginning" sincedb\_path =\> "/appl/log/sincedbloc/sincedbfile.txt" } } filter{ … --- ## [Collectd install](https://discuss.elastic.co/t/collectd-install/68283)
**Author:** [@imaad](https://discuss.elastic.co/u/imaad)\ **Replies:** 3\ **Last updated:** [December 8, 2016, 7:18pm UTC](https://discuss.elastic.co/t/collectd-install/68283 "2016-12-08T19:18:23Z")
Hi guys, I'm trying to install collectd on centos from source. How can I do it? Thank you. --- ## [Add year to timestamp](https://discuss.elastic.co/t/add-year-to-timestamp/68195)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 5\ **Last updated:** [December 8, 2016, 6:58pm UTC](https://discuss.elastic.co/t/add-year-to-timestamp/68195 "2016-12-08T18:58:20Z")
Hi, The time stamp in my log looks like that: \[Mar 01 14:55:11\] . No year available. I want to add it to, to be the current year if possible or the previous year. I need to have the year, I assume int he timestamp, to… --- ## [Replacing @timestamp with timestamp of my log](https://discuss.elastic.co/t/replacing-timestamp-with-timestamp-of-my-log/68413)
**Author:** [@taurs](https://discuss.elastic.co/u/taurs)\ **Replies:** 17\ **Last updated:** [December 8, 2016, 6:55pm UTC](https://discuss.elastic.co/t/replacing-timestamp-with-timestamp-of-my-log/68413 "2016-12-08T18:55:22Z")
I want to replace timestamp with the timestamp of my log.. In place of @timestamp, I want to replace with timestamp\_match. input { beats { port =\> "5044" } } filter { if \[type\] == "log" { grok { match =\> { "message" =\> … --- ## [Filters empty messages](https://discuss.elastic.co/t/filters-empty-messages/68461)
**Author:** [@webofmars](https://discuss.elastic.co/u/webofmars)\ **Replies:** 1\ **Last updated:** [December 8, 2016, 5:37pm UTC](https://discuss.elastic.co/t/filters-empty-messages/68461 "2016-12-08T17:37:04Z")
Hello, I'm trying to filter some empty messages sent from badly formated logs. By empty i mean they can contain just an unknown number of newlines charachters and/or just some spaces. i tried something like that : … --- ## [SunCertPathBuilderException: unable to find valid certification](https://discuss.elastic.co/t/suncertpathbuilderexception-unable-to-find-valid-certification/68396)
**Author:** [@Shubham\_Agrawal](https://discuss.elastic.co/u/Shubham_Agrawal)\ **Replies:** 0\ **Last updated:** [December 8, 2016, 7:10am UTC](https://discuss.elastic.co/t/suncertpathbuilderexception-unable-to-find-valid-certification/68396 "2016-12-08T07:10:03Z")
Hi, I have setup logstash to parse logs and then later push them to elasticsearch. here is my config file: elasticsearch { hosts =\> \["https://\_my\_hosts\_:443"\] ssl =\> true ssl\_certificate\_varification =\> false } … --- ## [Latitude longitude conversation](https://discuss.elastic.co/t/latitude-longitude-conversation/68302)
**Author:** [@arkash20](https://discuss.elastic.co/u/arkash20)\ **Replies:** 4\ **Last updated:** [December 8, 2016, 10:06am UTC](https://discuss.elastic.co/t/latitude-longitude-conversation/68302 "2016-12-08T10:06:51Z")
Hi Huys, We need to convert 180,000,000 coordinates of Manhattan to addresses and then store the all the data in a database for statistical analyze. how can we use elastic for this purpose? thank in advance, Arkady … --- ## [Anyone please help on create keys/certs for filebeat 5 to logstash?](https://discuss.elastic.co/t/anyone-please-help-on-create-keys-certs-for-filebeat-5-to-logstash/68353)
**Author:** [@robertchen117](https://discuss.elastic.co/u/robertchen117)\ **Replies:** 1\ **Last updated:** [December 8, 2016, 6:36am UTC](https://discuss.elastic.co/t/anyone-please-help-on-create-keys-certs-for-filebeat-5-to-logstash/68353 "2016-12-08T06:36:20Z")
The doc (https://www.elastic.co/guide/en/beats/filebeat/5.0/configuring-ssl-logstash.html )did not say which way to create these keys/certificates, I tried to create with openssl two-way SSL but does not work, error is: … --- ## [Cluster ID "68ed60: JSON plugin error in Logstash 5 but no error in Logstash 2.3](https://discuss.elastic.co/t/cluster-id-68ed60-json-plugin-error-in-logstash-5-but-no-error-in-logstash-2-3/68200)
**Author:** [@Charles\_Lee](https://discuss.elastic.co/u/Charles_Lee)\ **Replies:** 1\ **Last updated:** [December 8, 2016, 6:34am UTC](https://discuss.elastic.co/t/cluster-id-68ed60-json-plugin-error-in-logstash-5-but-no-error-in-logstash-2-3/68200 "2016-12-08T06:34:39Z")
We got this error "JSON parse error, original data now in message field {:error=\>#\ **Author:** [@Geezer](https://discuss.elastic.co/u/Geezer)\ **Replies:** 1\ **Last updated:** [December 5, 2016, 10:56pm UTC](https://discuss.elastic.co/t/convert-decimal-ip-to-ipv4-dotted-decimal/68109 "2016-12-05T22:56:51Z") Hello, I am reading data from an SQL table which contains IP addresses in decimal format, and I need to convert these to dotted decimal. e.g. convert this: 1978211278 to this: 117.233.27.206 I am trying … --- ## [Error start Logstash 5](https://discuss.elastic.co/t/error-start-logstash-5/63965)
**Author:** [@axil\_76](https://discuss.elastic.co/u/axil_76)\ **Replies:** 3\ **Last updated:** [December 7, 2016, 4:47pm UTC](https://discuss.elastic.co/t/error-start-logstash-5/63965 "2016-12-07T16:47:52Z")
Hello, I just installed logstash 5 and when I run it with the following configuration I have an error message /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d -l /var/log/logstash --verbose --path.settings=/et… --- ## [How to handle \_grokparsefailure with multiple inputs](https://discuss.elastic.co/t/how-to-handle--grokparsefailure-with-multiple-inputs/68260)
**Author:** [@hunterchris](https://discuss.elastic.co/u/hunterchris)\ **Replies:** 2\ **Last updated:** [December 7, 2016, 3:50pm UTC](https://discuss.elastic.co/t/how-to-handle--grokparsefailure-with-multiple-inputs/68260 "2016-12-07T15:50:46Z")
Hello, I have logstash with two inputs: beats and gelf. Logs that come from beats are syslog messages, logs from gelf are docker logs. I created a filter to match multiline logs from docker that works perfect, however I… --- ## [How to remove field with logstash5.0.2?](https://discuss.elastic.co/t/how-to-remove-field-with-logstash5-0-2/68290)
**Author:** [@kaybinwong](https://discuss.elastic.co/u/kaybinwong)\ **Replies:** 2\ **Last updated:** [December 7, 2016, 12:12pm UTC](https://discuss.elastic.co/t/how-to-remove-field-with-logstash5-0-2/68290 "2016-12-07T12:12:50Z")
i remove field be by the below way, but it seems to remove all the event, what‘s the different when using remove\_field between drop and grok? remove all the event work --- ## [Reload the same file from the beginning without restarting logstash](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156)
**Author:** [@mehdi-aouadi](https://discuss.elastic.co/u/mehdi-aouadi)\ **Replies:** 4\ **Last updated:** [December 7, 2016, 9:36am UTC](https://discuss.elastic.co/t/reload-the-same-file-from-the-beginning-without-restarting-logstash/68156 "2016-12-07T09:36:29Z")
I managed to force Logstash reloading the whole file by pointing the sincedb\_path to NUL (Windows environment) and setting the start\_position at the beginning. Here is my file input configuration: input { file {… --- ## [How to remove duplicate events in logstash](https://discuss.elastic.co/t/how-to-remove-duplicate-events-in-logstash/67859)
**Author:** [@Priyanka\_Marihal](https://discuss.elastic.co/u/Priyanka_Marihal)\ **Replies:** 2\ **Last updated:** [December 7, 2016, 6:09am UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-events-in-logstash/67859 "2016-12-07T06:09:29Z")
Hi, I really need your help. I am using filebeat for transferring logs and processing using logstash and storing into elasticsearch as well as in CSV file but i can see duplicate events in both CSV file and Elastic… --- ## [LogStash querying from Multiple DB](https://discuss.elastic.co/t/logstash-querying-from-multiple-db/68251)
**Author:** [@Vinaya.Thimmappa](https://discuss.elastic.co/u/Vinaya.Thimmappa)\ **Replies:** 1\ **Last updated:** [December 7, 2016, 7:01am UTC](https://discuss.elastic.co/t/logstash-querying-from-multiple-db/68251 "2016-12-07T07:01:54Z")
I have a use case where in I have multiple Database in mysql. i.e DB1, DB2, DB3, DB4 .. and this is growing I want to query on a particular table T1 in each of DB and index to ES using log-stash I have one table in… --- ## [Split field and reverse join](https://discuss.elastic.co/t/split-field-and-reverse-join/68169)
**Author:** [@illxjasper](https://discuss.elastic.co/u/illxjasper)\ **Replies:** 1\ **Last updated:** [December 7, 2016, 6:48am UTC](https://discuss.elastic.co/t/split-field-and-reverse-join/68169 "2016-12-07T06:48:57Z")
I am trying to parse appache accesslogs with logstash for a project. To let elasticsearch search efficiënt I want to reverse the domainname. So for example: www.example.com Becomes com.example.www I tried to split … --- ## [How to handle different logs in logstash server](https://discuss.elastic.co/t/how-to-handle-different-logs-in-logstash-server/68247)
**Author:** [@houfan](https://discuss.elastic.co/u/houfan)\ **Replies:** 3\ **Last updated:** [December 7, 2016, 6:46am UTC](https://discuss.elastic.co/t/how-to-handle-different-logs-in-logstash-server/68247 "2016-12-07T06:46:00Z")
currently,,I want to send nginx\_access.log and nginx\_error.log to logstash server.here is the filebeat.yml configuration,I don't know whether I am right or not. and this is the nginx\_access log format: eeeeee.....… --- ## [Loading large table from mysql into ES with logstash](https://discuss.elastic.co/t/loading-large-table-from-mysql-into-es-with-logstash/67905)
**Author:** [@inancarin](https://discuss.elastic.co/u/inancarin)\ **Replies:** 3\ **Last updated:** [December 7, 2016, 6:43am UTC](https://discuss.elastic.co/t/loading-large-table-from-mysql-into-es-with-logstash/67905 "2016-12-07T06:43:31Z")
Hi guys, I have a large table in mysql which contains 58 millions of tweet records and it's size is continuously increasing as I am streaming from twitter. The schema of the table as follows: CREATE TABLE IF NOT EXIS… --- ## [Oracle listener regexp/grok pattern help](https://discuss.elastic.co/t/oracle-listener-regexp-grok-pattern-help/67886)
**Author:** [@FrankC](https://discuss.elastic.co/u/FrankC)\ **Replies:** 1\ **Last updated:** [December 7, 2016, 6:40am UTC](https://discuss.elastic.co/t/oracle-listener-regexp-grok-pattern-help/67886 "2016-12-07T06:40:54Z")
Hi, I need to get oracle listener logs loaded using LS. I was able to extract the timestamp and replace the LS @timestamp successfully as well as some other info. I am fairly new to regexp/grok, but came across this p… --- ## [Stops processing TCP Output - Requires restart of service](https://discuss.elastic.co/t/stops-processing-tcp-output-requires-restart-of-service/68235)
**Author:** [@brad.bow](https://discuss.elastic.co/u/brad.bow)\ **Replies:** 0\ **Last updated:** [December 7, 2016, 2:13am UTC](https://discuss.elastic.co/t/stops-processing-tcp-output-requires-restart-of-service/68235 "2016-12-07T02:13:12Z")
Hi all, We run Logstash across multiple API Windows servers. We have 3 file inputs, 1 TCP output, and a bunch of filters. Our inputs are: input { file { path =\> \[some file path\] type =\> "iis" … --- ## [Logstash output redis plugin infinite loop](https://discuss.elastic.co/t/logstash-output-redis-plugin-infinite-loop/68231)
**Author:** [@Sawyer\_Zhu](https://discuss.elastic.co/u/Sawyer_Zhu)\ **Replies:** 0\ **Last updated:** [December 7, 2016, 1:23am UTC](https://discuss.elastic.co/t/logstash-output-redis-plugin-infinite-loop/68231 "2016-12-07T01:23:11Z")
If some exception raised in receive method, it will goto infinite loop with 'retry' statement. Should add the max retry times limit. --- ## [Logstash output redis plugin add SSL suport](https://discuss.elastic.co/t/logstash-output-redis-plugin-add-ssl-suport/68232)
**Author:** [@Sawyer\_Zhu](https://discuss.elastic.co/u/Sawyer_Zhu)\ **Replies:** 0\ **Last updated:** [December 7, 2016, 1:26am UTC](https://discuss.elastic.co/t/logstash-output-redis-plugin-add-ssl-suport/68232 "2016-12-07T01:26:42Z")
Is there any plan for addding SSL support for redis output plugin? --- ## [Logstash hangs every time](https://discuss.elastic.co/t/logstash-hangs-every-time/68121)
**Author:** [@dremspider](https://discuss.elastic.co/u/dremspider)\ **Replies:** 4\ **Last updated:** [December 6, 2016, 10:11pm UTC](https://discuss.elastic.co/t/logstash-hangs-every-time/68121 "2016-12-06T22:11:00Z")
I have Centos 7 and installed from the repositories. I am using openjdk 1.8. I have been messing with getting logstash open for a while, and I can't even make it work from the command line.. When I launch it from the c… --- ## [Error logstash setup](https://discuss.elastic.co/t/error-logstash-setup/68191)
**Author:** [@bob-bza](https://discuss.elastic.co/u/bob-bza)\ **Replies:** 4\ **Last updated:** [December 6, 2016, 8:18pm UTC](https://discuss.elastic.co/t/error-logstash-setup/68191 "2016-12-06T20:18:59Z")
I was trying to configure logstash on my instance i was getting facing issue. i dont know if i was doing anything wrong I tried on both ubuntu & centos i was facing same issue with 5.0.1 but 2.4 works for me.I wan't to… --- ## [Only write a particular field of an event to a tcp socket](https://discuss.elastic.co/t/only-write-a-particular-field-of-an-event-to-a-tcp-socket/68217)
**Author:** [@niteshjain92](https://discuss.elastic.co/u/niteshjain92)\ **Replies:** 1\ **Last updated:** [December 6, 2016, 8:16pm UTC](https://discuss.elastic.co/t/only-write-a-particular-field-of-an-event-to-a-tcp-socket/68217 "2016-12-06T20:16:44Z")
Using the latest logstash and my current config is : input { kafka {....} } output { tcp { port =\> 10001 } } I can see with this config, the data written to the socket is: $ nc -l -p 10001 {"@timestamp":… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1066) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1068)