# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=746 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 747 --- ## [Grok filter log](https://discuss.elastic.co/t/grok-filter-log/132361)
**Author:** [@rouchad\_rouchad](https://discuss.elastic.co/u/rouchad_rouchad)\ **Replies:** 7\ **Last updated:** [June 6, 2018, 11:05am UTC](https://discuss.elastic.co/t/grok-filter-log/132361 "2018-06-06T11:05:58Z")
hi every one i need your help please : i want parse a log file with grok filter but it doesn't work i don't know why : the log file : Aug 24 14:26:49 192.168.0.61 firewall 514865 11:12:13:14:15:16 the logstash confi… --- ## [Kv filter for solving key value pair type of situation and avoiding some values in between](https://discuss.elastic.co/t/kv-filter-for-solving-key-value-pair-type-of-situation-and-avoiding-some-values-in-between/134744)
**Author:** [@Shivom\_Kumar](https://discuss.elastic.co/u/Shivom_Kumar)\ **Replies:** 2\ **Last updated:** [June 6, 2018, 10:12am UTC](https://discuss.elastic.co/t/kv-filter-for-solving-key-value-pair-type-of-situation-and-avoiding-some-values-in-between/134744 "2018-06-06T10:12:58Z")
I have my log something like this : May 23 2018 06:32:47 GMT: INFO (xdr): (xdr.c:607) \[030\]: dc-state CLUSTER\_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) shlat… --- ## [Debug input plugin](https://discuss.elastic.co/t/debug-input-plugin/134776)
**Author:** [@verboese](https://discuss.elastic.co/u/verboese)\ **Replies:** 0\ **Last updated:** [June 6, 2018, 9:47am UTC](https://discuss.elastic.co/t/debug-input-plugin/134776 "2018-06-06T09:47:30Z")
Hi. We send entries from a logfile via filebeats to logstash and forward it to elastic. logstash has a "beats" plugin configured. Some lines don't get forwarded though. I'm able to see filebeat sending them to logstash… --- ## [How to use elasticsearch input plugin in logstash to get more than 10000 results from ES](https://discuss.elastic.co/t/how-to-use-elasticsearch-input-plugin-in-logstash-to-get-more-than-10000-results-from-es/134586)
**Author:** [@Moka](https://discuss.elastic.co/u/Moka)\ **Replies:** 10\ **Last updated:** [June 6, 2018, 8:38am UTC](https://discuss.elastic.co/t/how-to-use-elasticsearch-input-plugin-in-logstash-to-get-more-than-10000-results-from-es/134586 "2018-06-06T08:38:53Z")
Hello, I want to copy the data in a distant Es (which I have a limited access ) to another ES where I have full rights. So,the input Es index contains 50000 documents , and I manage to get only 10000 hits using the fo… --- ## [Unable to launch logstash and ingest data - Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/\_xpack'](https://discuss.elastic.co/t/unable-to-launch-logstash-and-ingest-data-got-response-code-401-contacting-elasticsearch-at-url-http-localhost-9200-xpack/134522)
**Author:** [@soumya\_sahoo](https://discuss.elastic.co/u/soumya_sahoo)\ **Replies:** 4\ **Last updated:** [June 6, 2018, 6:59am UTC](https://discuss.elastic.co/t/unable-to-launch-logstash-and-ingest-data-got-response-code-401-contacting-elasticsearch-at-url-http-localhost-9200-xpack/134522 "2018-06-06T06:59:34Z")
Hi team, Let me provide a short background to this - I was able to successfully ingest data to the engine using logstash. It is only after installing x-pack that I run into the following error stack upon passing the fol… --- ## [Logs and database](https://discuss.elastic.co/t/logs-and-database/134730)
**Author:** [@hitman](https://discuss.elastic.co/u/hitman)\ **Replies:** 1\ **Last updated:** [June 6, 2018, 6:34am UTC](https://discuss.elastic.co/t/logs-and-database/134730 "2018-06-06T06:34:10Z")
i have logfiles which has user id and i dont have user name.user name is available in database.now i have to visualize usernames based on the userid present in logs.how? --- ## [Logstash Jdbc\_Streaming Plugin Example?](https://discuss.elastic.co/t/logstash-jdbc-streaming-plugin-example/134726)
**Author:** [@kartheek91](https://discuss.elastic.co/u/kartheek91)\ **Replies:** 0\ **Last updated:** [June 6, 2018, 6:08am UTC](https://discuss.elastic.co/t/logstash-jdbc-streaming-plugin-example/134726 "2018-06-06T06:08:12Z")
Hi.. Can any one tell me how to use logstash jdbc\_streaming plugin and please basic configuration file. --- ## [Use analyzer on existing index field to extract keywords](https://discuss.elastic.co/t/use-analyzer-on-existing-index-field-to-extract-keywords/134725)
**Author:** [@hvarun](https://discuss.elastic.co/u/hvarun)\ **Replies:** 0\ **Last updated:** [June 6, 2018, 6:05am UTC](https://discuss.elastic.co/t/use-analyzer-on-existing-index-field-to-extract-keywords/134725 "2018-06-06T06:05:32Z")
Hello, I am looking to apply language analyzer to an existing index which i have created in elastic. can anyone suggest how i can mention the index column in the text field instead of static text currently i am trying w… --- ## [Multiple beats with different filtering](https://discuss.elastic.co/t/multiple-beats-with-different-filtering/134719)
**Author:** [@laurentiusoica](https://discuss.elastic.co/u/laurentiusoica)\ **Replies:** 0\ **Last updated:** [June 6, 2018, 4:30am UTC](https://discuss.elastic.co/t/multiple-beats-with-different-filtering/134719 "2018-06-06T04:30:01Z")
Hi, I have multiple beats (multiple filebeats with different prospectors that require different filters, metricbeat, a custom beat) sending data to a logstash instance. What is the best approach for logstash pipeline c… --- ## [Elasticsearch xml file](https://discuss.elastic.co/t/elasticsearch-xml-file/134468)
**Author:** [@Hana\_Ne](https://discuss.elastic.co/u/Hana_Ne)\ **Replies:** 20\ **Last updated:** [June 5, 2018, 9:58pm UTC](https://discuss.elastic.co/t/elasticsearch-xml-file/134468 "2018-06-05T21:58:31Z")
Hi i need index xml file to elasticsearch my xml file is like this \ \ \00:00:12,884 --\> 00:00… --- ## [Logstash don't detect duplicated documents](https://discuss.elastic.co/t/logstash-dont-detect-duplicated-documents/134543)
**Author:** [@chrigui94](https://discuss.elastic.co/u/chrigui94)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 8:56pm UTC](https://discuss.elastic.co/t/logstash-dont-detect-duplicated-documents/134543 "2018-06-05T20:56:25Z")
I have to index dynamically multiple .json file in different folders but if I add the same json file that exists already logstash + Elasticsearch index it as a new file with the same documents. I need to find a solution … --- ## [Logstash Log file creation](https://discuss.elastic.co/t/logstash-log-file-creation/134602)
**Author:** [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 8:53pm UTC](https://discuss.elastic.co/t/logstash-log-file-creation/134602 "2018-06-05T20:53:08Z")
In the config file when are trying to read the log file in the log file there are multiline log information is there how can we handle that. Below is the log file i am giving you . 05 Jun 2018 12:20:59,171 | WARN | Ta… --- ## [Filebeat - Redis - Logstash: What Redis brings more?](https://discuss.elastic.co/t/filebeat-redis-logstash-what-redis-brings-more/134660)
**Author:** [@kuba72](https://discuss.elastic.co/u/kuba72)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 8:51pm UTC](https://discuss.elastic.co/t/filebeat-redis-logstash-what-redis-brings-more/134660 "2018-06-05T20:51:13Z")
Hi, What do I earn if I add Redis between Filebeat and Logstash? In this article (https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html) it does not provide a clear statement why Redis or Kafka is … --- ## [Logstash input jdbc :unknown error:](https://discuss.elastic.co/t/logstash-input-jdbc-unknown-error/134671)
**Author:** [@Olatunde\_Tokun](https://discuss.elastic.co/u/Olatunde_Tokun)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 8:06pm UTC](https://discuss.elastic.co/t/logstash-input-jdbc-unknown-error/134671 "2018-06-05T20:06:33Z")
Hey, I use multi pipeline to ingest logs from a database using jdbc driver. It works great, but every once (once in 2 hours) in a while I see errors from the jdbc input plugin. There are no missing events because it us… --- ## [Get worker id/info in pipeline](https://discuss.elastic.co/t/get-worker-id-info-in-pipeline/134683)
**Author:** [@Mikt25](https://discuss.elastic.co/u/Mikt25)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 7:21pm UTC](https://discuss.elastic.co/t/get-worker-id-info-in-pipeline/134683 "2018-06-05T19:21:30Z")
Is there a way to get a numeric workerId programmatically? For instance if "pipeline.workers: 4" then within a filter be able to retrieve a value 1, 2, 3, or 4 depending on what worker is processing. --- ## [Renaming nested fields](https://discuss.elastic.co/t/renaming-nested-fields/134462)
**Author:** [@Tiago\_Pinto](https://discuss.elastic.co/u/Tiago_Pinto)\ **Replies:** 11\ **Last updated:** [June 5, 2018, 7:15pm UTC](https://discuss.elastic.co/t/renaming-nested-fields/134462 "2018-06-05T19:15:22Z")
Hello, I'm trying to rename the fields coming from a jdbc\_streaming and has a target like \[field1\]\[nestedField\], in my document I get a list with the result of the query, since the jdbc\_streaming filter plugin don't ha… --- ## [Question: Handling Dynamic Logstash Provisioning with Kafka Input](https://discuss.elastic.co/t/question-handling-dynamic-logstash-provisioning-with-kafka-input/134664)
**Author:** [@yangas](https://discuss.elastic.co/u/yangas)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 3:58pm UTC](https://discuss.elastic.co/t/question-handling-dynamic-logstash-provisioning-with-kafka-input/134664 "2018-06-05T15:58:23Z")
Hey, Background: I wanted to confirm a behavior which would determine my ability to automate provisioning Logstash for a varying number of data sources. I currently set up some automation through Ansible which splits a… --- ## [Elastic\> Logstash\>Elastic dup messages](https://discuss.elastic.co/t/elastic-logstash-elastic-dup-messages/134672)
**Author:** [@Sahana\_athreya](https://discuss.elastic.co/u/Sahana_athreya)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 6:06pm UTC](https://discuss.elastic.co/t/elastic-logstash-elastic-dup-messages/134672 "2018-06-05T18:06:42Z")
Hi, I am trying to fetch data from elastic, add some fields and send it back to elastic on a new index. I want to make logstash continuously listen to elastic input plugin. I figured that's not possible. Now I set a … --- ## [Include field names before docs (data)](https://discuss.elastic.co/t/include-field-names-before-docs-data/134663)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 4:16pm UTC](https://discuss.elastic.co/t/include-field-names-before-docs-data/134663 "2018-06-05T16:16:46Z")
In order to export docs from Elasticsearch, I'm trying to utilize Csv output plugin | Logstash Reference for that, however I stumble upon small issue.. Is there a way to add field names as first line (before data)? --- ## [Timestamp grok pattern \[SOLVED\]](https://discuss.elastic.co/t/timestamp-grok-pattern-solved/134657)
**Author:** [@dr01](https://discuss.elastic.co/u/dr01)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 3:59pm UTC](https://discuss.elastic.co/t/timestamp-grok-pattern-solved/134657 "2018-06-05T15:59:16Z")
I need to parse a cloud-init logfile: 122.81 - Tue, 05 Jun 2018 11:53:21 +0000 - v. 0.7.9 Here's the grok pattern I wrote: %{NUMBER:time} - %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601\_TIMEZONE} - v. %{NUMBE… --- ## [Extract data with Logstash and Xpath](https://discuss.elastic.co/t/extract-data-with-logstash-and-xpath/134617)
**Author:** [@neo](https://discuss.elastic.co/u/neo)\ **Replies:** 2\ **Last updated:** [June 5, 2018, 3:05pm UTC](https://discuss.elastic.co/t/extract-data-with-logstash-and-xpath/134617 "2018-06-05T15:05:44Z")
Hi , I want to extract data (timestamp and message) via Xpath plugin in Logstash from XML files to display only them in fields in kibana. here is my LOGSTASH conf : input { beats { port =\> 5044 } } filter{ xml{ … --- ## [Logstash dynamic elasticsearch hosts output](https://discuss.elastic.co/t/logstash-dynamic-elasticsearch-hosts-output/134615)
**Author:** [@EloaCardoso](https://discuss.elastic.co/u/EloaCardoso)\ **Replies:** 6\ **Last updated:** [June 5, 2018, 2:46pm UTC](https://discuss.elastic.co/t/logstash-dynamic-elasticsearch-hosts-output/134615 "2018-06-05T14:46:44Z")
output { elasticsearch { ... hosts =\> \["%{ELASTICSEARCH\_URL}"\] } } Is that possible? Because it is not working within Logstash 6.1.4. --- ## [Pipelines running {:count=\>1, :pipelines=\>\["main"\]}](https://discuss.elastic.co/t/pipelines-running-count-1-pipelines-main/134605)
**Author:** [@vemula.pranavi](https://discuss.elastic.co/u/vemula.pranavi)\ **Replies:** 3\ **Last updated:** [June 5, 2018, 1:40pm UTC](https://discuss.elastic.co/t/pipelines-running-count-1-pipelines-main/134605 "2018-06-05T13:40:38Z")
when I execute the command, I get this. I created my test.conf file in bin folder itself. C:\\Users\\vemula.pranavi\\Desktop\\ELK\\logstash-6.2.4\\bin\>logstash -f test.conf Sending Logstash's logs to C:/Users/vemula.pranavi/… --- ## [Jms input plugin connect to mq server with ccdt](https://discuss.elastic.co/t/jms-input-plugin-connect-to-mq-server-with-ccdt/134621)
**Author:** [@GabayAvi](https://discuss.elastic.co/u/GabayAvi)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 12:56pm UTC](https://discuss.elastic.co/t/jms-input-plugin-connect-to-mq-server-with-ccdt/134621 "2018-06-05T12:56:26Z")
Hay i'm Tring to connect to MQ server with CCDT URL FILE . i don't know how to configure it? connection to SERVER that not request CCDT works Fine. below is my config file and yml file. what should i do to config t… --- ## [Issue with logstash on aarch64](https://discuss.elastic.co/t/issue-with-logstash-on-aarch64/133008)
**Author:** [@doctorhey](https://discuss.elastic.co/u/doctorhey)\ **Replies:** 16\ **Last updated:** [June 5, 2018, 12:27pm UTC](https://discuss.elastic.co/t/issue-with-logstash-on-aarch64/133008 "2018-06-05T12:27:08Z")
Hello, I'm having an issue with logstash running on aarch64. At this point, I don't believe it to be config based, as I ran config test on the various files in conf.d AND these files run on other hosts (that are x86\_64)… --- ## [Extract last 2 characters of a given string](https://discuss.elastic.co/t/extract-last-2-characters-of-a-given-string/134587)
**Author:** [@kkrg](https://discuss.elastic.co/u/kkrg)\ **Replies:** 3\ **Last updated:** [June 5, 2018, 11:40am UTC](https://discuss.elastic.co/t/extract-last-2-characters-of-a-given-string/134587 "2018-06-05T11:40:54Z")
Hello, I have a requirement to extract last 2 characters from a given string. Please let me know which logstash function I can use to achieve this. Thanks, Kiran --- ## [Which Logstash plugins are not thread safe?](https://discuss.elastic.co/t/which-logstash-plugins-are-not-thread-safe/134600)
**Author:** [@dr01](https://discuss.elastic.co/u/dr01)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 11:10am UTC](https://discuss.elastic.co/t/which-logstash-plugins-are-not-thread-safe/134600 "2018-06-05T11:10:22Z")
We just upgraded our Logstash server to a machine with multiple CPU cores, so we increased the number of pipeline workers from 1 to 4. Which Logstash plugins are not thread safe, and can therefore cause problems with pa… --- ## [Very huge amount of traffic by src IP. How handle it?](https://discuss.elastic.co/t/very-huge-amount-of-traffic-by-src-ip-how-handle-it/134584)
**Author:** [@Vladimir\_Fomin](https://discuss.elastic.co/u/Vladimir_Fomin)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 10:46am UTC](https://discuss.elastic.co/t/very-huge-amount-of-traffic-by-src-ip-how-handle-it/134584 "2018-06-05T10:46:26Z")
I've start NetFlow by using this command /usr/share/logstash/bin/logstash --modules netflow -M netflow.var.input.udp.port=9966 and it works. But when I look at overview traffic dashboard I see that some host send huge a… --- ## [Custom Filter for remote Command](https://discuss.elastic.co/t/custom-filter-for-remote-command/134546)
**Author:** [@shubham\_kumar1](https://discuss.elastic.co/u/shubham_kumar1)\ **Replies:** 0\ **Last updated:** [June 5, 2018, 7:31am UTC](https://discuss.elastic.co/t/custom-filter-for-remote-command/134546 "2018-06-05T07:31:45Z")
Hi, I made a custom filter which to perform some commands on a remote server using net-ssh and return the output. The filter works fine for 3 to 4 events but after that it starts to fail. Filter code:- # encoding: utf… --- ## [Javascript library similar to grok plugin](https://discuss.elastic.co/t/javascript-library-similar-to-grok-plugin/134534)
**Author:** [@jkburges](https://discuss.elastic.co/u/jkburges)\ **Replies:** 1\ **Last updated:** [June 5, 2018, 10:19am UTC](https://discuss.elastic.co/t/javascript-library-similar-to-grok-plugin/134534 "2018-06-05T10:19:08Z")
I am researching the possibility of developing a desktop app that allows parsing of log files. As a personal learning exercise, I'd like to do it using the electronjs framework. I really like the logstash-grok DSL for c… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=745) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=747)