# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=875 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 876 --- ## [Where should logstash live? Cloud vs OnPrem](https://discuss.elastic.co/t/where-should-logstash-live-cloud-vs-onprem/107472)
**Author:** [@Mike\_Hakopyan](https://discuss.elastic.co/u/Mike_Hakopyan)\ **Replies:** 3\ **Last updated:** [November 14, 2017, 9:47pm UTC](https://discuss.elastic.co/t/where-should-logstash-live-cloud-vs-onprem/107472 "2017-11-14T21:47:04Z")
I'm very new to LogStash and Elasticsearch. I'd like to know where to install Logstash; on prem or cloud? I have multiple locations where I'd be collecting log files with Beats and shipping to Logstash, then to Elastic… --- ## [Date Filter Plugin Not Working With Elasticsearch](https://discuss.elastic.co/t/date-filter-plugin-not-working-with-elasticsearch/107620)
**Author:** [@BinitShrestha](https://discuss.elastic.co/u/BinitShrestha)\ **Replies:** 5\ **Last updated:** [November 14, 2017, 9:20pm UTC](https://discuss.elastic.co/t/date-filter-plugin-not-working-with-elasticsearch/107620 "2017-11-14T21:20:45Z")
Hello Everyone! I am having trouble getting the Date Filter Plugin properly working with my Elasticsearch. Below is my Logstash configuration file: input { beats { port =\> "5043" } } filter { grok { match =\> \["m… --- ## [How do I do conditional processing based on input](https://discuss.elastic.co/t/how-do-i-do-conditional-processing-based-on-input/107621)
**Author:** [@sdussin](https://discuss.elastic.co/u/sdussin)\ **Replies:** 2\ **Last updated:** [November 14, 2017, 8:44pm UTC](https://discuss.elastic.co/t/how-do-i-do-conditional-processing-based-on-input/107621 "2017-11-14T20:44:17Z")
I have a case where my pipeline contains inputs from two different JDBC sources. In my filter, I want to run some things but not others. My input config looks like this: input { jdbc { jdbc\_connection\_strin… --- ## [Couchdb\_changes doesn't exist in 6.0](https://discuss.elastic.co/t/couchdb-changes-doesnt-exist-in-6-0/107600)
**Author:** [@wkpv68](https://discuss.elastic.co/u/wkpv68)\ **Replies:** 2\ **Last updated:** [November 14, 2017, 8:36pm UTC](https://discuss.elastic.co/t/couchdb-changes-doesnt-exist-in-6-0/107600 "2017-11-14T20:36:33Z")
Hi, I download the latest 6.0 and couchdb\_changes is missing. How can I add it Thanks --- ## [Remove common keyword from a filed value](https://discuss.elastic.co/t/remove-common-keyword-from-a-filed-value/107159)
**Author:** [@tharu85](https://discuss.elastic.co/u/tharu85)\ **Replies:** 1\ **Last updated:** [November 14, 2017, 8:09pm UTC](https://discuss.elastic.co/t/remove-common-keyword-from-a-filed-value/107159 "2017-11-14T20:09:10Z")
After I have apply the filter, I received output like below command\_set ="CmdAV=show ip policy \" But I need a output like below. command\_set ="show ip policy" For the filed command\_set, CmdAV= is common and I nee… --- ## [Logstash nested object aggregation is not working](https://discuss.elastic.co/t/logstash-nested-object-aggregation-is-not-working/107619)
**Author:** [@Farah\_Alrabee](https://discuss.elastic.co/u/Farah_Alrabee)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 7:59pm UTC](https://discuss.elastic.co/t/logstash-nested-object-aggregation-is-not-working/107619 "2017-11-14T19:59:32Z")
I have 3 tables : user, post, comment are joined with left join. I created a nested object (posts) for each user. I'm able to aggregate the posts object but i want also to aggregate the comments object inside each posts … --- ## [JBOSS EAP 6 Admin Audit Logging](https://discuss.elastic.co/t/jboss-eap-6-admin-audit-logging/106254)
**Author:** [@josh\_jacques](https://discuss.elastic.co/u/josh_jacques)\ **Replies:** 8\ **Last updated:** [November 14, 2017, 7:54pm UTC](https://discuss.elastic.co/t/jboss-eap-6-admin-audit-logging/106254 "2017-11-14T19:54:14Z")
I'm looking for a little help here, who am I kidding, I am looking for a lot of help here and if my inquiry has already been asked, I apologize in advance. Looking at the following two log entries, I'd like to take some… --- ## [How does String search in Logstash filter work?](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 12\ **Last updated:** [November 14, 2017, 7:47pm UTC](https://discuss.elastic.co/t/how-does-string-search-in-logstash-filter-work/107467 "2017-11-14T19:47:07Z")
Hi, I am using following filter in Logstash, if "INFO" in \[message\] and "instance" in \[message\] { mutate{ remove\_field =\> \["name"\] } } And it's doing not… --- ## [Trouble parsing following message through logstash](https://discuss.elastic.co/t/trouble-parsing-following-message-through-logstash/107611)
**Author:** [@PiyushPasari](https://discuss.elastic.co/u/PiyushPasari)\ **Replies:** 2\ **Last updated:** [November 14, 2017, 7:49pm UTC](https://discuss.elastic.co/t/trouble-parsing-following-message-through-logstash/107611 "2017-11-14T19:49:16Z")
ts=2017-04-19-19-12-58-881|result=SUCCESS | requestId=1617b0ff-0770-4da3-af4-ce3641637 |action=BestController-getData | elapsed=1241| ip=115.23.71.21 |method=GET | status=200 | uri=/ngwms/best/initial | userId=info I … --- ## [Logstash pipe creates screenshot cursor](https://discuss.elastic.co/t/logstash-pipe-creates-screenshot-cursor/107612)
**Author:** [@Mat\_S](https://discuss.elastic.co/u/Mat_S)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 7:40pm UTC](https://discuss.elastic.co/t/logstash-pipe-creates-screenshot-cursor/107612 "2017-11-14T19:40:35Z")
The project pipeline that I am working on involves grabbing Twitter data from a ZMQ publisher .py script, then ZMQ subscriber.py | logstash - f simple-config.rb (json), then running elasticsearch. An issue I am running … --- ## [How to parse date field into @timestamp](https://discuss.elastic.co/t/how-to-parse-date-field-into-timestamp/107396)
**Author:** [@sdussin](https://discuss.elastic.co/u/sdussin)\ **Replies:** 17\ **Last updated:** [November 14, 2017, 7:26pm UTC](https://discuss.elastic.co/t/how-to-parse-date-field-into-timestamp/107396 "2017-11-14T19:26:26Z")
I'm trying to replace the @timestamp that's generated by logstash with the contents of an existing field in my data. I've been fighting with this all day, and I'm nowhere. My data looks like this { "start\_time" : "201… --- ## [\[ERROR\]\[logstash.outputs.elasticsearch\]](https://discuss.elastic.co/t/error-logstash-outputs-elasticsearch/107604)
**Author:** [@Cristiano\_Alves](https://discuss.elastic.co/u/Cristiano_Alves)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 6:27pm UTC](https://discuss.elastic.co/t/error-logstash-outputs-elasticsearch/107604 "2017-11-14T18:27:30Z")
Hi, my logstash show ERROR: \[2017-11-14T16:19:04,713\]\[WARN \]\[logstash.outputs.elasticsearch\] Marking url as dead. Last error: \[LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError\] Elasticsearch Unre… --- ## [Index not created](https://discuss.elastic.co/t/index-not-created/107562)
**Author:** [@Sumit\_Gupta1](https://discuss.elastic.co/u/Sumit_Gupta1)\ **Replies:** 1\ **Last updated:** [November 14, 2017, 5:44pm UTC](https://discuss.elastic.co/t/index-not-created/107562 "2017-11-14T17:44:06Z")
Hi, My Config file is following: input { jdbc { jdbc\_driver\_library =\> "/app/sqljdbc4-2.0.jar" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc\_connection\_string =\> "jdbc:sqlserver://10.124.… --- ## [Break a single line into one multiline event](https://discuss.elastic.co/t/break-a-single-line-into-one-multiline-event/107253)
**Author:** [@seth.yes](https://discuss.elastic.co/u/seth.yes)\ **Replies:** 3\ **Last updated:** [November 14, 2017, 3:39pm UTC](https://discuss.elastic.co/t/break-a-single-line-into-one-multiline-event/107253 "2017-11-14T15:39:42Z")
I need to break down a single event into one multi-line event. e.g. single one-line event: {"node47-domain":{"name":"node47-domain"},"node1208-domain":{"name":"node1208-domain"},"node170-domain":{"name":"node170-domai… --- ## [Logstash RabbitMQ Wrap Incoming Message](https://discuss.elastic.co/t/logstash-rabbitmq-wrap-incoming-message/105925)
**Author:** [@Jeff\_Bursik](https://discuss.elastic.co/u/Jeff_Bursik)\ **Replies:** 6\ **Last updated:** [November 14, 2017, 3:34pm UTC](https://discuss.elastic.co/t/logstash-rabbitmq-wrap-incoming-message/105925 "2017-11-14T15:34:17Z")
I'm trying to wrap the incoming message off of Rabbit and then have some extra metadata outside of the wrapped message. Can this be done? Hoping for an end result like: errorMessage:{ ... }, trackingID:{ ... } header… --- ## [File output with Header like CSV](https://discuss.elastic.co/t/file-output-with-header-like-csv/107564)
**Author:** [@krishna\_chaitanya](https://discuss.elastic.co/u/krishna_chaitanya)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 3:19pm UTC](https://discuss.elastic.co/t/file-output-with-header-like-csv/107564 "2017-11-14T15:19:38Z")
I am using logstash 5.4 and trying to create a CSV file from the events processed. With csv output plugin, I got into same issue as this one: https://discuss.elastic.co/t/csv-output-failing-in-logstash-5-1-2/73751 The… --- ## [Hive JDBC](https://discuss.elastic.co/t/hive-jdbc/107478)
**Author:** [@Jeff\_Bursik](https://discuss.elastic.co/u/Jeff_Bursik)\ **Replies:** 1\ **Last updated:** [November 14, 2017, 3:19pm UTC](https://discuss.elastic.co/t/hive-jdbc/107478 "2017-11-14T15:19:23Z")
I've seen a couple posts asking for a Hive JDBC Input configuration however I haven't seen a successful configuration posted. Im curious what JDBC Driver is needed to make the Hive connection work? Thanks! --- ## [Getting \_grokparsefailure for an unkown reason](https://discuss.elastic.co/t/getting-grokparsefailure-for-an-unkown-reason/107556)
**Author:** [@ou\_snaaksie](https://discuss.elastic.co/u/ou_snaaksie)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 2:06pm UTC](https://discuss.elastic.co/t/getting-grokparsefailure-for-an-unkown-reason/107556 "2017-11-14T14:06:42Z")
I am tryinf to make a grok filter for the Synology NAS syslog messages, but I keep on getting a \_grokparsefailure even though my filter works according to http://grokdebug.herokuapp.com/. Filter: "\<%{POSINT:syslog\_pri}\>… --- ## [Logstash date parsing error (bug?)](https://discuss.elastic.co/t/logstash-date-parsing-error-bug/107503)
**Author:** [@Mojster](https://discuss.elastic.co/u/Mojster)\ **Replies:** 1\ **Last updated:** [November 14, 2017, 2:44pm UTC](https://discuss.elastic.co/t/logstash-date-parsing-error-bug/107503 "2017-11-14T14:44:33Z")
I'm using this pattern to parse dates from strings to date objects: date { locale =\> "en" match =\> \["firstsignUpDate", "dd.MM.YYYY"\] timezone =\> "Europe/Ljubljana" target =\> "firstsignUpDate" } When a pars a l… --- ## [Why is date filter in Logstash used?](https://discuss.elastic.co/t/why-is-date-filter-in-logstash-used/107549)
**Author:** [@amruth](https://discuss.elastic.co/u/amruth)\ **Replies:** 2\ **Last updated:** [November 14, 2017, 2:24pm UTC](https://discuss.elastic.co/t/why-is-date-filter-in-logstash-used/107549 "2017-11-14T14:24:34Z")
I am extracting fields from grok expression out of which one field is a timestamp field. Later I am using date filter and matching with timestamp field. But the problem is timestamp field is not consistent always. It may… --- ## [Aggregated Index name vs Index name - define in Logstash output](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531)
**Author:** [@ssasporta](https://discuss.elastic.co/u/ssasporta)\ **Replies:** 7\ **Last updated:** [November 14, 2017, 2:06pm UTC](https://discuss.elastic.co/t/aggregated-index-name-vs-index-name-define-in-logstash-output/107531 "2017-11-14T14:06:04Z")
My Logstash conf output looks like that: output { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "logstash%{\[fields\]\[index\]}%{+YYYY.MM.dd}" } stdout { codec … --- ## [Grok filter to Replace @timestamp with my log-time](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364)
**Author:** [@Toni\_Riera](https://discuss.elastic.co/u/Toni_Riera)\ **Replies:** 12\ **Last updated:** [November 14, 2017, 2:05pm UTC](https://discuss.elastic.co/t/grok-filter-to-replace-timestamp-with-my-log-time/105364 "2017-11-14T14:05:36Z")
Hi there, I'm trying to find out what to do, but got stucked in there. I've got ELK runnning on a single machine, and i'm trying to analyze certain logs using Logstash. Currently i've got an entry like this: Jan 22 15… --- ## [KV filter - for parsing my syslog ng](https://discuss.elastic.co/t/kv-filter-for-parsing-my-syslog-ng/107553)
**Author:** [@Raj\_Kumar](https://discuss.elastic.co/u/Raj_Kumar)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 1:34pm UTC](https://discuss.elastic.co/t/kv-filter-for-parsing-my-syslog-ng/107553 "2017-11-14T13:34:18Z")
Hi All, Iam using ELastic stack to store our syslog-ng logs, so I was able to get the logs into elasticsearch, which is perfect. So I thought I could go one further more to parse the message field , but the problem is … --- ## [How to handle Multiple Config files in logstash](https://discuss.elastic.co/t/how-to-handle-multiple-config-files-in-logstash/107520)
**Author:** [@Arun\_Raj](https://discuss.elastic.co/u/Arun_Raj)\ **Replies:** 8\ **Last updated:** [November 14, 2017, 12:18pm UTC](https://discuss.elastic.co/t/how-to-handle-multiple-config-files-in-logstash/107520 "2017-11-14T12:18:39Z")
Hi All I am trying to load two different configurations (different data filters for each ) at a time on single command line I tried the below : from command line ( sudo bin/logstash -f /etc/logstash/conf.d/logstash.c… --- ## [Running Logstash from .NET Service](https://discuss.elastic.co/t/running-logstash-from-net-service/106946)
**Author:** [@sn001](https://discuss.elastic.co/u/sn001)\ **Replies:** 12\ **Last updated:** [November 14, 2017, 12:02pm UTC](https://discuss.elastic.co/t/running-logstash-from-net-service/106946 "2017-11-14T12:02:28Z")
Hi there, I am working on large scale data ingestion into Elastic Search indexes. We have a number of database records that we want to ingest into ES after we have detected delta changes in the database. We have .NET S… --- ## [A plugin error](https://discuss.elastic.co/t/a-plugin-error/107488)
**Author:** [@Shaswata\_Bisi](https://discuss.elastic.co/u/Shaswata_Bisi)\ **Replies:** 2\ **Last updated:** [November 14, 2017, 11:28am UTC](https://discuss.elastic.co/t/a-plugin-error/107488 "2017-11-14T11:28:48Z")
I have ES 5.6.3 and logstash 5.4 installed on my local system whereas I have filebeat 5.4 installed in another system. Whenever I am trying to run logstash, I get the following error \[\[main\]\ **Author:** [@dsakpal](https://discuss.elastic.co/u/dsakpal)\ **Replies:** 9\ **Last updated:** [November 14, 2017, 10:10am UTC](https://discuss.elastic.co/t/create-json-object-in-logstash-pass-to-the-elastic-search/106665 "2017-11-14T10:10:29Z") My log format is like, 02.11.2017,11:33:13,DDIC,6,192.168.2.110,PFCG,SAPMSYST SAPMSYST1,Logon Successful (Type=A) My filter is like, filter { grok{ match=\>{ "message"=\>"%{NOTSPACE:date},%{NOTSPACE:time},%{WORD:use… --- ## [Logstash don't create index](https://discuss.elastic.co/t/logstash-dont-create-index/107441)
**Author:** [@ahmedcharef](https://discuss.elastic.co/u/ahmedcharef)\ **Replies:** 5\ **Last updated:** [November 14, 2017, 8:15am UTC](https://discuss.elastic.co/t/logstash-dont-create-index/107441 "2017-11-14T08:15:55Z")
Hi, I run ELK services on ubuntu 16.04. I have a csv file and I want to add it to elasticsearch, I use logstash for that: First this is my conf file: input { file { path =\> "/home/ahmed/Desktop/PopulationGenre.csv… --- ## [Logstash JSON conversion can we control](https://discuss.elastic.co/t/logstash-json-conversion-can-we-control/107379)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 3\ **Last updated:** [November 14, 2017, 7:45am UTC](https://discuss.elastic.co/t/logstash-json-conversion-can-we-control/107379 "2017-11-14T07:45:00Z")
When logstash fetch data from mysql....then it convert into json kv form....Can I control this conversion and what field I want it make in nested json form....is this possible in logstash.... Bcz I want some fields of m… --- ## [Logstash Aggregate plugin](https://discuss.elastic.co/t/logstash-aggregate-plugin/107501)
**Author:** [@Shekhar\_Pandey](https://discuss.elastic.co/u/Shekhar_Pandey)\ **Replies:** 0\ **Last updated:** [November 14, 2017, 7:42am UTC](https://discuss.elastic.co/t/logstash-aggregate-plugin/107501 "2017-11-14T07:42:46Z")
This example from elastic documentation: Given that you have this SQL query : SELECT country\_name, town\_name FROM town { "country\_name": "France", "town\_name": "Paris" } { "country\_name": "France", "town\_name": "Marse… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=874) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=876)