# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=561 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 562 --- ## [Dump mssql data to elasticsearch](https://discuss.elastic.co/t/dump-mssql-data-to-elasticsearch/177796)
**Author:** [@Barney\_Liang](https://discuss.elastic.co/u/Barney_Liang)\ **Replies:** 0\ **Last updated:** [April 22, 2019, 5:15am UTC](https://discuss.elastic.co/t/dump-mssql-data-to-elasticsearch/177796 "2019-04-22T05:15:01Z")
Hello I want to copy mssql data to elasticsearch, I have setup the logstash in windows according the online course. And the logs show that it running no error, but when there is no relate index in easticsearch. Could … --- ## [How to parse snmptrap "message" part](https://discuss.elastic.co/t/how-to-parse-snmptrap-message-part/177789)
**Author:** [@jasony](https://discuss.elastic.co/u/jasony)\ **Replies:** 1\ **Last updated:** [April 22, 2019, 4:23am UTC](https://discuss.elastic.co/t/how-to-parse-snmptrap-message-part/177789 "2019-04-22T04:23:41Z")
hello, i’m trying to filter a field from snmptrap. as i posted below, the "message" value seems to be creepy and not clear to grok. how can i parse them. please advise. GET vcenter\_alarm\_20190422/\_search { "query": … --- ## [Logstash CSV - How to concatenate 1 column's all values into a new field?](https://discuss.elastic.co/t/logstash-csv-how-to-concatenate-1-columns-all-values-into-a-new-field/177548)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 2\ **Last updated:** [April 22, 2019, 1:02am UTC](https://discuss.elastic.co/t/logstash-csv-how-to-concatenate-1-columns-all-values-into-a-new-field/177548 "2019-04-22T01:02:07Z")
Hi team, I have below records in CSV. timestamp,module,page,userId,actionType,httpsessionId 2019-02-13 02:56:05.356,succession,talentsearch,cgrant1,scm.ts.list\_saved\_search,sid1 2019-02-13 02:56:05.358,succession,talen… --- ## [Delete field based on value with variable fieldnames](https://discuss.elastic.co/t/delete-field-based-on-value-with-variable-fieldnames/177570)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 2\ **Last updated:** [April 21, 2019, 11:33pm UTC](https://discuss.elastic.co/t/delete-field-based-on-value-with-variable-fieldnames/177570 "2019-04-21T23:33:31Z")
Hi, I was wondering if it is possible to remove a field based on the field value without knowing the field name? In my case I know all unwanted data will contain the value "0000" but the name of the field containing th… --- ## [Logstach Aggregation filter not working. Data is not displayed in nested columns](https://discuss.elastic.co/t/logstach-aggregation-filter-not-working-data-is-not-displayed-in-nested-columns/177525)
**Author:** [@aldol](https://discuss.elastic.co/u/aldol)\ **Replies:** 5\ **Last updated:** [April 21, 2019, 11:01pm UTC](https://discuss.elastic.co/t/logstach-aggregation-filter-not-working-data-is-not-displayed-in-nested-columns/177525 "2019-04-21T23:01:46Z")
I have created the following aggregation in logstach. The data is not aggregated and is displayed next to the parent columns whereas they are supposed to be displayed in the nested columns: Here is the mapping before ru… --- ## [Splitting multiple logs combined as a single message to multiple event log entries](https://discuss.elastic.co/t/splitting-multiple-logs-combined-as-a-single-message-to-multiple-event-log-entries/177756)
**Author:** [@SISA\_Nirmal\_kumar\_B](https://discuss.elastic.co/u/SISA_Nirmal_kumar_B)\ **Replies:** 0\ **Last updated:** [April 21, 2019, 2:17pm UTC](https://discuss.elastic.co/t/splitting-multiple-logs-combined-as-a-single-message-to-multiple-event-log-entries/177756 "2019-04-21T14:17:56Z")
I have installed filebeat to forward apache logs to logstash to parse and store it in elasticsearch so that i can view it in Kibana, but i'm facing the below issue Multiple logs are getting parsed as a single message - … --- ## [Logstash file input \[roblem after upgrade to JRE 1.8 211](https://discuss.elastic.co/t/logstash-file-input-roblem-after-upgrade-to-jre-1-8-211/177752)
**Author:** [@RonGros](https://discuss.elastic.co/u/RonGros)\ **Replies:** 2\ **Last updated:** [April 21, 2019, 2:12pm UTC](https://discuss.elastic.co/t/logstash-file-input-roblem-after-upgrade-to-jre-1-8-211/177752 "2019-04-21T14:12:43Z")
Hi, I updated my java runtime today from build 1.8 181 to 1.8 211 and suddenly started seeing errors. it might or might not be related, as at the same time I started using a new pipeline, but I have never had such error… --- ## [Logstash Error start error](https://discuss.elastic.co/t/logstash-error-start-error/177749)
**Author:** [@MikoMan](https://discuss.elastic.co/u/MikoMan)\ **Replies:** 1\ **Last updated:** [April 21, 2019, 12:49pm UTC](https://discuss.elastic.co/t/logstash-error-start-error/177749 "2019-04-21T12:49:25Z")
Hi there, Just tried to config logstash to work with elatstich and i got some problems. running on same linux machine i got : elasticsearch 7 kibana 7 logstash 7 while trying to start logstash i got this error: \[… --- ## [Sending data to cloud Logstash pipeline](https://discuss.elastic.co/t/sending-data-to-cloud-logstash-pipeline/177688)
**Author:** [@rcoundon](https://discuss.elastic.co/u/rcoundon)\ **Replies:** 2\ **Last updated:** [April 20, 2019, 10:35pm UTC](https://discuss.elastic.co/t/sending-data-to-cloud-logstash-pipeline/177688 "2019-04-20T22:35:32Z")
I have setup a Logstash pipeline in Elastic Cloud but I can't find any information on how to send data to it e.g. URL, port etc. Please can someone point me at any docs or let me know? --- ## [Ignor incorrect json format (jsonparsefailure)](https://discuss.elastic.co/t/ignor-incorrect-json-format-jsonparsefailure/177728)
**Author:** [@arp220](https://discuss.elastic.co/u/arp220)\ **Replies:** 3\ **Last updated:** [April 20, 2019, 9:25pm UTC](https://discuss.elastic.co/t/ignor-incorrect-json-format-jsonparsefailure/177728 "2019-04-20T21:25:57Z")
Hi, I send a log with rsyslog to Kafka and my rsyslog config incorrect, so my json is wrong in Kafka topic. now, I want to consume with logstash and i want ignor all incorrect json message and only read correct json form… --- ## [Multiple separators when importing CSV files in logstash](https://discuss.elastic.co/t/multiple-separators-when-importing-csv-files-in-logstash/177722)
**Author:** [@flavienbwk](https://discuss.elastic.co/u/flavienbwk)\ **Replies:** 3\ **Last updated:** [April 20, 2019, 6:50pm UTC](https://discuss.elastic.co/t/multiple-separators-when-importing-csv-files-in-logstash/177722 "2019-04-20T18:50:40Z")
Hello, I have a bunch of CSV files I want to import with each, data of this format : first\_name;last\_name The problem is that sometimes, the format will be : first\_name:last\_name (the ; is :) Is there a way to provi… --- ## [Logstash performance with stdout](https://discuss.elastic.co/t/logstash-performance-with-stdout/177724)
**Author:** [@flavienbwk](https://discuss.elastic.co/u/flavienbwk)\ **Replies:** 0\ **Last updated:** [April 20, 2019, 6:13pm UTC](https://discuss.elastic.co/t/logstash-performance-with-stdout/177724 "2019-04-20T18:13:57Z")
Hello, I am importing a large set of data through multiple files with logstash, and I've parametrized the logstash configuration so all the data imported are redirected to stdout. But I've seen in previous development … --- ## [Match log.file.path in Grok](https://discuss.elastic.co/t/match-log-file-path-in-grok/177682)
**Author:** [@Gelinski](https://discuss.elastic.co/u/Gelinski)\ **Replies:** 2\ **Last updated:** [April 20, 2019, 4:31pm UTC](https://discuss.elastic.co/t/match-log-file-path-in-grok/177682 "2019-04-20T16:31:48Z")
How can i get info from log.file.path on grok to input in other custom field? log.file.path /app/prd1bea/logs/domselftst1prd/mng7node/access.log Grok (don't working): match =\> { "log.file.path" =\> "%{GREEDYDATA:teste}… --- ## [While reading one CSV file i want to lookup another CSV file for required data and that data i need to add it into my log](https://discuss.elastic.co/t/while-reading-one-csv-file-i-want-to-lookup-another-csv-file-for-required-data-and-that-data-i-need-to-add-it-into-my-log/177695)
**Author:** [@Kishore\_Kumar1](https://discuss.elastic.co/u/Kishore_Kumar1)\ **Replies:** 1\ **Last updated:** [April 20, 2019, 1:15pm UTC](https://discuss.elastic.co/t/while-reading-one-csv-file-i-want-to-lookup-another-csv-file-for-required-data-and-that-data-i-need-to-add-it-into-my-log/177695 "2019-04-20T13:15:01Z")
any one has any idea? example :: one.csv sno, name, address 1,kishore,dvk 2,rajesh,hyd 3,geetha,nlg two.csv sno, phoneNumber 1,45678 2,59787 3,78974 output ::: 1, kishore, dvk,45678 2,rajesh,hyd,59787 3,ge… --- ## [Recursive not working in kv filter in logstash](https://discuss.elastic.co/t/recursive-not-working-in-kv-filter-in-logstash/177706)
**Author:** [@Smack\_Alpha](https://discuss.elastic.co/u/Smack_Alpha)\ **Replies:** 1\ **Last updated:** [April 20, 2019, 1:13pm UTC](https://discuss.elastic.co/t/recursive-not-working-in-kv-filter-in-logstash/177706 "2019-04-20T13:13:11Z")
I want to know about the use of recursive function in kv filter. I am using a csv file. I uploaded the file to ES using logstash. After reading the guide from this link https://www.elastic.co/guide/en/logstash/current/p… --- ## [Upgrading to 7.0](https://discuss.elastic.co/t/upgrading-to-7-0/177712)
**Author:** [@peter1992](https://discuss.elastic.co/u/peter1992)\ **Replies:** 0\ **Last updated:** [April 20, 2019, 12:18pm UTC](https://discuss.elastic.co/t/upgrading-to-7-0/177712 "2019-04-20T12:18:11Z")
We currently have version 6.0.0 of the elk stack which is coming eol mid may. We are looking at the possibility of upgrading to v7. We do have some fields that are not in version 7 (type). Do anyone have any suggestions … --- ## [Mutate split not working as expected](https://discuss.elastic.co/t/mutate-split-not-working-as-expected/177710)
**Author:** [@Srikanth\_Reddy](https://discuss.elastic.co/u/Srikanth_Reddy)\ **Replies:** 1\ **Last updated:** [April 20, 2019, 11:17am UTC](https://discuss.elastic.co/t/mutate-split-not-working-as-expected/177710 "2019-04-20T11:17:35Z")
Hi, I am trying to split message by new new line mutate { copy =\> { "message" =\> "message\_val" } } mutate { split =\> { "message\_val" =\> "\\n" } add\_field =\> { "fullQuery" =\> "%{message\_val\[4\]}" } } the output is … --- ## [OutOfDirectMemoryError Crush loop](https://discuss.elastic.co/t/outofdirectmemoryerror-crush-loop/177611)
**Author:** [@luminance](https://discuss.elastic.co/u/luminance)\ **Replies:** 0\ **Last updated:** [April 19, 2019, 11:30am UTC](https://discuss.elastic.co/t/outofdirectmemoryerror-crush-loop/177611 "2019-04-19T11:30:05Z")
I keep facing these error right after couple of minutes of running and watch how the actual mem usage of the logstash container explodes and hangs like that until I restart it manually. \[2019-04-19T11:11:19,778\]\[INFO \]… --- ## [IF Statement With RegEx](https://discuss.elastic.co/t/if-statement-with-regex/173538)
**Author:** [@martb](https://discuss.elastic.co/u/martb)\ **Replies:** 4\ **Last updated:** [April 20, 2019, 7:16am UTC](https://discuss.elastic.co/t/if-statement-with-regex/173538 "2019-04-20T07:16:57Z")
Hi, I'm using an IF statement with RegEx but it's never true and seems to skip the whole IF statement. It doesn't get a far as "\[message\] =~ /(\\QChanges\\E)/" so I assume that probably doesn't work either as the first c… --- ## [Logstash Error due to libyaml](https://discuss.elastic.co/t/logstash-error-due-to-libyaml/177685)
**Author:** [@johndowe](https://discuss.elastic.co/u/johndowe)\ **Replies:** 0\ **Last updated:** [April 20, 2019, 6:33am UTC](https://discuss.elastic.co/t/logstash-error-due-to-libyaml/177685 "2019-04-20T06:33:20Z")
I have setup logstash together with Filebeat on Raspberry Pi and found another stumbling block: Apr 19 13:49:32 raspberrypi logstash\[22454\]: org/jruby/RubyKernel.java:984: warning: It seems your ruby installation is mis… --- ## [Mutate Convert XML Multiple Layers](https://discuss.elastic.co/t/mutate-convert-xml-multiple-layers/177678)
**Author:** [@MarkBeach](https://discuss.elastic.co/u/MarkBeach)\ **Replies:** 1\ **Last updated:** [April 20, 2019, 12:25am UTC](https://discuss.elastic.co/t/mutate-convert-xml-multiple-layers/177678 "2019-04-20T00:25:00Z")
I'm running into an issue trying to use mutate to convert and XML file. I can convert the first level, but if the XML file has multiple elements then they output as string. Logstash Configuration filter { if \[fiel… --- ## [Shibboleth - Log Improvements, looking for feedback|suggestions](https://discuss.elastic.co/t/shibboleth-log-improvements-looking-for-feedback-suggestions/177671)
**Author:** [@Dr\_Natas](https://discuss.elastic.co/u/Dr_Natas)\ **Replies:** 2\ **Last updated:** [April 19, 2019, 9:53pm UTC](https://discuss.elastic.co/t/shibboleth-log-improvements-looking-for-feedback-suggestions/177671 "2019-04-19T21:53:20Z")
Looking for some feedback and/or improvement notes. The only way for me to get my shib logs is to do a rsync of my shib logs. This caused an issue that would cause my logs to be re-read causing duplicate entries for 24 h… --- ## [Http\_poller and dynamic time variable](https://discuss.elastic.co/t/http-poller-and-dynamic-time-variable/177663)
**Author:** [@lgwapnitsky](https://discuss.elastic.co/u/lgwapnitsky)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 7:47pm UTC](https://discuss.elastic.co/t/http-poller-and-dynamic-time-variable/177663 "2019-04-19T19:47:25Z")
I have need to pull from an API where the time variables (start and end) need to be specified for a specific range in UTC format. What is the proper way to pull these from the system, as they are not environmental varia… --- ## [Logstash not loading jdbc\_driver\_class](https://discuss.elastic.co/t/logstash-not-loading-jdbc-driver-class/177661)
**Author:** [@Dede\_Pessu](https://discuss.elastic.co/u/Dede_Pessu)\ **Replies:** 0\ **Last updated:** [April 19, 2019, 7:01pm UTC](https://discuss.elastic.co/t/logstash-not-loading-jdbc-driver-class/177661 "2019-04-19T19:01:11Z")
\-I have the following configuration in my jdbc pipeline input { jdbc { jdbc\_driver\_library =\> "/usr/share/logstash/mysql-connector-java-5.1.46.jar" jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" … --- ## [How to rectify this error?](https://discuss.elastic.co/t/how-to-rectify-this-error/177588)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 5\ **Last updated:** [April 19, 2019, 5:26pm UTC](https://discuss.elastic.co/t/how-to-rectify-this-error/177588 "2019-04-19T17:26:17Z")
\[INFO \] 2019-04-19 14:44:01.440 \[\[apachelogs\]\>worker10\] elasticsearch - retrying failed action with response code: 429 ({"type"=\>"es\_rejected\_execution\_exception", "reason"=\>"rejected execution of processing of \[17928\]\[i… --- ## [Logstash failing with large pretty JSON files](https://discuss.elastic.co/t/logstash-failing-with-large-pretty-json-files/176110)
**Author:** [@John\_Michael\_Burke](https://discuss.elastic.co/u/John_Michael_Burke)\ **Replies:** 9\ **Last updated:** [April 19, 2019, 4:28pm UTC](https://discuss.elastic.co/t/logstash-failing-with-large-pretty-json-files/176110 "2019-04-19T16:28:10Z")
I am getting a failure in my stack... Filebeats -\> Logstash (Filter JSON) -\> Elasticsearch. My JSON files are valid as I have checked on jsonlint.com (I get 'Valid JSON'). My JSON files are roughly ~1300 lines. I have a… --- ## [Can a conf string be passed into logstash instead of a file location?](https://discuss.elastic.co/t/can-a-conf-string-be-passed-into-logstash-instead-of-a-file-location/177632)
**Author:** [@snowboardcto](https://discuss.elastic.co/u/snowboardcto)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 3:48pm UTC](https://discuss.elastic.co/t/can-a-conf-string-be-passed-into-logstash-instead-of-a-file-location/177632 "2019-04-19T15:48:04Z")
Can a string be passed into logstash that represents a conf file? I'd like to dynamically create a conf on demand. I know we can bin/logstash -f test.conf, but I'd like to know if we can pass in a string instead of ref… --- ## [Multiple consecutive lines with no unique identifier](https://discuss.elastic.co/t/multiple-consecutive-lines-with-no-unique-identifier/177600)
**Author:** [@agiorgi](https://discuss.elastic.co/u/agiorgi)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 3:44pm UTC](https://discuss.elastic.co/t/multiple-consecutive-lines-with-no-unique-identifier/177600 "2019-04-19T15:44:24Z")
Hello everyone! This is my first post here on elastic fourm :smile: so please excuse me if i'm breaking any rules... I'm trying to parse a log file from a message broker. The logs are very heterogeneus, and have no uni… --- ## [Problem with multiple aggregate function](https://discuss.elastic.co/t/problem-with-multiple-aggregate-function/177583)
**Author:** [@Alankrit\_Aman\_Mishra](https://discuss.elastic.co/u/Alankrit_Aman_Mishra)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 3:22pm UTC](https://discuss.elastic.co/t/problem-with-multiple-aggregate-function/177583 "2019-04-19T15:22:20Z")
Hello, I have logs that have a similar format. INFO - 98765 - http\_Method - POST WARN - 12345 - latency - 55 ERROR - 12345 - status\_code - 200 WARN - 98765 - latency - 99 XYZ - 12345 - country - XYZ ERROR - 98765 … --- ## [Grok not parsit](https://discuss.elastic.co/t/grok-not-parsit/177607)
**Author:** [@darksmoke](https://discuss.elastic.co/u/darksmoke)\ **Replies:** 1\ **Last updated:** [April 19, 2019, 3:07pm UTC](https://discuss.elastic.co/t/grok-not-parsit/177607 "2019-04-19T15:07:22Z")
I have PostgreSQL logs in stderr format (csv format is not possible to do). The message does not fall into the grok. I think that will fall into the last rule where the tag "foo5" 2019-04-01 06:59:22 EEST \[3103294\]: d… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=560) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=562)