# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=658 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 659 --- ## [Install ingest-user-agent on ELK](https://discuss.elastic.co/t/install-ingest-user-agent-on-elk/154429)
**Author:** [@rodrigue](https://discuss.elastic.co/u/rodrigue)\ **Replies:** 2\ **Last updated:** [October 29, 2018, 11:45am UTC](https://discuss.elastic.co/t/install-ingest-user-agent-on-elk/154429 "2018-10-29T11:45:15Z")
Hello, I would like, in my elk stack, to be able to know which browser my users are using. I've added the user-agent header in my logs and I send it to elastic via filebeat and logstash. Everything works fine. Now I ha… --- ## [Logstash: handle url with pipes in them when parsing leef format](https://discuss.elastic.co/t/logstash-handle-url-with-pipes-in-them-when-parsing-leef-format/154428)
**Author:** [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Replies:** 0\ **Last updated:** [October 29, 2018, 10:45am UTC](https://discuss.elastic.co/t/logstash-handle-url-with-pipes-in-them-when-parsing-leef-format/154428 "2018-10-29T10:45:14Z")
We are handling leef format logs from syslog which are presumably easy to handle with the kv fillter that breaks the fields with the pipe (|) delimiter; However the whole process breaks when the url itself containes pip… --- ## [Logstash is dying with an Exception](https://discuss.elastic.co/t/logstash-is-dying-with-an-exception/154422)
**Author:** [@Ashish\_Khare](https://discuss.elastic.co/u/Ashish_Khare)\ **Replies:** 0\ **Last updated:** [October 29, 2018, 10:10am UTC](https://discuss.elastic.co/t/logstash-is-dying-with-an-exception/154422 "2018-10-29T10:10:07Z")
Hello, I am running my logstash with 20g of jvm memory. But suddenly logstash is dying with following exception : Oct 29 09:54:11 prod-1-ap-southeast-1b-elk-node-006 logstash: \[ERROR\] 2018-10-29 09:54:11.708 \[Ruby-0-T… --- ## [How to assign the json values to fields](https://discuss.elastic.co/t/how-to-assign-the-json-values-to-fields/154418)
**Author:** [@manoj\_gundam](https://discuss.elastic.co/u/manoj_gundam)\ **Replies:** 0\ **Last updated:** [October 29, 2018, 10:01am UTC](https://discuss.elastic.co/t/how-to-assign-the-json-values-to-fields/154418 "2018-10-29T10:01:07Z")
hi my log message looks like this {"applicationName":test, "FlowName":test, "key":"EmpID", "Value":"E103", "Domain": 0,"hostname":localhost,"cluster":, "LogStatus":"InProcess", "CurrentPayload":"E103", "FlowID":null, "L… --- ## [How to define the shard settings in the logstash output configuration?](https://discuss.elastic.co/t/how-to-define-the-shard-settings-in-the-logstash-output-configuration/154407)
**Author:** [@Bharath\_Pusuluri](https://discuss.elastic.co/u/Bharath_Pusuluri)\ **Replies:** 3\ **Last updated:** [October 29, 2018, 9:40am UTC](https://discuss.elastic.co/t/how-to-define-the-shard-settings-in-the-logstash-output-configuration/154407 "2018-10-29T09:40:33Z")
The Elasticsearch and Logstash version using is 5.2.1, where can i define the shards settings in the below elasticsearch output pattern. elasticsearch { index =\> "yarn-apps-logs-%{+YYYY.MM}" document\_type =\> "do… --- ## [Http\_Poller plugin giving circular redirect "other URL"](https://discuss.elastic.co/t/http-poller-plugin-giving-circular-redirect-other-url/154410)
**Author:** [@priti](https://discuss.elastic.co/u/priti)\ **Replies:** 0\ **Last updated:** [October 29, 2018, 9:36am UTC](https://discuss.elastic.co/t/http-poller-plugin-giving-circular-redirect-other-url/154410 "2018-10-29T09:36:34Z")
Hi, I am trying to hit URL through http\_poller plugin but getting message circular redirect to login page. How can i pass this login page ? --- ## [Didn't immediately reflect or update the elasticsearch have insert last record data in table through logstash](https://discuss.elastic.co/t/didnt-immediately-reflect-or-update-the-elasticsearch-have-insert-last-record-data-in-table-through-logstash/153820)
**Author:** [@karthick\_tgi](https://discuss.elastic.co/u/karthick_tgi)\ **Replies:** 3\ **Last updated:** [October 29, 2018, 9:26am UTC](https://discuss.elastic.co/t/didnt-immediately-reflect-or-update-the-elasticsearch-have-insert-last-record-data-in-table-through-logstash/153820 "2018-10-29T09:26:29Z")
Using logstash 6.4.2 I have using the statement on select query for last insert the data in table. I run with a schedule ( schedule =\> "\* \* \* \* \*" ) for logstash instead of running,but the data is didn't update or refl… --- ## [Failed to execute action error in Logstash](https://discuss.elastic.co/t/failed-to-execute-action-error-in-logstash/154384)
**Author:** [@groverjatin17](https://discuss.elastic.co/u/groverjatin17)\ **Replies:** 2\ **Last updated:** [October 29, 2018, 9:07am UTC](https://discuss.elastic.co/t/failed-to-execute-action-error-in-logstash/154384 "2018-10-29T09:07:55Z")
Hi All, I have a logstash\_sample.conf file which have :- input { jdbc { jdbc\_connection\_string =\> "jdbc:oracle:thin:@host:port/XE" jdbc\_user =\> "username" jdbc\_password =\> "password" jdbc\_d… --- ## [Logstash-plugins/logstash-output-slack](https://discuss.elastic.co/t/logstash-plugins-logstash-output-slack/154332)
**Author:** [@Eladku](https://discuss.elastic.co/u/Eladku)\ **Replies:** 1\ **Last updated:** [October 29, 2018, 9:00am UTC](https://discuss.elastic.co/t/logstash-plugins-logstash-output-slack/154332 "2018-10-29T09:00:13Z")
So this might seem like a dumb question..... How do I install this output filter on logstash 2.3? Thanks, --- ## [Source perl script in logstash input conf file](https://discuss.elastic.co/t/source-perl-script-in-logstash-input-conf-file/153955)
**Author:** [@niraj\_pandey](https://discuss.elastic.co/u/niraj_pandey)\ **Replies:** 1\ **Last updated:** [October 29, 2018, 8:16am UTC](https://discuss.elastic.co/t/source-perl-script-in-logstash-input-conf-file/153955 "2018-10-29T08:16:36Z")
I have a perl script when I source it on my Linux shell (through the command eval a.pl) it set few env variables. Now I need to source this file with logstash input file inside exec plugin. I tried the following but it … --- ## [Use eval with exec pluging](https://discuss.elastic.co/t/use-eval-with-exec-pluging/154400)
**Author:** [@niraj\_pandey](https://discuss.elastic.co/u/niraj_pandey)\ **Replies:** 1\ **Last updated:** [October 29, 2018, 8:26am UTC](https://discuss.elastic.co/t/use-eval-with-exec-pluging/154400 "2018-10-29T08:26:36Z")
I have a perl script when I source it on my Linux shell (through the command eval a.pl ) it set few env variables. Now I need to source this file with logstash input file inside exec plugin. I tried the following but it… --- ## [What is logstash, Kibana in simple term?](https://discuss.elastic.co/t/what-is-logstash-kibana-in-simple-term/154191)
**Author:** [@inandi](https://discuss.elastic.co/u/inandi)\ **Replies:** 7\ **Last updated:** [October 29, 2018, 8:10am UTC](https://discuss.elastic.co/t/what-is-logstash-kibana-in-simple-term/154191 "2018-10-29T08:10:08Z")
I am learning to use logstash to connect ES with MySQL, JSON file. I am a bit confused. what Logstash really is. some sort of connector or something else. PS: I know its very immature question, but I want to learn the d… --- ## [Is kafka output plugin v7.2.0 officially available ..?](https://discuss.elastic.co/t/is-kafka-output-plugin-v7-2-0-officially-available/154395)
**Author:** [@nikhil1991](https://discuss.elastic.co/u/nikhil1991)\ **Replies:** 0\ **Last updated:** [October 29, 2018, 7:48am UTC](https://discuss.elastic.co/t/is-kafka-output-plugin-v7-2-0-officially-available/154395 "2018-10-29T07:48:03Z")
Just wanted to know whether kafka output plugin version 7.2.0 is official available or not..? Because on official page : https://www.elastic.co/guide/en/logstash/master/plugins-outputs-kafka.html it is showing "You are… --- ## [Elapsed\_expired\_error event is not generated](https://discuss.elastic.co/t/elapsed-expired-error-event-is-not-generated/154390)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 0\ **Last updated:** [October 29, 2018, 6:58am UTC](https://discuss.elastic.co/t/elapsed-expired-error-event-is-not-generated/154390 "2018-10-29T06:58:31Z")
hi community The elapsed\_expired\_error is not generated after waiting 300 secs here is elapsed filter config if "t1\_req" in \[tags\] or "t1\_res" in \[tags\] { elapsed { start\_tag =\> "t1\_req" end\_tag =\> "t1\_res" uniqu… --- ## [Not getting every datas in nested json filter](https://discuss.elastic.co/t/not-getting-every-datas-in-nested-json-filter/153386)
**Author:** [@Balamurali](https://discuss.elastic.co/u/Balamurali)\ **Replies:** 1\ **Last updated:** [October 29, 2018, 6:07am UTC](https://discuss.elastic.co/t/not-getting-every-datas-in-nested-json-filter/153386 "2018-10-29T06:07:49Z")
Hi Team, We are using logstash 6.4 version and not getting every data in nested JSON filter. Sample data: {"from":0,"size":1000,"query":{"bool":{"must":\[{"term":{"companyId":12484}},{"term":{"aroundid":22}},{"term":{… --- ## [Java::org.mariadb.jdbc.Driver not loaded](https://discuss.elastic.co/t/java-org-mariadb-jdbc-driver-not-loaded/154367)
**Author:** [@sally.c](https://discuss.elastic.co/u/sally.c)\ **Replies:** 1\ **Last updated:** [October 29, 2018, 3:39am UTC](https://discuss.elastic.co/t/java-org-mariadb-jdbc-driver-not-loaded/154367 "2018-10-29T03:39:54Z")
Hi, I am using Logstach to migrate data from mariadb to elasticSearch, but I always get this error, 'Error: Java::org.mariadb.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc\_driver… --- ## [Creating a single output from log file](https://discuss.elastic.co/t/creating-a-single-output-from-log-file/153900)
**Author:** [@shivabharat](https://discuss.elastic.co/u/shivabharat)\ **Replies:** 5\ **Last updated:** [October 29, 2018, 3:37am UTC](https://discuss.elastic.co/t/creating-a-single-output-from-log-file/153900 "2018-10-29T03:37:47Z")
Hi All, I would like to know if its possible to read the entire log file and then parse messages and produce a single JASON document / CSV file (single entry)? Appreciate any pointers on this. Thanks --- ## [How to tag when elapsed\_expired\_error raise](https://discuss.elastic.co/t/how-to-tag-when-elapsed-expired-error-raise/154362)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 0\ **Last updated:** [October 29, 2018, 2:32am UTC](https://discuss.elastic.co/t/how-to-tag-when-elapsed-expired-error-raise/154362 "2018-10-29T02:32:42Z")
hi community may i ask one thing. i use elapsed filter plugin to get duration time between different log types. when elapsed\_expired\_error event is generated, i want to add field or tag on it. is that possible? plz g… --- ## [Logstash 6.4.1 Conditional Filter Problem](https://discuss.elastic.co/t/logstash-6-4-1-conditional-filter-problem/154193)
**Author:** [@uher3805](https://discuss.elastic.co/u/uher3805)\ **Replies:** 2\ **Last updated:** [October 28, 2018, 8:09pm UTC](https://discuss.elastic.co/t/logstash-6-4-1-conditional-filter-problem/154193 "2018-10-28T20:09:22Z")
Hi, I am suddently confronted with this error in my ELK Installation. I have set up a filter based on the Content of an extra field I add in filebeat. I cannot get this to work, help is appreciated! Thanks, Udo /var/log/… --- ## [Logstash:ERROR\]\[logstash.config.sourceloader](https://discuss.elastic.co/t/logstash-error-logstash-config-sourceloader/154294)
**Author:** [@ralimaye](https://discuss.elastic.co/u/ralimaye)\ **Replies:** 5\ **Last updated:** [October 28, 2018, 4:42pm UTC](https://discuss.elastic.co/t/logstash-error-logstash-config-sourceloader/154294 "2018-10-28T16:42:33Z")
Not able to push the data through logstash. Getting below error. \[2018-10-27T22:00:34,851\]\[INFO \]\[logstash.modules.scaffold\] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"C:/logstash/modules/fb\_apache/con… --- ## [Urgent help for elasticsearch and logstash demo](https://discuss.elastic.co/t/urgent-help-for-elasticsearch-and-logstash-demo/154306)
**Author:** [@murali12180](https://discuss.elastic.co/u/murali12180)\ **Replies:** 3\ **Last updated:** [October 27, 2018, 11:18pm UTC](https://discuss.elastic.co/t/urgent-help-for-elasticsearch-and-logstash-demo/154306 "2018-10-27T23:18:43Z")
Hi, My CURL XPUT command works fine using its mapping however, i'm having trouble loading the same file from s3 to AWS Elasticsearch. Using logstash conf file (below), dumps the s3 json data file into "message" field in… --- ## [Customer pattern to parse specific data](https://discuss.elastic.co/t/customer-pattern-to-parse-specific-data/154303)
**Author:** [@bab](https://discuss.elastic.co/u/bab)\ **Replies:** 1\ **Last updated:** [October 27, 2018, 10:02pm UTC](https://discuss.elastic.co/t/customer-pattern-to-parse-specific-data/154303 "2018-10-27T22:02:00Z")
hey, i have a problem with my logfiles i want to create a grok pattern which can chek if this log-entry have after sev field a string starrt whit com than parse it in package. i try this pattern but not lukcy: log-entr… --- ## [CSV::MalformedCSVError: Unquoted fields do not allow \\r or \\n](https://discuss.elastic.co/t/csv-unquoted-fields-do-not-allow-r-or-n/154292)
**Author:** [@ffknob](https://discuss.elastic.co/u/ffknob)\ **Replies:** 1\ **Last updated:** [October 27, 2018, 3:47pm UTC](https://discuss.elastic.co/t/csv-unquoted-fields-do-not-allow-r-or-n/154292 "2018-10-27T15:47:14Z")
Hello, I'm getting a CSV::MalformedCSVError exception for events where the (possible) multilined field has line breaks. I think I managed to join all the event's lines in one single event (at least that's what I unders… --- ## [Logstash "sniffing" argument in the output configuration](https://discuss.elastic.co/t/logstash-sniffing-argument-in-the-output-configuration/154252)
**Author:** [@ggoodenough](https://discuss.elastic.co/u/ggoodenough)\ **Replies:** 3\ **Last updated:** [October 27, 2018, 9:41am UTC](https://discuss.elastic.co/t/logstash-sniffing-argument-in-the-output-configuration/154252 "2018-10-27T09:41:30Z")
I'm currently in the throws of implementation and come across something confusing. Hopefully one of you experts can help me out. First, my output code block which is in a separate file from the inputs: 90-ES\_output.conf … --- ## [How to forward logstash output to ES cluster and not to one host](https://discuss.elastic.co/t/how-to-forward-logstash-output-to-es-cluster-and-not-to-one-host/153816)
**Author:** [@Parvatayya\_Malimath](https://discuss.elastic.co/u/Parvatayya_Malimath)\ **Replies:** 4\ **Last updated:** [October 27, 2018, 4:11am UTC](https://discuss.elastic.co/t/how-to-forward-logstash-output-to-es-cluster-and-not-to-one-host/153816 "2018-10-27T04:11:53Z")
Hi I have to collect logs from around 20 servers(micro services are running there ), I have 3 elasticserach hosts, which are running in cluster formation. (2 masters and 1 data ). now when i trying to forward logs from … --- ## [How to Run Multiple conf on same machine?](https://discuss.elastic.co/t/how-to-run-multiple-conf-on-same-machine/154201)
**Author:** [@vikas\_gopal](https://discuss.elastic.co/u/vikas_gopal)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 8:09pm UTC](https://discuss.elastic.co/t/how-to-run-multiple-conf-on-same-machine/154201 "2018-10-26T20:09:26Z")
Hi Experts, It seems this question has been asked multiple times like here . I was aware that I can use a separate logstash folder all together and can run another conf . But this is really not acceptable in my case bec… --- ## [Help with ELK and geo\_point for Filebeat](https://discuss.elastic.co/t/help-with-elk-and-geo-point-for-filebeat/154249)
**Author:** [@Lee\_Lilleorg-Meilleu](https://discuss.elastic.co/u/Lee_Lilleorg-Meilleu)\ **Replies:** 0\ **Last updated:** [October 26, 2018, 7:01pm UTC](https://discuss.elastic.co/t/help-with-elk-and-geo-point-for-filebeat/154249 "2018-10-26T19:01:37Z")
Hi, I have an ELK setup with Filebeat -\> Redis -\> Logstash -\> Elasticsearch using the latest 6.X. I have scoured the net about the cryptic geo\_point type (that won't let me make a coordinate map) I have my indexes cre… --- ## [Never ending story: how to check and remove empty fields, arrays etc](https://discuss.elastic.co/t/never-ending-story-how-to-check-and-remove-empty-fields-arrays-etc/154243)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 0\ **Last updated:** [October 26, 2018, 5:38pm UTC](https://discuss.elastic.co/t/never-ending-story-how-to-check-and-remove-empty-fields-arrays-etc/154243 "2018-10-26T17:38:35Z")
season I season II season III season IV (thanks to season III): ruby { # Initialize the suspects once: init =\> "SuspectFields = \['tags'\]" code =\> " SuspectFields.each do |fie… --- ## [Help to parse snmp inpunt](https://discuss.elastic.co/t/help-to-parse-snmp-inpunt/154212)
**Author:** [@Vomitar](https://discuss.elastic.co/u/Vomitar)\ **Replies:** 1\ **Last updated:** [October 26, 2018, 4:01pm UTC](https://discuss.elastic.co/t/help-to-parse-snmp-inpunt/154212 "2018-10-26T16:01:56Z")
Hallo to all , i need a help to parsing an snmp input , extract some variable and put them in to a files. SNMP INPUT : { "SNMPv2-SMI::enterprises.193.110.2.666.1.1.2.1.6.10.65.67.84.73.86.65.84.73.79.78.2101.5" =\> … --- ## [Logstash config settings](https://discuss.elastic.co/t/logstash-config-settings/154194)
**Author:** [@emailshak](https://discuss.elastic.co/u/emailshak)\ **Replies:** 7\ **Last updated:** [October 26, 2018, 3:13pm UTC](https://discuss.elastic.co/t/logstash-config-settings/154194 "2018-10-26T15:13:44Z")
I have logstash 6.0 installed on a 4 CPU node with 32GB memory. I would like to improve the performance of my ingest to elasticsearch. At present i am ingesting a 8 GB file that contains 42 columns of text data. With t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=657) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=659)