# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=433 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 434 --- ## [Grab data from mysql using more than one tracking column](https://discuss.elastic.co/t/grab-data-from-mysql-using-more-than-one-tracking-column/211477)
**Author:** [@fim](https://discuss.elastic.co/u/fim)\ **Replies:** 1\ **Last updated:** [December 11, 2019, 3:44pm UTC](https://discuss.elastic.co/t/grab-data-from-mysql-using-more-than-one-tracking-column/211477 "2019-12-11T15:44:54Z")
Hi Community I grab data from a mysql database using logstash' on a regular basis. In the mysql database the primary key is based on three columns instead of a single id column as primary key. How can I set more than … --- ## [Logstash not appending date to ES indices](https://discuss.elastic.co/t/logstash-not-appending-date-to-es-indices/211349)
**Author:** [@scriner](https://discuss.elastic.co/u/scriner)\ **Replies:** 6\ **Last updated:** [December 11, 2019, 2:31pm UTC](https://discuss.elastic.co/t/logstash-not-appending-date-to-es-indices/211349 "2019-12-11T14:31:46Z")
Hello, I'm currently working through upgrading a 2.6 ELK setup to 7.5 on a new server using what I can from the old config. The goal is to send rsyslog data from clients to Logstash. Previously, this worked (mostly) fin… --- ## [Convert String to date and replace @timestamp](https://discuss.elastic.co/t/convert-string-to-date-and-replace-timestamp/211455)
**Author:** [@ginu](https://discuss.elastic.co/u/ginu)\ **Replies:** 2\ **Last updated:** [December 11, 2019, 1:27pm UTC](https://discuss.elastic.co/t/convert-string-to-date-and-replace-timestamp/211455 "2019-12-11T13:27:15Z")
Hi all I have a custom field msg.message.timestamp which have the date in the format yyyy-MM-ddThh:mm:ss.nnnZ format eg. 2019-12-11T10:17:54.961Z I would like to use this as @timestamp. I tied a couple of things wit… --- ## [Logstash output to elasticsearch - SSL](https://discuss.elastic.co/t/logstash-output-to-elasticsearch-ssl/211305)
**Author:** [@RichardH](https://discuss.elastic.co/u/RichardH)\ **Replies:** 1\ **Last updated:** [December 11, 2019, 12:27pm UTC](https://discuss.elastic.co/t/logstash-output-to-elasticsearch-ssl/211305 "2019-12-11T12:27:07Z")
Hello, I am trying to get a Logstash output to Elasticsearch going. My output conf looks like this: output { elasticsearch { hosts =\> "https://mdbxd1veel1pr..net:9200" ssl =\> true ssl\_certificate\_verification =\> fa… --- ## [File input not reading last line (Logstash)](https://discuss.elastic.co/t/file-input-not-reading-last-line-logstash/211464)
**Author:** [@Ganesh999](https://discuss.elastic.co/u/Ganesh999)\ **Replies:** 0\ **Last updated:** [December 11, 2019, 12:17pm UTC](https://discuss.elastic.co/t/file-input-not-reading-last-line-logstash/211464 "2019-12-11T12:17:41Z")
Hi , I have been facing issue in parsing log file. As the documentation says a new line will be needed for logstash/filebeat to read the last line. i have added a new line but still it is not reading the last line. Als… --- ## [Apache log with grok](https://discuss.elastic.co/t/apache-log-with-grok/211460)
**Author:** [@onelaseth](https://discuss.elastic.co/u/onelaseth)\ **Replies:** 0\ **Last updated:** [December 11, 2019, 12:03pm UTC](https://discuss.elastic.co/t/apache-log-with-grok/211460 "2019-12-11T12:03:27Z")
Hi, I was wondering how i can structure my apache log file with grok? an example of my log file is: 93.114.45.13 - - \[17/May/2015:10:05:04 +0000\] "GET /reset.css HTTP/1.1" 200 1015 "http://www.semicomplete.com/article… --- ## [Multiple pipelines: Failed to read pipelines yaml file](https://discuss.elastic.co/t/multiple-pipelines-failed-to-read-pipelines-yaml-file/210589)
**Author:** [@WalterLeff](https://discuss.elastic.co/u/WalterLeff)\ **Replies:** 11\ **Last updated:** [December 11, 2019, 11:00am UTC](https://discuss.elastic.co/t/multiple-pipelines-failed-to-read-pipelines-yaml-file/210589 "2019-12-11T11:00:25Z")
Hi, i've managed to run elastic search with kibana and logstash with JDBC mssql driver for ingesting data. Now i'm trying to run multiple pipelines but i keep getting an error. ELK : 7.4.0 version pipeline.yml : # L… --- ## [Dissector mapping, field not found](https://discuss.elastic.co/t/dissector-mapping-field-not-found/211427)
**Author:** [@Chunbo\_Liao](https://discuss.elastic.co/u/Chunbo_Liao)\ **Replies:** 1\ **Last updated:** [December 11, 2019, 10:22am UTC](https://discuss.elastic.co/t/dissector-mapping-field-not-found/211427 "2019-12-11T10:22:13Z")
my input file as: file{ path =\> \["/home/11thone/caselog/logmonster///IPDU\*/GtpSnapshot\*.log", "/home/11thone/caselog/logmonster///IPDU\*/mmeDiameter\*.log", "/home/11thone/caselog/logmonster///IPDU\*/s1snapshot\*.log"\] … --- ## [ELK logstash config statement error for mongodb input-jdbc plugin](https://discuss.elastic.co/t/elk-logstash-config-statement-error-for-mongodb-input-jdbc-plugin/210690)
**Author:** [@Shantaram\_Tupe](https://discuss.elastic.co/u/Shantaram_Tupe)\ **Replies:** 7\ **Last updated:** [December 11, 2019, 9:48am UTC](https://discuss.elastic.co/t/elk-logstash-config-statement-error-for-mongodb-input-jdbc-plugin/210690 "2019-12-11T09:48:11Z")
Hi All I am new to ELK and MongoDB too, trying to view MongoDB data into Kibana. But but index is not getting created due to following error \[2019-12-05T16:49:14,073\]\[DEBUG\]\[org.mongodb.driver.protocol.command\]\[main\]… --- ## [Logstash unable to create custom indices in Elastisearch](https://discuss.elastic.co/t/logstash-unable-to-create-custom-indices-in-elastisearch/211269)
**Author:** [@tritri](https://discuss.elastic.co/u/tritri)\ **Replies:** 4\ **Last updated:** [December 11, 2019, 9:09am UTC](https://discuss.elastic.co/t/logstash-unable-to-create-custom-indices-in-elastisearch/211269 "2019-12-11T09:09:57Z")
I am trying to configure ELK locally on Windows. I have been able to create a pipeline, detect, parse and view logs in Kibana. I now want to define indexation conditionally in the pipeline config, but this is causing som… --- ## [Multiple sql\_last\_value logstash](https://discuss.elastic.co/t/multiple-sql-last-value-logstash/211423)
**Author:** [@sandy\_rizqi](https://discuss.elastic.co/u/sandy_rizqi)\ **Replies:** 0\ **Last updated:** [December 11, 2019, 7:03am UTC](https://discuss.elastic.co/t/multiple-sql-last-value-logstash/211423 "2019-12-11T07:03:11Z")
Hi, i've some issues in pipelining data from oracle to elasticsearch using logstash. Whereas in oracle have composite key , so i need two or more sql\_last\_value for pipelining data from oracle. Is there any feature fr… --- ## [Trying to replace ‘host’ and port field](https://discuss.elastic.co/t/trying-to-replace-host-and-port-field/211366)
**Author:** [@Arnodl](https://discuss.elastic.co/u/Arnodl)\ **Replies:** 2\ **Last updated:** [December 11, 2019, 6:56am UTC](https://discuss.elastic.co/t/trying-to-replace-host-and-port-field/211366 "2019-12-11T06:56:10Z")
I'm trying to do is remove the "host" field which is created by the "tcp" input plugin. I can not do: mutate { rename =\> { "host" =\> "\[source\]\[ip\]" } } because I use ECS, and I have fields such as: \[host\]\[hostname\] . … --- ## [Can we compare two strings with logstash or elastic search?](https://discuss.elastic.co/t/can-we-compare-two-strings-with-logstash-or-elastic-search/211406)
**Author:** [@Blason](https://discuss.elastic.co/u/Blason)\ **Replies:** 0\ **Last updated:** [December 11, 2019, 3:21am UTC](https://discuss.elastic.co/t/can-we-compare-two-strings-with-logstash-or-elastic-search/211406 "2019-12-11T03:21:45Z")
Hi Folks, Here is my use case; I have BIND logs being parsed thus extracting queries domain; however I am collecting DNS queries from end users as well using sysmon. So my query is Since I have BIND server sitting in f… --- ## [Overriding HTTP\_Poller SSL Input](https://discuss.elastic.co/t/overriding-http-poller-ssl-input/211384)
**Author:** [@RunningSmurf](https://discuss.elastic.co/u/RunningSmurf)\ **Replies:** 0\ **Last updated:** [December 10, 2019, 10:07pm UTC](https://discuss.elastic.co/t/overriding-http-poller-ssl-input/211384 "2019-12-10T22:07:38Z")
Good afternoon, I am trying to setup a pipeline using HTTP\_Poller, and would like to override the SSL requirement, but to no avail. I have tried several option found in forums, but still get the following error: PKIX … --- ## [Something wrong with my logstash.conf on the rolling index configuration... Need help](https://discuss.elastic.co/t/something-wrong-with-my-logstash-conf-on-the-rolling-index-configuration-need-help/208545)
**Author:** [@shylendran](https://discuss.elastic.co/u/shylendran)\ **Replies:** 2\ **Last updated:** [December 10, 2019, 6:15pm UTC](https://discuss.elastic.co/t/something-wrong-with-my-logstash-conf-on-the-rolling-index-configuration-need-help/208545 "2019-12-10T18:15:13Z")
Hi There, I am facing an issue on creating rolling index in one of our environment. Even if I mention the index with Date & Time, still it is not creating the index daily. I see only one index named as logstash-2019.11.… --- ## [Logstash remove\_field not working in order to upload csv to elasticsearch](https://discuss.elastic.co/t/logstash-remove-field-not-working-in-order-to-upload-csv-to-elasticsearch/211206)
**Author:** [@carlman001](https://discuss.elastic.co/u/carlman001)\ **Replies:** 0\ **Last updated:** [December 10, 2019, 12:58am UTC](https://discuss.elastic.co/t/logstash-remove-field-not-working-in-order-to-upload-csv-to-elasticsearch/211206 "2019-12-10T00:58:33Z")
I'm using elasticsearch, kibana and logstash 6.0.1. I wish to upload csv data to elasticsearch by logstash and removing fields (path, @timestamp, @version, host and message). I'm showing logstash.conf and emp.csv files … --- ## [Logstash syslog help](https://discuss.elastic.co/t/logstash-syslog-help/211186)
**Author:** [@f1wing](https://discuss.elastic.co/u/f1wing)\ **Replies:** 2\ **Last updated:** [December 10, 2019, 4:02pm UTC](https://discuss.elastic.co/t/logstash-syslog-help/211186 "2019-12-10T16:02:18Z")
Hi All, I am having problem with logstash with grok. I am successfully to use the NetScreen firewall logs %{NETSCREENSESSIONLOG} in grok debugger. But when I run it, I get grokpasefailure. I notice the log is differ… --- ## [When sending data to Logstash instead of directly to ES we get...InvalidFrameProtocolException: Invalid Frame Type, received: 83](https://discuss.elastic.co/t/when-sending-data-to-logstash-instead-of-directly-to-es-we-get-invalidframeprotocolexception-invalid-frame-type-received-83/211323)
**Author:** [@Dvorst](https://discuss.elastic.co/u/Dvorst)\ **Replies:** 0\ **Last updated:** [December 10, 2019, 3:27pm UTC](https://discuss.elastic.co/t/when-sending-data-to-logstash-instead-of-directly-to-es-we-get-invalidframeprotocolexception-invalid-frame-type-received-83/211323 "2019-12-10T15:27:31Z")
Hello, we have a problem with sending data from our own Ensemble environment to (our local) Logstash instance -\> (our local) ElasticSearch server (to port 5044) ......InvalidFrameProtocolException: Invalid Frame Type, … --- ## [Help with match date 2019-12-04 06:34:33.850000+00:00](https://discuss.elastic.co/t/help-with-match-date-2019-12-04-0633-850000-00-00/211152)
**Author:** [@LLhborda](https://discuss.elastic.co/u/LLhborda)\ **Replies:** 7\ **Last updated:** [December 10, 2019, 3:13pm UTC](https://discuss.elastic.co/t/help-with-match-date-2019-12-04-0633-850000-00-00/211152 "2019-12-10T15:13:05Z")
I have date values like this 2019-12-04 06:34:33.850000+00:00 I have trouble describing the format on "match". So far the closet I came up is "yyyy-mm-dd hh:mm:ss.sss" should I use: timezone =\> GMT locale - en-us T… --- ## [Invalid IP network, skipping {:network=\>"10.13.7.0/10.13.7.24"](https://discuss.elastic.co/t/invalid-ip-network-skipping-network-10-13-7-0-10-13-7-24/211252)
**Author:** [@Vishnu\_mk](https://discuss.elastic.co/u/Vishnu_mk)\ **Replies:** 1\ **Last updated:** [December 10, 2019, 3:06pm UTC](https://discuss.elastic.co/t/invalid-ip-network-skipping-network-10-13-7-0-10-13-7-24/211252 "2019-12-10T15:06:58Z")
input { file { path =\> "/home/a/Documents/iplist.txt" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { grok { match =\> { "message" =\> "%{IP:src\_ip}" } } … --- ## [Index not rolling over automatically each day](https://discuss.elastic.co/t/index-not-rolling-over-automatically-each-day/211207)
**Author:** [@amhulli](https://discuss.elastic.co/u/amhulli)\ **Replies:** 1\ **Last updated:** [December 10, 2019, 2:58pm UTC](https://discuss.elastic.co/t/index-not-rolling-over-automatically-each-day/211207 "2019-12-10T14:58:05Z")
Hi, I've been reading many of the similar topics here regarding creating daily indexes with the name appearing with the current date (YYYY:MM:dd) appended to the alias name. I am having difficulty getting my elasticsear… --- ## [Http poller output](https://discuss.elastic.co/t/http-poller-output/211166)
**Author:** [@raged](https://discuss.elastic.co/u/raged)\ **Replies:** 1\ **Last updated:** [December 10, 2019, 1:34pm UTC](https://discuss.elastic.co/t/http-poller-output/211166 "2019-12-10T13:34:34Z")
I have an API that returns several objects. Currently the output is getting each item and sending it individually. Is there a way to send the entire JSON from the initial http poller? I am trying to utilize the http p… --- ## [Log Parser for multiple fields and files](https://discuss.elastic.co/t/log-parser-for-multiple-fields-and-files/209218)
**Author:** [@gangireddy\_l](https://discuss.elastic.co/u/gangireddy_l)\ **Replies:** 1\ **Last updated:** [December 10, 2019, 10:38am UTC](https://discuss.elastic.co/t/log-parser-for-multiple-fields-and-files/209218 "2019-12-10T10:38:56Z")
I need to fetch data which occurs in multiple files and multiple times in the same file.which is presented in the below format. 19-10-25 Name Succ Fail Reject Thrput Response time (ms) (… --- ## [Logstash merge events](https://discuss.elastic.co/t/logstash-merge-events/211265)
**Author:** [@Ram\_29](https://discuss.elastic.co/u/Ram_29)\ **Replies:** 0\ **Last updated:** [December 10, 2019, 9:50am UTC](https://discuss.elastic.co/t/logstash-merge-events/211265 "2019-12-10T09:50:57Z")
Hi I want to merge two events. First one has few fields, which I will merge with second event using aggregate plugin. But at times, first event comes little late and second event has null values for those fields. I want… --- ## [How do i aggregate these documets](https://discuss.elastic.co/t/how-do-i-aggregate-these-documets/210874)
**Author:** [@Aayush\_Agrawal](https://discuss.elastic.co/u/Aayush_Agrawal)\ **Replies:** 5\ **Last updated:** [December 10, 2019, 9:38am UTC](https://discuss.elastic.co/t/how-do-i-aggregate-these-documets/210874 "2019-12-10T09:38:07Z")
I have logs of around thousand lines in a file, from which only three lines are useful for me. I am using grok filter to identify and filter those logs, now i want them to be in same document. How do i do that? I am run… --- ## [Logstash JDBC class cannot found](https://discuss.elastic.co/t/logstash-jdbc-class-cannot-found/209807)
**Author:** [@Gunawan\_Kevin](https://discuss.elastic.co/u/Gunawan_Kevin)\ **Replies:** 2\ **Last updated:** [December 10, 2019, 8:39am UTC](https://discuss.elastic.co/t/logstash-jdbc-class-cannot-found/209807 "2019-12-10T08:39:05Z")
Hi All, Can you help me to solve this problem I'm using elasticsearch-7.4.2 kibana-7.4.2 logstash-7.4.2 windows 10 Error: com.mysql.cj.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver i… --- ## [CSV lookup (multiple input, multiple output)](https://discuss.elastic.co/t/csv-lookup-multiple-input-multiple-output/211229)
**Author:** [@a.kuz](https://discuss.elastic.co/u/a.kuz)\ **Replies:** 0\ **Last updated:** [December 10, 2019, 6:42am UTC](https://discuss.elastic.co/t/csv-lookup-multiple-input-multiple-output/211229 "2019-12-10T06:42:09Z")
Hello. There is the way to lookup from file with multiple fields input and multiple fileds output? For example: I have csv like field name field1 field2 field3 field4 field5 field data 1 2 3 4 5 And input data… --- ## [Parsing csv and conditions](https://discuss.elastic.co/t/parsing-csv-and-conditions/210865)
**Author:** [@Khuman](https://discuss.elastic.co/u/Khuman)\ **Replies:** 1\ **Last updated:** [December 10, 2019, 6:27am UTC](https://discuss.elastic.co/t/parsing-csv-and-conditions/210865 "2019-12-10T06:27:22Z")
I'm using elastic stack 7.3.2. I need to use working logstash config on new version. This one construction work perfect on version 6.5.0, but don't work on 7.3.2 filter { csv { columns =\> \["column1","column2","column3… --- ## [I want to Index IPs in elasticsearch from my text document](https://discuss.elastic.co/t/i-want-to-index-ips-in-elasticsearch-from-my-text-document/211217)
**Author:** [@Vishnu\_mk](https://discuss.elastic.co/u/Vishnu_mk)\ **Replies:** 0\ **Last updated:** [December 10, 2019, 5:32am UTC](https://discuss.elastic.co/t/i-want-to-index-ips-in-elasticsearch-from-my-text-document/211217 "2019-12-10T05:32:18Z")
Hi, My text document contains IPs in below format: 1)10.12.7.2 - 10.7.24.25 2}10.5.44.1 - 10.5.44.255 I want to index all the IPs Lying between the IPs ranges from reading the text file --- ## [I got 'Error registering plugin', 'Could not execute action' Error](https://discuss.elastic.co/t/i-got-error-registering-plugin-could-not-execute-action-error/210952)
**Author:** [@John\_Baek](https://discuss.elastic.co/u/John_Baek)\ **Replies:** 2\ **Last updated:** [December 10, 2019, 5:27am UTC](https://discuss.elastic.co/t/i-got-error-registering-plugin-could-not-execute-action-error/210952 "2019-12-10T05:27:20Z")
Hi, everyone :slight\_smile: I'm a beginner on ELK, and I'm making a conf file that processes snort logs. when i run "./logstash -f /etc/logstash/conf.d/selk.conf", i got error \[ERROR\] 2019-12-07 16:19:57.858 \[\[main\]-p… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=432) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=434)