# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=353 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 354 --- ## [Logstash crashes after some time](https://discuss.elastic.co/t/logstash-crashes-after-some-time/231958)
**Author:** [@kagashe](https://discuss.elastic.co/u/kagashe)\ **Replies:** 2\ **Last updated:** [May 25, 2020, 11:19am UTC](https://discuss.elastic.co/t/logstash-crashes-after-some-time/231958 "2020-05-25T11:19:59Z")
Hi all, I seem to have a problem with my logstash but but not sure what's causing it. I have 2 Elasticsearch machines in my cluster, one data and one client/master. Initially I had one logstash instance running on the c… --- ## [Sync mongodb and elastic using logstash](https://discuss.elastic.co/t/sync-mongodb-and-elastic-using-logstash/234075)
**Author:** [@farshid](https://discuss.elastic.co/u/farshid)\ **Replies:** 2\ **Last updated:** [May 25, 2020, 9:17am UTC](https://discuss.elastic.co/t/sync-mongodb-and-elastic-using-logstash/234075 "2020-05-25T09:17:33Z")
I have same problem with logstash issue https://discuss.elastic.co/t/sync-mongodb-data-to-elasticsearch-update-if-exists-or-insert-using-logstash/206897/2 https://discuss.elastic.co/t/sync-mongodb-data-to-elasticsearc… --- ## [Facing Error while connecting Mariadb with elk through jdbc plugin](https://discuss.elastic.co/t/facing-error-while-connecting-mariadb-with-elk-through-jdbc-plugin/233341)
**Author:** [@Karthik\_Yadav](https://discuss.elastic.co/u/Karthik_Yadav)\ **Replies:** 5\ **Last updated:** [May 25, 2020, 8:26am UTC](https://discuss.elastic.co/t/facing-error-while-connecting-mariadb-with-elk-through-jdbc-plugin/233341 "2020-05-25T08:26:08Z")
Logstash configuration file input { jdbc { jdbc\_validate\_connection =\> true jdbc\_driver\_library =\> "/etc/java-libs/mariadb-java-client-2.4.2.jar" jdbc\_driver\_class =\> "Java::org.mariadb.jdbc.Driver" jdbc\_connection… --- ## [Accessing/comparing fields from two different input plugins in filter, simultaneously](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917)
**Author:** [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Replies:** 10\ **Last updated:** [May 25, 2020, 8:21am UTC](https://discuss.elastic.co/t/accessing-comparing-fields-from-two-different-input-plugins-in-filter-simultaneously/233917 "2020-05-25T08:21:37Z")
I want to use file input along with Elasticsearch query input and then in filter I want to drop the Elasticsearch query results that don't quite match my file input. Now here is my problem: I can't seem to access and c… --- ## [Use of variables in a query in logstash](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706)
**Author:** [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\ **Replies:** 12\ **Last updated:** [May 25, 2020, 8:20am UTC](https://discuss.elastic.co/t/use-of-variables-in-a-query-in-logstash/233706 "2020-05-25T08:20:57Z")
Hi, I'm using ELK 7.4.0. I want to use dynamic variables in logstash configuration file. Plan A: There is an input plugin called elasticsearch for querying elasticsearch, can I use input from a file and then put its o… --- ## [Failed to parse field \[host\]](https://discuss.elastic.co/t/failed-to-parse-field-host/234079)
**Author:** [@PraveenKT](https://discuss.elastic.co/u/PraveenKT)\ **Replies:** 7\ **Last updated:** [May 25, 2020, 7:29am UTC](https://discuss.elastic.co/t/failed-to-parse-field-host/234079 "2020-05-25T07:29:39Z")
I am getting below error. May I know the reason and resolve the issue. Below is the logstash config. ++++++++++++++++++++++++++=== \[2020-05-25T09:35:25,844\]\[WARN \]\[logstash.outputs.elasticsearch\]\[main\]\[17dee1c8bb9821d… --- ## [Error in parsing date field](https://discuss.elastic.co/t/error-in-parsing-date-field/233986)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 8\ **Last updated:** [May 24, 2020, 12:07pm UTC](https://discuss.elastic.co/t/error-in-parsing-date-field/233986 "2020-05-24T12:07:36Z")
Hi All, I'm trying to ingest data from database into elasticsearch, for which i have written mapping in kibana as PUT agent5 { "mappings" : { "properties": { "logged\_date": { "type": "date", … --- ## [Using certificates on logstash output plugin](https://discuss.elastic.co/t/using-certificates-on-logstash-output-plugin/233967)
**Author:** [@francisaugusto](https://discuss.elastic.co/u/francisaugusto)\ **Replies:** 4\ **Last updated:** [May 24, 2020, 11:47am UTC](https://discuss.elastic.co/t/using-certificates-on-logstash-output-plugin/233967 "2020-05-24T11:47:51Z")
Hi, Is it possible to use certificates to connect logstash to elasticsearch, the same way it is possible with the beat input plugin? Best, Francis --- ## [Fields are merge grok output](https://discuss.elastic.co/t/fields-are-merge-grok-output/233988)
**Author:** [@himalc](https://discuss.elastic.co/u/himalc)\ **Replies:** 8\ **Last updated:** [May 24, 2020, 8:18am UTC](https://discuss.elastic.co/t/fields-are-merge-grok-output/233988 "2020-05-24T08:18:20Z")
I have a log like this: 2020-05-22T11:45:21.297418 H 20 mytest.cpp:175 stdlog sql\_execute 38533 73 testdb admin 100-xxy {"query\_str","client","execution\_time\_ms","total\_time\_ms"} {"select \* from mydatabase;","tcp:l… --- ## [ERROR IN CONNECTING SQL SERVER TO ELASTIC SEARCH](https://discuss.elastic.co/t/error-in-connecting-sql-server-to-elastic-search/233982)
**Author:** [@saeem](https://discuss.elastic.co/u/saeem)\ **Replies:** 1\ **Last updated:** [May 24, 2020, 4:48am UTC](https://discuss.elastic.co/t/error-in-connecting-sql-server-to-elastic-search/233982 "2020-05-24T04:48:17Z")
CONF FILE : input { jdbc { jdbc\_connection\_string =\> "jdbc:sqlserver://HQTSTW2\\SQLEXPRESS;databaseName=sample;integratedSecurity=false;" jdbc\_user =\> "root" jdbc\_driver\_library =\> "C:/Program Files/sqljdbc\_6.2/enu/jre… --- ## [Combining Data in Multiple CSVs to Single Index](https://discuss.elastic.co/t/combining-data-in-multiple-csvs-to-single-index/233999)
**Author:** [@rukmalf](https://discuss.elastic.co/u/rukmalf)\ **Replies:** 1\ **Last updated:** [May 23, 2020, 8:14pm UTC](https://discuss.elastic.co/t/combining-data-in-multiple-csvs-to-single-index/233999 "2020-05-23T20:14:51Z")
Hi, I have 3 CSVs which I use to create 3 indexes. But the problem is the 1st and 3rd Indexes are connected via the 2nd Index (refer to the image) End of the day I need to get a search result like A1, A2, A3,B1,B2,B3.… --- ## [\[2020-05-22T09:04:47,691\]\[WARN \]\[logstash.outputs.amazonelasticsearch\] Could not index event to Elasticsearch](https://discuss.elastic.co/t/2020-05-22t0947-691-warn-logstash-outputs-amazonelasticsearch-could-not-index-event-to-elasticsearch/234011)
**Author:** [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Replies:** 1\ **Last updated:** [May 23, 2020, 7:45pm UTC](https://discuss.elastic.co/t/2020-05-22t0947-691-warn-logstash-outputs-amazonelasticsearch-could-not-index-event-to-elasticsearch/234011 "2020-05-23T19:45:49Z")
\[2020-05-22T09:04:47,691\]\[WARN \]\[logstash.outputs.amazonelasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\> \["index", {:\_id=\>nil, :\_index=\>"\*\*\*\*-2020.05.22", :\_type=\>"\_doc", :\_routing=\>nil}… --- ## [How to configure different indexes in logstash](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/58665)
**Author:** [@rocky4bmw](https://discuss.elastic.co/u/rocky4bmw)\ **Replies:** 37\ **Last updated:** [May 23, 2020, 3:07pm UTC](https://discuss.elastic.co/t/how-to-configure-different-indexes-in-logstash/58665 "2020-05-23T15:07:11Z")
Hi Team, Can anyone help me in confugiring multiple indexes so that logs are shipped to different indices based on the environment type(PROD,SIT & DEV). Currently my stepup is working with default filebeat-\* index L… --- ## [Filebeat to Logstash TLS](https://discuss.elastic.co/t/filebeat-to-logstash-tls/233990)
**Author:** [@ethical20](https://discuss.elastic.co/u/ethical20)\ **Replies:** 0\ **Last updated:** [May 23, 2020, 10:36am UTC](https://discuss.elastic.co/t/filebeat-to-logstash-tls/233990 "2020-05-23T10:36:35Z")
Hi, I'm trying to set TLS communication between filebeat and logstash. I can see the documentation here https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html but this is in case of crt and… --- ## [Types removal warning](https://discuss.elastic.co/t/types-removal-warning/233916)
**Author:** [@irishwill2008](https://discuss.elastic.co/u/irishwill2008)\ **Replies:** 9\ **Last updated:** [May 22, 2020, 9:14pm UTC](https://discuss.elastic.co/t/types-removal-warning/233916 "2020-05-22T21:14:51Z")
Hey guys, Inside "elasticsearch\_deprecation.log" i see the odd warning stating: \[types removal\] Specifying types in bulk requests is deprecated.. I want to remove this, i am running my Elastic Stack on v7.7. I looked o… --- ## [Geoip.location.lat and geoip.location.lon showing as number, cannot convert to geoip type](https://discuss.elastic.co/t/geoip-location-lat-and-geoip-location-lon-showing-as-number-cannot-convert-to-geoip-type/233475)
**Author:** [@mostafa\_kamal](https://discuss.elastic.co/u/mostafa_kamal)\ **Replies:** 10\ **Last updated:** [May 22, 2020, 6:22pm UTC](https://discuss.elastic.co/t/geoip-location-lat-and-geoip-location-lon-showing-as-number-cannot-convert-to-geoip-type/233475 "2020-05-22T18:22:10Z")
I actually want to make a user-location based map in kibana, but I cannot do that because geoip.location.lat and geoip.location.lon showing as number, I cannot convert to geoip type. this is my logstash conf filter { … --- ## [Extracting data from multiline error message](https://discuss.elastic.co/t/extracting-data-from-multiline-error-message/233935)
**Author:** [@guas5](https://discuss.elastic.co/u/guas5)\ **Replies:** 0\ **Last updated:** [May 22, 2020, 4:59pm UTC](https://discuss.elastic.co/t/extracting-data-from-multiline-error-message/233935 "2020-05-22T16:59:22Z")
Hello everyone, would be very gratefull for a hint. I have a multiline message and I need to extract some info from multiline error (Producer.scala:77): one column Producer.scala and the next one is number of error (77)… --- ## [Using XPath to Create Field Names](https://discuss.elastic.co/t/using-xpath-to-create-field-names/233892)
**Author:** [@Jordan\_Muncaster](https://discuss.elastic.co/u/Jordan_Muncaster)\ **Replies:** 4\ **Last updated:** [May 22, 2020, 3:28pm UTC](https://discuss.elastic.co/t/using-xpath-to-create-field-names/233892 "2020-05-22T15:28:05Z")
Trying to use xpath to create the field name in a filter. filter { xml { source =\> "message" store\_xml =\> false xpath =\> \[ "//XmlTag/text()", "//XmlTag/@name" … --- ## [Convert multiline csv data to single line during filter](https://discuss.elastic.co/t/convert-multiline-csv-data-to-single-line-during-filter/233741)
**Author:** [@rkhapre](https://discuss.elastic.co/u/rkhapre)\ **Replies:** 2\ **Last updated:** [May 22, 2020, 2:07pm UTC](https://discuss.elastic.co/t/convert-multiline-csv-data-to-single-line-during-filter/233741 "2020-05-22T14:07:12Z")
HI All Currently after input i get data in this format { "csvdata" : "col1","col2","col3" } { "csvdata" : "col1row1","col2row1","col3row1" } { "csvdata" : "col1row2","col2row2","col3row2" } How should i club this muti… --- ## [How JDBC input get data without schedule?](https://discuss.elastic.co/t/how-jdbc-input-get-data-without-schedule/233790)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 2\ **Last updated:** [May 22, 2020, 2:12pm UTC](https://discuss.elastic.co/t/how-jdbc-input-get-data-without-schedule/233790 "2020-05-22T14:12:17Z")
A coworker told me that you can ommit the schedule in the JDBC input, and still will get data, if this is true, how and when the SQL query get triggered? does logtash check for changes in the DB periodically? --- ## [How to edit logrotate for logstash?](https://discuss.elastic.co/t/how-to-edit-logrotate-for-logstash/233905)
**Author:** [@aerodynamic](https://discuss.elastic.co/u/aerodynamic)\ **Replies:** 0\ **Last updated:** [May 22, 2020, 1:27pm UTC](https://discuss.elastic.co/t/how-to-edit-logrotate-for-logstash/233905 "2020-05-22T13:27:41Z")
Hello, I'm trying to increase the log retention for logstashs logs. I'm unsure where I'm supposed to edit these settings. Is it it the log4j2 file? Or is it somewhere in /etc/logrotate? --- ## [GROK Pattern not working with IIS Logs](https://discuss.elastic.co/t/grok-pattern-not-working-with-iis-logs/233851)
**Author:** [@muthucse](https://discuss.elastic.co/u/muthucse)\ **Replies:** 1\ **Last updated:** [May 22, 2020, 12:32pm UTC](https://discuss.elastic.co/t/grok-pattern-not-working-with-iis-logs/233851 "2020-05-22T12:32:43Z")
Hi , Applying the below grok pattern is throwing error. How to parse the \[object+object\] at end of the url ? Sample Pattern: %{TIMESTAMP\_ISO8601:iis.access.time} %{WORD:iis.access.site\_name} %{WORD:http.request.method… --- ## [Multiple SNMP Inputs or Pipelines](https://discuss.elastic.co/t/multiple-snmp-inputs-or-pipelines/233858)
**Author:** [@Offi83](https://discuss.elastic.co/u/Offi83)\ **Replies:** 1\ **Last updated:** [May 22, 2020, 12:20pm UTC](https://discuss.elastic.co/t/multiple-snmp-inputs-or-pipelines/233858 "2020-05-22T12:20:03Z")
Hi, I need to use multiple SNMP Inputs but always running into the same problem. Whether I use one pipeline with two inputs or two pipelines with one input - I always get an error for one input like: \[2020-05-22T08:12:… --- ## [Logstash and cloud.auth not working correctly](https://discuss.elastic.co/t/logstash-and-cloud-auth-not-working-correctly/233862)
**Author:** [@clasyc](https://discuss.elastic.co/u/clasyc)\ **Replies:** 1\ **Last updated:** [May 22, 2020, 12:00pm UTC](https://discuss.elastic.co/t/logstash-and-cloud-auth-not-working-correctly/233862 "2020-05-22T12:00:49Z")
I spent a whole day trying to figure out why my logstash (running on ec2) could not connect into Elastic Cloud endpoint. Added cloud.id and cloud.auth in my logstash.yml, but every time I was getting 401 response. After… --- ## [Logstash parallel processing](https://discuss.elastic.co/t/logstash-parallel-processing/233814)
**Author:** [@andrezenun](https://discuss.elastic.co/u/andrezenun)\ **Replies:** 2\ **Last updated:** [May 22, 2020, 3:37am UTC](https://discuss.elastic.co/t/logstash-parallel-processing/233814 "2020-05-22T03:37:34Z")
Hi, I'm developing a POC in a costumer to process huge amount of csv files. So far I ingested more than 100mi of documents. I need more throughput on the processing the files and ingesting on elastic. I'm using docker an… --- ## [JVM settings for Logstash container](https://discuss.elastic.co/t/jvm-settings-for-logstash-container/233838)
**Author:** [@worapojc](https://discuss.elastic.co/u/worapojc)\ **Replies:** 0\ **Last updated:** [May 22, 2020, 2:56am UTC](https://discuss.elastic.co/t/jvm-settings-for-logstash-container/233838 "2020-05-22T02:56:38Z")
Hi Elastic team, Currently, I'm using many Logstash instances to parse the customer logs. There are around 8 instances / server with 4GB JVM settings (-Xmx 4GB -Xms 4GB). The server have CPU 16 and Memory for 64GB. In … --- ## [Basic sql to ES config](https://discuss.elastic.co/t/basic-sql-to-es-config/233770)
**Author:** [@steevhise](https://discuss.elastic.co/u/steevhise)\ **Replies:** 2\ **Last updated:** [May 21, 2020, 7:42pm UTC](https://discuss.elastic.co/t/basic-sql-to-es-config/233770 "2020-05-21T19:42:38Z")
I am trying to do just the basic configuration of trying to get some data from our mysql database into elasticsearch in order to do fulltext searches in a web application. We have the basic install done and in fact have… --- ## [Increase amount of debug logs](https://discuss.elastic.co/t/increase-amount-of-debug-logs/233766)
**Author:** [@aerodynamic](https://discuss.elastic.co/u/aerodynamic)\ **Replies:** 2\ **Last updated:** [May 21, 2020, 5:55pm UTC](https://discuss.elastic.co/t/increase-amount-of-debug-logs/233766 "2020-05-21T17:55:41Z")
Hello, I've set my logstast.yml's log.level to debug. However there are only 9 log files actually being produced missing-files/log-6/logstash-slowlog-plain.log missing-files/log-6/logstash-plain-2020-05-21-1.log.gz m… --- ## [Email output not being sent to multiple Gmail IDs](https://discuss.elastic.co/t/email-output-not-being-sent-to-multiple-gmail-ids/233776)
**Author:** [@san\_sun](https://discuss.elastic.co/u/san_sun)\ **Replies:** 0\ **Last updated:** [May 21, 2020, 4:55pm UTC](https://discuss.elastic.co/t/email-output-not-being-sent-to-multiple-gmail-ids/233776 "2020-05-21T16:55:07Z")
I have Logstash email output that is configured as: email { port =\> "587" address =\> "smtp.gmail.com" username =\> "a@gmail.com" password =\> "pass… --- ## [Identifying Unique Log Sources](https://discuss.elastic.co/t/identifying-unique-log-sources/233380)
**Author:** [@phinkel](https://discuss.elastic.co/u/phinkel)\ **Replies:** 3\ **Last updated:** [May 21, 2020, 3:26pm UTC](https://discuss.elastic.co/t/identifying-unique-log-sources/233380 "2020-05-21T15:26:50Z")
Hello, We are attempting to start collecting from our first ESXi host and getting the logs to logstash is any easy process however, our goal is to take multiple log source types (asa, cisco switch, esxi, etc.) all on th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=352) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=354)