# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=726 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 727 --- ## [Is it possible to get the records from last run time to now](https://discuss.elastic.co/t/is-it-possible-to-get-the-records-from-last-run-time-to-now/139094)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 2\ **Last updated:** [July 9, 2018, 7:37am UTC](https://discuss.elastic.co/t/is-it-possible-to-get-the-records-from-last-run-time-to-now/139094 "2018-07-09T07:37:21Z")
is it possible to get the records from last run time to now eg:"SELECT \* from contacts where date between 'last\_run\_time and CURRENT\_TIMESTAMP " --- ## [Enable\_auto\_commit was false for input-kafka](https://discuss.elastic.co/t/enable-auto-commit-was-false-for-input-kafka/139084)
**Author:** [@tyb1222](https://discuss.elastic.co/u/tyb1222)\ **Replies:** 0\ **Last updated:** [July 9, 2018, 6:07am UTC](https://discuss.elastic.co/t/enable-auto-commit-was-false-for-input-kafka/139084 "2018-07-09T06:07:18Z")
I use logstash-5.6.8 。Now I configured enable\_auto\_commit =fasle and wanted to commit kafka offset by myself . How should I do? --- ## [Filebeat adds a log and Logstash reports it twice to elasticsearch. (Does not include old data, only upload new newly added data repeatedly)](https://discuss.elastic.co/t/filebeat-adds-a-log-and-logstash-reports-it-twice-to-elasticsearch-does-not-include-old-data-only-upload-new-newly-added-data-repeatedly/138837)
**Author:** [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\ **Replies:** 0\ **Last updated:** [July 6, 2018, 7:24am UTC](https://discuss.elastic.co/t/filebeat-adds-a-log-and-logstash-reports-it-twice-to-elasticsearch-does-not-include-old-data-only-upload-new-newly-added-data-repeatedly/138837 "2018-07-06T07:24:54Z")
Add a new data to the filebeat detection log file，Filebeat debugging information output is as follows: 2018-07-06T15:12:21.543+0800 DEBUG \[publish\] pipeline/processor.go:291 Publish event: { "@timestamp": "2018-07-06T… --- ## [Problem with Multiple Lat/Lon fields in a CSV](https://discuss.elastic.co/t/problem-with-multiple-lat-lon-fields-in-a-csv/139066)
**Author:** [@Celtics2018](https://discuss.elastic.co/u/Celtics2018)\ **Replies:** 1\ **Last updated:** [July 9, 2018, 2:14am UTC](https://discuss.elastic.co/t/problem-with-multiple-lat-lon-fields-in-a-csv/139066 "2018-07-09T02:14:12Z")
Hello, I am trying to index a csv data set on Divvy bikeshare ride data. I want to be able to visualize the locations of the bike docks on a map, so both the location of the beginning of the rental and the location of t… --- ## [Possible bug in Kafka metadata](https://discuss.elastic.co/t/possible-bug-in-kafka-metadata/139059)
**Author:** [@crickes](https://discuss.elastic.co/u/crickes)\ **Replies:** 0\ **Last updated:** [July 8, 2018, 3:28pm UTC](https://discuss.elastic.co/t/possible-bug-in-kafka-metadata/139059 "2018-07-08T15:28:54Z")
Hi, I' have been playing around with using the Kafka input plugin and I wanted to use some of the metadata to assist with debugging some issues. I output the metadata to stdout with the metadata option set to true, and … --- ## [Add new field and updating index by changes in another index](https://discuss.elastic.co/t/add-new-field-and-updating-index-by-changes-in-another-index/137492)
**Author:** [@akapit](https://discuss.elastic.co/u/akapit)\ **Replies:** 9\ **Last updated:** [July 8, 2018, 9:55am UTC](https://discuss.elastic.co/t/add-new-field-and-updating-index-by-changes-in-another-index/137492 "2018-07-08T09:55:44Z")
Hi, I'm importing two csv files into two different indexes in elastic. One of them is business data and the other are config parameters. When importing the business data csv in elastic, I need that index to have added… --- ## [Data type can not change from float to long issue](https://discuss.elastic.co/t/data-type-can-not-change-from-float-to-long-issue/138917)
**Author:** [@ajit](https://discuss.elastic.co/u/ajit)\ **Replies:** 1\ **Last updated:** [July 7, 2018, 7:34am UTC](https://discuss.elastic.co/t/data-type-can-not-change-from-float-to-long-issue/138917 "2018-07-07T07:34:38Z")
Hi, I am using ELK 6.3.0 version. I am pushing data to ES using logstash JDBC. I have one query which is fetching 188 records in Oracle. But from logstash my 8 records getting missed. I am getting data type convert ex… --- ## [How to create a geo\_point field in logstash for a default index logstash-\*](https://discuss.elastic.co/t/how-to-create-a-geo-point-field-in-logstash-for-a-default-index-logstash/138830)
**Author:** [@George\_Cherian](https://discuss.elastic.co/u/George_Cherian)\ **Replies:** 5\ **Last updated:** [July 7, 2018, 7:13am UTC](https://discuss.elastic.co/t/how-to-create-a-geo-point-field-in-logstash-for-a-default-index-logstash/138830 "2018-07-07T07:13:37Z")
Hi How to make a field geo\_point type in logstash itself , so that I don't need to initial create a index manually with geo\_point temple and the use it in logstash , which is double work. Thanks George --- ## [Understanding SYSLOGBASE2 pattern](https://discuss.elastic.co/t/understanding-syslogbase2-pattern/138993)
**Author:** [@dss](https://discuss.elastic.co/u/dss)\ **Replies:** 0\ **Last updated:** [July 7, 2018, 1:05am UTC](https://discuss.elastic.co/t/understanding-syslogbase2-pattern/138993 "2018-07-07T01:05:58Z")
SYSLOGBASE2 (?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP\_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource}+(?: %{SYSLOGPROG}:|) Could I get a hand with the meaning of a few constructs in this pattern… --- ## [Logstash goes idle after around 20 mins](https://discuss.elastic.co/t/logstash-goes-idle-after-around-20-mins/138987)
**Author:** [@amilewalk](https://discuss.elastic.co/u/amilewalk)\ **Replies:** 0\ **Last updated:** [July 6, 2018, 10:26pm UTC](https://discuss.elastic.co/t/logstash-goes-idle-after-around-20-mins/138987 "2018-07-06T22:26:46Z")
My logstash is not staying stable. Once i start the logstash process, I can view logs flowing to kibana for around 20-30 mins max and it then stops. However I see the logstash process is still running and the last logsta… --- ## [Grok match and divide string to multiple fields](https://discuss.elastic.co/t/grok-match-and-divide-string-to-multiple-fields/138864)
**Author:** [@victor.nilsson](https://discuss.elastic.co/u/victor.nilsson)\ **Replies:** 5\ **Last updated:** [July 6, 2018, 7:28pm UTC](https://discuss.elastic.co/t/grok-match-and-divide-string-to-multiple-fields/138864 "2018-07-06T19:28:25Z")
Hi I want to match a syslog string and match into corresponding fields. Here is the message in question: Jul 06 2018 08:04:18: %ASA-6-305011: Built dynamic UDP translation from any:123.123.123.123/35439 to OUTSIDE-VRF… --- ## [Logstash exec input plugin - only some remote nodes](https://discuss.elastic.co/t/logstash-exec-input-plugin-only-some-remote-nodes/138784)
**Author:** [@MikaelLindstrom](https://discuss.elastic.co/u/MikaelLindstrom)\ **Replies:** 5\ **Last updated:** [July 6, 2018, 7:26pm UTC](https://discuss.elastic.co/t/logstash-exec-input-plugin-only-some-remote-nodes/138784 "2018-07-06T19:26:25Z")
I've been using metricbeat and filebeat and it works well and I'm trying to use exec plugin to run a command (netstat...) on certain nodes only. I have 4 different types of remote nodes I'm ingesting data from and I want… --- ## [Filtering json with embedded xml](https://discuss.elastic.co/t/filtering-json-with-embedded-xml/138955)
**Author:** [@jspeer](https://discuss.elastic.co/u/jspeer)\ **Replies:** 7\ **Last updated:** [July 6, 2018, 7:03pm UTC](https://discuss.elastic.co/t/filtering-json-with-embedded-xml/138955 "2018-07-06T19:03:29Z")
I'm pulling json messages from a jms queue, where most of them filter properly. However there are specific messages with XML payload. What would be the best way to filter this type of message? { "envCode": "RWCS"… --- ## [How to create a logstash filter for index the logs with specific word only](https://discuss.elastic.co/t/how-to-create-a-logstash-filter-for-index-the-logs-with-specific-word-only/138948)
**Author:** [@rijinmp](https://discuss.elastic.co/u/rijinmp)\ **Replies:** 3\ **Last updated:** [July 6, 2018, 4:34pm UTC](https://discuss.elastic.co/t/how-to-create-a-logstash-filter-for-index-the-logs-with-specific-word-only/138948 "2018-07-06T16:34:33Z")
I want to index only the logs contain the word "production" , anywhere in the log . if the word " production" not present in log ... i dont want to index that line --- ## [Logstash grok second row is parsed incorrectly](https://discuss.elastic.co/t/logstash-grok-second-row-is-parsed-incorrectly/138805)
**Author:** [@databeata](https://discuss.elastic.co/u/databeata)\ **Replies:** 2\ **Last updated:** [July 6, 2018, 4:09pm UTC](https://discuss.elastic.co/t/logstash-grok-second-row-is-parsed-incorrectly/138805 "2018-07-06T16:09:46Z")
The first row is parsed correctly. The second row is parsed incorrectly. What is wrong with my grok pattern? These are the two rows. 2018-07-05 17:57:11,373 ERROR Failed 2018-07-05 17:57:11,373 ERROR \[Timer-Driven Pro… --- ## [How to find missing logs in Logstash?](https://discuss.elastic.co/t/how-to-find-missing-logs-in-logstash/138947)
**Author:** [@colm](https://discuss.elastic.co/u/colm)\ **Replies:** 0\ **Last updated:** [July 6, 2018, 3:19pm UTC](https://discuss.elastic.co/t/how-to-find-missing-logs-in-logstash/138947 "2018-07-06T15:19:36Z")
I am running logstash 6.3 with 6 file inputs, applying some basic filtering and sending output to elasticsearch. Logstash reports that it is processing 20 - 50e/s. Comparing my input logs and elasticsearch output, I co… --- ## [Problem to create index in Elasticsearch:cannot be changed from type \[float\] to \[long\]:](https://discuss.elastic.co/t/problem-to-create-index-in-elasticsearch-cannot-be-changed-from-type-float-to-long/138885)
**Author:** [@Akhilesh-Tiwari](https://discuss.elastic.co/u/Akhilesh-Tiwari)\ **Replies:** 0\ **Last updated:** [July 6, 2018, 9:45am UTC](https://discuss.elastic.co/t/problem-to-create-index-in-elasticsearch-cannot-be-changed-from-type-float-to-long/138885 "2018-07-06T09:45:06Z")
Hi Team, I have an issue in pushing data into elasticsearch using my database table through logstash JDBC,some row's data is missing and throws an Exception. The Exception is is. \[2018-07-06T12:37:24,587\]\[WARN \]\[logst… --- ## [Last run reflect in db is possible or not?](https://discuss.elastic.co/t/last-run-reflect-in-db-is-possible-or-not/138895)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 10\ **Last updated:** [July 6, 2018, 1:27pm UTC](https://discuss.elastic.co/t/last-run-reflect-in-db-is-possible-or-not/138895 "2018-07-06T13:27:12Z")
i have created config file in that i have coded such that for every 2mins my db is called ,it works perfectly... and i have have column name "date(TImestamp) in db : eg:2018-07-06 11:11:19.754371", i have to overwrite … --- ## [DIRTY READs from microsoft sql server](https://discuss.elastic.co/t/dirty-reads-from-microsoft-sql-server/138881)
**Author:** [@titus](https://discuss.elastic.co/u/titus)\ **Replies:** 1\ **Last updated:** [July 6, 2018, 12:49pm UTC](https://discuss.elastic.co/t/dirty-reads-from-microsoft-sql-server/138881 "2018-07-06T12:49:08Z")
Hy guys, This was my initial concern: "I am new to elastic and logstash, I tried importing some data from sql in order to learn some stuff. I've got a table with 13k rows. After importing it to elastic using logstash … --- ## [Connection logstash to filebeat](https://discuss.elastic.co/t/connection-logstash-to-filebeat/138913)
**Author:** [@Labibme](https://discuss.elastic.co/u/Labibme)\ **Replies:** 4\ **Last updated:** [July 6, 2018, 12:58pm UTC](https://discuss.elastic.co/t/connection-logstash-to-filebeat/138913 "2018-07-06T12:58:35Z")
Hi Team, I'm looking for a way to connect a logstash to filebeat but i can't find any way. In documentation neither in internet, may be someone already tried that before. Destination is to have: kafka stream to logst… --- ## [Unknown setting 'endpoint' for s3](https://discuss.elastic.co/t/unknown-setting-endpoint-for-s3/138852)
**Author:** [@krisko](https://discuss.elastic.co/u/krisko)\ **Replies:** 1\ **Last updated:** [July 6, 2018, 12:05pm UTC](https://discuss.elastic.co/t/unknown-setting-endpoint-for-s3/138852 "2018-07-06T12:05:16Z")
Hello, I'm trying to use s3 input plug with logstash: input { s3 { access\_key\_id =\> "" secret\_access\_key =\> "" endpoint =\> "my.endpoint.de" bucket =\> "mybucket" type =\> "s3test" } } but logstash has issue wi… --- ## [Promote \`message\` json properties to root level](https://discuss.elastic.co/t/promote-message-json-properties-to-root-level/138888)
**Author:** [@mrpmorris](https://discuss.elastic.co/u/mrpmorris)\ **Replies:** 1\ **Last updated:** [July 6, 2018, 11:28am UTC](https://discuss.elastic.co/t/promote-message-json-properties-to-root-level/138888 "2018-07-06T11:28:48Z")
The information I am logging is all ending up as a JSON string in a top-level item named message. How can I promote the properties of that message entry to top-level entries? Please be aware that I currently know nothin… --- ## [How to read CSV file ... error as below](https://discuss.elastic.co/t/how-to-read-csv-file-error-as-below/138645)
**Author:** [@Raju.Chavan](https://discuss.elastic.co/u/Raju.Chavan)\ **Replies:** 4\ **Last updated:** [July 6, 2018, 10:00am UTC](https://discuss.elastic.co/t/how-to-read-csv-file-error-as-below/138645 "2018-07-06T10:00:42Z")
D:\\kibana\\latest\\logstash-6.2.2\\logstash-6.2.2\\bin\>logstash -f D:\\kibana\\latest risk\_analysis\\incidentConfigFile.conf Sending Logstash's logs to D:/kibana/latest/logstash-6.2.2/logstash-6.2.2/logs w hich is now config… --- ## [Logstash config issue for a custom log file](https://discuss.elastic.co/t/logstash-config-issue-for-a-custom-log-file/138539)
**Author:** [@kaushik.vankayala](https://discuss.elastic.co/u/kaushik.vankayala)\ **Replies:** 8\ **Last updated:** [July 6, 2018, 9:55am UTC](https://discuss.elastic.co/t/logstash-config-issue-for-a-custom-log-file/138539 "2018-07-06T09:55:14Z")
Hi There, I am a newbie to logstash and working on a grok pattern as below. I am only able to partially achieve my goal. Please help with my issue as described below; My logs look like; \[2018-06-28 11:49:41.257\] INF… --- ## [Logstash Plugin error: When I use double as datatype in mutate filter](https://discuss.elastic.co/t/logstash-plugin-error-when-i-use-double-as-datatype-in-mutate-filter/138703)
**Author:** [@rahulnama](https://discuss.elastic.co/u/rahulnama)\ **Replies:** 5\ **Last updated:** [July 6, 2018, 9:42am UTC](https://discuss.elastic.co/t/logstash-plugin-error-when-i-use-double-as-datatype-in-mutate-filter/138703 "2018-07-06T09:42:39Z")
Hello Below are the series of values i'm indexing with logstash. 0.00483871, 0.003225806, 0.006451613, 0.008064516 I'm converting it to float using mutate filter. But in kibana I see values as 0.004, 0.003, 0.006 etc. … --- ## [Logstash 数据直接发送到 es 后，index 对应的 path 不正确](https://discuss.elastic.co/t/logstash-es-index-path/138872)
**Author:** [@qiyongxiao](https://discuss.elastic.co/u/qiyongxiao)\ **Replies:** 2\ **Last updated:** [July 6, 2018, 9:20am UTC](https://discuss.elastic.co/t/logstash-es-index-path/138872 "2018-07-06T09:20:52Z")
如下面这样，我配置的处理规则为： input { file { path =\> "/data/logs/entry\_view\_storage\_api/entry\_view\_storage\_api.juejin.im.public.access.log" } } filter { grok { match =\> {"message" =\> "%{COMMONAPACHELOG}"} } mutate { remove… --- ## [Logstash jdbc input update](https://discuss.elastic.co/t/logstash-jdbc-input-update/138870)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 0\ **Last updated:** [July 6, 2018, 9:12am UTC](https://discuss.elastic.co/t/logstash-jdbc-input-update/138870 "2018-07-06T09:12:13Z")
i have postgres db from that i have to update the last run field after run execution ,,i am new to elk ,,many have suggested metadata\_path .. any suggestion r some brief idea??? --- ## [Convert EPOCH time to Time Duration](https://discuss.elastic.co/t/convert-epoch-time-to-time-duration/138857)
**Author:** [@sahadevgd](https://discuss.elastic.co/u/sahadevgd)\ **Replies:** 0\ **Last updated:** [July 6, 2018, 8:42am UTC](https://discuss.elastic.co/t/convert-epoch-time-to-time-duration/138857 "2018-07-06T08:42:58Z")
I have an application which outputs the below data in csv format. The timeStamp format is the EPOCH time. We want this to be shown as Elapsed Time in Kibana. (Example Below). Actual Output from the Application - timeSt… --- ## [Logstash how to synchronize MySQL one to many relational table data and implement nested object storage in elasticsearch?](https://discuss.elastic.co/t/logstash-how-to-synchronize-mysql-one-to-many-relational-table-data-and-implement-nested-object-storage-in-elasticsearch/138847)
**Author:** [@wm1994](https://discuss.elastic.co/u/wm1994)\ **Replies:** 0\ **Last updated:** [July 6, 2018, 8:10am UTC](https://discuss.elastic.co/t/logstash-how-to-synchronize-mysql-one-to-many-relational-table-data-and-implement-nested-object-storage-in-elasticsearch/138847 "2018-07-06T08:10:11Z")
Hi,I am a novice for Logstash,My project needs full text search, and then I chose logstash synchronous MySQL database, but I encountered some problems in the process. \*\*question：\*\*Logstash how to synchronize MySQL one t… --- ## [Problem in Logstash Starting](https://discuss.elastic.co/t/problem-in-logstash-starting/138577)
**Author:** [@Akhilesh-Tiwari](https://discuss.elastic.co/u/Akhilesh-Tiwari)\ **Replies:** 5\ **Last updated:** [July 6, 2018, 7:22am UTC](https://discuss.elastic.co/t/problem-in-logstash-starting/138577 "2018-07-06T07:22:28Z")
HI Team, Actually i am suffering a issue in pushing data from Filebeat to logstash. After trying to run the logstash,Logstash does not give any Error but PIpeLine is not Starting.Here is the output of logstash after st… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=725) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=727)