# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=460 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 461 --- ## [Unable to create index per day](https://discuss.elastic.co/t/unable-to-create-index-per-day/203582)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 4\ **Last updated:** [October 18, 2019, 6:21pm UTC](https://discuss.elastic.co/t/unable-to-create-index-per-day/203582 "2019-10-18T18:21:33Z")
Hi, my output config is: output { elasticsearch { hosts =\> \["localhost"\] manage\_template =\> true index =\> "logstash-%{+YYYY.MM.dd}" } } It creates a index like "logstash-2019.10.14-000001" and doesn't create new … --- ## [Import data vs. Manage index / index pattern](https://discuss.elastic.co/t/import-data-vs-manage-index-index-pattern/204245)
**Author:** [@Alexandre.Bernier](https://discuss.elastic.co/u/Alexandre.Bernier)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 3:10pm UTC](https://discuss.elastic.co/t/import-data-vs-manage-index-index-pattern/204245 "2019-10-18T15:10:29Z")
Hi, I don't know if I'm in the good section of the forum and I'm sorry about that but I need help to setup my import pattern. We actually import data daily at 6 AM using Logstash (7.3) into Elasticsearch (7.4) in the i… --- ## [Import double array data from csv file](https://discuss.elastic.co/t/import-double-array-data-from-csv-file/203982)
**Author:** [@newkate12](https://discuss.elastic.co/u/newkate12)\ **Replies:** 2\ **Last updated:** [October 18, 2019, 1:50pm UTC](https://discuss.elastic.co/t/import-double-array-data-from-csv-file/203982 "2019-10-18T13:50:19Z")
Hello I'm new to kibana and I've got two questions about visualizing double array data. I have a csv file including time series sensor data which consists of double array. timestamp | sensor\_id | sensor\_data 2019-10-1… --- ## [Strip part of message Logstash](https://discuss.elastic.co/t/strip-part-of-message-logstash/204232)
**Author:** [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 1:12pm UTC](https://discuss.elastic.co/t/strip-part-of-message-logstash/204232 "2019-10-18T13:12:00Z")
Hi, I am trying to strip part of message, with below configuration but getting mutate error, could you please check what I am doing wrong. Message \[\:\ \:\,\:\ \ **Author:** [@cperzrt10](https://discuss.elastic.co/u/cperzrt10)\ **Replies:** 3\ **Last updated:** [October 18, 2019, 12:48pm UTC](https://discuss.elastic.co/t/set-geoloacation-variable-type-in-json/204200 "2019-10-18T12:48:07Z") I need to convert data type in logstash to send it to kibana , right now kibana recieve it like long type y want to convert into geolocation to use kibana´s map i have this : { "varname":\[ { "n… --- ## [Keep Elasticsearch synchronized with database using Logstash and JDBC](https://discuss.elastic.co/t/keep-elasticsearch-synchronized-with-database-using-logstash-and-jdbc/204223)
**Author:** [@MAvD](https://discuss.elastic.co/u/MAvD)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 12:22pm UTC](https://discuss.elastic.co/t/keep-elasticsearch-synchronized-with-database-using-logstash-and-jdbc/204223 "2019-10-18T12:22:03Z")
Hello, Currently, we are updating our data in Elasticsearch by uploading a lot of data once in a while from a relational database which uses quite some data. We want this this to be synchronized by uploading only new an… --- ## [How to change @timestamp field from UTC to local times](https://discuss.elastic.co/t/how-to-change-timestamp-field-from-utc-to-local-times/93329)
**Author:** [@Thanadol\_Thadasade](https://discuss.elastic.co/u/Thanadol_Thadasade)\ **Replies:** 7\ **Last updated:** [October 18, 2019, 12:01pm UTC](https://discuss.elastic.co/t/how-to-change-timestamp-field-from-utc-to-local-times/93329 "2019-10-18T12:01:05Z")
Hi, @magnusbaeck. I having problems with output beats, because output beats doesn't match with my local times. Example output from metricbeat 2017-07-17T03:52:38.878Z And my local times. 2017-07-17, 10.52 Can I… --- ## [Remove commas from integer filter](https://discuss.elastic.co/t/remove-commas-from-integer-filter/204148)
**Author:** [@sud0](https://discuss.elastic.co/u/sud0)\ **Replies:** 1\ **Last updated:** [October 18, 2019, 11:57am UTC](https://discuss.elastic.co/t/remove-commas-from-integer-filter/204148 "2019-10-18T11:57:00Z")
Hi guys. I am filtering customer id using the below: (%{NUMBER:customer-id:int} On my Kibana dashboard, I see it with , and I would like to take the commas out. Example: \*\*#\*\* customer-id 389,542,059 Expected: \*\*#… --- ## [Grokparse error](https://discuss.elastic.co/t/grokparse-error/204171)
**Author:** [@yash\_kumar](https://discuss.elastic.co/u/yash_kumar)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 6:22am UTC](https://discuss.elastic.co/t/grokparse-error/204171 "2019-10-18T06:22:49Z")
I am using logstash configuration file to read log file and then extract pattern: \[ERROR\] 2000023:Making status off as an error \*\*\*Objective: \*\*\*: to extract pattern whether pattern is there in file or not and if pat… --- ## [Create Logstash output files in a docker image](https://discuss.elastic.co/t/create-logstash-output-files-in-a-docker-image/204195)
**Author:** [@basavyr](https://discuss.elastic.co/u/basavyr)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 9:39am UTC](https://discuss.elastic.co/t/create-logstash-output-files-in-a-docker-image/204195 "2019-10-18T09:39:34Z")
I'm trying to integrate my Logstash development project with Docker (I am quite new to Docker as well, so my knowledge of Docker is quite basic). I am able to create an image with a simple Dockerfile and then successful… --- ## [\[Logstash7.0.1\] lError occurs when connecting to database](https://discuss.elastic.co/t/logstash7-0-1-lerror-occurs-when-connecting-to-database/202630)
**Author:** [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)\ **Replies:** 8\ **Last updated:** [October 18, 2019, 9:32am UTC](https://discuss.elastic.co/t/logstash7-0-1-lerror-occurs-when-connecting-to-database/202630 "2019-10-18T09:32:33Z")
Hi Team, I have below logstash configuration file by trying to connect to SAP HANA database(HANA DB and elk stack are not on same laptop). But there's error when executing logstash -f ../config/logstashJDBC.conf. inpu… --- ## [Configure fortigate](https://discuss.elastic.co/t/configure-fortigate/203603)
**Author:** [@Asmaa\_Sarih](https://discuss.elastic.co/u/Asmaa_Sarih)\ **Replies:** 4\ **Last updated:** [October 18, 2019, 8:28am UTC](https://discuss.elastic.co/t/configure-fortigate/203603 "2019-10-18T08:28:13Z")
How to configure fortigate to send logs to logstash ? --- ## [I want to insert json file via api into client url](https://discuss.elastic.co/t/i-want-to-insert-json-file-via-api-into-client-url/204168)
**Author:** [@elk3](https://discuss.elastic.co/u/elk3)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 5:47am UTC](https://discuss.elastic.co/t/i-want-to-insert-json-file-via-api-into-client-url/204168 "2019-10-18T05:47:48Z")
input { file { type =\> "json" start\_position=\>"beginning" path =\> "C:/softwARE/ELK/logstash-7.4.0/logstash-7.4.0/bin/api/API Load/LocalEducationAgency.json" sincedb\_path=\>"nul" } } filter{ } output { h… --- ## [Wondering that can we lookup on a list to remove the field not on the list](https://discuss.elastic.co/t/wondering-that-can-we-lookup-on-a-list-to-remove-the-field-not-on-the-list/204018)
**Author:** [@Izek](https://discuss.elastic.co/u/Izek)\ **Replies:** 2\ **Last updated:** [October 18, 2019, 2:19am UTC](https://discuss.elastic.co/t/wondering-that-can-we-lookup-on-a-list-to-remove-the-field-not-on-the-list/204018 "2019-10-18T02:19:37Z")
Hi folks, We have an index with so many data form different application. we are currently looking for a way like schema restriction and not allow people to generate new field. Is there any way to use logstash to looku… --- ## [Multiple Instances of Logstash JDBC input](https://discuss.elastic.co/t/multiple-instances-of-logstash-jdbc-input/204147)
**Author:** [@bnered](https://discuss.elastic.co/u/bnered)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 1:00am UTC](https://discuss.elastic.co/t/multiple-instances-of-logstash-jdbc-input/204147 "2019-10-18T01:00:16Z")
My Requirement is to fetch updated/new records from the table. I used the sql\_last\_value to fetch the latest the records. I want to run logstash in multiple instances to avoid os/hardware crashes and get continuous updat… --- ## [Logstash implementation API](https://discuss.elastic.co/t/logstash-implementation-api/204144)
**Author:** [@Brendow\_Adriel](https://discuss.elastic.co/u/Brendow_Adriel)\ **Replies:** 0\ **Last updated:** [October 18, 2019, 12:36am UTC](https://discuss.elastic.co/t/logstash-implementation-api/204144 "2019-10-18T00:36:53Z")
Hello guys, someone have a example what how to implements using logstash, an java api that sending data from a database on elasticsearch? --- ## [Multiple Databases and Indexes Logstash](https://discuss.elastic.co/t/multiple-databases-and-indexes-logstash/204129)
**Author:** [@gucarletto](https://discuss.elastic.co/u/gucarletto)\ **Replies:** 0\ **Last updated:** [October 17, 2019, 8:56pm UTC](https://discuss.elastic.co/t/multiple-databases-and-indexes-logstash/204129 "2019-10-17T20:56:49Z")
I am using Logstash to sync 4 tables from Mysql to Elasticsearch. My problem is how to scale this to 200 hundred customers. Each customer is in a different database schema and possibly other server too. Every customer sh… --- ## [Ruby filter code for parsing JSON column sometimes insert values twice](https://discuss.elastic.co/t/ruby-filter-code-for-parsing-json-column-sometimes-insert-values-twice/204032)
**Author:** [@xibal](https://discuss.elastic.co/u/xibal)\ **Replies:** 3\ **Last updated:** [October 17, 2019, 8:11pm UTC](https://discuss.elastic.co/t/ruby-filter-code-for-parsing-json-column-sometimes-insert-values-twice/204032 "2019-10-17T20:11:33Z")
Hi! I'm importing a large (20M rows) table into elasticsearch and sometimes these rows have a column notes that is either empty or holding a JSON string. As this is a legacy app, this notes column may hold two differen… --- ## [GeoIP - locate ip address from text file](https://discuss.elastic.co/t/geoip-locate-ip-address-from-text-file/204039)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 2\ **Last updated:** [October 17, 2019, 7:04pm UTC](https://discuss.elastic.co/t/geoip-locate-ip-address-from-text-file/204039 "2019-10-17T19:04:22Z")
Hey guys, I was wondering if I can see location of IP address when insert an IP address from text file via logstash with filter for geoip in it. I was trying it with Apache log line sample from GeoIP in the Elastic St… --- ## [Netflow from Watchguard](https://discuss.elastic.co/t/netflow-from-watchguard/204109)
**Author:** [@Vladx](https://discuss.elastic.co/u/Vladx)\ **Replies:** 0\ **Last updated:** [October 17, 2019, 5:44pm UTC](https://discuss.elastic.co/t/netflow-from-watchguard/204109 "2019-10-17T17:44:32Z")
Hi, I'm trying to use logstash as netflow collector to forward netflow to Qradar. It is pretty straightforward and I think my config is ok #this is for netflow udp { port =\> 2081 type =\> netflow codec =\> netflow { … --- ## [Single key (field) with multiple values to multiple key (fields) with single fields convertion](https://discuss.elastic.co/t/single-key-field-with-multiple-values-to-multiple-key-fields-with-single-fields-convertion/204090)
**Author:** [@Andres\_Man](https://discuss.elastic.co/u/Andres_Man)\ **Replies:** 2\ **Last updated:** [October 17, 2019, 3:50pm UTC](https://discuss.elastic.co/t/single-key-field-with-multiple-values-to-multiple-key-fields-with-single-fields-convertion/204090 "2019-10-17T15:50:41Z")
I need to split a log into multiple fields however the log has the same field name for multiple values and when I use the fiedl\_split option of kv it will put all the the fields with the same key in the same field. How c… --- ## [Logstash adding date at the beginning of the each record](https://discuss.elastic.co/t/logstash-adding-date-at-the-beginning-of-the-each-record/203936)
**Author:** [@sekharvijju](https://discuss.elastic.co/u/sekharvijju)\ **Replies:** 1\ **Last updated:** [October 17, 2019, 3:44pm UTC](https://discuss.elastic.co/t/logstash-adding-date-at-the-beginning-of-the-each-record/203936 "2019-10-17T15:44:32Z")
Hi Logstash gurus, I m trying to setup a basic logstash client to push server logs into S3 bucket using Input plugin & s3 output plugin. input { file { path =\> "/logs/sys\*" exclude =\> "\*.tar" start\_position =\> "beg… --- ## [Logstash: installing plugin results in certificate verify failed error](https://discuss.elastic.co/t/logstash-installing-plugin-results-in-certificate-verify-failed-error/204076)
**Author:** [@an.schall](https://discuss.elastic.co/u/an.schall)\ **Replies:** 1\ **Last updated:** [October 17, 2019, 3:31pm UTC](https://discuss.elastic.co/t/logstash-installing-plugin-results-in-certificate-verify-failed-error/204076 "2019-10-17T15:31:51Z")
Hi, I am behind a corporate firewall with SSL interception. As many people here in this forum, I receive an SSL-related error when installing a logstash plugin: root@kali:/etc/ssl/certs# /usr/share/logstash/bin/logstash… --- ## [Writing grok filter which ignores some field and captures time and date](https://discuss.elastic.co/t/writing-grok-filter-which-ignores-some-field-and-captures-time-and-date/204028)
**Author:** [@Aayush\_Agrawal](https://discuss.elastic.co/u/Aayush_Agrawal)\ **Replies:** 1\ **Last updated:** [October 17, 2019, 3:12pm UTC](https://discuss.elastic.co/t/writing-grok-filter-which-ignores-some-field-and-captures-time-and-date/204028 "2019-10-17T15:12:37Z")
I am trying to capture start time and end time for this log, but every time i am having different errors. Kindly help. Offline batch Start Time -\>Oct 8, 2019 11:47:33 PM, End time -\>Oct 8, 2019 11:49:43 PM --- ## [Logstash elasticsearch input with \_source disabled](https://discuss.elastic.co/t/logstash-elasticsearch-input-with-source-disabled/204034)
**Author:** [@raviep](https://discuss.elastic.co/u/raviep)\ **Replies:** 2\ **Last updated:** [October 17, 2019, 12:24pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-input-with-source-disabled/204034 "2019-10-17T12:24:49Z")
Hi, I'm using logstash elastic search input plugin to export elastic search indexes into csv file. Please note that \_source is disabled for indices. The issue here is export works only when \_source is enabled otherwise … --- ## [ParserError with Code 111](https://discuss.elastic.co/t/parsererror-with-code-111/204044)
**Author:** [@akihu](https://discuss.elastic.co/u/akihu)\ **Replies:** 0\ **Last updated:** [October 17, 2019, 12:00pm UTC](https://discuss.elastic.co/t/parsererror-with-code-111/204044 "2019-10-17T12:00:54Z")
Dear Community, I've got server sending following json data via filebeat to logstash. Logstash version is 6.2.4. For some unknown reason to me, logstash returns following error during json filtering: Error parsing js… --- ## [Cannot add new field from event type](https://discuss.elastic.co/t/cannot-add-new-field-from-event-type/203875)
**Author:** [@francescouk](https://discuss.elastic.co/u/francescouk)\ **Replies:** 2\ **Last updated:** [October 17, 2019, 10:37am UTC](https://discuss.elastic.co/t/cannot-add-new-field-from-event-type/203875 "2019-10-17T10:37:42Z")
Hello guys, I´m new to ELK stack and trying to add custom field to kibana using the filter in logstash. Can someone point me to right direction?? I want to add the field "Logon Activity" like "message" showing in the p… --- ## [GROK parsing totally different log lines](https://discuss.elastic.co/t/grok-parsing-totally-different-log-lines/203692)
**Author:** [@Vladpov](https://discuss.elastic.co/u/Vladpov)\ **Replies:** 4\ **Last updated:** [October 17, 2019, 6:56am UTC](https://discuss.elastic.co/t/grok-parsing-totally-different-log-lines/203692 "2019-10-17T06:56:25Z")
Hey, I have following log lines: 111.222.3.11 - - \[15/Oct/2019:08:40:49 +0200\] "asaa /ssss/asf sss/saaa" 200 12833 "-" "sada(asd; fffff sff ff f asdasd) ssssss/6sadasf (ddddd, sssssss) ddd/sdasds/2222" 222.333.3.11 - -… --- ## [The index pattern logstash1\* does not contain any of the following compatible field types: geo\_point](https://discuss.elastic.co/t/the-index-pattern-logstash1-does-not-contain-any-of-the-following-compatible-field-types-geo-point/203963)
**Author:** [@TheSun](https://discuss.elastic.co/u/TheSun)\ **Replies:** 0\ **Last updated:** [October 17, 2019, 4:10am UTC](https://discuss.elastic.co/t/the-index-pattern-logstash1-does-not-contain-any-of-the-following-compatible-field-types-geo-point/203963 "2019-10-17T04:10:02Z")
Hi, I am using logstash to process the data of 2 pfsense, I create 2 indexes pfsense1 and pfsense2. But when creating visualization on geo.location it reported an error: The index pattern pfsense1 \* does not contain any… --- ## [Always get SIGTERM when starting Logstash with s3 output](https://discuss.elastic.co/t/always-get-sigterm-when-starting-logstash-with-s3-output/203331)
**Author:** [@Qingkun\_Li](https://discuss.elastic.co/u/Qingkun_Li)\ **Replies:** 1\ **Last updated:** [October 17, 2019, 12:56am UTC](https://discuss.elastic.co/t/always-get-sigterm-when-starting-logstash-with-s3-output/203331 "2019-10-17T00:56:27Z")
I use stable/logstash helm chart (chart version 2.3.0, app version 7.1.1) to deploy logstash. I have no issue using elasticsearch output, but always get SIGTERM with s3 output. I don't see any obvious cause of this in th… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=459) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=461)