# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=212 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 213 --- ## [If my logstash input is File, do I still need to enable the persistent queue](https://discuss.elastic.co/t/if-my-logstash-input-is-file-do-i-still-need-to-enable-the-persistent-queue/278351)
**Author:** [@asasas234](https://discuss.elastic.co/u/asasas234)\ **Replies:** 2\ **Last updated:** [July 12, 2021, 1:22am UTC](https://discuss.elastic.co/t/if-my-logstash-input-is-file-do-i-still-need-to-enable-the-persistent-queue/278351 "2021-07-12T01:22:59Z")
As the question, my Logstash reads data from a file and eventually writes it to elasticsearch, I need to ensure that the data will not be lost, do I need to enable persistent queues? My question is whether I need to enab… --- ## [Logstash jdbc input for sql express](https://discuss.elastic.co/t/logstash-jdbc-input-for-sql-express/276712)
**Author:** [@IsAa](https://discuss.elastic.co/u/IsAa)\ **Replies:** 1\ **Last updated:** [July 11, 2021, 2:35pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-for-sql-express/276712 "2021-07-11T14:35:21Z")
Hi, Does anyone know a good example showing how to configure input for jdbc driver with sqlexpress. input { jdbc { jdbc\_driver\_library =\> "C:\\JDBC8.4\\sqljdbc\_8.4\\enu\\mssql-jdbc-8.4.1.jre14.jar" jdbc\_driver\_class =\> … --- ## [Logstash 5 not running](https://discuss.elastic.co/t/logstash-5-not-running/64449)
**Author:** [@Vishal\_Sharma1](https://discuss.elastic.co/u/Vishal_Sharma1)\ **Replies:** 71\ **Last updated:** [July 11, 2021, 11:39am UTC](https://discuss.elastic.co/t/logstash-5-not-running/64449 "2021-07-11T11:39:04Z")
hey there, I am following the official document and seems like it's not working. I am using a freshly installed CentOS 7 system and have Java 1.8 installed. As per the document when i run below \[Foo@Elastic logstash\]… --- ## [Can't find grok match](https://discuss.elastic.co/t/cant-find-grok-match/278240)
**Author:** [@Badb0y](https://discuss.elastic.co/u/Badb0y)\ **Replies:** 4\ **Last updated:** [July 11, 2021, 2:57am UTC](https://discuss.elastic.co/t/cant-find-grok-match/278240 "2021-07-11T02:57:07Z")
Hi, I'm almost at the solution but not sure how to make this optimal and how to parse the ? marked part. These are my log lines: 2021-07-09T12:11:40.917+0700 7fce8a240700 1 beast: 0x7fceee9b06b0: 10.111.111.111 - - \[… --- ## [Logstash don't send logs to elasticsearch](https://discuss.elastic.co/t/logstash-dont-send-logs-to-elasticsearch/278339)
**Author:** [@diegz](https://discuss.elastic.co/u/diegz)\ **Replies:** 20\ **Last updated:** [July 11, 2021, 12:50am UTC](https://discuss.elastic.co/t/logstash-dont-send-logs-to-elasticsearch/278339 "2021-07-11T00:50:43Z")
Hi community, I configured ssl on the whole stack (elasticsearch, kibana, logstash, filebeat) When I manually launch logstash to test my configuration, I have this return. I think it's good but I don't see anything in… --- ## [Listen loop error: #\](https://discuss.elastic.co/t/listen-loop-error-ioerror-closed-stream/278338)
**Author:** [@Carlos\_Velasquez](https://discuss.elastic.co/u/Carlos_Velasquez)\ **Replies:** 0\ **Last updated:** [July 10, 2021, 3:41pm UTC](https://discuss.elastic.co/t/listen-loop-error-ioerror-closed-stream/278338 "2021-07-10T15:41:10Z")
Hi, When I start the logstash (manual invocation), I receive the folowing error. the version i'm using is 7.12.1 and Centos 8 Operating System: CentOS Linux 8 CPE OS Name: cpe:/o:centos:centos:8 Ke… --- ## [I am unable to sync data from postgresql, Basically I am not able to start logstash](https://discuss.elastic.co/t/i-am-unable-to-sync-data-from-postgresql-basically-i-am-not-able-to-start-logstash/278331)
**Author:** [@NETESH\_KUMAR](https://discuss.elastic.co/u/NETESH_KUMAR)\ **Replies:** 2\ **Last updated:** [July 10, 2021, 2:50pm UTC](https://discuss.elastic.co/t/i-am-unable-to-sync-data-from-postgresql-basically-i-am-not-able-to-start-logstash/278331 "2021-07-10T14:50:47Z")
\#Here is my config file #--------- config file start---------------- Sample Logstash configuration for creating a simple Beats -\> Logstash -\> Elasticsearch pipeline. input{ jdbc{ jdbc\_connection\_string =\> "jdbc:postg… --- ## [Removing item from array based on string value from another field](https://discuss.elastic.co/t/removing-item-from-array-based-on-string-value-from-another-field/278279)
**Author:** [@mad\_dog](https://discuss.elastic.co/u/mad_dog)\ **Replies:** 6\ **Last updated:** [July 9, 2021, 6:27pm UTC](https://discuss.elastic.co/t/removing-item-from-array-based-on-string-value-from-another-field/278279 "2021-07-09T18:27:00Z")
I have a ruby filter that puts all mac addresses (access points and stations macs) in a log to mac\_addresses field and this works fine. I also have another grok filter which puts only station mac addresses to station\_ma… --- ## [Create KV-pairs for field not in key:value format and use ruby filter on KV pairs](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220)
**Author:** [@parinitha.nagaraja](https://discuss.elastic.co/u/parinitha.nagaraja)\ **Replies:** 3\ **Last updated:** [July 9, 2021, 4:44pm UTC](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220 "2021-07-09T16:44:39Z")
Hello, I am trying to apply KV filter to a field that does not have "key:value key:value" format. The format is "value(key) value(key) ...". Below are the details. 2021-06-28 19:06:15.848 \[0000-XYZ\] \[appLog … --- ## [Importing text from text files and merging to existing documents](https://discuss.elastic.co/t/importing-text-from-text-files-and-merging-to-existing-documents/278269)
**Author:** [@stashing\_logs](https://discuss.elastic.co/u/stashing_logs)\ **Replies:** 1\ **Last updated:** [July 9, 2021, 4:43pm UTC](https://discuss.elastic.co/t/importing-text-from-text-files-and-merging-to-existing-documents/278269 "2021-07-09T16:43:39Z")
I have an existing elastic engine containing documents with an ID and some metadata (date, original file path, author etc.). I now have the text files which have the extracted text for these documents. The text files ar… --- ## [Does the lumberjack output plugin do the loadbalacing](https://discuss.elastic.co/t/does-the-lumberjack-output-plugin-do-the-loadbalacing/278252)
**Author:** [@lemahdois](https://discuss.elastic.co/u/lemahdois)\ **Replies:** 1\ **Last updated:** [July 9, 2021, 4:40pm UTC](https://discuss.elastic.co/t/does-the-lumberjack-output-plugin-do-the-loadbalacing/278252 "2021-07-09T16:40:07Z")
Dear all, I am using lumberjack output plugin to send logs from logstash to logstash. I use this code lumberjack { codec =\> json hosts =\> \["XXX","YYY"\] port =\> ZZZ … --- ## [Logstash .Conf file](https://discuss.elastic.co/t/logstash-conf-file/278194)
**Author:** [@Ashwin\_Patil1](https://discuss.elastic.co/u/Ashwin_Patil1)\ **Replies:** 9\ **Last updated:** [July 9, 2021, 2:40pm UTC](https://discuss.elastic.co/t/logstash-conf-file/278194 "2021-07-09T14:40:33Z")
I am trying to collect logs from multiple inputs and write them into 2 indices. Please refer the below conf file, am able to create a index called syslog but am not able to create index called winlog. All the logs are g… --- ## [Pipeline continues when stopping Logstash service](https://discuss.elastic.co/t/pipeline-continues-when-stopping-logstash-service/277936)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 2\ **Last updated:** [July 9, 2021, 1:58pm UTC](https://discuss.elastic.co/t/pipeline-continues-when-stopping-logstash-service/277936 "2021-07-09T13:58:26Z")
Logstash v.7.13.2 Hi all, I'm successfully running 3 logstash instances via the following pipelines.yml file to keep an elasticsearch index up-to-date with changes to Oracle database records and windows logs: - pipeli… --- ## [Keystore Issue on Suse and windows](https://discuss.elastic.co/t/keystore-issue-on-suse-and-windows/278275)
**Author:** [@Ashwin\_Patil1](https://discuss.elastic.co/u/Ashwin_Patil1)\ **Replies:** 0\ **Last updated:** [July 9, 2021, 11:21am UTC](https://discuss.elastic.co/t/keystore-issue-on-suse-and-windows/278275 "2021-07-09T11:21:55Z")
I am trying to create logstash keystore and winlogbeat keystore, but I am getting the below error. However I was able to create filebeat keystore. Below is the error for logstash on Suse linux, localhost:/usr/share/lo… --- ## [Desired fields did get create after modifying logstash config](https://discuss.elastic.co/t/desired-fields-did-get-create-after-modifying-logstash-config/278039)
**Author:** [@cool999](https://discuss.elastic.co/u/cool999)\ **Replies:** 6\ **Last updated:** [July 9, 2021, 10:28am UTC](https://discuss.elastic.co/t/desired-fields-did-get-create-after-modifying-logstash-config/278039 "2021-07-09T10:28:12Z")
Hi All, I have created two different elastic search clusters. In first cluster (created newly) ,(v 7.13.2), I have created below config in pipeline.conf under conf.d for logstash and it worked as I see expected fields … --- ## [Daily index not created](https://discuss.elastic.co/t/daily-index-not-created/278215)
**Author:** [@prat](https://discuss.elastic.co/u/prat)\ **Replies:** 6\ **Last updated:** [July 9, 2021, 8:39am UTC](https://discuss.elastic.co/t/daily-index-not-created/278215 "2021-07-09T08:39:39Z")
Hi, I see below indices getting created daily without an issue. yellow open access\_server-2021.07.06 HGHg9wiBTz-42qxzIAuV3A 1 1 203 0 341.5kb 341.5kb yellow open access\_serv… --- ## [Kibana custome timestamp and @timeestamp are different](https://discuss.elastic.co/t/kibana-custome-timestamp-and-timeestamp-are-different/277561)
**Author:** [@mrunalini](https://discuss.elastic.co/u/mrunalini)\ **Replies:** 22\ **Last updated:** [July 8, 2021, 1:14pm UTC](https://discuss.elastic.co/t/kibana-custome-timestamp-and-timeestamp-are-different/277561 "2021-07-08T13:14:00Z")
Hi Team, I am using below xml/xpath statement to convert my string timestamp to date type timestamp in logstash input - 2021-06-30-20:11:23 xpath - xpath =\>\[ "concat(substring(/AdMsg/AdLels/AuditLevel/TS/text(), 1,10)… --- ## [Changing Time Format](https://discuss.elastic.co/t/changing-time-format/278115)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 3\ **Last updated:** [July 8, 2021, 12:53pm UTC](https://discuss.elastic.co/t/changing-time-format/278115 "2021-07-08T12:53:25Z")
I have a filter that is reading 2 fields (STARTTIME and ENDTIME) from a CSV file and then changing to deviceCustomString1 and deviceCustomString2 and I need to change the time layout. I am trying to change the time form… --- ## [Splitting csv files send to elasticsearch via logstash http input](https://discuss.elastic.co/t/splitting-csv-files-send-to-elasticsearch-via-logstash-http-input/278144)
**Author:** [@reynavan](https://discuss.elastic.co/u/reynavan)\ **Replies:** 0\ **Last updated:** [July 8, 2021, 7:19am UTC](https://discuss.elastic.co/t/splitting-csv-files-send-to-elasticsearch-via-logstash-http-input/278144 "2021-07-08T07:19:45Z")
Hello! I am new to ELK stack and I'm looking for some advice. I have 3 different types of csv's files which are generated every day, I'm sending them to IP address where logstash is configured, than i want to visualize … --- ## [Logstash can't write logs in the path](https://discuss.elastic.co/t/logstash-cant-write-logs-in-the-path/278155)
**Author:** [@Andrea\_Colangelo](https://discuss.elastic.co/u/Andrea_Colangelo)\ **Replies:** 1\ **Last updated:** [July 8, 2021, 9:25am UTC](https://discuss.elastic.co/t/logstash-cant-write-logs-in-the-path/278155 "2021-07-08T09:25:04Z")
Hi, i am having an error on logstash when i start it. that's my log: Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will like… --- ## [Logstash uses current year for timestamp without year and shows events with future dates](https://discuss.elastic.co/t/logstash-uses-current-year-for-timestamp-without-year-and-shows-events-with-future-dates/277877)
**Author:** [@theirfan](https://discuss.elastic.co/u/theirfan)\ **Replies:** 4\ **Last updated:** [July 8, 2021, 8:26am UTC](https://discuss.elastic.co/t/logstash-uses-current-year-for-timestamp-without-year-and-shows-events-with-future-dates/277877 "2021-07-08T08:26:53Z")
Hi Community !! I have logs wherein there's no year mentioned, and while parsing, logstash adds current year and makes the event appear in future dates. i checked on forums for a similar issue but unfortunately, there … --- ## [Any sane way to filter/remove fields in an array such as \[0\], \[1\]?](https://discuss.elastic.co/t/any-sane-way-to-filter-remove-fields-in-an-array-such-as-0-1/278140)
**Author:** [@Sjaak01](https://discuss.elastic.co/u/Sjaak01)\ **Replies:** 0\ **Last updated:** [July 8, 2021, 6:21am UTC](https://discuss.elastic.co/t/any-sane-way-to-filter-remove-fields-in-an-array-such-as-0-1/278140 "2021-07-08T06:21:44Z")
Hi, My data contains an array from which I only want to save the message fields. But I don't think you can do regexp on fields in prune, grok or mutate filters (I tried, it didn't work). The problem is that I do not kno… --- ## [Logstash insufficient memory for java runtime env](https://discuss.elastic.co/t/logstash-insufficient-memory-for-java-runtime-env/276705)
**Author:** [@IsAa](https://discuss.elastic.co/u/IsAa)\ **Replies:** 5\ **Last updated:** [July 7, 2021, 9:23pm UTC](https://discuss.elastic.co/t/logstash-insufficient-memory-for-java-runtime-env/276705 "2021-07-07T21:23:10Z")
Hi ELK Users, I am wondering if anyone has come across this issue when running logstash PS C:\\Program Files\\ELK\\logstash-7.13.1\\bin\> .\\logstash.bat -f C:\\Program Files\\ELK\\logstash-7.13.1\\config\\logstash-test-std.conf … --- ## [How to drop data in logstash if ES index is not available](https://discuss.elastic.co/t/how-to-drop-data-in-logstash-if-es-index-is-not-available/278071)
**Author:** [@prabhakar\_talari](https://discuss.elastic.co/u/prabhakar_talari)\ **Replies:** 1\ **Last updated:** [July 7, 2021, 5:35pm UTC](https://discuss.elastic.co/t/how-to-drop-data-in-logstash-if-es-index-is-not-available/278071 "2021-07-07T17:35:10Z")
Hi All, We do not want logstash to create indexes since we are creating indexes with custom settings / mappings. some times we are getting the data from beats to logstash but the es index is not available. if the index… --- ## [Logstash elastic HostUnreachableError while elastic is available](https://discuss.elastic.co/t/logstash-elastic-hostunreachableerror-while-elastic-is-available/278067)
**Author:** [@JoranDox](https://discuss.elastic.co/u/JoranDox)\ **Replies:** 0\ **Last updated:** [July 7, 2021, 12:13pm UTC](https://discuss.elastic.co/t/logstash-elastic-hostunreachableerror-while-elastic-is-available/278067 "2021-07-07T12:13:04Z")
logstash (7.13.1) doesn't work without clear reason why: we get "LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError" but we can curl from the same pod with the same url & credentials it works in … --- ## [Yet another logstash and json issue, unable to ingest a "basic" json](https://discuss.elastic.co/t/yet-another-logstash-and-json-issue-unable-to-ingest-a-basic-json/277991)
**Author:** [@cibernicola](https://discuss.elastic.co/u/cibernicola)\ **Replies:** 4\ **Last updated:** [July 7, 2021, 10:09am UTC](https://discuss.elastic.co/t/yet-another-logstash-and-json-issue-unable-to-ingest-a-basic-json/277991 "2021-07-07T10:09:56Z")
I have a dir with tons of subdirs with a json file like this: { "id": "2234", "name": "Text", "url": "https://url.com", "param": "string", "last": "2021-07-06 20:05:49.458724", "url2": "url.com… --- ## [Problems in Elasticsearch combining two types of logs using two pipelines (Logstash)](https://discuss.elastic.co/t/problems-in-elasticsearch-combining-two-types-of-logs-using-two-pipelines-logstash/278048)
**Author:** [@francesco96](https://discuss.elastic.co/u/francesco96)\ **Replies:** 0\ **Last updated:** [July 7, 2021, 9:06am UTC](https://discuss.elastic.co/t/problems-in-elasticsearch-combining-two-types-of-logs-using-two-pipelines-logstash/278048 "2021-07-07T09:06:57Z")
Hi everyone, I need some help. I have configured my ES stack inside a docker. I need to show two types of logs (syslog, log), in an Elastic index. I have created two configuration files logstash.cong logstash-syslog.c… --- ## [Jdbc input not writing index](https://discuss.elastic.co/t/jdbc-input-not-writing-index/277815)
**Author:** [@charles97](https://discuss.elastic.co/u/charles97)\ **Replies:** 0\ **Last updated:** [July 5, 2021, 10:53am UTC](https://discuss.elastic.co/t/jdbc-input-not-writing-index/277815 "2021-07-05T10:53:33Z")
Hello, here is my sample input.conf: input { jdbc { jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc\_connection\_string =\> "jdbc:sqlserver://x.x.x.x;databa… --- ## [Im stuck at formatting data while trying to send xml file to Elasticsearch](https://discuss.elastic.co/t/im-stuck-at-formatting-data-while-trying-to-send-xml-file-to-elasticsearch/278002)
**Author:** [@Sourtoast](https://discuss.elastic.co/u/Sourtoast)\ **Replies:** 1\ **Last updated:** [July 6, 2021, 11:23pm UTC](https://discuss.elastic.co/t/im-stuck-at-formatting-data-while-trying-to-send-xml-file-to-elasticsearch/278002 "2021-07-06T23:23:30Z")
I'm trying to add data from xml to Elasticsearch but I want only specific fields with its names changed. I'm using mutate filter to map xml fields to my own fields. My problem is with photos input { http\_poller { ur… --- ## [Mysql-slow multiline](https://discuss.elastic.co/t/mysql-slow-multiline/277757)
**Author:** [@bz\_Os](https://discuss.elastic.co/u/bz_Os)\ **Replies:** 2\ **Last updated:** [July 6, 2021, 9:40pm UTC](https://discuss.elastic.co/t/mysql-slow-multiline/277757 "2021-07-06T21:40:34Z")
Hello, I am parsing the mysql-slow logs using the parser: input{ pipeline { address =\> input } file{ path =\> "/etc/logstash/logsamples/all.log" start\_position =\> "beginnin… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=211) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=213)