# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=459 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 460 --- ## [Synch pgsql with elastic with jdbc plugin](https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665)
**Author:** [@skowron-line](https://discuss.elastic.co/u/skowron-line)\ **Replies:** 2\ **Last updated:** [October 22, 2019, 2:47pm UTC](https://discuss.elastic.co/t/synch-pgsql-with-elastic-with-jdbc-plugin/204665 "2019-10-22T14:47:00Z")
Hi I having trouble with syncing data from db to elastic, the problem is that when Iam loading data, sometimes Im missing 3, 4 records sometimes more in my elastic index I have 10 workers which load data to database. M… --- ## [Question on filebeat + logstash log filename](https://discuss.elastic.co/t/question-on-filebeat-logstash-log-filename/204666)
**Author:** [@lingfei\_ouyang](https://discuss.elastic.co/u/lingfei_ouyang)\ **Replies:** 0\ **Last updated:** [October 22, 2019, 2:24pm UTC](https://discuss.elastic.co/t/question-on-filebeat-logstash-log-filename/204666 "2019-10-22T14:24:06Z")
Hello All, I had a question regards on the filebeat + logstash, here is what I had been tested: case1: filebeat (linux) + logstash (linux), when I tested to renaming the source(filebeat) log file name, and then udpated… --- ## [Logstash, filebeat and http poller](https://discuss.elastic.co/t/logstash-filebeat-and-http-poller/202672)
**Author:** [@dub](https://discuss.elastic.co/u/dub)\ **Replies:** 4\ **Last updated:** [October 22, 2019, 2:20pm UTC](https://discuss.elastic.co/t/logstash-filebeat-and-http-poller/202672 "2019-10-22T14:20:17Z")
Hi all We are currently using filebeat and grok filter to extract a document number from a text file. now we would like to receive additional information about the document that is not in the same file. is there a way … --- ## [Logstash 7.1.0 won't start](https://discuss.elastic.co/t/logstash-7-1-0-wont-start/201695)
**Author:** [@robq](https://discuss.elastic.co/u/robq)\ **Replies:** 1\ **Last updated:** [October 22, 2019, 11:48am UTC](https://discuss.elastic.co/t/logstash-7-1-0-wont-start/201695 "2019-10-22T11:48:29Z")
I had logstash 7.1.0 running on two servers last week, but today systemctl keeps restarting it. I su'd the logstash user to see what it's doing, and when starting it manually, I get this error: \[ERROR\] 2019-09-30 15:25:… --- ## [Unable to get filename pattern from log.file.path value](https://discuss.elastic.co/t/unable-to-get-filename-pattern-from-log-file-path-value/204541)
**Author:** [@durgaram](https://discuss.elastic.co/u/durgaram)\ **Replies:** 2\ **Last updated:** [October 22, 2019, 9:42am UTC](https://discuss.elastic.co/t/unable-to-get-filename-pattern-from-log-file-path-value/204541 "2019-10-22T09:42:40Z")
Hi The following pattern in logstash conf not extracting the logappname value. grok { match =\> { "log.file.path" =\> "%{GREEDYDATA:logappname}"} } filebeat value is "log.file.path: C:\\Data\\Logs\\Test\_1.log" --- ## [Beats input ssl problem](https://discuss.elastic.co/t/beats-input-ssl-problem/204606)
**Author:** [@Bartekk](https://discuss.elastic.co/u/Bartekk)\ **Replies:** 0\ **Last updated:** [October 22, 2019, 8:46am UTC](https://discuss.elastic.co/t/beats-input-ssl-problem/204606 "2019-10-22T08:46:20Z")
Info : Ubuntu 18.04 ELK : 7.4.0 OpenSSL : 1.1.1 I just followed the tutorial Configuring SSL, TLS, and HTTPS to secure Elasticsearch, Kibana, Beats, and Logstash | Elastic Blog Connected logstash + kibana + elastic n… --- ## [Auto-update Logstash on Insert on SQL Server](https://discuss.elastic.co/t/auto-update-logstash-on-insert-on-sql-server/204522)
**Author:** [@sam.cook](https://discuss.elastic.co/u/sam.cook)\ **Replies:** 2\ **Last updated:** [October 22, 2019, 7:40am UTC](https://discuss.elastic.co/t/auto-update-logstash-on-insert-on-sql-server/204522 "2019-10-22T07:40:43Z")
Hi, I am working with Logstash and Elasticsearch but I have a problem. In my case I want Elasticsearch to be updated with the new data added into a SQL Database, all done via Logstash. I set the configuration file as b… --- ## [Transfer json format data from external service to elasticseach](https://discuss.elastic.co/t/transfer-json-format-data-from-external-service-to-elasticseach/204464)
**Author:** [@namali](https://discuss.elastic.co/u/namali)\ **Replies:** 2\ **Last updated:** [October 22, 2019, 5:54am UTC](https://discuss.elastic.co/t/transfer-json-format-data-from-external-service-to-elasticseach/204464 "2019-10-22T05:54:38Z")
trying to transfer data in json format which in external service to elasticseach. data provide page by page. next page define in previous page how to transfer data from all pages to elasticseach sample json file coun… --- ## [Logstash installation on older version](https://discuss.elastic.co/t/logstash-installation-on-older-version/199771)
**Author:** [@skyluke.1987](https://discuss.elastic.co/u/skyluke.1987)\ **Replies:** 4\ **Last updated:** [October 22, 2019, 3:38am UTC](https://discuss.elastic.co/t/logstash-installation-on-older-version/199771 "2019-10-22T03:38:11Z")
Hi how can I install Logstash 7.1.1 on Centos 6.10 ? So far cant get any guide on this. Reason, my other servers are using Kibana and ES version 7.1.1. Thanks --- ## [SIEM Custom Data](https://discuss.elastic.co/t/siem-custom-data/203723)
**Author:** [@JDMurph](https://discuss.elastic.co/u/JDMurph)\ **Replies:** 1\ **Last updated:** [October 21, 2019, 7:36pm UTC](https://discuss.elastic.co/t/siem-custom-data/203723 "2019-10-21T19:36:31Z")
I am just trying to get some custom data into SIEM using Logstash reading a csv file. I have no trouble getting the data into Elastic Search but it does not show up in the SIEM dashboard. Even after I add the index under… --- ## [Logstash in docker](https://discuss.elastic.co/t/logstash-in-docker/204503)
**Author:** [@Asmaa\_Sarih](https://discuss.elastic.co/u/Asmaa_Sarih)\ **Replies:** 5\ **Last updated:** [October 21, 2019, 5:48pm UTC](https://discuss.elastic.co/t/logstash-in-docker/204503 "2019-10-21T17:48:50Z")
Hello everyone! I am having some problem when delploying logstash in docker. When I run container and go inside, I don't see /etc/logstash/ directory. How can I configure inputs, filters, and outputs? --- ## [Cannot find my index in Kibana](https://discuss.elastic.co/t/cannot-find-my-index-in-kibana/202225)
**Author:** [@Barushka](https://discuss.elastic.co/u/Barushka)\ **Replies:** 2\ **Last updated:** [October 21, 2019, 5:12pm UTC](https://discuss.elastic.co/t/cannot-find-my-index-in-kibana/202225 "2019-10-21T17:12:49Z")
Hi, I am practicing and learning ELK. I am using a new version of ELK 7.1.1 I have successfully created conf file and I used a simple log file ( few lines). I want to use it to practice coordinate maps. Although I do no… --- ## [Inner join and config of Logstash and JDBC](https://discuss.elastic.co/t/inner-join-and-config-of-logstash-and-jdbc/204406)
**Author:** [@MAvD](https://discuss.elastic.co/u/MAvD)\ **Replies:** 1\ **Last updated:** [October 21, 2019, 1:32pm UTC](https://discuss.elastic.co/t/inner-join-and-config-of-logstash-and-jdbc/204406 "2019-10-21T13:32:17Z")
Hello, Currently, we are updating our data in Elasticsearch by uploading a lot of data once in a while from a relational database which uses quite some data. We want this this to be synchronized by uploading only new an… --- ## [Logstash without NSSM](https://discuss.elastic.co/t/logstash-without-nssm/204480)
**Author:** [@philu](https://discuss.elastic.co/u/philu)\ **Replies:** 3\ **Last updated:** [October 21, 2019, 1:47pm UTC](https://discuss.elastic.co/t/logstash-without-nssm/204480 "2019-10-21T13:47:42Z")
Is there a way to install Logstash (7.x) on Windows WITHOUT nssm (non su\*\*ing Service Manager? Every tutorial is based on nssm. Same for Kibana. Does SC.exe not provided enough possiblities? Why is that so? Also anothe… --- ## [Logstash grok filter for firewall traffic logs](https://discuss.elastic.co/t/logstash-grok-filter-for-firewall-traffic-logs/204374)
**Author:** [@smgh](https://discuss.elastic.co/u/smgh)\ **Replies:** 4\ **Last updated:** [October 21, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-grok-filter-for-firewall-traffic-logs/204374 "2019-10-21T13:41:54Z")
First I have to mention that this is my first ELK experience. I setup ELK 7.04 on Ubuntu 18.04.3 LTS and trying to ingest my gateway/firewall logs into logstash. filebeat is also setup on the box. I see the traffic logs… --- ## [Use filter in output](https://discuss.elastic.co/t/use-filter-in-output/204444)
**Author:** [@NicolasJ](https://discuss.elastic.co/u/NicolasJ)\ **Replies:** 2\ **Last updated:** [October 21, 2019, 1:09pm UTC](https://discuss.elastic.co/t/use-filter-in-output/204444 "2019-10-21T13:09:54Z")
Hi all, I wonder if it is possible to use filter value in output. For example, my output is formatted like this : output { mongodb { id =\> "mongodb\_plugin\_id" collection =\> "" database =\> "my\_mogodb\_database" uri… --- ## [Error: com.microsoft.sqlserver.jdbc.SQLServerDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc\_driver\_library?](https://discuss.elastic.co/t/error-com-microsoft-sqlserver-jdbc-sqlserverdriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/204440)
**Author:** [@adarsh-c](https://discuss.elastic.co/u/adarsh-c)\ **Replies:** 1\ **Last updated:** [October 21, 2019, 12:01pm UTC](https://discuss.elastic.co/t/error-com-microsoft-sqlserver-jdbc-sqlserverdriver-not-loaded-are-you-sure-youve-included-the-correct-jdbc-driver-in-jdbc-driver-library/204440 "2019-10-21T12:01:55Z")
I am using logstash-7.4.0 , jdk-11.0.5 and Microsoft JDBC Driver 7.4 for SQL Server. I am getting an jdbc connection issue . Any idea ? batch.delay"=\>50, "pipeline.max\_inflight"=\>500, :thread=\>"#\… --- ## [Parse elastic response json into CSV](https://discuss.elastic.co/t/parse-elastic-response-json-into-csv/204347)
**Author:** [@raviep](https://discuss.elastic.co/u/raviep)\ **Replies:** 2\ **Last updated:** [October 21, 2019, 5:59am UTC](https://discuss.elastic.co/t/parse-elastic-response-json-into-csv/204347 "2019-10-21T05:59:29Z")
Hi, I need to parse elastic search response JSON to CSV using logstash. Here I need only fields parameters to go into CSV file. { "took" : 11, "timed\_out" : false, "\_shards" : { "total" : 3, "successful" : 3, "sk… --- ## [Issues with grok and beats input](https://discuss.elastic.co/t/issues-with-grok-and-beats-input/204261)
**Author:** [@562uned](https://discuss.elastic.co/u/562uned)\ **Replies:** 3\ **Last updated:** [October 21, 2019, 5:26am UTC](https://discuss.elastic.co/t/issues-with-grok-and-beats-input/204261 "2019-10-21T05:26:01Z")
Hello everyone, I have been racking my brain trying to figure out what im doing wrong here.. i have a really simple input im testing with that i cannot seem to get working correctly. I have this log file that outputs "… --- ## [Error while load logstash config](https://discuss.elastic.co/t/error-while-load-logstash-config/203967)
**Author:** [@dean92](https://discuss.elastic.co/u/dean92)\ **Replies:** 2\ **Last updated:** [October 21, 2019, 2:36am UTC](https://discuss.elastic.co/t/error-while-load-logstash-config/203967 "2019-10-21T02:36:10Z")
hi expert, I'm still newbie in ES. I have problem with my logstash where there are some failed that no index in my ES after load the config. Below is my log file and show the failed to execute action. Please advice furt… --- ## [Logstash split string issues when parsing text](https://discuss.elastic.co/t/logstash-split-string-issues-when-parsing-text/204308)
**Author:** [@Srikanth\_Reddy](https://discuss.elastic.co/u/Srikanth_Reddy)\ **Replies:** 1\ **Last updated:** [October 20, 2019, 8:57pm UTC](https://discuss.elastic.co/t/logstash-split-string-issues-when-parsing-text/204308 "2019-10-20T20:57:05Z")
Hi, I have message is in bellow format TIME: 2019-10-19T11:48:40.896687Z USER@HOST: TEST\[TEST\] @ \[172.30.1.71\] ID: 1130779 QUERY\_TIME: 1.611727 LOCK\_TIME: 0.000079 ROWS\_SENT: 5 ROWS\_EXAMINED: 13056 SET TIMESTAMP=15… --- ## [Change type field in logstash](https://discuss.elastic.co/t/change-type-field-in-logstash/204368)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 1\ **Last updated:** [October 20, 2019, 6:02pm UTC](https://discuss.elastic.co/t/change-type-field-in-logstash/204368 "2019-10-20T18:02:27Z")
Hi. I wanna have sum of network usage per client in nginx. This is the logstash before changes: grok { match =\> { "message" =\> "%{IP:clientip} ... %{NUMBER:bytes}" } } so at first I changed the NUMBER to INT: grok { … --- ## [Changing timezone](https://discuss.elastic.co/t/changing-timezone/204240)
**Author:** [@Necus](https://discuss.elastic.co/u/Necus)\ **Replies:** 2\ **Last updated:** [October 20, 2019, 11:21am UTC](https://discuss.elastic.co/t/changing-timezone/204240 "2019-10-20T11:21:31Z")
Hi, I'm trying to change timezone for time logs, but for some reason after configuration nothing has changed. What I have done: copying default date/time field: mutate { copy =\> {"time" =\> "new\_time"} } changin… --- ## [Unable to read email from outlook](https://discuss.elastic.co/t/unable-to-read-email-from-outlook/204187)
**Author:** [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\ **Replies:** 1\ **Last updated:** [October 19, 2019, 10:00pm UTC](https://discuss.elastic.co/t/unable-to-read-email-from-outlook/204187 "2019-10-19T22:00:26Z")
I am trying to read the emails from the outlook. input { imap { host =\> "outlook.office365.com" password =\> "\*\*\*\*\*\*\*\*" user =\> "\*\*\*\*\*\*\*\*\*" port =\> 993 secure =\> true check\_interval =\> 10 folder =\> "XYZ" } } out… --- ## [Move processed from one S3 bucket to other bucket](https://discuss.elastic.co/t/move-processed-from-one-s3-bucket-to-other-bucket/204199)
**Author:** [@pami](https://discuss.elastic.co/u/pami)\ **Replies:** 3\ **Last updated:** [October 19, 2019, 7:20pm UTC](https://discuss.elastic.co/t/move-processed-from-one-s3-bucket-to-other-bucket/204199 "2019-10-19T19:20:59Z")
can anyone can suggest how to move the processed files from one bucket to other. scenario: Bucket 1 2019/10/ will S3 plugin move the processed files in the same structure ? --- ## [JSON LInes to JSON ARRAY](https://discuss.elastic.co/t/json-lines-to-json-array/204300)
**Author:** [@Sanjay\_Agnani](https://discuss.elastic.co/u/Sanjay_Agnani)\ **Replies:** 1\ **Last updated:** [October 19, 2019, 1:07pm UTC](https://discuss.elastic.co/t/json-lines-to-json-array/204300 "2019-10-19T13:07:32Z")
Hi All, We receive json lines from logs. would need to aggregate this json lines to json Array Sample: { "source" : "Request" "target": "Request\_parsed" } { "source" : "Request1" "target": "Request\_parsed1" } … --- ## [Using regex in gsub raises CPU usage significantly](https://discuss.elastic.co/t/using-regex-in-gsub-raises-cpu-usage-significantly/202408)
**Author:** [@Ged](https://discuss.elastic.co/u/Ged)\ **Replies:** 5\ **Last updated:** [October 19, 2019, 12:55pm UTC](https://discuss.elastic.co/t/using-regex-in-gsub-raises-cpu-usage-significantly/202408 "2019-10-19T12:55:39Z")
Hi guys, I want to share my observations about using gsub with regex. I use gsub for extracting application instance name from log file path. Example path: /logs/pppas/logs\_tcc/app619\_pppas-prod1/payload.log where ap… --- ## [How to use includes in Logstash conf files](https://discuss.elastic.co/t/how-to-use-includes-in-logstash-conf-files/204283)
**Author:** [@kmiklas](https://discuss.elastic.co/u/kmiklas)\ **Replies:** 1\ **Last updated:** [October 19, 2019, 8:09am UTC](https://discuss.elastic.co/t/how-to-use-includes-in-logstash-conf-files/204283 "2019-10-19T08:09:01Z")
Can includes be used in logstash config files? Minimal, Complete, and Verifiable Example Can I replace this... file: beats.conf input { beats { port =\> 5044 } } filter { date { match =\> \["myTimest… --- ## [Logstash not reading files input](https://discuss.elastic.co/t/logstash-not-reading-files-input/203908)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 3\ **Last updated:** [October 18, 2019, 8:13pm UTC](https://discuss.elastic.co/t/logstash-not-reading-files-input/203908 "2019-10-18T20:13:21Z")
Hi I have configured an elasticsearch cluster with 4 nodes (3 are master/data nodes, and 1 is a coordinating node). In the coordinating node i have logstash and kibana. But i'm cannot make logstash to read files from i… --- ## [Unable to ingest NEXUS 3000 sFlow with Logstash](https://discuss.elastic.co/t/unable-to-ingest-nexus-3000-sflow-with-logstash/204098)
**Author:** [@TitanMerluza](https://discuss.elastic.co/u/TitanMerluza)\ **Replies:** 1\ **Last updated:** [October 18, 2019, 7:59pm UTC](https://discuss.elastic.co/t/unable-to-ingest-nexus-3000-sflow-with-logstash/204098 "2019-10-18T19:59:46Z")
Hi, Hope you can help me with the issue I'm having. Currrent scenario: We are currently integrating both netflow and sflow in Elastic with logstash version 7.3.2 on a CenOS 7 server. We've managed to visualize netflow… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=458) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=460)