# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=211 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 212 --- ## [Environment setup for modifying logstash plugin](https://discuss.elastic.co/t/environment-setup-for-modifying-logstash-plugin/278712)
**Author:** [@pjohnson](https://discuss.elastic.co/u/pjohnson)\ **Replies:** 0\ **Last updated:** [July 14, 2021, 7:52pm UTC](https://discuss.elastic.co/t/environment-setup-for-modifying-logstash-plugin/278712 "2021-07-14T19:52:47Z")
Is there a tutorial for setting up your environment for modifying an existing logstash ruby plugin i.e. what version of java to use, jruby to use, how to install jars etc. How to write a Logstash output plugin | Logstash… --- ## [Need Help : To Parse XML and Json data received via TCP syslog in logstash](https://discuss.elastic.co/t/need-help-to-parse-xml-and-json-data-received-via-tcp-syslog-in-logstash/278650)
**Author:** [@Shruti\_Gupta](https://discuss.elastic.co/u/Shruti_Gupta)\ **Replies:** 3\ **Last updated:** [July 14, 2021, 5:45pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-and-json-data-received-via-tcp-syslog-in-logstash/278650 "2021-07-14T17:45:43Z")
I am receiving google apigee logs in syslog at 5444 port. Logs are being generating via multiple applications and format is also different like XML and Json. Need help how can we configure logstash.conf to parse both ty… --- ## [Logstash crashing](https://discuss.elastic.co/t/logstash-crashing/278590)
**Author:** [@cdhgold](https://discuss.elastic.co/u/cdhgold)\ **Replies:** 2\ **Last updated:** [July 14, 2021, 5:28pm UTC](https://discuss.elastic.co/t/logstash-crashing/278590 "2021-07-14T17:28:19Z")
I'm using OSS version 7.10 of both filebeat and logstash on RHEL 7.3 Elastisearch "destination" is a Elastisearch 7.10 AWS Elastisearch domain I can run a curl and insert data into the AWS ES however logstash keeps cra… --- ## [Is it possible to pass variables from init to script\_params?](https://discuss.elastic.co/t/is-it-possible-to-pass-variables-from-init-to-script-params/278680)
**Author:** [@davidpellerin](https://discuss.elastic.co/u/davidpellerin)\ **Replies:** 1\ **Last updated:** [July 14, 2021, 4:45pm UTC](https://discuss.elastic.co/t/is-it-possible-to-pass-variables-from-init-to-script-params/278680 "2021-07-14T16:45:28Z")
Just wondering if it's possible to pass along variables from the "init" section of the Ruby filter plugin to the script\_params? In my case the data I am looking up is something I only want to do once: ruby { init =\>… --- ## [Connection refused port 5044](https://discuss.elastic.co/t/connection-refused-port-5044/277149)
**Author:** [@diegz](https://discuss.elastic.co/u/diegz)\ **Replies:** 1\ **Last updated:** [July 14, 2021, 4:35pm UTC](https://discuss.elastic.co/t/connection-refused-port-5044/277149 "2021-07-14T16:35:43Z")
Hi, When i want to validate the logstash server certificate with this command : curl -v --cacert /etc/logstash/config/certs/logstash.crt https://logstash.ad-it.fr:5044 Rebuilt URL to: https://192.168.1.30:5044/ \* Try… --- ## [ELK and Outsystems logs](https://discuss.elastic.co/t/elk-and-outsystems-logs/277504)
**Author:** [@IsAa](https://discuss.elastic.co/u/IsAa)\ **Replies:** 1\ **Last updated:** [July 14, 2021, 4:31pm UTC](https://discuss.elastic.co/t/elk-and-outsystems-logs/277504 "2021-07-14T16:31:18Z")
Hi, I am just wondering if anyone has integrated outsystems logs with ELK stack. What is the best approach for this? I have come across logstash accelerators on Github which runs pipelines. But if I am already runnin… --- ## [Logstash email output](https://discuss.elastic.co/t/logstash-email-output/277538)
**Author:** [@Khairul\_Anuar\_bin\_Ab](https://discuss.elastic.co/u/Khairul_Anuar_bin_Ab)\ **Replies:** 1\ **Last updated:** [July 14, 2021, 4:29pm UTC](https://discuss.elastic.co/t/logstash-email-output/277538 "2021-07-14T16:29:30Z")
I'm trying to send an email using logstash, however logstash keep sending the same email every minutes. The output configuration as per below. Any ideas what went wrong? output { email { to =\> "zyx@gmail.com" body =\>… --- ## [Logstash running but not listening on input port](https://discuss.elastic.co/t/logstash-running-but-not-listening-on-input-port/277799)
**Author:** [@cuongnp](https://discuss.elastic.co/u/cuongnp)\ **Replies:** 1\ **Last updated:** [July 14, 2021, 4:24pm UTC](https://discuss.elastic.co/t/logstash-running-but-not-listening-on-input-port/277799 "2021-07-14T16:24:03Z")
Hi community, I have a logstash pipeline, It works well when I start the pipeline with command line. But when start as a service , there is no input port listening even the service is running. Please have a look This i… --- ## [Why logstash shut down when I use output to log.txt?](https://discuss.elastic.co/t/why-logstash-shut-down-when-i-use-output-to-log-txt/277837)
**Author:** [@111418](https://discuss.elastic.co/u/111418)\ **Replies:** 1\ **Last updated:** [July 14, 2021, 4:18pm UTC](https://discuss.elastic.co/t/why-logstash-shut-down-when-i-use-output-to-log-txt/277837 "2021-07-14T16:18:06Z")
I have a log index "log\_index\_0701" and there is about 9000000 logs in this index. I want to output this index log as a txt file but logstash always shut down when I output. Here is my config input{ elasti… --- ## [Winlogbeat failed to connect Logstash server after ssl configuration](https://discuss.elastic.co/t/winlogbeat-failed-to-connect-logstash-server-after-ssl-configuration/278542)
**Author:** [@Rojal\_Paul](https://discuss.elastic.co/u/Rojal_Paul)\ **Replies:** 1\ **Last updated:** [July 14, 2021, 4:09pm UTC](https://discuss.elastic.co/t/winlogbeat-failed-to-connect-logstash-server-after-ssl-configuration/278542 "2021-07-14T16:09:43Z")
I iam posting teh photos of my logstash pipeline, and result of winlogbear output when i tried the test, it's refusing the connection. cd .. --- ## [Logstash resarting with error](https://discuss.elastic.co/t/logstash-resarting-with-error/278666)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 0\ **Last updated:** [July 14, 2021, 1:17pm UTC](https://discuss.elastic.co/t/logstash-resarting-with-error/278666 "2021-07-14T13:17:12Z")
Hey, I have 3 logstash instances running but each gets restarted after a while with this error: \[2021-07-14T14:35:23,972\]\[FATAL\]\[logstash.runner \] An unexpected error occurred! {:error=\>java.lang.IllegalStateEx… --- ## [Filebeat fails to process kibana json logs “failed to format message from \*json-.log “in a docker enviroment with logstash](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295)
**Author:** [@Annette1](https://discuss.elastic.co/u/Annette1)\ **Replies:** 7\ **Last updated:** [July 14, 2021, 11:40am UTC](https://discuss.elastic.co/t/filebeat-fails-to-process-kibana-json-logs-failed-to-format-message-from-json-log-in-a-docker-enviroment-with-logstash/278295 "2021-07-14T11:40:32Z")
Problem description - since I have installed logstash I am seeing the following in kibana logs - failed to format message from /var/lib/docker/.containers/xxx-json.log If I remove logstash and send directly through ela… --- ## [Force! Logstash keeps restarting](https://discuss.elastic.co/t/force-logstash-keeps-restarting/278624)
**Author:** [@rojin](https://discuss.elastic.co/u/rojin)\ **Replies:** 0\ **Last updated:** [July 14, 2021, 7:37am UTC](https://discuss.elastic.co/t/force-logstash-keeps-restarting/278624 "2021-07-14T07:37:24Z")
Hi! Two of the logstash instances keep restarting giving this error in the log. The other is OK and does not. Can someone help me? \[2021-07-14T11:58:40,333\]\[FATAL\]\[logstash.runner \] An unexpected error occurred… --- ## [Logstash S3 plugin cannot assume a across-account role with external\_id](https://discuss.elastic.co/t/logstash-s3-plugin-cannot-assume-a-across-account-role-with-external-id/278623)
**Author:** [@wilsonwang](https://discuss.elastic.co/u/wilsonwang)\ **Replies:** 0\ **Last updated:** [July 14, 2021, 7:33am UTC](https://discuss.elastic.co/t/logstash-s3-plugin-cannot-assume-a-across-account-role-with-external-id/278623 "2021-07-14T07:33:03Z")
I am using logstash S3 plugin which installed on an EC2 to ingest log from S3 in another account B. It runs well we I configure the role\_arn in conf of S3 plugin, but when a external\_ID is added to the role, the problem… --- ## [How can I see which line of the csv file I am on?](https://discuss.elastic.co/t/how-can-i-see-which-line-of-the-csv-file-i-am-on/278613)
**Author:** [@Busra\_Duygu](https://discuss.elastic.co/u/Busra_Duygu)\ **Replies:** 4\ **Last updated:** [July 14, 2021, 6:39am UTC](https://discuss.elastic.co/t/how-can-i-see-which-line-of-the-csv-file-i-am-on/278613 "2021-07-14T06:39:43Z")
Hello friends, I have a question. When transferring data to elasticsearch with Logstash, can I see which line of the csv file it is in? For example, the field named path shows me which csv file I am in. So, is there a fi… --- ## [How to use nested JSON fields to search for content and drop it](https://discuss.elastic.co/t/how-to-use-nested-json-fields-to-search-for-content-and-drop-it/278576)
**Author:** [@Alex\_Pilon](https://discuss.elastic.co/u/Alex_Pilon)\ **Replies:** 11\ **Last updated:** [July 13, 2021, 7:13pm UTC](https://discuss.elastic.co/t/how-to-use-nested-json-fields-to-search-for-content-and-drop-it/278576 "2021-07-13T19:13:28Z")
Hi all, Been stuck on this since yesterday, and now I'm at a loss. Basically: Using the Azure EventHub input plugin, I'm ingesting Azure logs from various sources, one of the being Azure SQL Audit logs, which are all i… --- ## [Logstash record\_last\_run updated on output issue](https://discuss.elastic.co/t/logstash-record-last-run-updated-on-output-issue/278570)
**Author:** [@spawnrider](https://discuss.elastic.co/u/spawnrider)\ **Replies:** 1\ **Last updated:** [July 13, 2021, 5:10pm UTC](https://discuss.elastic.co/t/logstash-record-last-run-updated-on-output-issue/278570 "2021-07-13T17:10:21Z")
Hi, Why Logstash (JDBC input plugin) is recording the "Record Last Run" timestamp in the metadata file even if there is an issue on the output ? I had an connection issue (due to network) to the Elasticsearch instance … --- ## [Grokparsefailure when %{TIMESTAMP\_ISO8601} pattern is in the first position parser](https://discuss.elastic.co/t/grokparsefailure-when-timestamp-iso8601-pattern-is-in-the-first-position-parser/278308)
**Author:** [@Adixon\_Diaz](https://discuss.elastic.co/u/Adixon_Diaz)\ **Replies:** 5\ **Last updated:** [July 13, 2021, 3:38pm UTC](https://discuss.elastic.co/t/grokparsefailure-when-timestamp-iso8601-pattern-is-in-the-first-position-parser/278308 "2021-07-13T15:38:25Z")
Hi Elastic Team, I have a issue when in my grok filter y put first the pattern %{TIMESTAMP\_ISO8601}. I share a example: With %{TIMESTAMP\_ISO8601} at first position: grok Debugger: I got \[1\] "\_grokparsefailure" … --- ## [Aggregate filter plugin session duration](https://discuss.elastic.co/t/aggregate-filter-plugin-session-duration/278099)
**Author:** [@gentle\_ghost](https://discuss.elastic.co/u/gentle_ghost)\ **Replies:** 13\ **Last updated:** [July 13, 2021, 1:51pm UTC](https://discuss.elastic.co/t/aggregate-filter-plugin-session-duration/278099 "2021-07-13T13:51:28Z")
Hello, Reviewing ways to calculate session duration and using transforms I was able to calculate session duration by finding the min and max time between timestamps. This is fine except in cases where there is no uniqu… --- ## [Filter for parsing Forticlient, Fortiweb, Fortisandbox, Fortimail](https://discuss.elastic.co/t/filter-for-parsing-forticlient-fortiweb-fortisandbox-fortimail/278549)
**Author:** [@Aleksei](https://discuss.elastic.co/u/Aleksei)\ **Replies:** 0\ **Last updated:** [July 13, 2021, 1:47pm UTC](https://discuss.elastic.co/t/filter-for-parsing-forticlient-fortiweb-fortisandbox-fortimail/278549 "2021-07-13T13:47:34Z")
Hello everyone. Who ever integration to Elastic Fortinet production - Forticlient AV, FortiWeb, Fortisandbox, Fortimail ? --- ## [CEF codec outputting multiple CEF logs into single CEF l](https://discuss.elastic.co/t/cef-codec-outputting-multiple-cef-logs-into-single-cef-l/278477)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 4\ **Last updated:** [July 13, 2021, 12:35pm UTC](https://discuss.elastic.co/t/cef-codec-outputting-multiple-cef-logs-into-single-cef-l/278477 "2021-07-13T12:35:13Z")
Hey Everyone, I am having an issue when I output to a syslog sever using the CEF codec from the logs we receive from Filbeat. The logs contain mulitple individual CEF longs into one log, so sometimes you can get 3 diff… --- ## [Logstash line codec not visible](https://discuss.elastic.co/t/logstash-line-codec-not-visible/278529)
**Author:** [@chapmantrain](https://discuss.elastic.co/u/chapmantrain)\ **Replies:** 0\ **Last updated:** [July 13, 2021, 11:35am UTC](https://discuss.elastic.co/t/logstash-line-codec-not-visible/278529 "2021-07-13T11:35:26Z")
Not real where to go from where. I can see the codec: C:\\logstash-7.11.0\\bin\>logstash-plugin list Using JAVA\_HOME defined java: C:\\Program Files\\Java\\jdk-14.0.1 WARNING, using JAVA\_HOME while Logstash distribution comes… --- ## [Logstash "Duplicate field 'match'](https://discuss.elastic.co/t/logstash-duplicate-field-match/278524)
**Author:** [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Replies:** 0\ **Last updated:** [July 13, 2021, 10:42am UTC](https://discuss.elastic.co/t/logstash-duplicate-field-match/278524 "2021-07-13T10:42:39Z")
Hi all I am using following input in logstash conf file to checking matching of a field. input { elasticsearch { hosts =\> \["http://localhost:9200/"\] index =\> "log\*" query =\> '{ "query": { "bool" : { … --- ## [How to prevent duplicate and has null value documents with fingerprint](https://discuss.elastic.co/t/how-to-prevent-duplicate-and-has-null-value-documents-with-fingerprint/278150)
**Author:** [@Busra\_Duygu](https://discuss.elastic.co/u/Busra_Duygu)\ **Replies:** 16\ **Last updated:** [July 13, 2021, 10:33am UTC](https://discuss.elastic.co/t/how-to-prevent-duplicate-and-has-null-value-documents-with-fingerprint/278150 "2021-07-13T10:33:20Z")
my csv file =\> name,surname,age,email,phone Harry,Potter,18,NULL,NULL Harry,Potter,NULL,harrypotter@gmail.com,+955555555 Harry,Potter,NULL,harrypotter@gmail.com,NULL Harry,Potter,NULL,NULL,+955555555 When I want to dete… --- ## [Block in initialize'", block in healthcheck!' \[LOGSTASH\]](https://discuss.elastic.co/t/block-in-initialize-block-in-healthcheck-logstash/278432)
**Author:** [@Mesut\_Yilmaz](https://discuss.elastic.co/u/Mesut_Yilmaz)\ **Replies:** 4\ **Last updated:** [July 13, 2021, 12:32am UTC](https://discuss.elastic.co/t/block-in-initialize-block-in-healthcheck-logstash/278432 "2021-07-13T00:32:34Z")
I installed brand new logstash and I got the same error. logstash.conf is working config and systax is ok but I got pipeline error. systemctl start logstash - is running but on logstash-plain.log there are some errors as… --- ## [How to remove duplicate document and field with null value](https://discuss.elastic.co/t/how-to-remove-duplicate-document-and-field-with-null-value/278424)
**Author:** [@Busra\_Duygu](https://discuss.elastic.co/u/Busra_Duygu)\ **Replies:** 0\ **Last updated:** [July 12, 2021, 12:04pm UTC](https://discuss.elastic.co/t/how-to-remove-duplicate-document-and-field-with-null-value/278424 "2021-07-12T12:04:29Z")
Hello friends, there are duplicate lines with empty values in the csv file. I want to delete the null fields in these records and overwrite them with other records. i managed to delete fields with null value. However, I … --- ## [Unable to handshake Logstash and Elastic after TLS encryption](https://discuss.elastic.co/t/unable-to-handshake-logstash-and-elastic-after-tls-encryption/278319)
**Author:** [@Debarati\_Goswami](https://discuss.elastic.co/u/Debarati_Goswami)\ **Replies:** 1\ **Last updated:** [July 12, 2021, 9:37am UTC](https://discuss.elastic.co/t/unable-to-handshake-logstash-and-elastic-after-tls-encryption/278319 "2021-07-12T09:37:05Z")
Hi All , This is the first time I am building up an ELK stack for my customer . After enabling minimal , basic and basic+HTTPs security for Elastic Stack , I am unable to run pipelines due to certification related issue… --- ## [Logstash is not showing in kibana UI](https://discuss.elastic.co/t/logstash-is-not-showing-in-kibana-ui/278395)
**Author:** [@MdRashid](https://discuss.elastic.co/u/MdRashid)\ **Replies:** 0\ **Last updated:** [July 12, 2021, 9:00am UTC](https://discuss.elastic.co/t/logstash-is-not-showing-in-kibana-ui/278395 "2021-07-12T09:00:20Z")
\#\[Couldn’t find any Elasticsearch data Hi All, i have install elk as a docker container but issue not able to see es data in kibana here is the http://95.217.144.48:9200/ in browser it show me { "name" : "92bbcfa73… --- ## [Rolling pipelines.separate\_logs](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925)
**Author:** [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\ **Replies:** 3\ **Last updated:** [July 12, 2021, 7:20am UTC](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925 "2021-07-12T07:20:06Z")
Logstash v.7.13.2 Hi all, I'm successfully running 3 logstash instances via the following pipelines.yml file: - pipeline.id: upsert queue.type: persisted queue.checkpoint.writes: 1 path.config: "\ **Author:** [@gangireddy\_l](https://discuss.elastic.co/u/gangireddy_l)\ **Replies:** 1\ **Last updated:** [July 12, 2021, 7:14am UTC](https://discuss.elastic.co/t/logstash-file-watcher-logs/278236 "2021-07-12T07:14:34Z") Hi team, I configured logstash to read to read /var/log/app/app.log file and it is able to read and send logs but data is not continuous. Is there any way to monitor file watcher in logstash. Like HARVESTER in filebe… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=210) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=212)