# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=257 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 258 --- ## [GSUB & Regular Expression](https://discuss.elastic.co/t/gsub-regular-expression/263442)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 1\ **Last updated:** [February 5, 2021, 6:12pm UTC](https://discuss.elastic.co/t/gsub-regular-expression/263442 "2021-02-05T18:12:39Z")
I have a text field being ingested that I want to remove an email disclaimer from, unfortunately, I can't figure out the correct syntax to remove it. I've got the mutate filter setup in the following way: mutate { gs… --- ## [Running logstash pipeline is not working](https://discuss.elastic.co/t/running-logstash-pipeline-is-not-working/263394)
**Author:** [@Srinivas\_RK](https://discuss.elastic.co/u/Srinivas_RK)\ **Replies:** 2\ **Last updated:** [February 5, 2021, 4:48pm UTC](https://discuss.elastic.co/t/running-logstash-pipeline-is-not-working/263394 "2021-02-05T16:48:01Z")
input { jdbc { id =\> "my\_plugin\_id" jdbc\_connection\_string =\> "jdbc:sqlserver://abc.database.windows.net:1433;databaseName=db;" jdbc\_driver\_library =\> "" jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.… --- ## [Changing file path name every five minutes in output plugin](https://discuss.elastic.co/t/changing-file-path-name-every-five-minutes-in-output-plugin/263383)
**Author:** [@ritusingh](https://discuss.elastic.co/u/ritusingh)\ **Replies:** 1\ **Last updated:** [February 5, 2021, 4:36pm UTC](https://discuss.elastic.co/t/changing-file-path-name-every-five-minutes-in-output-plugin/263383 "2021-02-05T16:36:01Z")
Hi, My requirement is my file name should change every 5 minutes . Currently I am using below configuration to change it every minute. Please tell me some way to change it every 5 minute. output { stdout { codec =\> ru… --- ## [How to filter data from beat in logstash](https://discuss.elastic.co/t/how-to-filter-data-from-beat-in-logstash/263418)
**Author:** [@IL\_Mare](https://discuss.elastic.co/u/IL_Mare)\ **Replies:** 0\ **Last updated:** [February 5, 2021, 3:31pm UTC](https://discuss.elastic.co/t/how-to-filter-data-from-beat-in-logstash/263418 "2021-02-05T15:31:17Z")
Hello. I need to add data comes from beats to ElasticSearch. But before I need to filter to keep only records that have "status" and status is not "sent". I wrote next configuration for it. input { beats { port =\> … --- ## [Duplicate records in Elasticsearch](https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658)
**Author:** [@sachin1112](https://discuss.elastic.co/u/sachin1112)\ **Replies:** 2\ **Last updated:** [February 5, 2021, 2:22pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658 "2021-02-05T14:22:18Z")
Hello Elastic Team, We are creating our Agile dashboards using ELK Stack , we are able to extract data and push it to ES without any issues for the first time. but when we are pushing the incremental data , we are getti… --- ## [Too many logstash processes are running on server](https://discuss.elastic.co/t/too-many-logstash-processes-are-running-on-server/263391)
**Author:** [@hariskhalique](https://discuss.elastic.co/u/hariskhalique)\ **Replies:** 0\ **Last updated:** [February 5, 2021, 1:53pm UTC](https://discuss.elastic.co/t/too-many-logstash-processes-are-running-on-server/263391 "2021-02-05T13:53:46Z")
Hi, I am using logstash pipeline to dump data in elastic search. Only two configuration define in pipeline. But when I htop my server there are lots of processes. following are screen short of htop. Also getting fol… --- ## [Kafka Topics](https://discuss.elastic.co/t/kafka-topics/263379)
**Author:** [@Abderrahmen\_Ridha](https://discuss.elastic.co/u/Abderrahmen_Ridha)\ **Replies:** 0\ **Last updated:** [February 5, 2021, 11:51am UTC](https://discuss.elastic.co/t/kafka-topics/263379 "2021-02-05T11:51:18Z")
how can i get all Kafka Topics in Logstash ? --- ## [Snmp agent configuration for logstash](https://discuss.elastic.co/t/snmp-agent-configuration-for-logstash/261339)
**Author:** [@shehzadshaikh](https://discuss.elastic.co/u/shehzadshaikh)\ **Replies:** 2\ **Last updated:** [February 5, 2021, 11:24am UTC](https://discuss.elastic.co/t/snmp-agent-configuration-for-logstash/261339 "2021-02-05T11:24:56Z")
I'm trying to pull data from snmp agent directly to logstash, i went through online documents but none of the are descriptive and show what exactly configuration to put on the smp agent. Also while there is plugins avail… --- ## [@timestamp format while exporting from elasticsearch to csv](https://discuss.elastic.co/t/timestamp-format-while-exporting-from-elasticsearch-to-csv/263137)
**Author:** [@ritusingh](https://discuss.elastic.co/u/ritusingh)\ **Replies:** 5\ **Last updated:** [February 5, 2021, 10:15am UTC](https://discuss.elastic.co/t/timestamp-format-while-exporting-from-elasticsearch-to-csv/263137 "2021-02-05T10:15:12Z")
Hi, I want to know how can I change the format of @timestamp while exporting from elasticsearch to csv via logstash. In Kibana I have kept the format as "Feb-03-2021 20:03:04.603" using advance settings. I want same for… --- ## [Ssl not working in elasticsearch input plugin](https://discuss.elastic.co/t/ssl-not-working-in-elasticsearch-input-plugin/263357)
**Author:** [@ritusingh](https://discuss.elastic.co/u/ritusingh)\ **Replies:** 1\ **Last updated:** [February 5, 2021, 10:13am UTC](https://discuss.elastic.co/t/ssl-not-working-in-elasticsearch-input-plugin/263357 "2021-02-05T10:13:59Z")
Hi, I am fetching logs from elasticsearch via logstash but if I use ssl I get the following error. 2056 error: 2056 2056 2056 Manticore::ClientProtocolException 2056 PKIX path building failed: sun.… --- ## [Filebeat not publishing logs to Logstash on sebp/elk docker image](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-logstash-on-sebp-elk-docker-image/263351)
**Author:** [@Pavitar\_Dua](https://discuss.elastic.co/u/Pavitar_Dua)\ **Replies:** 0\ **Last updated:** [February 5, 2021, 8:44am UTC](https://discuss.elastic.co/t/filebeat-not-publishing-logs-to-logstash-on-sebp-elk-docker-image/263351 "2021-02-05T08:44:36Z")
I run filebeat with this command 'sudo ./filebeat -e -c filebeat.yml'. 2021-02-05T14:10:29.915+0530 INFO instance/beat.go:640 Home path: \[/Users/pavitardua/filebeat-7.9.2-darwin-x86\_64\] Config path: \[/Users/pavitardua/… --- ## [How to send syslog to multiple destinations?](https://discuss.elastic.co/t/how-to-send-syslog-to-multiple-destinations/263354)
**Author:** [@kelk](https://discuss.elastic.co/u/kelk)\ **Replies:** 0\ **Last updated:** [February 5, 2021, 8:52am UTC](https://discuss.elastic.co/t/how-to-send-syslog-to-multiple-destinations/263354 "2021-02-05T08:52:23Z")
I'm using logstash syslog module to send customized outputs to multiple destinations/port I've done as below, but somehow only the 1st destination receives message. Is there another format to have multiple syslog outpu… --- ## [Syslog-ng logstash date\_time\_parse\_exception](https://discuss.elastic.co/t/syslog-ng-logstash-date-time-parse-exception/263104)
**Author:** [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\ **Replies:** 1\ **Last updated:** [February 5, 2021, 7:57am UTC](https://discuss.elastic.co/t/syslog-ng-logstash-date-time-parse-exception/263104 "2021-02-05T07:57:56Z")
Hello! I am trying to send logs from Syslog-ng to logstash and no logs are indexed in elasticsearch because of date time parse exception. I cannot use Beats on the devices, so the only option is Syslog-ng. Could anyone … --- ## [How to customize the name of the s3 part file](https://discuss.elastic.co/t/how-to-customize-the-name-of-the-s3-part-file/263334)
**Author:** [@kaushik.vankayala](https://discuss.elastic.co/u/kaushik.vankayala)\ **Replies:** 0\ **Last updated:** [February 5, 2021, 7:00am UTC](https://discuss.elastic.co/t/how-to-customize-the-name-of-the-s3-part-file/263334 "2021-02-05T07:00:33Z")
Hi There, I use the S3 output plugin for persisting the raw data of our Mulesoft applicaiton. I have following configuration is my s3 output plugin; output { s3 { id =\> "experience\_rt4\_s3\_backup" codec =\> line {… --- ## [Got error in logstash. help me!](https://discuss.elastic.co/t/got-error-in-logstash-help-me/262587)
**Author:** [@Farmer\_J](https://discuss.elastic.co/u/Farmer_J)\ **Replies:** 2\ **Last updated:** [February 5, 2021, 6:39am UTC](https://discuss.elastic.co/t/got-error-in-logstash-help-me/262587 "2021-02-05T06:39:37Z")
hi guys . i got some error code in AMI when i excute logstash config file for indexing these is error codes=\> Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash… --- ## [Delete doc.delete](https://discuss.elastic.co/t/delete-doc-delete/263333)
**Author:** [@Farmer\_J](https://discuss.elastic.co/u/Farmer_J)\ **Replies:** 0\ **Last updated:** [February 5, 2021, 6:38am UTC](https://discuss.elastic.co/t/delete-doc-delete/263333 "2021-02-05T06:38:08Z")
good after noon mates. im using logstash on AMI i have question about doc.delete data Im trying to delete doc.delete datas is it properly command shell mate? curl -X POST "https://search-everon-jkgsx53bmgr3lhvcwpgyl… --- ## [Field reference with a split event](https://discuss.elastic.co/t/field-reference-with-a-split-event/263315)
**Author:** [@jiacob](https://discuss.elastic.co/u/jiacob)\ **Replies:** 1\ **Last updated:** [February 5, 2021, 12:20am UTC](https://discuss.elastic.co/t/field-reference-with-a-split-event/263315 "2021-02-05T00:20:25Z")
Hello, I'm looking into trying to split a nested field that uses a field reference and I'm not sure which is the right way of doing it. split { field =\> "\[field1\]\[field2\]\['%{\[field3\]}'\]" } This gives me a field refere… --- ## [Date format error from logstash](https://discuss.elastic.co/t/date-format-error-from-logstash/263299)
**Author:** [@earlsanchez](https://discuss.elastic.co/u/earlsanchez)\ **Replies:** 4\ **Last updated:** [February 4, 2021, 11:26pm UTC](https://discuss.elastic.co/t/date-format-error-from-logstash/263299 "2021-02-04T23:26:33Z")
Hello, I'm getting a date format error from logstash but the format seems to match the posted date object. Can some one see specifically what it is complaining about? Logstash error: {"error":{"root\_cause":\[{"type":"m… --- ## [Drop documents when similar to previous processed ones](https://discuss.elastic.co/t/drop-documents-when-similar-to-previous-processed-ones/262484)
**Author:** [@Ernesto\_Guerra](https://discuss.elastic.co/u/Ernesto_Guerra)\ **Replies:** 2\ **Last updated:** [February 4, 2021, 5:01pm UTC](https://discuss.elastic.co/t/drop-documents-when-similar-to-previous-processed-ones/262484 "2021-02-04T17:01:15Z")
Hello, I'm pretty new to this but I would like to know if there is the possibility to drop one document in a logstash configuration file if the timestamp is similar to a previous one processed. For example, I would like… --- ## [Logstash JVM Heap Errors (Not Allocating to Xms)](https://discuss.elastic.co/t/logstash-jvm-heap-errors-not-allocating-to-xms/263270)
**Author:** [@randyitguy](https://discuss.elastic.co/u/randyitguy)\ **Replies:** 0\ **Last updated:** [February 4, 2021, 4:57pm UTC](https://discuss.elastic.co/t/logstash-jvm-heap-errors-not-allocating-to-xms/263270 "2021-02-04T16:57:00Z")
I'm using logstash 7.10. I have 3 pipelines with 3 different input sources. Two out of the three stop producing output after a short number of documents. One of them runs fine. If I try to run logstash with all 3 specifi… --- ## [Logstash elasticsearch output plugin - Populating api\_key from metadata field does not work](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-populating-api-key-from-metadata-field-does-not-work/263259)
**Author:** [@vigneshr35](https://discuss.elastic.co/u/vigneshr35)\ **Replies:** 0\ **Last updated:** [February 4, 2021, 3:13pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-populating-api-key-from-metadata-field-does-not-work/263259 "2021-02-04T15:13:50Z")
Hi All, I am using the elasticsearch output plugin of logstash to post my events to elasticsearch. I am using the api\_key authentication method. It is all working fine until I have the api\_key parameter value hardcoded.… --- ## [How to convert to requried JSON](https://discuss.elastic.co/t/how-to-convert-to-requried-json/263254)
**Author:** [@onkarborade](https://discuss.elastic.co/u/onkarborade)\ **Replies:** 0\ **Last updated:** [February 4, 2021, 2:40pm UTC](https://discuss.elastic.co/t/how-to-convert-to-requried-json/263254 "2021-02-04T14:40:17Z")
Hi, Getting following logstash output "rosterentry" : { "rosterdaterange" : \[ "100,01-JUL-20,01-AUG-22", "100,02-JUL-20,02-AUG-22", "101,03-JUL-19,0… --- ## [Beats require setup with elasticsearch as the output before using logstash as the output](https://discuss.elastic.co/t/beats-require-setup-with-elasticsearch-as-the-output-before-using-logstash-as-the-output/263249)
**Author:** [@bzak](https://discuss.elastic.co/u/bzak)\ **Replies:** 0\ **Last updated:** [February 4, 2021, 2:10pm UTC](https://discuss.elastic.co/t/beats-require-setup-with-elasticsearch-as-the-output-before-using-logstash-as-the-output/263249 "2021-02-04T14:10:10Z")
I have elk running in k8s. When I configure the beats to ship logs straight to elasticsearch, logs flow smoothly. However, I can't get the beats to ship logs to Logstash. There is a work around that I use to get this wor… --- ## [How to use logstash to index file with suggester mapping](https://discuss.elastic.co/t/how-to-use-logstash-to-index-file-with-suggester-mapping/263238)
**Author:** [@ahmed\_hussiney](https://discuss.elastic.co/u/ahmed_hussiney)\ **Replies:** 0\ **Last updated:** [February 4, 2021, 12:32pm UTC](https://discuss.elastic.co/t/how-to-use-logstash-to-index-file-with-suggester-mapping/263238 "2021-02-04T12:32:10Z")
Hello, I am new to Elastic search , and I want to implement auto complete search functionality by indexing data from JSON file , however all the samples I can find for "completion" suggester is using curl or developer t… --- ## [Input with file and pipeline plugin at the sametime](https://discuss.elastic.co/t/input-with-file-and-pipeline-plugin-at-the-sametime/263233)
**Author:** [@hzarrabi](https://discuss.elastic.co/u/hzarrabi)\ **Replies:** 0\ **Last updated:** [February 4, 2021, 11:28am UTC](https://discuss.elastic.co/t/input-with-file-and-pipeline-plugin-at-the-sametime/263233 "2021-02-04T11:28:57Z")
Hi Guys, I'm trying to use in my input.conf file and pipeline plugin at the sametime but logstash crash with the following error message \[FATAL\]\[logstash.runner\] An unexpected error occurred! {:error=\>#\bad request](https://discuss.elastic.co/t/kibana-7-10-0-visualization-throwing-error-on-aggregation-esaggs-bad-request/263086)
**Author:** [@Rakesh\_Rayabarapu](https://discuss.elastic.co/u/Rakesh_Rayabarapu)\ **Replies:** 1\ **Last updated:** [February 4, 2021, 6:23am UTC](https://discuss.elastic.co/t/kibana-7-10-0-visualization-throwing-error-on-aggregation-esaggs-bad-request/263086 "2021-02-04T06:23:09Z")
Hello Team, While we creating a visualization with any aggregation except count renders an error. please find error details below. Thanks in advance. search\_phase\_execution\_exception all shards failed Error: Bad Req… --- ## [Need help to get logstash output in following format](https://discuss.elastic.co/t/need-help-to-get-logstash-output-in-following-format/263198)
**Author:** [@onkarborade](https://discuss.elastic.co/u/onkarborade)\ **Replies:** 0\ **Last updated:** [February 4, 2021, 5:57am UTC](https://discuss.elastic.co/t/need-help-to-get-logstash-output-in-following-format/263198 "2021-02-04T05:57:03Z")
Hi, I am struggling to get bellow format "tch\_id" : 201, "rosterentry": \[ { "rolerosterid": 100, "rosterdaterange": \[ … --- ## [Is there a graphical interface for Logstash?](https://discuss.elastic.co/t/is-there-a-graphical-interface-for-logstash/263024)
**Author:** [@SUMANT\_MISHRA](https://discuss.elastic.co/u/SUMANT_MISHRA)\ **Replies:** 2\ **Last updated:** [February 4, 2021, 3:27am UTC](https://discuss.elastic.co/t/is-there-a-graphical-interface-for-logstash/263024 "2021-02-04T03:27:11Z")
Is there a graphical interface for Logstash? If no, can I get a documentation for highly scalable setup for logstash? As I read another post, it was somewhere down the line in 2015. I am not able to get a full-proof s… --- ## [S\_s\_l\_handshake\_exception error](https://discuss.elastic.co/t/s-s-l-handshake-exception-error/263041)
**Author:** [@edster](https://discuss.elastic.co/u/edster)\ **Replies:** 2\ **Last updated:** [February 4, 2021, 2:42am UTC](https://discuss.elastic.co/t/s-s-l-handshake-exception-error/263041 "2021-02-04T02:42:57Z")
I am getting the following error { "error" : { "root\_cause" : \[ { "type" : "s\_s\_l\_handshake\_exception", "reason" : "Received fatal alert: handshake\_failure" } \], "type" : "s\_s\_l\_handshake\_exception", "reason" … --- ## [Logstash - How can i get the data from a key-value pair?](https://discuss.elastic.co/t/logstash-how-can-i-get-the-data-from-a-key-value-pair/263134)
**Author:** [@vinodhini](https://discuss.elastic.co/u/vinodhini)\ **Replies:** 2\ **Last updated:** [February 3, 2021, 6:00pm UTC](https://discuss.elastic.co/t/logstash-how-can-i-get-the-data-from-a-key-value-pair/263134 "2021-02-03T18:00:38Z")
I have a field which is like "host":{"name":"abcdef"}, How can i get the value "abcdef" alone from the host field? gsub is not allowing me to remove the {} or "name": I tried to get the value by referencing as %{host}{n… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=256) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=258)