# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=365 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 366 --- ## [Inconvenient size limitations in Logstash server](https://discuss.elastic.co/t/inconvenient-size-limitations-in-logstash-server/230535)
**Author:** [@Razan\_Abdel\_Qader](https://discuss.elastic.co/u/Razan_Abdel_Qader)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 11:57am UTC](https://discuss.elastic.co/t/inconvenient-size-limitations-in-logstash-server/230535 "2020-04-30T11:57:09Z")
Logstash is not receiving logs using UDP protocol (tiny limit on the message size) the connection between Logstash server and my staging server is failing in sending out data , after investigating it appears that it’s a … --- ## [How to extract part of a string from a field in json format and put it into a self added field?](https://discuss.elastic.co/t/how-to-extract-part-of-a-string-from-a-field-in-json-format-and-put-it-into-a-self-added-field/230532)
**Author:** [@iammanmale](https://discuss.elastic.co/u/iammanmale)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 11:52am UTC](https://discuss.elastic.co/t/how-to-extract-part-of-a-string-from-a-field-in-json-format-and-put-it-into-a-self-added-field/230532 "2020-04-30T11:52:14Z")
As the title, I successfully use json filter to extract the items from the data which is in json format, but i cannot extract part of a string from one of the field extracted and put it into another self added field. I … --- ## [Logstash connectively issue to elasticsearch](https://discuss.elastic.co/t/logstash-connectively-issue-to-elasticsearch/230501)
**Author:** [@hello4usharath](https://discuss.elastic.co/u/hello4usharath)\ **Replies:** 14\ **Last updated:** [April 30, 2020, 11:35am UTC](https://discuss.elastic.co/t/logstash-connectively-issue-to-elasticsearch/230501 "2020-04-30T11:35:18Z")
\[2020-04-30T00:45:37,515\]\[ERROR\]\[logstash.outputs.elasticsearch\] Encountered a retryable error. Will Retry with exponential backoff {:code=\>400, :url=\>"http://es-stage-c1-vip.xxxxxx.com:9200/\_bulk"} \[2020-04-30T00:45:4… --- ## [Exception mechanism of handling db failure conection](https://discuss.elastic.co/t/exception-mechanism-of-handling-db-failure-conection/230519)
**Author:** [@jagan](https://discuss.elastic.co/u/jagan)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 10:32am UTC](https://discuss.elastic.co/t/exception-mechanism-of-handling-db-failure-conection/230519 "2020-04-30T10:32:26Z")
Hi mates, I have a issue related to database connection failure.Some attributes comes from kafka and some other attributes comes from different databases and ingest to ES via logstash. All the events gets ingested very… --- ## [Grok stalling logstash](https://discuss.elastic.co/t/grok-stalling-logstash/230431)
**Author:** [@ksarpong](https://discuss.elastic.co/u/ksarpong)\ **Replies:** 3\ **Last updated:** [April 30, 2020, 10:01am UTC](https://discuss.elastic.co/t/grok-stalling-logstash/230431 "2020-04-30T10:01:43Z")
I am trying to parse via grok , a palo log in the format \<14\>Apr 29 19:48:21 XXX-PANO-LOG2 1,2020/04/29 19:48:20,013101003497,TRAFFIC,end,2049,2020/04/29 19:48:19,123.456.78.910,X.X.X.X,0.0.0.0,0.0.0.0,Skype Stun Traffi… --- ## [Duplicate entries from logstash exec plugin](https://discuss.elastic.co/t/duplicate-entries-from-logstash-exec-plugin/230510)
**Author:** [@Saravana37](https://discuss.elastic.co/u/Saravana37)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 9:30am UTC](https://discuss.elastic.co/t/duplicate-entries-from-logstash-exec-plugin/230510 "2020-04-30T09:30:41Z")
Hello All , We are using exec plugin in logstash to run a batch file for every 5 mins and send it to Elastic search. But weirdly we are observing duplicate entries intermittently . Kindly find the attached screenshot fo… --- ## [Split array with key+value fields to key: value](https://discuss.elastic.co/t/split-array-with-key-value-fields-to-key-value/230506)
**Author:** [@Rudolf\_Reddy\_Macejka](https://discuss.elastic.co/u/Rudolf_Reddy_Macejka)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 9:10am UTC](https://discuss.elastic.co/t/split-array-with-key-value-fields-to-key-value/230506 "2020-04-30T09:10:11Z")
Hello, I am trying to convert this input { "request": { "time": "2020-04-30T07:32:13.997Z", "size": "0", "headers": \[ { "key": "Correlation-Id", "value": "124f4cdf-… --- ## [Logstash to Oracle DB using SSL](https://discuss.elastic.co/t/logstash-to-oracle-db-using-ssl/230280)
**Author:** [@randheerpolepelly](https://discuss.elastic.co/u/randheerpolepelly)\ **Replies:** 1\ **Last updated:** [April 30, 2020, 8:41am UTC](https://discuss.elastic.co/t/logstash-to-oracle-db-using-ssl/230280 "2020-04-30T08:41:03Z")
Hi, I am new to Logstash and I didn't find documentation for connecting to oracle DB using SSL. Please help. Currently, I am using the below details in the pipeline. jdbc\_driver\_library =\> "/usr/share/logstash/drive… --- ## [Logasth jdbc connection](https://discuss.elastic.co/t/logasth-jdbc-connection/230489)
**Author:** [@Mateuszpk](https://discuss.elastic.co/u/Mateuszpk)\ **Replies:** 0\ **Last updated:** [April 30, 2020, 7:39am UTC](https://discuss.elastic.co/t/logasth-jdbc-connection/230489 "2020-04-30T07:39:59Z")
Hi How I can limit the connection to database in logstash jdbc plugin. Currently I have 250 SQL stmt and all SQL are excutting in the same time. Is it possible to change logstash beahaviou and let say execute all quer… --- ## [All fields are mapped as text](https://discuss.elastic.co/t/all-fields-are-mapped-as-text/230373)
**Author:** [@martin.s](https://discuss.elastic.co/u/martin.s)\ **Replies:** 4\ **Last updated:** [April 30, 2020, 5:29am UTC](https://discuss.elastic.co/t/all-fields-are-mapped-as-text/230373 "2020-04-30T05:29:52Z")
Hello, I wasn't sure if this is more an Elasticsearch or Logstash issue, hope this is the right place. As you can probably guess I am rather new to Elasticsearch. This is Elasticsearch 6.8 on CentOS 7.7. I have a CSV … --- ## [Logstash Help Config file](https://discuss.elastic.co/t/logstash-help-config-file/230451)
**Author:** [@vstevenson](https://discuss.elastic.co/u/vstevenson)\ **Replies:** 2\ **Last updated:** [April 30, 2020, 12:44am UTC](https://discuss.elastic.co/t/logstash-help-config-file/230451 "2020-04-30T00:44:58Z")
Hi, I am trying to ingest a bunch of logs into my Elastic cluster through Logstash, however I am having a hard time with the config file. Here is what my log files look like: #Software: Microsoft Internet Informat… --- ## [Kafka Input JSON Codec](https://discuss.elastic.co/t/kafka-input-json-codec/230406)
**Author:** [@lifeofguenter](https://discuss.elastic.co/u/lifeofguenter)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 11:52pm UTC](https://discuss.elastic.co/t/kafka-input-json-codec/230406 "2020-04-29T23:52:17Z")
From version 3.0.0 the default codec for kafka-input-plugin has been changed from json to plain (see: https://github.com/logstash-plugins/logstash-input-kafka/blob/master/CHANGELOG.md#300beta2-1). One possibility would … --- ## [Do I need to convert fields in logstash if I have a index template in elastic?](https://discuss.elastic.co/t/do-i-need-to-convert-fields-in-logstash-if-i-have-a-index-template-in-elastic/230434)
**Author:** [@ElasticLiver](https://discuss.elastic.co/u/ElasticLiver)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 11:45pm UTC](https://discuss.elastic.co/t/do-i-need-to-convert-fields-in-logstash-if-i-have-a-index-template-in-elastic/230434 "2020-04-29T23:45:52Z")
Hi, I've been converting some fields in logstash, but I was wondering if I really need to do this if I have a index template with those fields already created in elastic mutate { convert =\> { … --- ## [Assigning an aggregated event timestamp to @timestamp](https://discuss.elastic.co/t/assigning-an-aggregated-event-timestamp-to-timestamp/229381)
**Author:** [@xyz2](https://discuss.elastic.co/u/xyz2)\ **Replies:** 5\ **Last updated:** [April 29, 2020, 9:57pm UTC](https://discuss.elastic.co/t/assigning-an-aggregated-event-timestamp-to-timestamp/229381 "2020-04-29T21:57:58Z")
HI, I am aggragating events for each session from log I fetch the log time from event in aggregate filter, i want to have a field startdatetime and assign session open time to this, set timeout\_timestamp\_field =\> "St… --- ## [Logstash is not load-balanced properly](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021)
**Author:** [@Rakesh\_B](https://discuss.elastic.co/u/Rakesh_B)\ **Replies:** 4\ **Last updated:** [April 29, 2020, 7:49pm UTC](https://discuss.elastic.co/t/logstash-is-not-load-balanced-properly/230021 "2020-04-29T19:49:21Z")
Hi, We are using logstash version: 7.6.1, from the helm chart - https://github.com/elastic/helm-charts/tree/master/logstash. resources: requests: cpu: "2" memory: "6Gi" limits: cpu: "2" memory: "6G… --- ## [Translate filter with dictionary\_path and a special delimiter](https://discuss.elastic.co/t/translate-filter-with-dictionary-path-and-a-special-delimiter/230407)
**Author:** [@theo1991](https://discuss.elastic.co/u/theo1991)\ **Replies:** 3\ **Last updated:** [April 29, 2020, 6:55pm UTC](https://discuss.elastic.co/t/translate-filter-with-dictionary-path-and-a-special-delimiter/230407 "2020-04-29T18:55:20Z")
Hello, Is it possible to specify a delimiter in a translate filter and how ? I have a dictionnary in csv where the delimiter is | This is my current filter : translate { field =\> \[field\] destination =\> \[destination\] … --- ## [Logstash start error 7.3.0 version](https://discuss.elastic.co/t/logstash-start-error-7-3-0-version/230418)
**Author:** [@krishna\_bhargav](https://discuss.elastic.co/u/krishna_bhargav)\ **Replies:** 0\ **Last updated:** [April 29, 2020, 6:27pm UTC](https://discuss.elastic.co/t/logstash-start-error-7-3-0-version/230418 "2020-04-29T18:27:32Z")
:\\Logstash\\logstash-7.3.0\\bin\>logstash Thread.exclusive is deprecated, use Thread::Mutex \[ERROR\] 2020-04-29 14:23:40.972 \[main\] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an erro… --- ## [If and problem](https://discuss.elastic.co/t/if-and-problem/230222)
**Author:** [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\ **Replies:** 10\ **Last updated:** [April 29, 2020, 5:16pm UTC](https://discuss.elastic.co/t/if-and-problem/230222 "2020-04-29T17:16:50Z")
I have an erreur if use this condition in logstsh if \[job\] == "MNNATY0P02" and \[statut\] == "OK" { mutate { add\_field =\> { "statut\_globale" =\> "1" } } } --- ## [Help getting started](https://discuss.elastic.co/t/help-getting-started/230404)
**Author:** [@thenewguy](https://discuss.elastic.co/u/thenewguy)\ **Replies:** 0\ **Last updated:** [April 29, 2020, 4:00pm UTC](https://discuss.elastic.co/t/help-getting-started/230404 "2020-04-29T16:00:36Z")
I am trying to make sense of how to use logstash but I am not getting anywhere. I am trying to deploy the logstash container to receive gelf messages from the docker logger. How do I use the environment variable config… --- ## [Getting \_dateparsefailure tag](https://discuss.elastic.co/t/getting-dateparsefailure-tag/230375)
**Author:** [@Hari\_Krishna](https://discuss.elastic.co/u/Hari_Krishna)\ **Replies:** 4\ **Last updated:** [April 29, 2020, 2:54pm UTC](https://discuss.elastic.co/t/getting-dateparsefailure-tag/230375 "2020-04-29T14:54:56Z")
I have to replace the @timestamp in kibana with the timestamp in my log.But when i try to do that it doesn't replace, but it adds a \_dateparsefailure tag. Here is my filter filter { grok { break\_on\_match =\> false … --- ## [Http poller with integraion key](https://discuss.elastic.co/t/http-poller-with-integraion-key/230381)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 0\ **Last updated:** [April 29, 2020, 1:50pm UTC](https://discuss.elastic.co/t/http-poller-with-integraion-key/230381 "2020-04-29T13:50:16Z")
Hi All, We are trying to ingest using logstash from servicenow, generally we used to get restapi url and user credentials to configure the logstash config file. Now the scenario is like they have given us a restapi url… --- ## [Logstash - http input plugin](https://discuss.elastic.co/t/logstash-http-input-plugin/230284)
**Author:** [@nages](https://discuss.elastic.co/u/nages)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 1:39pm UTC](https://discuss.elastic.co/t/logstash-http-input-plugin/230284 "2020-04-29T13:39:59Z")
As of now, http input plugin acts as "server" and listens for incoming connections .Is it possible to configure to connect is as "client" like tcp plugin does. tcp plugin had a mode attribute which will enable to act it … --- ## [Lostash std err and out file oversize](https://discuss.elastic.co/t/lostash-std-err-and-out-file-oversize/230297)
**Author:** [@kopilim96](https://discuss.elastic.co/u/kopilim96)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 1:35pm UTC](https://discuss.elastic.co/t/lostash-std-err-and-out-file-oversize/230297 "2020-04-29T13:35:32Z")
Hi all, one of my colleagues called me yesterday and I was told that there are 2 logfile from Logstash are growing oversize, screenshot as below: So the scenario is I am running 2 logstash as background services 24/7 … --- ## [Logstash not running continously](https://discuss.elastic.co/t/logstash-not-running-continously/230312)
**Author:** [@john.wambua](https://discuss.elastic.co/u/john.wambua)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 1:27pm UTC](https://discuss.elastic.co/t/logstash-not-running-continously/230312 "2020-04-29T13:27:24Z")
Hi, Am having trouble to keep my logstash running continously.It runs the after a day or 2 it stops running so i have to manually restart it.What could be causing this someone help --- ## [Best practice for adding tag in an inexpensive way?](https://discuss.elastic.co/t/best-practice-for-adding-tag-in-an-inexpensive-way/230321)
**Author:** [@omersiar](https://discuss.elastic.co/u/omersiar)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 1:26pm UTC](https://discuss.elastic.co/t/best-practice-for-adding-tag-in-an-inexpensive-way/230321 "2020-04-29T13:26:24Z")
Hello All, Trying to understand structure of Logstash pipeline configuration, it seems almost every plugin (that are shipped by default) has some common functions like adding fields, removing tags, etc. My question is … --- ## [JSON Parse Failure using logstash](https://discuss.elastic.co/t/json-parse-failure-using-logstash/230013)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 4\ **Last updated:** [April 29, 2020, 1:25pm UTC](https://discuss.elastic.co/t/json-parse-failure-using-logstash/230013 "2020-04-29T13:25:18Z")
Hi All, I'm trying to ingest a json file using logstash adn i'm getting json parse failure message, can someone guide me where i'm doing the mistake My json FIle \[ { "Org": "ogr1", "date\_last": "2020-01-14 0… --- ## [Logstash not able to ingest csvv](https://discuss.elastic.co/t/logstash-not-able-to-ingest-csvv/230326)
**Author:** [@richylyq](https://discuss.elastic.co/u/richylyq)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 1:19pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-ingest-csvv/230326 "2020-04-29T13:19:31Z")
Hi all, I just started working with logstash yesterday and could ingest my file successfully, but since this morning, I could not ingest using the same config file that was used yesterday and also a new config file tha… --- ## [How can I parse nested json in message?](https://discuss.elastic.co/t/how-can-i-parse-nested-json-in-message/230348)
**Author:** [@Win32Sector](https://discuss.elastic.co/u/Win32Sector)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 1:10pm UTC](https://discuss.elastic.co/t/how-can-i-parse-nested-json-in-message/230348 "2020-04-29T13:10:35Z")
Hi All! Can u help me with parsing log messages, for example: \`{"log":"{"timestamp":"2020-04-20T10:07:17.483Z","version":"1","message":"request","logger\_name":"WebClientLoggingHandler","thread\_name":"reactor-http-epoll… --- ## [How to use aggregate function in logstash to calculate difference between 'elapsed' time fields?](https://discuss.elastic.co/t/how-to-use-aggregate-function-in-logstash-to-calculate-difference-between-elapsed-time-fields/230004)
**Author:** [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Replies:** 4\ **Last updated:** [April 28, 2020, 7:16pm UTC](https://discuss.elastic.co/t/how-to-use-aggregate-function-in-logstash-to-calculate-difference-between-elapsed-time-fields/230004 "2020-04-28T19:16:03Z")
I am using the latest version of logstash(7.6.2). I am trying to calculate the total time spent in system excluding time spent between Out2 and In2 i.e. \[(Out3-In1) - (Out2-In2)\] for a given ID. Find below my sample da… --- ## [Elastic search memory](https://discuss.elastic.co/t/elastic-search-memory/230355)
**Author:** [@Lynd](https://discuss.elastic.co/u/Lynd)\ **Replies:** 1\ **Last updated:** [April 29, 2020, 12:45pm UTC](https://discuss.elastic.co/t/elastic-search-memory/230355 "2020-04-29T12:45:03Z")
I tried to set the heap size of elastic search .so I have set it as -Xmx64m and -Xms64m.but when I restarted the es and check for the memory its still showing 2g.then I commented out those two setting written as follows… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=364) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=366)