# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=513 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 514 --- ## [Creating object with ruby](https://discuss.elastic.co/t/creating-object-with-ruby/189551)
**Author:** [@Rom1](https://discuss.elastic.co/u/Rom1)\ **Replies:** 2\ **Last updated:** [July 9, 2019, 9:48pm UTC](https://discuss.elastic.co/t/creating-object-with-ruby/189551 "2019-07-09T21:48:54Z")
Hi all, I have an event with fields containing tables: {"pfp":{"cfr":\["A", "B", "C"\], "name":\["X", "Y", "Z"\]}} I would like to transform them into objects: "pfp" =\> \[ \[0\] { "cfr" =\> "A", "name" =\> "X" }, \[1\] … --- ## [Logstash sincedb file not getting updated in ELK 6.8.0](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524)
**Author:** [@sureshrupnar](https://discuss.elastic.co/u/sureshrupnar)\ **Replies:** 7\ **Last updated:** [July 10, 2019, 6:48am UTC](https://discuss.elastic.co/t/logstash-sincedb-file-not-getting-updated-in-elk-6-8-0/188524 "2019-07-10T06:48:18Z")
Hi, recently we have migrated to ELK 6.8.0 and while ingesting data via logstash we are using sincedb\_path =\> "/export/vendor.sincedb" in input plugin inside the file component. This file was updated regularly after … --- ## [SOAP request with logstash](https://discuss.elastic.co/t/soap-request-with-logstash/188969)
**Author:** [@He\_Pamela](https://discuss.elastic.co/u/He_Pamela)\ **Replies:** 5\ **Last updated:** [July 9, 2019, 9:06pm UTC](https://discuss.elastic.co/t/soap-request-with-logstash/188969 "2019-07-09T21:06:29Z")
Hi, I'd need to recibe a Soap response from a web service. I coudn't find an oficial documentation and I don't now how to do that with the http poller input from logstash. Somebody nows how to do this? Thanks. --- ## [File\_completed\_action =\> "delete" - Works intermittently](https://discuss.elastic.co/t/file-completed-action-delete-works-intermittently/189609)
**Author:** [@rotem\_lom](https://discuss.elastic.co/u/rotem_lom)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 6:28pm UTC](https://discuss.elastic.co/t/file-completed-action-delete-works-intermittently/189609 "2019-07-09T18:28:37Z")
Hi There, I worked with logstash - 6.7.0 and ES - 6.7.1 I tried to use file\_completed\_action with " delete" and sometimes this action was work well and sometimes not. Does anyone know how "file\_completed\_action" work… --- ## [Logs not being indexed in Elasticsearch from Redis](https://discuss.elastic.co/t/logs-not-being-indexed-in-elasticsearch-from-redis/189489)
**Author:** [@proudboffin](https://discuss.elastic.co/u/proudboffin)\ **Replies:** 2\ **Last updated:** [July 9, 2019, 6:14pm UTC](https://discuss.elastic.co/t/logs-not-being-indexed-in-elasticsearch-from-redis/189489 "2019-07-09T18:14:11Z")
Hi all, Trying to build the following pipeline for apache access logs: Filebeat -\> Redis -\> Logstash -\> Elasticsearch. All components are configured (see below configs) and start successfully: Filebeat publishes the lo… --- ## [Logstash doesn't recognize zeep-python](https://discuss.elastic.co/t/logstash-doesnt-recognize-zeep-python/189601)
**Author:** [@He\_Pamela](https://discuss.elastic.co/u/He_Pamela)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 5:06pm UTC](https://discuss.elastic.co/t/logstash-doesnt-recognize-zeep-python/189601 "2019-07-09T17:06:59Z")
Hi , I have a python Script and I receive a SOAP web service data. in my logstash I call the python script with an exec input and then I send the data to Elasticsearch. When I run logstash manually the data is send it t… --- ## [Substring of column](https://discuss.elastic.co/t/substring-of-column/189121)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 8\ **Last updated:** [July 9, 2019, 4:53pm UTC](https://discuss.elastic.co/t/substring-of-column/189121 "2019-07-09T16:53:23Z")
Need to add a new column with substring/extracted value of available column. Example: Your XYZ Debit Card has been charged $ xxxxx.xx I need to add columns cardtype =\> Debit/Credit Amount =\> xxxxx.xx would great if… --- ## [Grok multiple matches reports failures in an unexpected way](https://discuss.elastic.co/t/grok-multiple-matches-reports-failures-in-an-unexpected-way/189593)
**Author:** [@Sous\_Lesquels](https://discuss.elastic.co/u/Sous_Lesquels)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 4:25pm UTC](https://discuss.elastic.co/t/grok-multiple-matches-reports-failures-in-an-unexpected-way/189593 "2019-07-09T16:25:45Z")
I have this filter: grok { break\_on\_match =\> false match =\> { 'message' =\> \[ 'a(?\\\d+)', 'b(?\\\d+)' \] } } and this input file: a11 b21 a31 b32 When run, this parses correctly, i.e. I get… --- ## [Multiline codec not picking up custom patterns](https://discuss.elastic.co/t/multiline-codec-not-picking-up-custom-patterns/189426)
**Author:** [@Sous\_Lesquels](https://discuss.elastic.co/u/Sous_Lesquels)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 4:18pm UTC](https://discuss.elastic.co/t/multiline-codec-not-picking-up-custom-patterns/189426 "2019-07-09T16:18:56Z")
With this config: input { file { codec =\> multiline { ... patterns\_dir =\> \[ '/path/to/patterns' \] ... } } } and the existing / readable file: /path/to/patter… --- ## [Logstash is not outputing my file with or without grok](https://discuss.elastic.co/t/logstash-is-not-outputing-my-file-with-or-without-grok/189571)
**Author:** [@Eddy\_Meds](https://discuss.elastic.co/u/Eddy_Meds)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 4:09pm UTC](https://discuss.elastic.co/t/logstash-is-not-outputing-my-file-with-or-without-grok/189571 "2019-07-09T16:09:12Z")
Hello, I'm trying to set a simple .txt file as an input. And I want a parsed .txt for the output/result. Inbetween, I'm using a Grok filter to match a specific pattern. This is a simple test and it doesn't work : My… --- ## [New filed is not showing After using GROK Command](https://discuss.elastic.co/t/new-filed-is-not-showing-after-using-grok-command/189580)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 1\ **Last updated:** [July 9, 2019, 3:27pm UTC](https://discuss.elastic.co/t/new-filed-is-not-showing-after-using-grok-command/189580 "2019-07-09T15:27:37Z")
I have a csv file in which i have only one record which is Insert\_time 13/05/2019 0:00:55 Mobile 923xxxxxxxxx Telco : Paktel Mask : 62222 Message : Your MCB Debit Card has been charged PKR x Now i am using grok me… --- ## [Upgrading Logstash from 7.1.1 to 7.2.0 breaks my parsing configuration](https://discuss.elastic.co/t/upgrading-logstash-from-7-1-1-to-7-2-0-breaks-my-parsing-configuration/189176)
**Author:** [@jbrowe](https://discuss.elastic.co/u/jbrowe)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 3:13pm UTC](https://discuss.elastic.co/t/upgrading-logstash-from-7-1-1-to-7-2-0-breaks-my-parsing-configuration/189176 "2019-07-09T15:13:49Z")
I have an appliance that sends JSON formatted logs over a TCP port. The "input" section in my Logstash pipeline configuration file looks like: input { tcp { host =\> "192.168.1.93" port =\> 5501 codec =\> json id … --- ## [KV Whitespace Question](https://discuss.elastic.co/t/kv-whitespace-question/189400)
**Author:** [@swiftingdarkness](https://discuss.elastic.co/u/swiftingdarkness)\ **Replies:** 6\ **Last updated:** [July 9, 2019, 2:01pm UTC](https://discuss.elastic.co/t/kv-whitespace-question/189400 "2019-07-09T14:01:50Z")
I am trying to figure out a way to either: Replace all spaces in the field with underscores OR Capture the entire string with the spaces into the field. Every solution that I have found does not work so far. I've lo… --- ## [Splitting based on different lines and generating multiple events](https://discuss.elastic.co/t/splitting-based-on-different-lines-and-generating-multiple-events/189439)
**Author:** [@Sous\_Lesquels](https://discuss.elastic.co/u/Sous_Lesquels)\ **Replies:** 3\ **Last updated:** [July 9, 2019, 1:56pm UTC](https://discuss.elastic.co/t/splitting-based-on-different-lines-and-generating-multiple-events/189439 "2019-07-09T13:56:15Z")
Say I have a file of this format: a11 a21 b22 c23 a31 a41 a51 b52 c53 I.e. lines can be either: aX aX bY cZ I want to generate: A signle event for aX lines (emitting {a: aX, c: 0}) Two events for aX bY cZ lines (e… --- ## [Deleted .logstash\_jdbc\_last\_run](https://discuss.elastic.co/t/deleted-logstash-jdbc-last-run/189563)
**Author:** [@LouisP](https://discuss.elastic.co/u/LouisP)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 1:52pm UTC](https://discuss.elastic.co/t/deleted-logstash-jdbc-last-run/189563 "2019-07-09T13:52:37Z")
I was using the JDBC plugin with the .logstash\_jdbc\_last\_run and :sql\_last\_value to query a database regularly and make sure that I only get new entries. Then I changed the .conf file and I don't need to use :sql\_last\_va… --- ## [I am facing issues while parsing date using grok plugin in Filter](https://discuss.elastic.co/t/i-am-facing-issues-while-parsing-date-using-grok-plugin-in-filter/189519)
**Author:** [@subrat1512](https://discuss.elastic.co/u/subrat1512)\ **Replies:** 1\ **Last updated:** [July 9, 2019, 1:23pm UTC](https://discuss.elastic.co/t/i-am-facing-issues-while-parsing-date-using-grok-plugin-in-filter/189519 "2019-07-09T13:23:27Z")
I have the below in my logs "2019-07-09\\t07:35:42.210\\t0.193\\t0\\tGET\\t{URI}\\t304\\t{ECID}\\t{someId}\\t-\\t{ip address} I am not able to figure out how should i frame my filter using GROK. I tried with something like the … --- ## [Include logstash-input-azureblob plugin to Elastic](https://discuss.elastic.co/t/include-logstash-input-azureblob-plugin-to-elastic/189556)
**Author:** [@Marusyk](https://discuss.elastic.co/u/Marusyk)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 1:10pm UTC](https://discuss.elastic.co/t/include-logstash-input-azureblob-plugin-to-elastic/189556 "2019-07-09T13:10:42Z")
Are there any plans to include logstash-input-azureblob plugin to Elastic? I found this information about the similar plugin logstash-input-azureeventhub As of April 25th 2018, Microsoft has transitioned the code and … --- ## [Issues with between](https://discuss.elastic.co/t/issues-with-between/185750)
**Author:** [@Harshet\_Jain](https://discuss.elastic.co/u/Harshet_Jain)\ **Replies:** 3\ **Last updated:** [July 9, 2019, 1:04pm UTC](https://discuss.elastic.co/t/issues-with-between/185750 "2019-07-09T13:04:22Z")
I am using the following ruby code but getting an exception about "ruby exception undefined method between? for nil:Nilclass"? event.set("\[weekInMonth\]", 2) if event.get("\[tempDay\]").between?(8, 15) What should I chang… --- ## [Conditional Grok Pattern](https://discuss.elastic.co/t/conditional-grok-pattern/189512)
**Author:** [@AkilaSachin](https://discuss.elastic.co/u/AkilaSachin)\ **Replies:** 1\ **Last updated:** [July 9, 2019, 12:57pm UTC](https://discuss.elastic.co/t/conditional-grok-pattern/189512 "2019-07-09T12:57:42Z")
Hi all, 172.26.76.68 - - \[05/Jul/2019:08:52:55 +0530\] GET /keyword1/rest/keyword2/keyword3/ keyword4/keyword5?moNo=123456 HTTP/1.1 200 40 7 172.26.76.69 - - \[05/Jul/2019:08:52:53 +0530\] GET /keyword1/keyword2/ 1234… --- ## [How to transform fields but dont insert them into elasticsearch](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504)
**Author:** [@Josep\_Soler](https://discuss.elastic.co/u/Josep_Soler)\ **Replies:** 3\ **Last updated:** [July 9, 2019, 12:49pm UTC](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504 "2019-07-09T12:49:15Z")
I would like to insert the document name into de index value via variable but i need to lowercase it, so i have to create a new field, also i would like to change the @timestamp to get %{+YYYY.MM.dd} variable. The probl… --- ## [The given configuration is invalid. Reason: undefined method \`-' for nil:NilClass](https://discuss.elastic.co/t/the-given-configuration-is-invalid-reason-undefined-method-for-nil-nilclass/189499)
**Author:** [@Duella](https://discuss.elastic.co/u/Duella)\ **Replies:** 1\ **Last updated:** [July 9, 2019, 12:21pm UTC](https://discuss.elastic.co/t/the-given-configuration-is-invalid-reason-undefined-method-for-nil-nilclass/189499 "2019-07-09T12:21:09Z")
Hi, I am receiving the below error and am not sure what is causing it. I am just trying to convert eastings and northings to co-ordinates to make the data useful in kibana. Anyone know what could be causing it? \[2019-0… --- ## [Aggregate](https://discuss.elastic.co/t/aggregate/189505)
**Author:** [@Andreasky](https://discuss.elastic.co/u/Andreasky)\ **Replies:** 2\ **Last updated:** [July 9, 2019, 10:51am UTC](https://discuss.elastic.co/t/aggregate/189505 "2019-07-09T10:51:41Z")
aggregate{ task\_id =\> "%{Response}" code =\>" if(event.get(Response) == "answered") map\['answers'\] ||= 0; map\['answers'\] += 1; end … --- ## [No suitable driver found for jdbc:mariadb](https://discuss.elastic.co/t/no-suitable-driver-found-for-jdbc-mariadb/189525)
**Author:** [@pkshara](https://discuss.elastic.co/u/pkshara)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 10:42am UTC](https://discuss.elastic.co/t/no-suitable-driver-found-for-jdbc-mariadb/189525 "2019-07-09T10:42:42Z")
Hi Team, I am trying to connect to mariaDB as input , but facing error as Java::JavaSql::SQLException: No suitable driver found for jdbc:mariadb below are the input configuration in logstash input { jdbc { jdbc\_dri… --- ## [Cloudwatch plugin not working on ApplicationELB Metrics](https://discuss.elastic.co/t/cloudwatch-plugin-not-working-on-applicationelb-metrics/189501)
**Author:** [@jm01065](https://discuss.elastic.co/u/jm01065)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 9:24am UTC](https://discuss.elastic.co/t/cloudwatch-plugin-not-working-on-applicationelb-metrics/189501 "2019-07-09T09:24:51Z")
Hi we use the cloudwatch plugin to get the metrics for ClassicELB's and ApplicationELB's/TargetGroups. For ClassicELB's it works perfect, but for ApplicationELB's/TargetGroups we can't seem to get it working. We want to… --- ## [Updating data doubles and then triples the hit count on kibana](https://discuss.elastic.co/t/updating-data-doubles-and-then-triples-the-hit-count-on-kibana/189458)
**Author:** [@tinamina12](https://discuss.elastic.co/u/tinamina12)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 9:14am UTC](https://discuss.elastic.co/t/updating-data-doubles-and-then-triples-the-hit-count-on-kibana/189458 "2019-07-09T09:14:31Z")
I am trying to parse logs. whenever new logs are written in input file, the logstash reads the whole log file again and adds them in the exisiting parsed logs. my configuration is below. please guide me what am i doing w… --- ## [(Updated 2019/06/19) Seeking suggestion for uploading data via logstash](https://discuss.elastic.co/t/updated-2019-06-19-seeking-suggestion-for-uploading-data-via-logstash/186144)
**Author:** [@leeyu](https://discuss.elastic.co/u/leeyu)\ **Replies:** 13\ **Last updated:** [July 9, 2019, 9:08am UTC](https://discuss.elastic.co/t/updated-2019-06-19-seeking-suggestion-for-uploading-data-via-logstash/186144 "2019-07-09T09:08:51Z")
Hi all, I have some data that I want to upload via logstash. The data is about 100MB a day and stored locally. I want to do the following: Upload the old data from the past 3 years (about 100GB). Upload the new … --- ## [Renaming fields wild card logstash](https://discuss.elastic.co/t/renaming-fields-wild-card-logstash/189396)
**Author:** [@Paulo64](https://discuss.elastic.co/u/Paulo64)\ **Replies:** 2\ **Last updated:** [July 9, 2019, 9:00am UTC](https://discuss.elastic.co/t/renaming-fields-wild-card-logstash/189396 "2019-07-09T09:00:08Z")
Hello, From what I have read, there is not yet wild card associated with fields to use with "mutate". In my logstash config, I am doing something like this : filter { mutate { rename =\> \["Site.Equipment.Me… --- ## [Why my ES mapping that automatically creating through logstash have so many unreadable types?](https://discuss.elastic.co/t/why-my-es-mapping-that-automatically-creating-through-logstash-have-so-many-unreadable-types/189274)
**Author:** [@Haitianisgood](https://discuss.elastic.co/u/Haitianisgood)\ **Replies:** 4\ **Last updated:** [July 9, 2019, 8:54am UTC](https://discuss.elastic.co/t/why-my-es-mapping-that-automatically-creating-through-logstash-have-so-many-unreadable-types/189274 "2019-07-09T08:54:25Z")
ENV: logstash 6.2.4 elasticsearch 6.2.4 My logstash config: filter { if \[type\] == "pro" { ............ prune { whitelist\_names =\> \[ "^@timestamp$","^index\_month$","^type$","^model$","^hos… --- ## [How to complete data in case of service outage?](https://discuss.elastic.co/t/how-to-complete-data-in-case-of-service-outage/189493)
**Author:** [@atis06](https://discuss.elastic.co/u/atis06)\ **Replies:** 0\ **Last updated:** [July 9, 2019, 8:53am UTC](https://discuss.elastic.co/t/how-to-complete-data-in-case-of-service-outage/189493 "2019-07-09T08:53:07Z")
Hi, I wonder if service outage happens e.g. Logstash stops, is there any common practice to complete/fill the data had not piped? BR, Attila --- ## [Add new Field in Logstash](https://discuss.elastic.co/t/add-new-field-in-logstash/189477)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 1\ **Last updated:** [July 9, 2019, 8:37am UTC](https://discuss.elastic.co/t/add-new-field-in-logstash/189477 "2019-07-09T08:37:16Z")
I have this csv file which look like this Insert Time :13/05/2019 0:00:54, Mobile : 923xxxxxxxxx, Telco: Mobilink, Mask : 6222 Message : Your MCB Debit Card has been charged PKR xx I want to find logstash the mess… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=512) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=514)