# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=43 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 44 --- ## [Mutate and gsub usage](https://discuss.elastic.co/t/mutate-and-gsub-usage/351769)
**Author:** [@Sara93](https://discuss.elastic.co/u/Sara93)\ **Replies:** 3\ **Last updated:** [January 29, 2024, 12:32pm UTC](https://discuss.elastic.co/t/mutate-and-gsub-usage/351769 "2024-01-29T12:32:11Z")
Hi, I was trying to fetch below fileds from the log 2024-01-10 04:21:52.018 -06:00 \[INF\] \[2696100223720240110042151-10\] {"Message":"Device\_Response\_2696100223720240110042151-10","ApiEndPoint":"ws://10.136.41.18:50000/"… --- ## [Log Configuration](https://discuss.elastic.co/t/log-configuration/351974)
**Author:** [@tejas.d](https://discuss.elastic.co/u/tejas.d)\ **Replies:** 3\ **Last updated:** [January 29, 2024, 12:18pm UTC](https://discuss.elastic.co/t/log-configuration/351974 "2024-01-29T12:18:52Z")
How can i define Application log files when i have a suffix as system date after .log in filebeat.yml folder Examle: C:/path/folder/\*.log29012024 --- ## [Deleting array fields logstash](https://discuss.elastic.co/t/deleting-array-fields-logstash/351790)
**Author:** [@andrejcoliveira](https://discuss.elastic.co/u/andrejcoliveira)\ **Replies:** 6\ **Last updated:** [January 29, 2024, 11:37am UTC](https://discuss.elastic.co/t/deleting-array-fields-logstash/351790 "2024-01-29T11:37:43Z")
Hi, In my company we're trying to process a json array sent by filiebeat to logstash. The json that logstash receives is: { "logFile": \[ { "line": 2, "elements": \[ { "line": 4, "name": "", "description": "", "type": ""… --- ## [The latest version of logstash 8.12.0. Reports vulnerable](https://discuss.elastic.co/t/the-latest-version-of-logstash-8-12-0-reports-vulnerable/351952)
**Author:** [@ranjini](https://discuss.elastic.co/u/ranjini)\ **Replies:** 2\ **Last updated:** [January 29, 2024, 4:54am UTC](https://discuss.elastic.co/t/the-latest-version-of-logstash-8-12-0-reports-vulnerable/351952 "2024-01-29T04:54:10Z")
How to fix vulnerabilities in the maven-core-3.3.9.jar, maven-compat-3.3.9.jar and derby-10.14.1.0.jar. This is reported vulnerable in the latest Logstash package? DetailedName org.apache.maven:maven-core 3.3.9 … --- ## [Logstash JDBC Output to Postgres case-sensitive issue](https://discuss.elastic.co/t/logstash-jdbc-output-to-postgres-case-sensitive-issue/351881)
**Author:** [@h\_ng\_d\_ng](https://discuss.elastic.co/u/h_ng_d_ng)\ **Replies:** 9\ **Last updated:** [January 28, 2024, 9:53am UTC](https://discuss.elastic.co/t/logstash-jdbc-output-to-postgres-case-sensitive-issue/351881 "2024-01-28T09:53:03Z")
Hi, i got issue with logstash when trying to import data to postgres table. here are my output config: and it always show error that :\[ERROR\] 2024-01-26 20:22:38.348 \[\[main\]\>worker0\] jdbc - JDBC - Exception. Not retr… --- ## [Input Varnish Logs to Logstash](https://discuss.elastic.co/t/input-varnish-logs-to-logstash/351898)
**Author:** [@ugola](https://discuss.elastic.co/u/ugola)\ **Replies:** 1\ **Last updated:** [January 26, 2024, 8:54pm UTC](https://discuss.elastic.co/t/input-varnish-logs-to-logstash/351898 "2024-01-26T20:54:23Z")
My goal is to send varnish logs to Logstash, both are running on different servers. Currently I am able to perform this using rsyslog (I also saw examples of FileBeat being used) on the machine where varnish logs are sto… --- ## [Getting \_grokparsefailure for grok pattern on \[audit\_data\]\[messages\] field for modsecurity json log?](https://discuss.elastic.co/t/getting-grokparsefailure-for-grok-pattern-on-audit-data-messages-field-for-modsecurity-json-log/351831)
**Author:** [@sunnysigara](https://discuss.elastic.co/u/sunnysigara)\ **Replies:** 4\ **Last updated:** [January 26, 2024, 5:52pm UTC](https://discuss.elastic.co/t/getting-grokparsefailure-for-grok-pattern-on-audit-data-messages-field-for-modsecurity-json-log/351831 "2024-01-26T17:52:51Z")
Input Json {"transaction":{"time":"26/Jan/2024:00:54:31 +0530","transaction\_id":"16645304250678661185","remote\_address":"141.98.7.28","remote\_port":80,"local\_address":"127.0.0.1","local\_port":80},"request":{"request\_lin… --- ## [Why \_grokparsefailure?](https://discuss.elastic.co/t/why-grokparsefailure/351859)
**Author:** [@emoxam](https://discuss.elastic.co/u/emoxam)\ **Replies:** 5\ **Last updated:** [January 26, 2024, 12:38pm UTC](https://discuss.elastic.co/t/why-grokparsefailure/351859 "2024-01-26T12:38:14Z")
Part of a config filter { if \[message\] =~ /actions/ or \[message\] =~ /172\\.16\\.10\\.78/ or \[message\] =~ /172\\.16\\.10\\.77/ { grok { match =\> \[ "message", "%{GREEDYDATA:timestamp}%{LOGLEVEL:level}%{GR… --- ## [Failed to install template {:message=\>"Got response code '400' contacting Elasticsearch at URL 'http://x.x.x.x:9200/\_template/ecs-logstash'",](https://discuss.elastic.co/t/failed-to-install-template-message-got-response-code-400-contacting-elasticsearch-at-url-http-x-x-x-x-9200-template-ecs-logstash/351578)
**Author:** [@sunnysigara](https://discuss.elastic.co/u/sunnysigara)\ **Replies:** 5\ **Last updated:** [January 26, 2024, 7:13am UTC](https://discuss.elastic.co/t/failed-to-install-template-message-got-response-code-400-contacting-elasticsearch-at-url-http-x-x-x-x-9200-template-ecs-logstash/351578 "2024-01-26T07:13:27Z")
Using a default mapping template {:es\_version=\>7, :ecs\_compatibility=\>:v8} gives me this error, \[2024-01-23T02:24:07,381\]\[INFO \]\[logstash.outputs.elasticsearch\]\[main\] Elasticsearch version determined (7.13.3) {:es\_versi… --- ## [Configuring Input file on Windows to get Output in 2024](https://discuss.elastic.co/t/configuring-input-file-on-windows-to-get-output-in-2024/351839)
**Author:** [@Will\_J](https://discuss.elastic.co/u/Will_J)\ **Replies:** 2\ **Last updated:** [January 25, 2024, 11:10pm UTC](https://discuss.elastic.co/t/configuring-input-file-on-windows-to-get-output-in-2024/351839 "2024-01-25T23:10:18Z")
I am trying a simple file input / output example. Using following conf file at the C:/logstash-8.11.3-windows-x86\_64/logstash-8.11.3/config/file-pipeline.conf input { file { path =\> \["C:/logstash-8.11.3-wind… --- ## [How to loop the jdbc streaming filter by pass the index of array of object?](https://discuss.elastic.co/t/how-to-loop-the-jdbc-streaming-filter-by-pass-the-index-of-array-of-object/351784)
**Author:** [@kishorkumar](https://discuss.elastic.co/u/kishorkumar)\ **Replies:** 1\ **Last updated:** [January 25, 2024, 5:59pm UTC](https://discuss.elastic.co/t/how-to-loop-the-jdbc-streaming-filter-by-pass-the-index-of-array-of-object/351784 "2024-01-25T17:59:31Z")
So here is the usecase i have the "mainSKU": \[ { "id": 102, }, { "id": 101, }, { "id": 100, } \] like this and i am using jdb\_streaming and every mainSKU has 3 to 4 SKU in order to that i need to loop get … --- ## [Sql escape character in logstash](https://discuss.elastic.co/t/sql-escape-character-in-logstash/351800)
**Author:** [@Rakesh\_Verma](https://discuss.elastic.co/u/Rakesh_Verma)\ **Replies:** 2\ **Last updated:** [January 25, 2024, 5:51pm UTC](https://discuss.elastic.co/t/sql-escape-character-in-logstash/351800 "2024-01-25T17:51:31Z")
I am getting below error in following sql . SELECT HA.HotelID AS HotelCode,'\[' + STUFF((SELECT ',' + '{"AmenityId": ' + CAST(HA\_inner.AmenityId AS VARCHAR(10)) + ', "AmenityName": ' + QUOTENAME(HAT\_inner.AmenityName, '"… --- ## [Import Emails into Elasticsearch using Logstash IMAP Input](https://discuss.elastic.co/t/import-emails-into-elasticsearch-using-logstash-imap-input/351798)
**Author:** [@frank\_esg](https://discuss.elastic.co/u/frank_esg)\ **Replies:** 0\ **Last updated:** [January 25, 2024, 12:38pm UTC](https://discuss.elastic.co/t/import-emails-into-elasticsearch-using-logstash-imap-input/351798 "2024-01-25T12:38:01Z")
Hi, we would like to import Emails into Elasticsearch to have a kind of Email Archive. We started with Logstash and the IMAP Input plugin. But it turned out that this plugin was not updated in the last years and has se… --- ## [How to convert HEXA field into ASCII field through logstash pipeline](https://discuss.elastic.co/t/how-to-convert-hexa-field-into-ascii-field-through-logstash-pipeline/351771)
**Author:** [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Replies:** 1\ **Last updated:** [January 25, 2024, 9:44am UTC](https://discuss.elastic.co/t/how-to-convert-hexa-field-into-ascii-field-through-logstash-pipeline/351771 "2024-01-25T09:44:11Z")
Hi All, we are getting one filed in the form of hexa i need to convert that filed into ASCII through logstash pipeline. Could you please guide me on this? eg:- "abc": "8a64756c656173654368616e" I need to convert "abc"… --- ## [Create Helper Functions within Pipeline](https://discuss.elastic.co/t/create-helper-functions-within-pipeline/351581)
**Author:** [@michael\_c\_michael](https://discuss.elastic.co/u/michael_c_michael)\ **Replies:** 3\ **Last updated:** [January 24, 2024, 11:54pm UTC](https://discuss.elastic.co/t/create-helper-functions-within-pipeline/351581 "2024-01-24T23:54:40Z")
I have a function in my pipeline that does some math and another that does some translation. Is there a way to create a function like in python, so that I don't have to retype the same code everytime I want to do this op… --- ## [Expected one of \[A-Za-z0-9\_-\], \[\\t\\n \\n\], "#","=\>" ... after input {](https://discuss.elastic.co/t/expected-one-of-a-za-z0-9-t-n-n-after-input/351694)
**Author:** [@ZinedineR](https://discuss.elastic.co/u/ZinedineR)\ **Replies:** 2\ **Last updated:** [January 24, 2024, 9:18pm UTC](https://discuss.elastic.co/t/expected-one-of-a-za-z0-9-t-n-n-after-input/351694 "2024-01-24T21:18:42Z")
The error comes in line 19 column 19 in after input{} I think the configuration is correct but the error shows there's unexpected character input { jdbc { jdbc\_driver\_library =\> "$DRIVER\_PATH" jdbc\_connection\_s… --- ## [Data not getting synced properly from SQL to elastic](https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742)
**Author:** [@abhishek\_agarwal](https://discuss.elastic.co/u/abhishek_agarwal)\ **Replies:** 2\ **Last updated:** [January 24, 2024, 5:10pm UTC](https://discuss.elastic.co/t/data-not-getting-synced-properly-from-sql-to-elastic/351742 "2024-01-24T17:10:09Z")
I am syncing data from sql to elastic and in filter in the code block I am appending to map eg features but what is happening is that when I am running the bulk sql query ,not all features are getting appended to each in… --- ## [Fileabeat in UAT is 8.2.0 filebeat in PROD is 7.9.3](https://discuss.elastic.co/t/fileabeat-in-uat-is-8-2-0-filebeat-in-prod-is-7-9-3/351682)
**Author:** [@Hanuma](https://discuss.elastic.co/u/Hanuma)\ **Replies:** 2\ **Last updated:** [January 24, 2024, 1:35pm UTC](https://discuss.elastic.co/t/fileabeat-in-uat-is-8-2-0-filebeat-in-prod-is-7-9-3/351682 "2024-01-24T13:35:42Z")
Hi Team, Can you please let us know the difference between filebeat version 7.9.3 and 8.2.0. Will this create issue in Available fields in Elastic search. we have issue in Available fields not showing in PROD those re… --- ## [Exception caught while applying mutate filter {:exception=\>"Could not set field 'product' on object 'x' to value 'y'.This is probably due to trying to set a field like \[foo\]\[bar\] = someValuewhen \[foo\] is not either a map or a string"}](https://discuss.elastic.co/t/exception-caught-while-applying-mutate-filter-exception-could-not-set-field-product-on-object-x-to-value-y-this-is-probably-due-to-trying-to-set-a-field-like-foo-bar-somevaluewhen-foo-is-not-either-a-map-or-a-string/351707)
**Author:** [@mwitsas](https://discuss.elastic.co/u/mwitsas)\ **Replies:** 1\ **Last updated:** [January 24, 2024, 1:22pm UTC](https://discuss.elastic.co/t/exception-caught-while-applying-mutate-filter-exception-could-not-set-field-product-on-object-x-to-value-y-this-is-probably-due-to-trying-to-set-a-field-like-foo-bar-somevaluewhen-foo-is-not-either-a-map-or-a-string/351707 "2024-01-24T13:22:41Z")
When attempting to rename fields as in the following example: mutate { rename =\> { "vendor" =\> "\[abc\]\[vendor\]" "vendor\_product" =\> "\[abc\]\[vendor\]\[product\]" "vendor\_product\_version" =\> "\[abc\]\[vend… --- ## [Problem with an IF statement not working](https://discuss.elastic.co/t/problem-with-an-if-statement-not-working/351608)
**Author:** [@FaisalParkar](https://discuss.elastic.co/u/FaisalParkar)\ **Replies:** 10\ **Last updated:** [January 24, 2024, 10:15am UTC](https://discuss.elastic.co/t/problem-with-an-if-statement-not-working/351608 "2024-01-24T10:15:42Z")
Hello Everyone, I hope someone is able to assist. I am running ELK stack 8.11.3 and am using Logstash to ingest Syslogs in a CEF format. I have everything working and it works nicely, however I want to add an IF stateme… --- ## [Logstash cannot upload to https Elasticsearch](https://discuss.elastic.co/t/logstash-cannot-upload-to-https-elasticsearch/351601)
**Author:** [@cisupport-zkb](https://discuss.elastic.co/u/cisupport-zkb)\ **Replies:** 13\ **Last updated:** [January 24, 2024, 7:13am UTC](https://discuss.elastic.co/t/logstash-cannot-upload-to-https-elasticsearch/351601 "2024-01-24T07:13:11Z")
Hi everyone, I'm having troubles on uploading data from a csv file to https Elasticsearch, using Logstash. This is the logstash.conf configured: input { file { path =\> "${pwd}/some-metrics.csv" fil… --- ## [Error reading empty line from CSV file with CSV codec](https://discuss.elastic.co/t/error-reading-empty-line-from-csv-file-with-csv-codec/351635)
**Author:** [@tsegars](https://discuss.elastic.co/u/tsegars)\ **Replies:** 2\ **Last updated:** [January 23, 2024, 7:59pm UTC](https://discuss.elastic.co/t/error-reading-empty-line-from-csv-file-with-csv-codec/351635 "2024-01-23T19:59:08Z")
My input CSV files will have empty lines interspersed in them. The CSV codec decoder generates the following error when encountering an empty line: \[ERROR\]\[filewatch.tailmode.handlers.grow\]\[main\]\[62dd5eb2b6763ff5522ed54… --- ## [Failed parsing date from field Oracle alert log](https://discuss.elastic.co/t/failed-parsing-date-from-field-oracle-alert-log/351590)
**Author:** [@Prabhu\_Athithan](https://discuss.elastic.co/u/Prabhu_Athithan)\ **Replies:** 19\ **Last updated:** [January 23, 2024, 7:00pm UTC](https://discuss.elastic.co/t/failed-parsing-date-from-field-oracle-alert-log/351590 "2024-01-23T19:00:26Z")
Failed parsing date from field {:field=\>"timestamp", :value=\>"%{year} %{month} %{monthday} %{time}", :exception=\>"Invalid format: "%{year} %{month} %{monthday} %{t..."", :config\_parsers=\>"yyyy MMM dd HH:mm:ss", :config\_l… --- ## [Help in capturing E2E timestamp from raw logs](https://discuss.elastic.co/t/help-in-capturing-e2e-timestamp-from-raw-logs/351335)
**Author:** [@Anurag101](https://discuss.elastic.co/u/Anurag101)\ **Replies:** 5\ **Last updated:** [January 23, 2024, 5:41pm UTC](https://discuss.elastic.co/t/help-in-capturing-e2e-timestamp-from-raw-logs/351335 "2024-01-23T17:41:55Z")
Hi All, I am new to ELK and am trying to achieve a real time monitoring framework which captures E2E timestamp of an ansync logback application. The ask is to capture the timestamp corresponding to a unique ID in the l… --- ## [Forcing a Logstash pipeline to restart](https://discuss.elastic.co/t/forcing-a-logstash-pipeline-to-restart/351639)
**Author:** [@intrepid1](https://discuss.elastic.co/u/intrepid1)\ **Replies:** 0\ **Last updated:** [January 23, 2024, 3:11pm UTC](https://discuss.elastic.co/t/forcing-a-logstash-pipeline-to-restart/351639 "2024-01-23T15:11:10Z")
Hi there, I have a pipeline that extracts documents from Elasticsearch and sends them to S3. I want to do some reconciliation on the process to prove that the number of documents extracted from Elasticsearch and the num… --- ## [I am trying to deduplicate my events one the basis of timestamp and operation field. But it did not work?](https://discuss.elastic.co/t/i-am-trying-to-deduplicate-my-events-one-the-basis-of-timestamp-and-operation-field-but-it-did-not-work/351599)
**Author:** [@Subrato1](https://discuss.elastic.co/u/Subrato1)\ **Replies:** 0\ **Last updated:** [January 23, 2024, 8:18am UTC](https://discuss.elastic.co/t/i-am-trying-to-deduplicate-my-events-one-the-basis-of-timestamp-and-operation-field-but-it-did-not-work/351599 "2024-01-23T08:18:42Z")
My Log event: { "priority" =\> 13, "host" =\> "172.31.63.35", "consistency" =\> "\\"ONE\\"", "source" =\> "\\"127.0.0.1", "type" =\> "scylladb", "severity" =\> 5… --- ## [Regarding parsing of data in logstash](https://discuss.elastic.co/t/regarding-parsing-of-data-in-logstash/351400)
**Author:** [@Ajay\_Kumar.S](https://discuss.elastic.co/u/Ajay_Kumar.S)\ **Replies:** 2\ **Last updated:** [January 23, 2024, 1:59am UTC](https://discuss.elastic.co/t/regarding-parsing-of-data-in-logstash/351400 "2024-01-23T01:59:48Z")
"message" =\> "{\\"namespace\\":\\"oci\_computeagent\\",\\"resourceGroup\\":null,\\"compartmentId\\":\\"ocid1.compartment.oc1..aaaaaaaacu54zo4clgrmfs3faxqqgfxyu2mjlufgslcem3venf2kon2ktmsq\\",\\"name\\":\\"DiskIopsWritten\\",\\"dimensions… --- ## [Filebeats not sending output to Logstash](https://discuss.elastic.co/t/filebeats-not-sending-output-to-logstash/351154)
**Author:** [@pmuno007](https://discuss.elastic.co/u/pmuno007)\ **Replies:** 1\ **Last updated:** [January 23, 2024, 12:33am UTC](https://discuss.elastic.co/t/filebeats-not-sending-output-to-logstash/351154 "2024-01-23T00:33:26Z")
I suspect filebeats is not sending output to Logstash since Logstash has not created an index in Elastic Search. There are no errors in logs. Filebeat logs also do not indicate nor mention Logstash connection (not sure i… --- ## [Logstash container gets closed automatically after installation of the "logstash-input-mongodb" plugin](https://discuss.elastic.co/t/logstash-container-gets-closed-automatically-after-installation-of-the-logstash-input-mongodb-plugin/351479)
**Author:** [@Vladyslav\_Googlya](https://discuss.elastic.co/u/Vladyslav_Googlya)\ **Replies:** 1\ **Last updated:** [January 23, 2024, 12:16am UTC](https://discuss.elastic.co/t/logstash-container-gets-closed-automatically-after-installation-of-the-logstash-input-mongodb-plugin/351479 "2024-01-23T00:16:14Z")
Hi everyone, I'm trying to run the Logstash container in the docker-compose file to sync my MongoDB data with Elastic. But, after the step of installation of the "logstash-input-MongoDB" plugin, the container gets exited… --- ## [Modsecurity log (split on audit\_data\[messages\]) Only String and Array types are splittable](https://discuss.elastic.co/t/modsecurity-log-split-on-audit-data-messages-only-string-and-array-types-are-splittable/351507)
**Author:** [@sunnysigara](https://discuss.elastic.co/u/sunnysigara)\ **Replies:** 4\ **Last updated:** [January 22, 2024, 8:50pm UTC](https://discuss.elastic.co/t/modsecurity-log-split-on-audit-data-messages-only-string-and-array-types-are-splittable/351507 "2024-01-22T20:50:57Z")
{ "transaction": { "time": "20/Jan/2024:00:10:51 +0530", "transaction\_id": "16717361827536742843", "remote\_address": "20.1.198.110", "remote\_port": 80, "local\_address": "127.0.… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=42) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=44)