# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=65 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 66 --- ## [Getting user id from logstash](https://discuss.elastic.co/t/getting-user-id-from-logstash/339504)
**Author:** [@frh](https://discuss.elastic.co/u/frh)\ **Replies:** 1\ **Last updated:** [July 31, 2023, 10:01am UTC](https://discuss.elastic.co/t/getting-user-id-from-logstash/339504 "2023-07-31T10:01:29Z")
Hi, I've been trying to get this output in kibana by modifying my logstash, but to no avail. I'm not sure what went wrong. Input: User 'xxxxxx' logged in with concurrent ALM My logstash looks something like this: matc… --- ## [Getting error when trying to run a filebeat](https://discuss.elastic.co/t/getting-error-when-trying-to-run-a-filebeat/339651)
**Author:** [@rkannan](https://discuss.elastic.co/u/rkannan)\ **Replies:** 3\ **Last updated:** [July 31, 2023, 9:00am UTC](https://discuss.elastic.co/t/getting-error-when-trying-to-run-a-filebeat/339651 "2023-07-31T09:00:36Z")
Exiting: fileset tomcat/error is configured but doesn't exist --- ## [Multiple JDBC input for different tables and output into separate indexes](https://discuss.elastic.co/t/multiple-jdbc-input-for-different-tables-and-output-into-separate-indexes/339596)
**Author:** [@Youdeep](https://discuss.elastic.co/u/Youdeep)\ **Replies:** 1\ **Last updated:** [July 31, 2023, 7:58am UTC](https://discuss.elastic.co/t/multiple-jdbc-input-for-different-tables-and-output-into-separate-indexes/339596 "2023-07-31T07:58:29Z")
Hello I'm new to ELK. Question - How do I use different index when importing tables from DB using logstash. I have used multiple JDBC input for different tables and separate output for each table in logstash. Logstash s… --- ## [How to forward ALL logs](https://discuss.elastic.co/t/how-to-forward-all-logs/339653)
**Author:** [@willsy](https://discuss.elastic.co/u/willsy)\ **Replies:** 0\ **Last updated:** [July 31, 2023, 7:20am UTC](https://discuss.elastic.co/t/how-to-forward-all-logs/339653 "2023-07-31T07:20:41Z")
I have the following logstash configuration file that successfully sends information to a third party location. Effectively what i am asking is, how do i constantly send ALL the data going into elastic to this third par… --- ## [Logstash querying elasticsearch timeout error](https://discuss.elastic.co/t/logstash-querying-elasticsearch-timeout-error/339085)
**Author:** [@willsy](https://discuss.elastic.co/u/willsy)\ **Replies:** 4\ **Last updated:** [July 31, 2023, 7:09am UTC](https://discuss.elastic.co/t/logstash-querying-elasticsearch-timeout-error/339085 "2023-07-31T07:09:33Z")
Hello, I have the following error; just seeing if anyone knows where i am setting this? i originally put the timeout setting in the testpipeline.conf for logstash. Any help is greatly appreciated \[2023-07-24T11:59:41,3… --- ## [Rename nested field in Logstash using Ruby filter](https://discuss.elastic.co/t/rename-nested-field-in-logstash-using-ruby-filter/339637)
**Author:** [@mario\_kazela](https://discuss.elastic.co/u/mario_kazela)\ **Replies:** 1\ **Last updated:** [July 31, 2023, 3:57am UTC](https://discuss.elastic.co/t/rename-nested-field-in-logstash-using-ruby-filter/339637 "2023-07-31T03:57:52Z")
Hi, I have some issues with rename a nested field in json. Example of nested field: test\_results.result.legacy.entities.user\_mentions.name then i want to rename it to displayname I have try this method, but unfortuna… --- ## [Pipeline Fail, failed to load pipeline. Error: Expected one of \[ \\\\t\\\\r\\\\n\], \\"#\\", \\"input\\", \\"filter\\", \\"output\\" at line 21](https://discuss.elastic.co/t/pipeline-fail-failed-to-load-pipeline-error-expected-one-of-t-r-n-input-filter-output-at-line-21/339615)
**Author:** [@Youdeep](https://discuss.elastic.co/u/Youdeep)\ **Replies:** 3\ **Last updated:** [July 31, 2023, 12:05am UTC](https://discuss.elastic.co/t/pipeline-fail-failed-to-load-pipeline-error-expected-one-of-t-r-n-input-filter-output-at-line-21/339615 "2023-07-31T00:05:17Z")
Running a pipeline with two config file: Error after running: logstash -f .\\config\\pipelines.yml Error: \[ERROR\]\[logstash.agent \] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_i… --- ## [Split type failure Logstash](https://discuss.elastic.co/t/split-type-failure-logstash/339594)
**Author:** [@Bharat\_Lahori](https://discuss.elastic.co/u/Bharat_Lahori)\ **Replies:** 2\ **Last updated:** [July 29, 2023, 5:58pm UTC](https://discuss.elastic.co/t/split-type-failure-logstash/339594 "2023-07-29T17:58:16Z")
Dear Team, I have configured below logstash conf file . Trying to give stdin input and getting an error as split type failure. PFB details. We need to create two events based on metricValues. Conf file input { stdi… --- ## [Logstash Json Parsing Error](https://discuss.elastic.co/t/logstash-json-parsing-error/339515)
**Author:** [@fizem](https://discuss.elastic.co/u/fizem)\ **Replies:** 1\ **Last updated:** [July 28, 2023, 9:03pm UTC](https://discuss.elastic.co/t/logstash-json-parsing-error/339515 "2023-07-28T21:03:46Z")
Hi, I have setup the following pipeline to consolidate my logs in Elastic Search Cluster : filebeat to gather nginx logs ==\> logstash to parse the log and mutate them if needed ==\> Elasticsearch cluster. I'm facing an… --- ## [How to validate a json value is numeric](https://discuss.elastic.co/t/how-to-validate-a-json-value-is-numeric/339562)
**Author:** [@sc5283](https://discuss.elastic.co/u/sc5283)\ **Replies:** 4\ **Last updated:** [July 28, 2023, 9:01pm UTC](https://discuss.elastic.co/t/how-to-validate-a-json-value-is-numeric/339562 "2023-07-28T21:01:27Z")
noob question I have a JSON as follows: {"attr1":"One", "attr2":"300"} {"attr1":"Two","attr2":45.0} {"attr1":"Three","attr2":"Not Set"} attr2 is a numeric value How do I check if attr2 is numeric, not a string befo… --- ## [I want send a duplicate or clone of my data throught logstash to another kibana/elastic adminitrador](https://discuss.elastic.co/t/i-want-send-a-duplicate-or-clone-of-my-data-throught-logstash-to-another-kibana-elastic-adminitrador/339228)
**Author:** [@hlcxpl](https://discuss.elastic.co/u/hlcxpl)\ **Replies:** 3\ **Last updated:** [July 28, 2023, 5:10pm UTC](https://discuss.elastic.co/t/i-want-send-a-duplicate-or-clone-of-my-data-throught-logstash-to-another-kibana-elastic-adminitrador/339228 "2023-07-28T17:10:56Z")
how could i duplicate the data or send de same data to another elastic, the logstash version is 7.17 while elastic version where i want to receive is 8.8.2, i try with the output configuration but i received this err… --- ## [Validate document before sending to elasticsearch](https://discuss.elastic.co/t/validate-document-before-sending-to-elasticsearch/337859)
**Author:** [@A\_Mightiev](https://discuss.elastic.co/u/A_Mightiev)\ **Replies:** 2\ **Last updated:** [July 28, 2023, 3:19pm UTC](https://discuss.elastic.co/t/validate-document-before-sending-to-elasticsearch/337859 "2023-07-28T15:19:00Z")
Hi, I have a strict mapping in my ES cluster and send documents via Logstash, sometimes the documents get dropped because they don't conform the strict mapping, is there a way to check if the document conforms or not to… --- ## [How to use the JSON filter correctly?](https://discuss.elastic.co/t/how-to-use-the-json-filter-correctly/339372)
**Author:** [@Chel\_Db](https://discuss.elastic.co/u/Chel_Db)\ **Replies:** 13\ **Last updated:** [July 28, 2023, 2:27pm UTC](https://discuss.elastic.co/t/how-to-use-the-json-filter-correctly/339372 "2023-07-28T14:27:51Z")
Application logs is of below JSON format and I'm unsure what should be the source field incase I'm using the JSON filter ? I would like to have all the fields appear on the Kibana output, particularly the message field,… --- ## [Need help dropping specific messages](https://discuss.elastic.co/t/need-help-dropping-specific-messages/339441)
**Author:** [@RJC](https://discuss.elastic.co/u/RJC)\ **Replies:** 4\ **Last updated:** [July 28, 2023, 12:31pm UTC](https://discuss.elastic.co/t/need-help-dropping-specific-messages/339441 "2023-07-28T12:31:15Z")
My logstash server generates the following messages every time it is restarted: {"syslog\_severity\_code":5,"syslog\_severity":"notice","syslog\_facility\_code":1,"message":"\\u0000\\u0016\\u0000\\u0014\\u0000\\u0017\\u0000\\u0018\\u… --- ## [Logstash - GCP cloud storage Output plugin](https://discuss.elastic.co/t/logstash-gcp-cloud-storage-output-plugin/339524)
**Author:** [@Luko](https://discuss.elastic.co/u/Luko)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 9:48am UTC](https://discuss.elastic.co/t/logstash-gcp-cloud-storage-output-plugin/339524 "2023-07-28T09:48:21Z")
Hello Does Losgatsh Output plugin - google\_cloud\_storage, support the action based on the content of the field. I want to do something like that: output { google\_cloud\_storage { bucket =\> "bucket\_name/%{… --- ## [Logstash- How to parse formatted JSON arrays in log files](https://discuss.elastic.co/t/logstash-how-to-parse-formatted-json-arrays-in-log-files/339498)
**Author:** [@fisher\_he](https://discuss.elastic.co/u/fisher_he)\ **Replies:** 0\ **Last updated:** [July 28, 2023, 6:15am UTC](https://discuss.elastic.co/t/logstash-how-to-parse-formatted-json-arrays-in-log-files/339498 "2023-07-28T06:15:33Z")
Logstash version: 7.17.10 Elasticsearch version:7.17.10 The logs are located in /var/logs directory and the format is as below: xxx.log \[ { "t": "SYS", "dt": "2023-04-17 19:46:40.147 GMT-04:00", "c": "M… --- ## [Data relation using logstash conf file](https://discuss.elastic.co/t/data-relation-using-logstash-conf-file/338925)
**Author:** [@Vinod\_Kumar2](https://discuss.elastic.co/u/Vinod_Kumar2)\ **Replies:** 6\ **Last updated:** [July 28, 2023, 4:20am UTC](https://discuss.elastic.co/t/data-relation-using-logstash-conf-file/338925 "2023-07-28T04:20:37Z")
Multiple csv files in data folder and one column in common to relate the data between files. created logstash conf file and running manually and logstash gets shutdown. Sample data: File1:sample\_orders.csv id,product… --- ## [Logstash HTTP client](https://discuss.elastic.co/t/logstash-http-client/339483)
**Author:** [@Arjun\_Nambiar](https://discuss.elastic.co/u/Arjun_Nambiar)\ **Replies:** 1\ **Last updated:** [July 27, 2023, 11:55pm UTC](https://discuss.elastic.co/t/logstash-http-client/339483 "2023-07-27T23:55:08Z")
I have an Elasticsearch cluster running on AWS which has an Elastic Load balancer(ELB) in front of it. I am scanning the ELB log file to find the clients writing to the Elasticsearch cluster. Logstash is also one of the … --- ## [Help with logstash output](https://discuss.elastic.co/t/help-with-logstash-output/339469)
**Author:** [@nach\_usal](https://discuss.elastic.co/u/nach_usal)\ **Replies:** 5\ **Last updated:** [July 27, 2023, 9:45pm UTC](https://discuss.elastic.co/t/help-with-logstash-output/339469 "2023-07-27T21:45:28Z")
Hi guys, I am trying to capture login and logout events in programs such as TeamViewer and AnyDesk, installed in a Windows virtual machine. Then I send them to my Logstash server via the same Filebeat node, here is the … --- ## [Split filter and add\_field encoding object to a JSON string](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590)
**Author:** [@MrOg](https://discuss.elastic.co/u/MrOg)\ **Replies:** 7\ **Last updated:** [July 27, 2023, 4:46pm UTC](https://discuss.elastic.co/t/split-filter-and-add-field-encoding-object-to-a-json-string/337590 "2023-07-27T16:46:34Z")
Hi, I have such a set of filters filter { json { source =\> "\[sql\_data\]\[response\]" } split { field =\> "docs" add\_field =\> { "id" =\> "%{\[docs\]\[id\]}" "names" =\> … --- ## [Extract fields from a field and add the total count](https://discuss.elastic.co/t/extract-fields-from-a-field-and-add-the-total-count/339386)
**Author:** [@joshuskarki](https://discuss.elastic.co/u/joshuskarki)\ **Replies:** 0\ **Last updated:** [July 27, 2023, 6:09am UTC](https://discuss.elastic.co/t/extract-fields-from-a-field-and-add-the-total-count/339386 "2023-07-27T06:09:12Z")
I have this logs (json format) imported into elastic via logstash. "fields.models": "{'msp': '1', 'tcl': '1'}", with logstash, how do we extract the model name and calculate the total count The desired output should a… --- ## [Struggling with '-' into field as value](https://discuss.elastic.co/t/struggling-with-into-field-as-value/339230)
**Author:** [@yquirion](https://discuss.elastic.co/u/yquirion)\ **Replies:** 3\ **Last updated:** [July 26, 2023, 10:51pm UTC](https://discuss.elastic.co/t/struggling-with-into-field-as-value/339230 "2023-07-26T22:51:09Z")
Greetings, For very long time, I'm struggling with those errors into my logstash server: \[2023-07-25T17:13:15,009\]\[WARN \]\[logstash.outputs.elasticsearch\]\[5555\_winlogbeat\]\[413af53fed5d62fe27389e3c6e0cc4781e6d3cffc048c8a… --- ## [Logstash pipeline is getting killed with jnr.enxio.channels.NativeException: Error closing fd 272: Stale file handle"](https://discuss.elastic.co/t/logstash-pipeline-is-getting-killed-with-jnr-enxio-channels-nativeexception-error-closing-fd-272-stale-file-handle/339328)
**Author:** [@jayanthi\_c](https://discuss.elastic.co/u/jayanthi_c)\ **Replies:** 2\ **Last updated:** [July 26, 2023, 6:23pm UTC](https://discuss.elastic.co/t/logstash-pipeline-is-getting-killed-with-jnr-enxio-channels-nativeexception-error-closing-fd-272-stale-file-handle/339328 "2023-07-26T18:23:59Z")
I am using filebeat to transfer log to logstash but we see that pipeline is getting killed with the below error jnr.enxio.channels.NativeException: Error closing fd 272: Stale file handle" Can someone please help --- ## [Logstash stops processing syslog messages when DNS server not available](https://discuss.elastic.co/t/logstash-stops-processing-syslog-messages-when-dns-server-not-available/339334)
**Author:** [@RJC](https://discuss.elastic.co/u/RJC)\ **Replies:** 0\ **Last updated:** [July 26, 2023, 5:07pm UTC](https://discuss.elastic.co/t/logstash-stops-processing-syslog-messages-when-dns-server-not-available/339334 "2023-07-26T17:07:09Z")
Running Logstash 8.5.2 on RHEL. I implemented DNS filter plugin to resolve IP addresses to hostnames for all syslog nodes reporting to this logstash server. I am using our local DNS server. It all worked perfectly unti… --- ## [Error logstash \[logstash.outputs.elasticsearch\] Encountered a retryable error code=\>503](https://discuss.elastic.co/t/error-logstash-logstash-outputs-elasticsearch-encountered-a-retryable-error-code-503/328331)
**Author:** [@San9](https://discuss.elastic.co/u/San9)\ **Replies:** 13\ **Last updated:** [July 26, 2023, 2:56pm UTC](https://discuss.elastic.co/t/error-logstash-logstash-outputs-elasticsearch-encountered-a-retryable-error-code-503/328331 "2023-07-26T14:56:45Z")
Hi all. I'm a beginner at this. When setting up another pipelayer, after starting it, the following errors started to appear in the log for all other pipelines: logstash\[363391\]: \[2023-03-23T08:05:49,424\]\[ERROR\]\[logsta… --- ## [Log Retention Issue - Only 10 Days of Logs Kept, Need Assistance](https://discuss.elastic.co/t/log-retention-issue-only-10-days-of-logs-kept-need-assistance/339307)
**Author:** [@7a6b6f](https://discuss.elastic.co/u/7a6b6f)\ **Replies:** 2\ **Last updated:** [July 26, 2023, 1:16pm UTC](https://discuss.elastic.co/t/log-retention-issue-only-10-days-of-logs-kept-need-assistance/339307 "2023-07-26T13:16:23Z")
Hi everyone, I am facing an issue with log retention in my Elastic Stack setup and could use some help in troubleshooting it. Currently, my system is only retaining logs for 10 days, and after that, the logs are being d… --- ## [Multiline Filter : How to group error logs with stacktrace to elastic search using logstash?](https://discuss.elastic.co/t/multiline-filter-how-to-group-error-logs-with-stacktrace-to-elastic-search-using-logstash/338952)
**Author:** [@karthi.charles](https://discuss.elastic.co/u/karthi.charles)\ **Replies:** 2\ **Last updated:** [July 26, 2023, 11:57am UTC](https://discuss.elastic.co/t/multiline-filter-how-to-group-error-logs-with-stacktrace-to-elastic-search-using-logstash/338952 "2023-07-26T11:57:45Z")
I am trying to group Error logs which having stacktrace information using multiline filter. Not sure how to set pattern correctly. Kindly help me to config the correct pattern. This is my logging pattern, INFO | 2023-… --- ## [Process logs of different formats to JSON](https://discuss.elastic.co/t/process-logs-of-different-formats-to-json/339258)
**Author:** [@Chel\_Db](https://discuss.elastic.co/u/Chel_Db)\ **Replies:** 0\ **Last updated:** [July 26, 2023, 6:45am UTC](https://discuss.elastic.co/t/process-logs-of-different-formats-to-json/339258 "2023-07-26T06:45:43Z")
I'm pretty new to ELK and I'm trying to push few of our service's logs to ES. Log funneling flow is --\> \` Fluentd --\> Logstash --\> ES --\> Kibana. \` A thing to note is that, each service has its own log format. Attach… --- ## [Multiple pipelines bug with pipe-to-pipe config and CEF codec](https://discuss.elastic.co/t/multiple-pipelines-bug-with-pipe-to-pipe-config-and-cef-codec/338889)
**Author:** [@Markenstein](https://discuss.elastic.co/u/Markenstein)\ **Replies:** 23\ **Last updated:** [July 25, 2023, 3:52pm UTC](https://discuss.elastic.co/t/multiple-pipelines-bug-with-pipe-to-pipe-config-and-cef-codec/338889 "2023-07-25T15:52:09Z")
Hi, everyone! I have faced with such problem: several CEF strings pushed into the following pipelines configuration causing \_cefparseerror in result cause to incorrect string in the input. It's break original message in… --- ## [Filtering messages from Logstash codec rubydebug output](https://discuss.elastic.co/t/filtering-messages-from-logstash-codec-rubydebug-output/339212)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 0\ **Last updated:** [July 25, 2023, 3:15pm UTC](https://discuss.elastic.co/t/filtering-messages-from-logstash-codec-rubydebug-output/339212 "2023-07-25T15:15:06Z")
Our logtsash conf file is using tcp input plugin to ingest messages from different ports. The output part is as follows: output { if \[@metadata\]\[indexPrefix\] { file { path =\> "/opt/total/l… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=64) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=66)