# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=280 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 281 --- ## [Create a conf file using grok or json filter](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903)
**Author:** [@Omair](https://discuss.elastic.co/u/Omair)\ **Replies:** 5\ **Last updated:** [November 19, 2020, 2:09pm UTC](https://discuss.elastic.co/t/create-a-conf-file-using-grok-or-json-filter/255903 "2020-11-19T14:09:23Z")
Hi.. I'm new to elastic stack and trying to learn using filters to structure my data but unfortunately not getting much help. Below is the application log 2020-11-19T00:08:34.627+0500 INFO 141799 com.l7tech.server.p… --- ## [In Kibana dashboard, latest logs are not showing](https://discuss.elastic.co/t/in-kibana-dashboard-latest-logs-are-not-showing/255993)
**Author:** [@shajimsh](https://discuss.elastic.co/u/shajimsh)\ **Replies:** 0\ **Last updated:** [November 19, 2020, 1:35pm UTC](https://discuss.elastic.co/t/in-kibana-dashboard-latest-logs-are-not-showing/255993 "2020-11-19T13:35:39Z")
Team, Even though we have latest records in the table, we are not able to see these latest records in kibana dashboard, below is the logstash jdbc query for fetching latest record. But it was working properly previously… --- ## [Logstash help with pattern for grok needed](https://discuss.elastic.co/t/logstash-help-with-pattern-for-grok-needed/255518)
**Author:** [@Christos\_Gitsis](https://discuss.elastic.co/u/Christos_Gitsis)\ **Replies:** 4\ **Last updated:** [November 19, 2020, 1:02pm UTC](https://discuss.elastic.co/t/logstash-help-with-pattern-for-grok-needed/255518 "2020-11-19T13:02:32Z")
Hello, I am trying to parse some data which is in one of the two following formats: Sample data: Data from service service-a loaded Data from service service-b not loaded With the following Grok Pattern: Data from se… --- ## [Parsing logs with logstash](https://discuss.elastic.co/t/parsing-logs-with-logstash/255960)
**Author:** [@Falikou1](https://discuss.elastic.co/u/Falikou1)\ **Replies:** 1\ **Last updated:** [November 19, 2020, 10:58am UTC](https://discuss.elastic.co/t/parsing-logs-with-logstash/255960 "2020-11-19T10:58:25Z")
I need to help on the parsing of my logs. This is my parser: input{ tcp { port =\> "5140" tags =\> "syslog" } } filter { grok { match =\> { "message" =\> "%{SYSLOG5424PRI:syslog\_index}%{SYSLOGHOST:syslog\_host} %{GR… --- ## [Get data from excel file into elasticsearch](https://discuss.elastic.co/t/get-data-from-excel-file-into-elasticsearch/255926)
**Author:** [@111411](https://discuss.elastic.co/u/111411)\ **Replies:** 1\ **Last updated:** [November 19, 2020, 5:49am UTC](https://discuss.elastic.co/t/get-data-from-excel-file-into-elasticsearch/255926 "2020-11-19T05:49:36Z")
hello. I'm try get data from excel file into elasticsearch. Message: Error: wrong number of arguments (given 0, expected 3) Exception: ArgumentError Stack: /Users/nguyenzuanbka/logstash-7.10.0/vendor/bundle/jruby/2.… --- ## [Date formatting](https://discuss.elastic.co/t/date-formatting/255898)
**Author:** [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\ **Replies:** 2\ **Last updated:** [November 18, 2020, 9:56pm UTC](https://discuss.elastic.co/t/date-formatting/255898 "2020-11-18T21:56:31Z")
I have a datafile I wish to ingest using Logstash. Historically I've always used CLI tools to manipulate the date/time fields to be easier to map using a .conf file (my imports tend to be CSV in nature). I was just won… --- ## [Transform date to ELK](https://discuss.elastic.co/t/transform-date-to-elk/255878)
**Author:** [@Matias\_Aguero\_Escoba](https://discuss.elastic.co/u/Matias_Aguero_Escoba)\ **Replies:** 1\ **Last updated:** [November 18, 2020, 9:26pm UTC](https://discuss.elastic.co/t/transform-date-to-elk/255878 "2020-11-18T21:26:20Z")
Hi everyone! my name is Matias Aguero, I need to transform a date value field with the format "2019-08-23 17:28:22 UTC" through a logstash pipeline to the format "2020-10-22T14:42:10". how i do this? Thanks! --- ## [Pipeline to Pipelines configuration](https://discuss.elastic.co/t/pipeline-to-pipelines-configuration/255343)
**Author:** [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Replies:** 3\ **Last updated:** [November 18, 2020, 8:11pm UTC](https://discuss.elastic.co/t/pipeline-to-pipelines-configuration/255343 "2020-11-18T20:11:11Z")
Hello, I want to use the pipeline to pipeline communication in pipeline creation in kibana, But i want the same input to send to 2 different pipelines. My code is the following but doesn't seem to work. input { be… --- ## [Unable to implement ssl\_enable in Logstash](https://discuss.elastic.co/t/unable-to-implement-ssl-enable-in-logstash/255838)
**Author:** [@Christer\_Palmen](https://discuss.elastic.co/u/Christer_Palmen)\ **Replies:** 1\ **Last updated:** [November 18, 2020, 3:00pm UTC](https://discuss.elastic.co/t/unable-to-implement-ssl-enable-in-logstash/255838 "2020-11-18T15:00:00Z")
I have a Nxlog agent sending windows event with om\_tcp module to a Logstash without errors. Now I am trying to implement more secure forwarding with SSL, there was no issues with the Nxlog agent, just configured with th… --- ## [HTTP output failure](https://discuss.elastic.co/t/http-output-failure/255858)
**Author:** [@chitreshg](https://discuss.elastic.co/u/chitreshg)\ **Replies:** 0\ **Last updated:** [November 18, 2020, 2:53pm UTC](https://discuss.elastic.co/t/http-output-failure/255858 "2020-11-18T14:53:18Z")
Hi all, I'm using logstash 7.10.0, I made the config file as input { stdin{} } output{ http { url =\> "{URL\_SET\_CLOG}" format =\> "json" http\_method =\> "post" headers =\> \["Authorization", "Bearer {CLOG\_TOKEN}… --- ## [Need Robust Grok filter for nginx error log format](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630)
**Author:** [@nitin194](https://discuss.elastic.co/u/nitin194)\ **Replies:** 4\ **Last updated:** [November 18, 2020, 2:40pm UTC](https://discuss.elastic.co/t/need-robust-grok-filter-for-nginx-error-log-format/255630 "2020-11-18T14:40:04Z")
We are facing an issue while filtering nginx error log format ... suppose below are two log snippet from the error log 2020/11/17 13:04:05 \[error\] 32237#32237: \*4185303 open() "/etc/nginx/html/favicon.ico" failed (2: … --- ## [Read multiple sqlite .db files in logstash](https://discuss.elastic.co/t/read-multiple-sqlite-db-files-in-logstash/255854)
**Author:** [@Deepak\_Kumar3](https://discuss.elastic.co/u/Deepak_Kumar3)\ **Replies:** 0\ **Last updated:** [November 18, 2020, 2:20pm UTC](https://discuss.elastic.co/t/read-multiple-sqlite-db-files-in-logstash/255854 "2020-11-18T14:20:01Z")
hi, trying to read sqlite .db files using jdbc plugin and here is the configuration file for logstash below. input { jdbc { jdbc\_driver\_library =\> "D:/etc/driver/sqlite-jdbc-3.32.3.2.jar" jdbc\_driver\_class… --- ## [Logstash-output-syslog plugin sends incorrect facility](https://discuss.elastic.co/t/logstash-output-syslog-plugin-sends-incorrect-facility/255734)
**Author:** [@jsteenkamp](https://discuss.elastic.co/u/jsteenkamp)\ **Replies:** 1\ **Last updated:** [November 18, 2020, 1:18pm UTC](https://discuss.elastic.co/t/logstash-output-syslog-plugin-sends-incorrect-facility/255734 "2020-11-18T13:18:57Z")
Linux Debian 10.6 Logstash 7.10.0 OpenJDK 11.0.8+10 rsyslogd 8.1901.0 plugin logstash-output-syslog example.conf: input { stdin {} } output { stdout {} syslog { id =\> "syslog\_output" appname =\> "logsta… --- ## [Logstash S3 input plugin - prefix usage](https://discuss.elastic.co/t/logstash-s3-input-plugin-prefix-usage/255819)
**Author:** [@david.preston](https://discuss.elastic.co/u/david.preston)\ **Replies:** 0\ **Last updated:** [November 18, 2020, 10:44am UTC](https://discuss.elastic.co/t/logstash-s3-input-plugin-prefix-usage/255819 "2020-11-18T10:44:59Z")
Hi - I'm using the logstash S3 input plugin - and need to implement "backup\_add\_prefix" - so that I can skip processed files - to improve performance. I assumed the first step to implementing this, was to get the prefix… --- ## [Logstash-output-syslog sends invalid json message from winlogbeat agent](https://discuss.elastic.co/t/logstash-output-syslog-sends-invalid-json-message-from-winlogbeat-agent/255754)
**Author:** [@jsteenkamp](https://discuss.elastic.co/u/jsteenkamp)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 7:05pm UTC](https://discuss.elastic.co/t/logstash-output-syslog-sends-invalid-json-message-from-winlogbeat-agent/255754 "2020-11-17T19:05:01Z")
Linux Debian 10.6 Logstash 7.10.0 OpenJDK 11.0.8+10 rsyslogd 8.1901.0 plugin logstash-output-syslog winlogbeat 7.10.0 logstash output.conf: output { elasticsearch { .... } syslog { id =\> "syslog\_output"… --- ## [Use Cisco MIB to fetch CPU and Memory using Logstash](https://discuss.elastic.co/t/use-cisco-mib-to-fetch-cpu-and-memory-using-logstash/255445)
**Author:** [@Ajay\_Singh2](https://discuss.elastic.co/u/Ajay_Singh2)\ **Replies:** 4\ **Last updated:** [November 18, 2020, 11:09am UTC](https://discuss.elastic.co/t/use-cisco-mib-to-fetch-cpu-and-memory-using-logstash/255445 "2020-11-18T11:09:38Z")
I have Cisco Switches and Routers. I have download their CPU and memory MIB's which are in .my format. How can i use logstash to utilize these MIB to fetch CPU and Memory status of cisco switches and routers ? --- ## [Logstash S3 output plugin with AWS emulation with Localstack](https://discuss.elastic.co/t/logstash-s3-output-plugin-with-aws-emulation-with-localstack/255822)
**Author:** [@Abolurah](https://discuss.elastic.co/u/Abolurah)\ **Replies:** 0\ **Last updated:** [November 18, 2020, 10:54am UTC](https://discuss.elastic.co/t/logstash-s3-output-plugin-with-aws-emulation-with-localstack/255822 "2020-11-18T10:54:58Z")
Hi All, after I spent a lot of time configuring and connecting Logstash to Localstack, I had many errors and no guide to follow . In my case I have 2 dockers , one is logstash docker with S3 output pluging , second dock… --- ## [Change format json file to metrics with Logstash and save it in Victoria Metrics](https://discuss.elastic.co/t/change-format-json-file-to-metrics-with-logstash-and-save-it-in-victoria-metrics/255732)
**Author:** [@recepbalibey](https://discuss.elastic.co/u/recepbalibey)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 3:56pm UTC](https://discuss.elastic.co/t/change-format-json-file-to-metrics-with-logstash-and-save-it-in-victoria-metrics/255732 "2020-11-17T15:56:07Z")
Here is only one example from my JSON file \>\> {"host":"ABCDEFASD","groups":\["ABVD","TEST"\],"applications": \["NETWORK"\],"itemid":143172,"name":"Operational status of interface Se0/1/0:17","clock":1604283792,"ns":926563… --- ## [Continuous data synchronization between mongodb and elasticsearch](https://discuss.elastic.co/t/continuous-data-synchronization-between-mongodb-and-elasticsearch/255788)
**Author:** [@Durga\_AK](https://discuss.elastic.co/u/Durga_AK)\ **Replies:** 1\ **Last updated:** [November 18, 2020, 7:18am UTC](https://discuss.elastic.co/t/continuous-data-synchronization-between-mongodb-and-elasticsearch/255788 "2020-11-18T07:18:02Z")
I have installed ELK stack in AWS EC2. I have created a logstash config file and I'm able to ingest existing data from mongodb to elastic search using the logstash -f config file. Now I have inserted few data into mongod… --- ## [Multiple pipelines in Logstash](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/255418)
**Author:** [@Cosmin\_Ciobanu1](https://discuss.elastic.co/u/Cosmin_Ciobanu1)\ **Replies:** 1\ **Last updated:** [November 17, 2020, 11:13pm UTC](https://discuss.elastic.co/t/multiple-pipelines-in-logstash/255418 "2020-11-17T23:13:25Z")
Hi! I'm trying to make multiple pipelines and I'm getting those errors every time: \[2020-11-15T00:33:36,799\]\[WARN \]\[org.logstash.execution.ShutdownWatcherExt\] {"inflight\_count"=\>0, "stalling\_threads\_info"=\>{"other"=\>\[{… --- ## [Logstash as daemon won't start](https://discuss.elastic.co/t/logstash-as-daemon-wont-start/255716)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [November 17, 2020, 10:05pm UTC](https://discuss.elastic.co/t/logstash-as-daemon-wont-start/255716 "2020-11-17T22:05:25Z")
I just upgraded system to 7.10.0 this is test system. and logstash daemon won't start. here is result of journalctl -xe start-up result is done. Nov 17 08:28:25 elkdev01 logstash\[13918\]: Using bundled JDK: /usr/sha… --- ## [Accessing nested fields in ruby filter for math](https://discuss.elastic.co/t/accessing-nested-fields-in-ruby-filter-for-math/255740)
**Author:** [@Andrew22](https://discuss.elastic.co/u/Andrew22)\ **Replies:** 4\ **Last updated:** [November 17, 2020, 7:30pm UTC](https://discuss.elastic.co/t/accessing-nested-fields-in-ruby-filter-for-math/255740 "2020-11-17T19:30:54Z")
Hello, I am having issues with this bit of code where I am trying to add 3 nested fields together and place them in a new nested field. here is my bit of ruby code ruby { code =\> "event.set('\[device\]\[disconnect\]\[dura… --- ## [Sort array before concatenated fingerprint hash?](https://discuss.elastic.co/t/sort-array-before-concatenated-fingerprint-hash/255602)
**Author:** [@sliddjur](https://discuss.elastic.co/u/sliddjur)\ **Replies:** 5\ **Last updated:** [November 17, 2020, 6:56pm UTC](https://discuss.elastic.co/t/sort-array-before-concatenated-fingerprint-hash/255602 "2020-11-17T18:56:02Z")
I have src\_ip and dst\_ip fields. I have copied both those values to "\[fw\]\[talkers\]" field to produce this: "fw": { "talkers": \[ "172.16.216.118", "172.23.253.22" \] Now I run fingerprint on this value to pr… --- ## [Trouble Mutating](https://discuss.elastic.co/t/trouble-mutating/255598)
**Author:** [@stevezemlicka](https://discuss.elastic.co/u/stevezemlicka)\ **Replies:** 6\ **Last updated:** [November 17, 2020, 5:36pm UTC](https://discuss.elastic.co/t/trouble-mutating/255598 "2020-11-17T17:36:19Z")
I am having trouble performing a mutate. I thought I had it working with a more simplified version of this config but cannot seem to get it going in this new one (though everything else seems to be working much better). … --- ## [Ruby Script in Logstash :: Enters a NULL into the Data?](https://discuss.elastic.co/t/ruby-script-in-logstash-enters-a-null-into-the-data/255727)
**Author:** [@redapplesonly](https://discuss.elastic.co/u/redapplesonly)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 3:41pm UTC](https://discuss.elastic.co/t/ruby-script-in-logstash-enters-a-null-into-the-data/255727 "2020-11-17T15:41:54Z")
Hi Logstash Jedi Masters, I’m running Logstash 7.7.1 in the Docker container. (Yes, I know I need to upgrade) In my Logstash config file, I’m running a snippet of Ruby code: ruby { init =\> " require 'sock… --- ## [Prune filter plugin - How whitelist a nested field?](https://discuss.elastic.co/t/prune-filter-plugin-how-whitelist-a-nested-field/255720)
**Author:** [@21908](https://discuss.elastic.co/u/21908)\ **Replies:** 3\ **Last updated:** [November 17, 2020, 3:16pm UTC](https://discuss.elastic.co/t/prune-filter-plugin-how-whitelist-a-nested-field/255720 "2020-11-17T15:16:14Z")
Documentation at has this example and it mentions that it would allow only "msg" through. filter { prune { whitelist\_names =\> \["^msg$"\] } } If I have a single record that has "source" … --- ## [Mutate rename filter not working on nested fields](https://discuss.elastic.co/t/mutate-rename-filter-not-working-on-nested-fields/255713)
**Author:** [@blastodorm](https://discuss.elastic.co/u/blastodorm)\ **Replies:** 2\ **Last updated:** [November 17, 2020, 2:33pm UTC](https://discuss.elastic.co/t/mutate-rename-filter-not-working-on-nested-fields/255713 "2020-11-17T14:33:03Z")
I'm trying to rename a field to work with my data. The field I'm attempting to rename is nested. My config: filter { mutate {rename =\> { "\[message\]\[context\]\[payload\]\[geolocation\]\[latitude\]" =\> "\[message\]\[context\]\[payl… --- ## [To create a log-level depends on string](https://discuss.elastic.co/t/to-create-a-log-level-depends-on-string/255712)
**Author:** [@Guhan\_S](https://discuss.elastic.co/u/Guhan_S)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 2:09pm UTC](https://discuss.elastic.co/t/to-create-a-log-level-depends-on-string/255712 "2020-11-17T14:09:46Z")
message:{"level":30",hostname":"someserver.com","pid":1627," like this i am getting logs, i have filter accoding to level in message. level:30 means info level log, 20 means debug level log --- ## [Logstash shut down after pipeline terminated](https://discuss.elastic.co/t/logstash-shut-down-after-pipeline-terminated/255478)
**Author:** [@Amir\_Almian](https://discuss.elastic.co/u/Amir_Almian)\ **Replies:** 2\ **Last updated:** [November 17, 2020, 1:24pm UTC](https://discuss.elastic.co/t/logstash-shut-down-after-pipeline-terminated/255478 "2020-11-17T13:24:27Z")
i,m running elastic stack on docker and with ssl security enable. this is my logstash.yml file http.host: "0.0.0.0" xpack.monitoring.elasticsearch.hosts: \[ "https://elasticsearch:9200" \] xpack.monitoring.enabled: true … --- ## [Logstash config to ingest data by filtering the data that is already present in another index in ES](https://discuss.elastic.co/t/logstash-config-to-ingest-data-by-filtering-the-data-that-is-already-present-in-another-index-in-es/255692)
**Author:** [@Sivajanani](https://discuss.elastic.co/u/Sivajanani)\ **Replies:** 0\ **Last updated:** [November 17, 2020, 12:29pm UTC](https://discuss.elastic.co/t/logstash-config-to-ingest-data-by-filtering-the-data-that-is-already-present-in-another-index-in-es/255692 "2020-11-17T12:29:46Z")
Hello All, I have a requirement to ingest only the data which already exists in another index. For example I have a index in ES like index1 =\[source=a,time=b,value=c,id=d\] and data set to be ingested like sample=\[value=… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=279) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=281)