# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=318 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 319 --- ## [Get service-now.com data](https://discuss.elastic.co/t/get-service-now-com-data/244320)
**Author:** [@netscape](https://discuss.elastic.co/u/netscape)\ **Replies:** 1\ **Last updated:** [August 9, 2020, 6:53pm UTC](https://discuss.elastic.co/t/get-service-now-com-data/244320 "2020-08-09T18:53:28Z")
Hi, we´re using servicenow.com system and I want to have stats of service-now tickets (INC|REQ) in our ELK. Service-now have an API, but how can I retrieve only new tickets since last sync? Anybody got an idea? --- ## [Logstash timestamp issue - the log line date is one hour ahead the actual date](https://discuss.elastic.co/t/logstash-timestamp-issue-the-log-line-date-is-one-hour-ahead-the-actual-date/244303)
**Author:** [@Muhammad\_Badawy](https://discuss.elastic.co/u/Muhammad_Badawy)\ **Replies:** 7\ **Last updated:** [August 9, 2020, 6:00pm UTC](https://discuss.elastic.co/t/logstash-timestamp-issue-the-log-line-date-is-one-hour-ahead-the-actual-date/244303 "2020-08-09T18:00:37Z")
Hello, I'm trying to read input from JDBC and applied date filter but logstash logs is one hour ahead the actual date. I don't where this difference came from. Logstash conf file input { jdbc { jdbc\_driver\_library =\> "… --- ## [Increasing heap size - Logstash](https://discuss.elastic.co/t/increasing-heap-size-logstash/244310)
**Author:** [@Natarajan](https://discuss.elastic.co/u/Natarajan)\ **Replies:** 0\ **Last updated:** [August 9, 2020, 4:13pm UTC](https://discuss.elastic.co/t/increasing-heap-size-logstash/244310 "2020-08-09T16:13:12Z")
I am using Logstash 7.6.2 and I set LS\_JAVA\_OPTS="-Xmx2g -Xms2g" as evirnoment variable in docker command. I am not sure it is reflected in docker container . How can we validate this once we set from command. --- ## [Grok filter not adding new fields](https://discuss.elastic.co/t/grok-filter-not-adding-new-fields/240825)
**Author:** [@vamshisiddarth](https://discuss.elastic.co/u/vamshisiddarth)\ **Replies:** 3\ **Last updated:** [August 9, 2020, 1:26pm UTC](https://discuss.elastic.co/t/grok-filter-not-adding-new-fields/240825 "2020-08-09T13:26:02Z")
We have the below grok filter configured for our journlabeat. The same was deployed on our local for filebeat was working fine but isn't adding the new fields on journalbeat. filter { grok { patterns\_dir =\> \[… --- ## [Logstash filter doesn't return correct results based on conditionals](https://discuss.elastic.co/t/logstash-filter-doesnt-return-correct-results-based-on-conditionals/244234)
**Author:** [@nmoham](https://discuss.elastic.co/u/nmoham)\ **Replies:** 2\ **Last updated:** [August 9, 2020, 1:15am UTC](https://discuss.elastic.co/t/logstash-filter-doesnt-return-correct-results-based-on-conditionals/244234 "2020-08-09T01:15:16Z")
I've have the following conf file in logstash; based on the incoming data from beats containing the environment field, I want to add a new field "\[@metadata\]\[shared\_env\_category\]". I am testing the it using stdin and wi… --- ## [Parsing complex xml](https://discuss.elastic.co/t/parsing-complex-xml/244271)
**Author:** [@zmundo](https://discuss.elastic.co/u/zmundo)\ **Replies:** 1\ **Last updated:** [August 8, 2020, 11:30pm UTC](https://discuss.elastic.co/t/parsing-complex-xml/244271 "2020-08-08T23:30:55Z")
I am working with cnc machine data reporting back in a complex xml structure. The data is only generated when the component changes value. If a machine sits ideal, things like the path component won't report any data,… --- ## [Logstash instantnly shut down after importing .csv file inside pipeline folder, also I dont have permisions inside CLI of logstash container or other containers](https://discuss.elastic.co/t/logstash-instantnly-shut-down-after-importing-csv-file-inside-pipeline-folder-also-i-dont-have-permisions-inside-cli-of-logstash-container-or-other-containers/244268)
**Author:** [@giganttus](https://discuss.elastic.co/u/giganttus)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 8:39pm UTC](https://discuss.elastic.co/t/logstash-instantnly-shut-down-after-importing-csv-file-inside-pipeline-folder-also-i-dont-have-permisions-inside-cli-of-logstash-container-or-other-containers/244268 "2020-08-08T20:39:25Z")
So I configured ELK - Stack with this config. And as result my logstash container shuts down and also while I was trying to fix it through CLI of logstash container I found out that I have no permission of any container… --- ## [Trouble connecting to Microsoft SQL Server in Logstash via jdbc](https://discuss.elastic.co/t/trouble-connecting-to-microsoft-sql-server-in-logstash-via-jdbc/244266)
**Author:** [@logicalblonde](https://discuss.elastic.co/u/logicalblonde)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 8:04pm UTC](https://discuss.elastic.co/t/trouble-connecting-to-microsoft-sql-server-in-logstash-via-jdbc/244266 "2020-08-08T20:04:06Z")
I'm having trouble connecting logstash to MS Sql Server via the jdbc driver. I'm using jdk 11.0.8 and MSS Sql Server jdbc driver 4.2 (jre8). I've had success setting up other machines with 11.0.7 and 4.2, but with this… --- ## [Can I use an existing field in the event to create the columns for a CSV filter?](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258)
**Author:** [@crickes](https://discuss.elastic.co/u/crickes)\ **Replies:** 2\ **Last updated:** [August 8, 2020, 5:10pm UTC](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258 "2020-08-08T17:10:27Z")
I've been struugling with this for far too long and I'm giving up and asking for help. I am being sent data via a Kafka topic which contains a field called 'headers' which is a CSV string of column headers, and a field c… --- ## [Dump RRD files into logstash to visualise in Kibana](https://discuss.elastic.co/t/dump-rrd-files-into-logstash-to-visualise-in-kibana/244250)
**Author:** [@Himani\_Agarwal](https://discuss.elastic.co/u/Himani_Agarwal)\ **Replies:** 0\ **Last updated:** [August 8, 2020, 12:28pm UTC](https://discuss.elastic.co/t/dump-rrd-files-into-logstash-to-visualise-in-kibana/244250 "2020-08-08T12:28:56Z")
Hello, I have RRD(Round Robin Database) files which contains the statistics of a remote server(collected using collectd). The requirement is to put this data on elastic search for analysis purpose. One of the solution … --- ## [How to remove repeting string in message?](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195)
**Author:** [@srbhklkrn](https://discuss.elastic.co/u/srbhklkrn)\ **Replies:** 11\ **Last updated:** [August 8, 2020, 7:20am UTC](https://discuss.elastic.co/t/how-to-remove-repeting-string-in-message/244195 "2020-08-08T07:20:28Z")
I have a syslog5424\_msg field which gets repeating JSON string, example log {"@t":"2020-08-07T15:32:19.7835920Z","@mt":"{@LogDetail}","ThreadId":32,"EventType":2890355315,"Application":"EsXYZ","Environment":"Development… --- ## [\[ERROR\]\[logstash.config.sourceloader\] No configuration found in the configured sources](https://discuss.elastic.co/t/error-logstash-config-sourceloader-no-configuration-found-in-the-configured-sources/244230)
**Author:** [@Umang\_Mahant](https://discuss.elastic.co/u/Umang_Mahant)\ **Replies:** 2\ **Last updated:** [August 7, 2020, 10:17pm UTC](https://discuss.elastic.co/t/error-logstash-config-sourceloader-no-configuration-found-in-the-configured-sources/244230 "2020-08-07T22:17:31Z")
Hi Masters, Please help me with this small query. Trying to load excel (Csv) to Kibana from logstash. I have not changed/ created any YML . geting the below error - Z:\\\>logstash -f logstash\_tc.conf --debug … --- ## [Getting new uuid every hour in logstash pipeline](https://discuss.elastic.co/t/getting-new-uuid-every-hour-in-logstash-pipeline/243406)
**Author:** [@Gopal\_Chauhan](https://discuss.elastic.co/u/Gopal_Chauhan)\ **Replies:** 2\ **Last updated:** [August 7, 2020, 4:18pm UTC](https://discuss.elastic.co/t/getting-new-uuid-every-hour-in-logstash-pipeline/243406 "2020-08-07T16:18:40Z")
Hi, I want to assign one unique id to all event/messages processed by logstash pipeline with in one hour. how can i do that? is there any timer which i can use to check that one hour has passes and now need to change t… --- ## [Logstash is sending variables to elasticsearch](https://discuss.elastic.co/t/logstash-is-sending-variables-to-elasticsearch/244082)
**Author:** [@shreddoctor](https://discuss.elastic.co/u/shreddoctor)\ **Replies:** 10\ **Last updated:** [August 7, 2020, 3:19pm UTC](https://discuss.elastic.co/t/logstash-is-sending-variables-to-elasticsearch/244082 "2020-08-07T15:19:49Z")
I have an issue where logstash is sending the variables to elasticsearch instead of enumerating the contents of the variable. The variables in question are coming from filebeat and I'm using these variables to create 2 d… --- ## [Azure blob plugin to logstash](https://discuss.elastic.co/t/azure-blob-plugin-to-logstash/244183)
**Author:** [@Amit\_Agarwal](https://discuss.elastic.co/u/Amit_Agarwal)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 1:46pm UTC](https://discuss.elastic.co/t/azure-blob-plugin-to-logstash/244183 "2020-08-07T13:46:03Z")
Hi, I am trying to send logs from azure blob storage to logstash. I am using azureblob input plugin but i dont see any logs being sent to logstash and I get no errors in debug logs. The files have extension of .log.gz. … --- ## [Multi pipelines](https://discuss.elastic.co/t/multi-pipelines/243676)
**Author:** [@peretz\_shlomi](https://discuss.elastic.co/u/peretz_shlomi)\ **Replies:** 4\ **Last updated:** [August 7, 2020, 1:24pm UTC](https://discuss.elastic.co/t/multi-pipelines/243676 "2020-08-07T13:24:39Z")
hi i have two pipeline that collect csv file from different folder each folder collect the data to a different index The file name is the same (Inside the file the information is a bit different) i have problem that… --- ## [JSON response parsing at attribute level](https://discuss.elastic.co/t/json-response-parsing-at-attribute-level/243552)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 9\ **Last updated:** [August 7, 2020, 12:09pm UTC](https://discuss.elastic.co/t/json-response-parsing-at-attribute-level/243552 "2020-08-07T12:09:27Z")
Hi, my logstash configuration for moving JSON response isinput { beats { port =\> 5044 } } filter {mutate { split =\> { "message" =\> "@$@" }} mutate { add\_field =\> { "response" =\> "%{\[message\]\[7\]}" "channel\_typ" =\> "%… --- ## [Logstash continues to restart itself on connection failures](https://discuss.elastic.co/t/logstash-continues-to-restart-itself-on-connection-failures/244172)
**Author:** [@VinayM30](https://discuss.elastic.co/u/VinayM30)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 12:10pm UTC](https://discuss.elastic.co/t/logstash-continues-to-restart-itself-on-connection-failures/244172 "2020-08-07T12:10:48Z")
I have a usecase where i need to ingest data from Oracle DB and send it to elastic search i have written a config file which is executing fine when connection is ok. but today when i ran it somehow password got wrong an… --- ## [Overwriting the @timestamp work on stdout but don't work on es](https://discuss.elastic.co/t/overwriting-the-timestamp-work-on-stdout-but-dont-work-on-es/244131)
**Author:** [@Tristan\_D](https://discuss.elastic.co/u/Tristan_D)\ **Replies:** 4\ **Last updated:** [August 7, 2020, 10:36am UTC](https://discuss.elastic.co/t/overwriting-the-timestamp-work-on-stdout-but-dont-work-on-es/244131 "2020-08-07T10:36:53Z")
I want to overwrite the @timestamp with date filter. The result in the stdout is ok, the @timestamp is written by the log time, but there is no document written into the elasticsearch. If I remove the date filter, the … --- ## [Support for SCADA protocols](https://discuss.elastic.co/t/support-for-scada-protocols/244138)
**Author:** [@dewell](https://discuss.elastic.co/u/dewell)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 8:43am UTC](https://discuss.elastic.co/t/support-for-scada-protocols/244138 "2020-08-07T08:43:27Z")
Hello! For testing and simulation I wrote several tools that create or translate 608070-5-104, GOOSE and Sampled Values messages. Are there any Beats or logstash plugins that can read those protocols? I have a hard tim… --- ## [Not able to see all fields when trying to create visualization](https://discuss.elastic.co/t/not-able-to-see-all-fields-when-trying-to-create-visualization/244060)
**Author:** [@Dinesh\_Potey](https://discuss.elastic.co/u/Dinesh_Potey)\ **Replies:** 5\ **Last updated:** [August 7, 2020, 8:11am UTC](https://discuss.elastic.co/t/not-able-to-see-all-fields-when-trying-to-create-visualization/244060 "2020-08-07T08:11:44Z")
Hi , I am trying to create error dashboard for which firstly I have to create a visualization out of my Index pattern. I am not able to see all the elements of my xml when I am trying to filter out to create the visuali… --- ## [How to run logstash service in the background?](https://discuss.elastic.co/t/how-to-run-logstash-service-in-the-background/243645)
**Author:** [@evacch](https://discuss.elastic.co/u/evacch)\ **Replies:** 11\ **Last updated:** [August 7, 2020, 7:58am UTC](https://discuss.elastic.co/t/how-to-run-logstash-service-in-the-background/243645 "2020-08-07T07:58:16Z")
I installed and configured ELK in a CentOS environment. I have been using the following command to run logstash in the foreground and it works perfectly. ./usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logst… --- ## [Finetuning CPU usage](https://discuss.elastic.co/t/finetuning-cpu-usage/244129)
**Author:** [@evacch](https://discuss.elastic.co/u/evacch)\ **Replies:** 0\ **Last updated:** [August 7, 2020, 7:17am UTC](https://discuss.elastic.co/t/finetuning-cpu-usage/244129 "2020-08-07T07:17:50Z")
Hi, I am new to ELK and I am running ELK in a CentOS 7 environment. I have 4 CPU cores in this machine and with the default configuration of Logstash, the CPU usage is always very high. Knowing that running Logstash … --- ## [Logstash 7.2.0 input jdbc plugin restarts itself and inserts data continuously in elastic](https://discuss.elastic.co/t/logstash-7-2-0-input-jdbc-plugin-restarts-itself-and-inserts-data-continuously-in-elastic/243966)
**Author:** [@VinayM30](https://discuss.elastic.co/u/VinayM30)\ **Replies:** 4\ **Last updated:** [August 7, 2020, 6:58am UTC](https://discuss.elastic.co/t/logstash-7-2-0-input-jdbc-plugin-restarts-itself-and-inserts-data-continuously-in-elastic/243966 "2020-08-07T06:58:53Z")
I am trying to import data from oracle db to elastic search, the process is happening with logstash in windows but when i run the same in a unix box it fetches the data from table then says \[Error\] \[logstash.javapipelin… --- ## [Error parsing JSON file](https://discuss.elastic.co/t/error-parsing-json-file/244014)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 4\ **Last updated:** [August 7, 2020, 6:53am UTC](https://discuss.elastic.co/t/error-parsing-json-file/244014 "2020-08-07T06:53:02Z")
I am trying to parse a json file which consists of multiple line but even though I am using multiline codec but still I am getting error. Sample of my Json file is as follows: \> \[ { "action": "drop", "clientAS… --- ## [Elasticsearch index not getting created when add\_field is used](https://discuss.elastic.co/t/elasticsearch-index-not-getting-created-when-add-field-is-used/243825)
**Author:** [@pranayv66](https://discuss.elastic.co/u/pranayv66)\ **Replies:** 8\ **Last updated:** [August 7, 2020, 4:49am UTC](https://discuss.elastic.co/t/elasticsearch-index-not-getting-created-when-add-field-is-used/243825 "2020-08-07T04:49:53Z")
Hi, I'm facing a problem when creating an elasticsearch index. I'm new to this, so please let me know what I could do to fix this. I'm pasting the two config files which I've used for reference. The one at the top is w… --- ## [Logstash is not loading the filter conf files](https://discuss.elastic.co/t/logstash-is-not-loading-the-filter-conf-files/241699)
**Author:** [@kittu9999](https://discuss.elastic.co/u/kittu9999)\ **Replies:** 7\ **Last updated:** [August 7, 2020, 3:43am UTC](https://discuss.elastic.co/t/logstash-is-not-loading-the-filter-conf-files/241699 "2020-08-07T03:43:55Z")
HI Team, we have upgrade our ELK stack from V5.0.1 to V6.8.1. Basically we moved to docker containers. we are having an issue with new logstash and the necessary fields are not showing in kibana. we have the same conf… --- ## [How to do Mathematical calculation on grok filter output](https://discuss.elastic.co/t/how-to-do-mathematical-calculation-on-grok-filter-output/244063)
**Author:** [@saurabhc](https://discuss.elastic.co/u/saurabhc)\ **Replies:** 2\ **Last updated:** [August 6, 2020, 5:52pm UTC](https://discuss.elastic.co/t/how-to-do-mathematical-calculation-on-grok-filter-output/244063 "2020-08-06T17:52:55Z")
My Input file looks like below. MemoryTotal: 100831888 kB MemoryFree: 1212388 kB MemoryAvailable: 49997588 kB I am filtering the values like "MemFree:\\s+%{NUMBER:mem\_free}","MemTotal:\\s+%{NUMBER:mem\_tot… --- ## [Logstash - match filename](https://discuss.elastic.co/t/logstash-match-filename/241792)
**Author:** [@elk6](https://discuss.elastic.co/u/elk6)\ **Replies:** 1\ **Last updated:** [August 6, 2020, 4:34pm UTC](https://discuss.elastic.co/t/logstash-match-filename/241792 "2020-08-06T16:34:26Z")
I have two servers. The first one hosts elastic stack. Both servers have a file /var/log/commands.log which is configured in the same way and are being shipped with filebeat to logstash. Using grok, I tried parsing the … --- ## [Logstash is not generating logs](https://discuss.elastic.co/t/logstash-is-not-generating-logs/241621)
**Author:** [@ramkumar.J](https://discuss.elastic.co/u/ramkumar.J)\ **Replies:** 1\ **Last updated:** [August 6, 2020, 4:26pm UTC](https://discuss.elastic.co/t/logstash-is-not-generating-logs/241621 "2020-08-06T16:26:57Z")
My Log stash service is running as a Windows service. No issue in the running status. But the Logs are not getting generated. I have Pipeline configured for this logstash service. I need a help to enable the Logs for t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=317) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=319)