# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=279 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 280 --- ## [Azure NSG Integration help](https://discuss.elastic.co/t/azure-nsg-integration-help/256342)
**Author:** [@Ameer\_Mukadam](https://discuss.elastic.co/u/Ameer_Mukadam)\ **Replies:** 1\ **Last updated:** [November 23, 2020, 10:19am UTC](https://discuss.elastic.co/t/azure-nsg-integration-help/256342 "2020-11-23T10:19:45Z")
Hi, I am trying to integrate Azure NSG Logs into Elastic, I have done 80% but there is 1 field is proving a little difficult for me and it has all the important flow information. Each log event contaains 5 flows inform… --- ## [How to shutdown logstash when jdbc or es connection failed?](https://discuss.elastic.co/t/how-to-shutdown-logstash-when-jdbc-or-es-connection-failed/256333)
**Author:** [@zhangyunkun](https://discuss.elastic.co/u/zhangyunkun)\ **Replies:** 0\ **Last updated:** [November 23, 2020, 9:46am UTC](https://discuss.elastic.co/t/how-to-shutdown-logstash-when-jdbc-or-es-connection-failed/256333 "2020-11-23T09:46:01Z")
I found that logstash won't shutdown when jdbc or es connection failed。 It troubles me because I'll never find this mistake unless I take the initiative to see it。 And i tried configue the jdbc plugin setting 'co… --- ## [Aggregate filter plugin](https://discuss.elastic.co/t/aggregate-filter-plugin/255729)
**Author:** [@Andex](https://discuss.elastic.co/u/Andex)\ **Replies:** 5\ **Last updated:** [November 23, 2020, 9:09am UTC](https://discuss.elastic.co/t/aggregate-filter-plugin/255729 "2020-11-23T09:09:25Z")
Hi, i create my aggregate filter plugin but it doesn't start with this error : \[ERROR\]\[logstash.filters.aggregate\] Missing a required setting for the aggregate filter plugin: filter { aggregate { code =\> # SETTING MI… --- ## [Sending New Relic Logs to ELK](https://discuss.elastic.co/t/sending-new-relic-logs-to-elk/256311)
**Author:** [@Edwin.v](https://discuss.elastic.co/u/Edwin.v)\ **Replies:** 0\ **Last updated:** [November 23, 2020, 7:10am UTC](https://discuss.elastic.co/t/sending-new-relic-logs-to-elk/256311 "2020-11-23T07:10:13Z")
Hello Everyone, I would like to know if there is a way I can send new relic logs to ELK stack. Any documentation will be really helpful. Thanks. --- ## [Set custom @timestamp from fields not working](https://discuss.elastic.co/t/set-custom-timestamp-from-fields-not-working/256262)
**Author:** [@nino](https://discuss.elastic.co/u/nino)\ **Replies:** 2\ **Last updated:** [November 22, 2020, 5:31pm UTC](https://discuss.elastic.co/t/set-custom-timestamp-from-fields-not-working/256262 "2020-11-22T17:31:48Z")
Hello all, i need help setting @timestamp from fields, why date plugin is not parsing my custom fill\_date correctly? mutate { add\_field =\> { "fill\_date" =\> "%{FillDate} %{FillTime}" } } date { timezone =\> "UCT… --- ## [Got response code '400' contacting Elasticsearch](https://discuss.elastic.co/t/got-response-code-400-contacting-elasticsearch/256276)
**Author:** [@Rnx](https://discuss.elastic.co/u/Rnx)\ **Replies:** 1\ **Last updated:** [November 22, 2020, 2:35pm UTC](https://discuss.elastic.co/t/got-response-code-400-contacting-elasticsearch/256276 "2020-11-22T14:35:08Z")
Using local instance of Logstash 7.10. with output to Cloud services made an horror hours to me as it is 'pretty bad not telling a thing' issue, however: If your Logstash starts properly and connects to a cloud; If you… --- ## [Logstash S3 Input Error](https://discuss.elastic.co/t/logstash-s3-input-error/256250)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 1\ **Last updated:** [November 22, 2020, 7:22am UTC](https://discuss.elastic.co/t/logstash-s3-input-error/256250 "2020-11-22T07:22:39Z")
Hi All, I am getting this error while running logstash with S3 input plugin. Is there anything wrong with my config. include\_object\_properties=\>false\> Error: Net::OpenTimeout Exception: Seahorse::Client::Networkin… --- ## [How to use S3 Input Plugin with assume role\_arn](https://discuss.elastic.co/t/how-to-use-s3-input-plugin-with-assume-role-arn/256248)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 0\ **Last updated:** [November 22, 2020, 1:52am UTC](https://discuss.elastic.co/t/how-to-use-s3-input-plugin-with-assume-role-arn/256248 "2020-11-22T01:52:08Z")
Hi, I am trying to find a way to use the S3 input plugin to ingest AWS Cloudtrail data. I have the config working but the way I am doing doesn't seem to be streamlined. So this is how I am able to make it work till now. … --- ## [\[Grok\] Array of patterns - only first pattern matched](https://discuss.elastic.co/t/grok-array-of-patterns-only-first-pattern-matched/256234)
**Author:** [@james64](https://discuss.elastic.co/u/james64)\ **Replies:** 2\ **Last updated:** [November 21, 2020, 5:19pm UTC](https://discuss.elastic.co/t/grok-array-of-patterns-only-first-pattern-matched/256234 "2020-11-21T17:19:21Z")
Hello, based on this documentation I am trying to use grok filter with this configuration: grok { match =\> { "message" =\> \[ ' volume\_path="%{UNIXPATH:csi\_volume\_path}', … --- ## [Alternative to running logstash from the command line](https://discuss.elastic.co/t/alternative-to-running-logstash-from-the-command-line/256231)
**Author:** [@jafri6](https://discuss.elastic.co/u/jafri6)\ **Replies:** 1\ **Last updated:** [November 21, 2020, 4:05pm UTC](https://discuss.elastic.co/t/alternative-to-running-logstash-from-the-command-line/256231 "2020-11-21T16:05:26Z")
I have a perfect working solution on ELK. The only issue that I am facing is that I want the logstash config to be run in the background as compared to running in the command line: /usr/share/logstash/bin/logstash --pat… --- ## [Parsing error Logstash with file as input](https://discuss.elastic.co/t/parsing-error-logstash-with-file-as-input/256227)
**Author:** [@Xor44](https://discuss.elastic.co/u/Xor44)\ **Replies:** 1\ **Last updated:** [November 21, 2020, 3:09pm UTC](https://discuss.elastic.co/t/parsing-error-logstash-with-file-as-input/256227 "2020-11-21T15:09:30Z")
Hi Folks , I'm new in ELK , i'm trying read and parse a file with logstash. But I get some errors during the parsing process Here is an example of logs // date="Nov 21 2020 14:36:00" AED="1.1.1.1" type=group\_location … --- ## [Ruby exception occurred: undefined method \`\[\]' for #\ **Author:** [@sujeetjha](https://discuss.elastic.co/u/sujeetjha)\ **Replies:** 3\ **Last updated:** [November 21, 2020, 2:02pm UTC](https://discuss.elastic.co/t/ruby-exception-occurred-undefined-method-for-logstash/256131 "2020-11-21T14:02:51Z") filter { ## Ignore the comments that IIS will add to the start of the W3C logs # if \[message\] =~ "^#" { drop {} } grok { ## Very helpful site for building these statements: # http://grokdebug.her… --- ## [GROK Date/time filter not working](https://discuss.elastic.co/t/grok-date-time-filter-not-working/256201)
**Author:** [@earlsanchez](https://discuss.elastic.co/u/earlsanchez)\ **Replies:** 2\ **Last updated:** [November 21, 2020, 2:26am UTC](https://discuss.elastic.co/t/grok-date-time-filter-not-working/256201 "2020-11-21T02:26:20Z")
Getting logstash failure when parsing the date/time stamp from my log. This seems to work in the GROK Debugger but fails when logstash filters it. Can someone please help to filter the date/time stamp into "date"? In GR… --- ## [Logstash not sending data to elastic](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic/256176)
**Author:** [@DedSec](https://discuss.elastic.co/u/DedSec)\ **Replies:** 5\ **Last updated:** [November 20, 2020, 6:50pm UTC](https://discuss.elastic.co/t/logstash-not-sending-data-to-elastic/256176 "2020-11-20T18:50:15Z")
Hello, I have a logstash file and when I am running the command sudo logstash -f bug.conf logstash starts but there is no data in ELK This is my bug.conf file: input { file { path =\> "/Users/siddharth/Desktop/program… --- ## [Date Joda parse does not work](https://discuss.elastic.co/t/date-joda-parse-does-not-work/256163)
**Author:** [@Jan\_Kabelka](https://discuss.elastic.co/u/Jan_Kabelka)\ **Replies:** 1\ **Last updated:** [November 20, 2020, 5:58pm UTC](https://discuss.elastic.co/t/date-joda-parse-does-not-work/256163 "2020-11-20T17:58:53Z")
Hi, I am trying to parse date to field @timestamp. My timestamp looks like below: "Timestamp" : "11/20/2020 3:32:01 PM" I tried: date { match =\> \[ "Timestamp", "MM/dd/yyyy HH:mm:ss a" \] remove\_field =\> \["Timestamp"\] … --- ## [Logstash add @version which is not in the original document](https://discuss.elastic.co/t/logstash-add-version-which-is-not-in-the-original-document/256144)
**Author:** [@amralieg](https://discuss.elastic.co/u/amralieg)\ **Replies:** 3\ **Last updated:** [November 20, 2020, 4:23pm UTC](https://discuss.elastic.co/t/logstash-add-version-which-is-not-in-the-original-document/256144 "2020-11-20T16:23:04Z")
Hi, I have this logstash config that moves data between 2 elasticsearch, when I run it I get this error, any idea how to solve this? input { elasticsearch { hosts =\> \["\"\] user =\> "\*\*" … --- ## [Logstash imap plugin read a lot of Duplication email](https://discuss.elastic.co/t/logstash-imap-plugin-read-a-lot-of-duplication-email/256150)
**Author:** [@shawn\_sun](https://discuss.elastic.co/u/shawn_sun)\ **Replies:** 0\ **Last updated:** [November 20, 2020, 3:59pm UTC](https://discuss.elastic.co/t/logstash-imap-plugin-read-a-lot-of-duplication-email/256150 "2020-11-20T15:59:28Z")
version： logstash7.8.0 and elasticsearch7.8.0 When I use Logstash imap input plugin to read email from IMAP server，The elasticsearch recive many repeat email this is logstash.conf input { imap { host =\> "imap.ma… --- ## [Grok filter not working](https://discuss.elastic.co/t/grok-filter-not-working/256077)
**Author:** [@Ajinkya\_1](https://discuss.elastic.co/u/Ajinkya_1)\ **Replies:** 1\ **Last updated:** [November 20, 2020, 2:20pm UTC](https://discuss.elastic.co/t/grok-filter-not-working/256077 "2020-11-20T14:20:55Z")
My Grok filter is not working in logstash Even though it is working in grok debugger below is the grok string: \\((?\%{DAY} %{MONTH} %{MONTHNUM} %{TIME} %{YEAR})\\) \\\[(?\(.\*))\\\] \\\[%{DATA:function}\\\] \\(%… --- ## [How can rename logstash fields](https://discuss.elastic.co/t/how-can-rename-logstash-fields/254867)
**Author:** [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\ **Replies:** 6\ **Last updated:** [November 20, 2020, 1:17pm UTC](https://discuss.elastic.co/t/how-can-rename-logstash-fields/254867 "2020-11-20T13:17:26Z")
rename fields: Thanks --- ## [Logstash with mongo input adding a default "document" field onto ElasticSearch Output index](https://discuss.elastic.co/t/logstash-with-mongo-input-adding-a-default-document-field-onto-elasticsearch-output-index/256120)
**Author:** [@MDuarte](https://discuss.elastic.co/u/MDuarte)\ **Replies:** 0\ **Last updated:** [November 20, 2020, 12:23pm UTC](https://discuss.elastic.co/t/logstash-with-mongo-input-adding-a-default-document-field-onto-elasticsearch-output-index/256120 "2020-11-20T12:23:47Z")
Hi there, I'm new to ELK stack. Although, I've done quite some extensive research and wasn't able to find the answer to my current problem, so I was hoping someone could help me. I've been trying to ingest documents fr… --- ## [Logstash on cifs mounted windows drive](https://discuss.elastic.co/t/logstash-on-cifs-mounted-windows-drive/256070)
**Author:** [@pk.241011](https://discuss.elastic.co/u/pk.241011)\ **Replies:** 0\ **Last updated:** [November 20, 2020, 7:14am UTC](https://discuss.elastic.co/t/logstash-on-cifs-mounted-windows-drive/256070 "2020-11-20T07:14:45Z")
I have to pull in some logs which will be dumped on a shared windows drive. The files are small. But their number will be high. There is no chance of files appearing again once they have been processed and deleted. The… --- ## [Split is not working on multivalued string fields](https://discuss.elastic.co/t/split-is-not-working-on-multivalued-string-fields/256068)
**Author:** [@Abdul\_Aziz](https://discuss.elastic.co/u/Abdul_Aziz)\ **Replies:** 0\ **Last updated:** [November 20, 2020, 7:06am UTC](https://discuss.elastic.co/t/split-is-not-working-on-multivalued-string-fields/256068 "2020-11-20T07:06:08Z")
I have record like below in my text file; id=1 name=Paint category=Decorative color=2345|red|5|| color=123|green|4|| color=345|yellow|8|| REC$$ I wanted output (stdout) as below using logstash configuration; "id… --- ## [Logstash Grok filters, split and path](https://discuss.elastic.co/t/logstash-grok-filters-split-and-path/256015)
**Author:** [@Priyanka3](https://discuss.elastic.co/u/Priyanka3)\ **Replies:** 1\ **Last updated:** [November 20, 2020, 5:46am UTC](https://discuss.elastic.co/t/logstash-grok-filters-split-and-path/256015 "2020-11-20T05:46:58Z")
Hi, I am new to Elastic search. So, can anyone explain me the difference between Split, xpath and Grok filters? and under what situation do we use these? Thanks, Priyanka --- ## [Modsecurity Log Grok Filter](https://discuss.elastic.co/t/modsecurity-log-grok-filter/256057)
**Author:** [@reza\_naipospos](https://discuss.elastic.co/u/reza_naipospos)\ **Replies:** 10\ **Last updated:** [November 20, 2020, 4:21am UTC](https://discuss.elastic.co/t/modsecurity-log-grok-filter/256057 "2020-11-20T04:21:28Z")
I make grok filter with this condition but noting show anything on grok debugger. Log format is Modsecurity audit log (?%{YEAR}\[./\]%{MONTHNUM}\[./\]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:thread… --- ## [Logstash.bat generates new .jar files every time it is run](https://discuss.elastic.co/t/logstash-bat-generates-new-jar-files-every-time-it-is-run/255747)
**Author:** [@Mefhisto1](https://discuss.elastic.co/u/Mefhisto1)\ **Replies:** 5\ **Last updated:** [November 19, 2020, 9:01pm UTC](https://discuss.elastic.co/t/logstash-bat-generates-new-jar-files-every-time-it-is-run/255747 "2020-11-19T21:01:10Z")
Hi, I'm running logstash every minute via a task scheduler. The task is to call logstash.bat with the appropriate .config file. This is done every minute. Logstash is generating a lot of files in TEMP folder every ti… --- ## [Grok cant detect change of line](https://discuss.elastic.co/t/grok-cant-detect-change-of-line/255796)
**Author:** [@Alexandros888](https://discuss.elastic.co/u/Alexandros888)\ **Replies:** 0\ **Last updated:** [November 18, 2020, 7:36am UTC](https://discuss.elastic.co/t/grok-cant-detect-change-of-line/255796 "2020-11-18T07:36:45Z")
Hello all, I have the following file that i want to grok: AG 100045 1FB702 27.12.2011 675337068 ZFA31200000...... AG 100045 X 17.11.1961 084260332 5352... AG 100046 1SF995 22.04.2016 217347660 TMBET6NH1G4...... AG 1… --- ## [Problem with split add field](https://discuss.elastic.co/t/problem-with-split-add-field/256018)
**Author:** [@gius78](https://discuss.elastic.co/u/gius78)\ **Replies:** 2\ **Last updated:** [November 19, 2020, 5:17pm UTC](https://discuss.elastic.co/t/problem-with-split-add-field/256018 "2020-11-19T17:17:15Z")
Hello, I am using logstash 7.6.2 I am trying to parse a CEF log, separated by pipes "|" I've done: mutate { copy =\> { "cefmessage" =\> "tmp\_message" } split =\> { "tmp\_message" =\> "|" } … --- ## [Influxdb ( Victoria Metrics ) Error + Filtering Logstash ( remove - split )](https://discuss.elastic.co/t/influxdb-victoria-metrics-error-filtering-logstash-remove-split/255994)
**Author:** [@recepbalibey](https://discuss.elastic.co/u/recepbalibey)\ **Replies:** 7\ **Last updated:** [November 19, 2020, 3:29pm UTC](https://discuss.elastic.co/t/influxdb-victoria-metrics-error-filtering-logstash-remove-split/255994 "2020-11-19T15:29:12Z")
Here is only one example from my JSON file \>\> {"host":"ABCDEFASD","groups":\["ABVD","TEST"\],"applications": \["NETWORK"\],"itemid":143172,"name":"Operational status of interface Se0/1/0:17","clock":1604283792,"ns":92656332… --- ## [\_grokparsefailure](https://discuss.elastic.co/t/grokparsefailure/255996)
**Author:** [@Jurilz](https://discuss.elastic.co/u/Jurilz)\ **Replies:** 2\ **Last updated:** [November 19, 2020, 2:51pm UTC](https://discuss.elastic.co/t/grokparsefailure/255996 "2020-11-19T14:51:03Z")
Good day. I'm trying to parse tomcat logs with logstash using the filebeat input plugin. My log-files look like: 2020-11-19 11:34:40,260 \[thread0-exec-1\] WARN org.springframework.web.servlet.PageNotFound - Request me… --- ## [How gsub parse the log line](https://discuss.elastic.co/t/how-gsub-parse-the-log-line/255803)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 5\ **Last updated:** [November 19, 2020, 2:32pm UTC](https://discuss.elastic.co/t/how-gsub-parse-the-log-line/255803 "2020-11-19T14:32:12Z")
I have a log file in json format having values and corresponding sub-values till 2 levels. All are encapsulated with double quotes. Normally the log is having simple key-value pairs having key and values encapsulated wit… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=278) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=280)