# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=374 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 375 --- ## [Accessing Metafield \_index in Logstash](https://discuss.elastic.co/t/accessing-metafield-index-in-logstash/228113)
**Author:** [@InfiniteLoops](https://discuss.elastic.co/u/InfiniteLoops)\ **Replies:** 3\ **Last updated:** [April 15, 2020, 5:12pm UTC](https://discuss.elastic.co/t/accessing-metafield-index-in-logstash/228113 "2020-04-15T17:12:10Z")
I am using this output for Elasticsearch index =\> "%{\[headers\]\[key\]}-data-%{+YYYY.MM}" So it creates a index name with the date added. I am trying to reindex this data through a logstash filter. So I am using the previ… --- ## [Logstash filter not working with Sysmon](https://discuss.elastic.co/t/logstash-filter-not-working-with-sysmon/228023)
**Author:** [@martinsrm](https://discuss.elastic.co/u/martinsrm)\ **Replies:** 2\ **Last updated:** [April 15, 2020, 4:36pm UTC](https://discuss.elastic.co/t/logstash-filter-not-working-with-sysmon/228023 "2020-04-15T16:36:11Z")
Hello everybody, I am trying to create a field called "Threat" just if I have a log from the "Microsoft-Windows-Sysmon" provider, but the "if" condition never works, could you guys give me an idea about it? Tks ######… --- ## [SNMP input plugin - SNMP V3 not working](https://discuss.elastic.co/t/snmp-input-plugin-snmp-v3-not-working/228164)
**Author:** [@Thelmo\_Henrique\_Sant](https://discuss.elastic.co/u/Thelmo_Henrique_Sant)\ **Replies:** 0\ **Last updated:** [April 15, 2020, 4:01pm UTC](https://discuss.elastic.co/t/snmp-input-plugin-snmp-v3-not-working/228164 "2020-04-15T16:01:43Z")
Hi , I am new with ELK and I'm trying to configure my logstash to get SNMP V3 information from my devices, but unfortunately, I don't know what I am doing wrong. I followed the procedures from the SNMP input plugin pag… --- ## [Transforming objects with key and value fields into simple fields](https://discuss.elastic.co/t/transforming-objects-with-key-and-value-fields-into-simple-fields/228152)
**Author:** [@wfhartford](https://discuss.elastic.co/u/wfhartford)\ **Replies:** 2\ **Last updated:** [April 15, 2020, 4:00pm UTC](https://discuss.elastic.co/t/transforming-objects-with-key-and-value-fields-into-simple-fields/228152 "2020-04-15T16:00:36Z")
Logstash is receiving events which include a list of json objects, each with two fields key and value: { "details": \[ { "key": "foo", "value": "foo value" }, { "key": "bar", "value": "bar value… --- ## [How to parse few XML files with different number of tags in different files using Logstash?](https://discuss.elastic.co/t/how-to-parse-few-xml-files-with-different-number-of-tags-in-different-files-using-logstash/228156)
**Author:** [@Gerych](https://discuss.elastic.co/u/Gerych)\ **Replies:** 1\ **Last updated:** [April 15, 2020, 3:43pm UTC](https://discuss.elastic.co/t/how-to-parse-few-xml-files-with-different-number-of-tags-in-different-files-using-logstash/228156 "2020-04-15T15:43:31Z")
Can someone tell me how you can parse these types of files through the logstash and after that index it in elasticsearch. Files can contain a variable number of internal tags. Here is my config file, but unfortunately… --- ## [Parsing data using logstash](https://discuss.elastic.co/t/parsing-data-using-logstash/228155)
**Author:** [@Ananya\_Mohiley](https://discuss.elastic.co/u/Ananya_Mohiley)\ **Replies:** 0\ **Last updated:** [April 15, 2020, 3:26pm UTC](https://discuss.elastic.co/t/parsing-data-using-logstash/228155 "2020-04-15T15:26:06Z")
hi am trying to parse text files using a suitable filter however getting stuck, the file looks as under file name@@@@@@@@@@@@@@@@@@@@@@@@@@dummy report 1.docx data = ROUTINE REPORT from = 17 to = 70 orig = main o… --- ## [Kv filter with spaces in values](https://discuss.elastic.co/t/kv-filter-with-spaces-in-values/227962)
**Author:** [@parosio](https://discuss.elastic.co/u/parosio)\ **Replies:** 3\ **Last updated:** [April 15, 2020, 3:20pm UTC](https://discuss.elastic.co/t/kv-filter-with-spaces-in-values/227962 "2020-04-15T15:20:29Z")
Hello, I've got a log that seems a perfect candidate for the kv filter: msg=Start MID 1242272 ICID 1632662 categorySignificance=/Informational catdt=Web Filtering categoryObject=/Host/Application/Service deviceSeverity… --- ## [No config files found in path \[Need Help\]](https://discuss.elastic.co/t/no-config-files-found-in-path-need-help/227915)
**Author:** [@elktail](https://discuss.elastic.co/u/elktail)\ **Replies:** 6\ **Last updated:** [April 15, 2020, 2:48pm UTC](https://discuss.elastic.co/t/no-config-files-found-in-path-need-help/227915 "2020-04-15T14:48:16Z")
I am very new to ELK stack. I have installed ELK stack via debian package on Ubuntu Server. I made two changes in logstash.yml: config.reload.automatic: true config.reload.interval: 3s Despite these two changes every… --- ## [Logstash PQ in conjuction with DLQ](https://discuss.elastic.co/t/logstash-pq-in-conjuction-with-dlq/228086)
**Author:** [@Kalilou\_Diaby](https://discuss.elastic.co/u/Kalilou_Diaby)\ **Replies:** 1\ **Last updated:** [April 15, 2020, 2:40pm UTC](https://discuss.elastic.co/t/logstash-pq-in-conjuction-with-dlq/228086 "2020-04-15T14:40:54Z")
Hello, Hope everyone is safe during this covid19 crisis. I have a logstash related question, today we are using Logstash Persistent Queue which seems to me enough to guarantee once an at-least-once delivery. Is there a… --- ## [Update elastic search document](https://discuss.elastic.co/t/update-elastic-search-document/228102)
**Author:** [@hariskhalique](https://discuss.elastic.co/u/hariskhalique)\ **Replies:** 1\ **Last updated:** [April 15, 2020, 2:38pm UTC](https://discuss.elastic.co/t/update-elastic-search-document/228102 "2020-04-15T14:38:28Z")
Hi, I have create following conf file input { jdbc { jdbc\_driver\_library =\> "/usr/share/java/mysql.jar" jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" jdbc\_connection\_string =\> "jdbc:mysql://… --- ## [How to give multiple files in different paths as input to logstash](https://discuss.elastic.co/t/how-to-give-multiple-files-in-different-paths-as-input-to-logstash/228094)
**Author:** [@poddraj](https://discuss.elastic.co/u/poddraj)\ **Replies:** 1\ **Last updated:** [April 15, 2020, 2:35pm UTC](https://discuss.elastic.co/t/how-to-give-multiple-files-in-different-paths-as-input-to-logstash/228094 "2020-04-15T14:35:28Z")
Hi All, I have a scenario where two log files are in different directory paths but the log structure of them is same. I want to pass both of them as input files in a single config file so that the filter in it will pars… --- ## [Invoke Logstash Ondemand dynamically like REST API invocation](https://discuss.elastic.co/t/invoke-logstash-ondemand-dynamically-like-rest-api-invocation/228135)
**Author:** [@subbareddy](https://discuss.elastic.co/u/subbareddy)\ **Replies:** 0\ **Last updated:** [April 15, 2020, 1:50pm UTC](https://discuss.elastic.co/t/invoke-logstash-ondemand-dynamically-like-rest-api-invocation/228135 "2020-04-15T13:50:21Z")
Hi, I have logstash pipeline running in vm which pulls historical data for x months. I want to run this pipeline on demand when ever i want. here i don't have direct access to vm. --- ## [Logstash -\> Elastic 403 permission issue](https://discuss.elastic.co/t/logstash-elastic-403-permission-issue/228131)
**Author:** [@Andrey\_Sidorenko](https://discuss.elastic.co/u/Andrey_Sidorenko)\ **Replies:** 0\ **Last updated:** [April 15, 2020, 1:29pm UTC](https://discuss.elastic.co/t/logstash-elastic-403-permission-issue/228131 "2020-04-15T13:29:27Z")
Hi, I am a new in ELK. At now i trying to configure and use my first elk server. I'm trying to configure security through basic realm and have a issue with logstash connection to elastic search. Following guide https:… --- ## [Logstash don't trust Elasticsearch's certificate](https://discuss.elastic.co/t/logstash-dont-trust-elasticsearchs-certificate/228062)
**Author:** [@delanobgt](https://discuss.elastic.co/u/delanobgt)\ **Replies:** 2\ **Last updated:** [April 15, 2020, 11:16am UTC](https://discuss.elastic.co/t/logstash-dont-trust-elasticsearchs-certificate/228062 "2020-04-15T11:16:16Z")
I am stuck for 1 week already.. :') Elasticsearch's cluster and Kibana is working fine except for logstash only. I kept getting these logs from elasticsearch: ''' es01 | {"type": "server", "timestamp": "2020-0… --- ## [Send multiple output 1 at a time to another pipeline in Logstash](https://discuss.elastic.co/t/send-multiple-output-1-at-a-time-to-another-pipeline-in-logstash/228022)
**Author:** [@newbie001](https://discuss.elastic.co/u/newbie001)\ **Replies:** 5\ **Last updated:** [April 15, 2020, 8:09am UTC](https://discuss.elastic.co/t/send-multiple-output-1-at-a-time-to-another-pipeline-in-logstash/228022 "2020-04-15T08:09:40Z")
Hi - I am very new to logstash. Is there a way on how I can send the output into another pipeline one at a time, with the unique field "somefield"? "somefield" values are from UUID function. CODE #input ODX files f… --- ## [Logstash - Date function - Mapping timestamp ends with error \_dateparsefailure when a hour is 02](https://discuss.elastic.co/t/logstash-date-function-mapping-timestamp-ends-with-error-dateparsefailure-when-a-hour-is-02/227679)
**Author:** [@vasek](https://discuss.elastic.co/u/vasek)\ **Replies:** 5\ **Last updated:** [April 15, 2020, 5:43am UTC](https://discuss.elastic.co/t/logstash-date-function-mapping-timestamp-ends-with-error-dateparsefailure-when-a-hour-is-02/227679 "2020-04-15T05:43:16Z")
Hello, I figured out one thing: Logstash 7.6.0 is not able map date to @timestamp field when timestamp has 02 in hour positio. In pipeline configuration file: if \[timestamp\] { date { match =\> \["timestamp", "yy… --- ## [How to add aggrigate map field to the original message as a additional field](https://discuss.elastic.co/t/how-to-add-aggrigate-map-field-to-the-original-message-as-a-additional-field/227671)
**Author:** [@Lal\_k](https://discuss.elastic.co/u/Lal_k)\ **Replies:** 4\ **Last updated:** [April 15, 2020, 4:42am UTC](https://discuss.elastic.co/t/how-to-add-aggrigate-map-field-to-the-original-message-as-a-additional-field/227671 "2020-04-15T04:42:17Z")
I have huge amount of logs coming from Cisco ASA firewall specially firewall deny. for example i have more than 10 messages from same source IP accessing same destination with same destination port. So the idea is , ins… --- ## [Sql\_last\_value resets to 0 upon restart](https://discuss.elastic.co/t/sql-last-value-resets-to-0-upon-restart/228017)
**Author:** [@priyankpatel22](https://discuss.elastic.co/u/priyankpatel22)\ **Replies:** 0\ **Last updated:** [April 15, 2020, 2:37am UTC](https://discuss.elastic.co/t/sql-last-value-resets-to-0-upon-restart/228017 "2020-04-15T02:37:34Z")
It seems the variable sql\_last\_value gets reset every time I restart logstash. As per the documentation and all other threads it should look at last\_run\_metadata\_path for the last saved value. I see the last\_run\_metada… --- ## [Error connecting Spring Microservices to elastic stack](https://discuss.elastic.co/t/error-connecting-spring-microservices-to-elastic-stack/228010)
**Author:** [@15051029\_TEC\_SALTILL](https://discuss.elastic.co/u/15051029_TEC_SALTILL)\ **Replies:** 0\ **Last updated:** [April 14, 2020, 11:44pm UTC](https://discuss.elastic.co/t/error-connecting-spring-microservices-to-elastic-stack/228010 "2020-04-14T23:44:48Z")
Im using the logstash docker images for a "simple" integration with java spring cloud microservices, at the end I made every container run smoothly, the problem came when I was trying to pass the logs from the microser… --- ## [Extract A, MX, and NS records based on a field](https://discuss.elastic.co/t/extract-a-mx-and-ns-records-based-on-a-field/228005)
**Author:** [@mnrdammu](https://discuss.elastic.co/u/mnrdammu)\ **Replies:** 0\ **Last updated:** [April 14, 2020, 10:20pm UTC](https://discuss.elastic.co/t/extract-a-mx-and-ns-records-based-on-a-field/228005 "2020-04-14T22:20:27Z")
Hi All, Is there a way to get A,MX and NS records based on a field ( which is essentially a domain name for eg: example.com) . I know we have logstash-dns-filter, but it can provide only "A" records. I would like to ge… --- ## [Use SSL with logstash syslog input plugin](https://discuss.elastic.co/t/use-ssl-with-logstash-syslog-input-plugin/227831)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 2\ **Last updated:** [April 14, 2020, 5:05pm UTC](https://discuss.elastic.co/t/use-ssl-with-logstash-syslog-input-plugin/227831 "2020-04-14T17:05:46Z")
Hi, I have an installation of ELK 7.6, and i have several devices that send information to logstash via syslog. I want to secure the communication, but in the logstash syslog input plugin i dont see any parameter for ss… --- ## [Time delay between logstash and elasticsearch](https://discuss.elastic.co/t/time-delay-between-logstash-and-elasticsearch/227749)
**Author:** [@honglei](https://discuss.elastic.co/u/honglei)\ **Replies:** 11\ **Last updated:** [April 14, 2020, 3:20pm UTC](https://discuss.elastic.co/t/time-delay-between-logstash-and-elasticsearch/227749 "2020-04-14T15:20:13Z")
My current pipeline is: filebeat-\>Logstash-\>ES(3 nodes). Filebeat and Logstash are deployed in the kubernetes cluster, both of the them are version-7.6.2. ES is deployed as a container on a virtual machine, images ver… --- ## [How to write to a multiple indexes using single elasticsearch output?](https://discuss.elastic.co/t/how-to-write-to-a-multiple-indexes-using-single-elasticsearch-output/227870)
**Author:** [@artur\_m](https://discuss.elastic.co/u/artur_m)\ **Replies:** 2\ **Last updated:** [April 14, 2020, 1:18pm UTC](https://discuss.elastic.co/t/how-to-write-to-a-multiple-indexes-using-single-elasticsearch-output/227870 "2020-04-14T13:18:38Z")
Hey folks, i'm pretty new guy for ELK stack, so don't judge me too much. So we have a basic fresh ELK setup v 7.6.2 with basic licence enabled. We are using filebeat to gather logs and send them to logstash. After tha… --- ## [Best way to parse events (filebeat module/logstash plugin/logstash input filter)](https://discuss.elastic.co/t/best-way-to-parse-events-filebeat-module-logstash-plugin-logstash-input-filter/227916)
**Author:** [@s34g4r](https://discuss.elastic.co/u/s34g4r)\ **Replies:** 0\ **Last updated:** [April 14, 2020, 12:55pm UTC](https://discuss.elastic.co/t/best-way-to-parse-events-filebeat-module-logstash-plugin-logstash-input-filter/227916 "2020-04-14T12:55:44Z")
I've installed and configured elasticstack on CentOS 8, using the packages in the elasticsearch repos. Once of the first log sources I'm toying with is mod\_security. Of course, the format of these logs is totally bruta… --- ## [Logstash ip data type](https://discuss.elastic.co/t/logstash-ip-data-type/227879)
**Author:** [@Mohammad\_Mousavi](https://discuss.elastic.co/u/Mohammad_Mousavi)\ **Replies:** 1\ **Last updated:** [April 14, 2020, 10:59am UTC](https://discuss.elastic.co/t/logstash-ip-data-type/227879 "2020-04-14T10:59:16Z")
Hi, I have this in grok : %{IP:clientip}, but in Kibana in index mapping I see: "clientip": { "type": "text", "norms": false, "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 I can't use ki… --- ## [Looking for a way, size of log event](https://discuss.elastic.co/t/looking-for-a-way-size-of-log-event/224805)
**Author:** [@shwesinhan](https://discuss.elastic.co/u/shwesinhan)\ **Replies:** 2\ **Last updated:** [April 14, 2020, 8:50am UTC](https://discuss.elastic.co/t/looking-for-a-way-size-of-log-event/224805 "2020-04-14T08:50:55Z")
Hello community, Is there any way to know message size in b or kb or mb or any unit of digital info in logstash? eg: logstash get message ABCDEFBADFASFD13241325POIYTUUBLABLABLA How can we know that message size? Is th… --- ## [The records in the elastic index are missing frequenlty](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738)
**Author:** [@tharak](https://discuss.elastic.co/u/tharak)\ **Replies:** 13\ **Last updated:** [April 14, 2020, 7:32am UTC](https://discuss.elastic.co/t/the-records-in-the-elastic-index-are-missing-frequenlty/227738 "2020-04-14T07:32:55Z")
I had a record in the elastic index which was indexed by logstash from a sql data source. This record which is already came to elastic index is not available after sometime. After updating the record in the database does… --- ## [Logstash doesn't loop log data to ES](https://discuss.elastic.co/t/logstash-doesnt-loop-log-data-to-es/227835)
**Author:** [@hibfnv](https://discuss.elastic.co/u/hibfnv)\ **Replies:** 1\ **Last updated:** [April 14, 2020, 6:07am UTC](https://discuss.elastic.co/t/logstash-doesnt-loop-log-data-to-es/227835 "2020-04-14T06:07:36Z")
Here's the logstash start process result: /opt/logstash/bin/logstash -f /opt/logstash/config/logstash.conf Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties \[2020-04-14T09:11:20,6… --- ## [Can we use logstash to gather information by SNMP Pool?](https://discuss.elastic.co/t/can-we-use-logstash-to-gather-information-by-snmp-pool/227850)
**Author:** [@psanggabuana](https://discuss.elastic.co/u/psanggabuana)\ **Replies:** 0\ **Last updated:** [April 14, 2020, 5:01am UTC](https://discuss.elastic.co/t/can-we-use-logstash-to-gather-information-by-snmp-pool/227850 "2020-04-14T05:01:46Z")
Hi Everyone, I want to gather information by SNMP Pool. Can we use logstash plugin for this? Thank you :grinning: --- ## [Extract server name from fqdn](https://discuss.elastic.co/t/extract-server-name-from-fqdn/224503)
**Author:** [@vijay\_kaali](https://discuss.elastic.co/u/vijay_kaali)\ **Replies:** 4\ **Last updated:** [April 14, 2020, 4:45am UTC](https://discuss.elastic.co/t/extract-server-name-from-fqdn/224503 "2020-04-14T04:45:22Z")
in beat.hostname, some are in CAPS / small , lower /upper case characters . To stabilize the search, i want to extract server name without fqdn and store in new field in lower case i tried following mutate { split … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=373) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=375)