# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=328 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 329 --- ## [Logstash multiple pipelines in k8s](https://discuss.elastic.co/t/logstash-multiple-pipelines-in-k8s/241497)
**Author:** [@digma89](https://discuss.elastic.co/u/digma89)\ **Replies:** 0\ **Last updated:** [July 16, 2020, 3:57pm UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-in-k8s/241497 "2020-07-16T15:57:32Z")
I'm trying to run multiple pipelines on my custom docker image in k8s, but doesn't seems to be working. On the config map in k8s I have: apiVersion: v1 kind: ConfigMap metadata: name: logstashffs-${trigger.resolvedEx… --- ## [Kafka input codec JSON parse error](https://discuss.elastic.co/t/kafka-input-codec-json-parse-error/240726)
**Author:** [@dikkini](https://discuss.elastic.co/u/dikkini)\ **Replies:** 4\ **Last updated:** [July 16, 2020, 3:16pm UTC](https://discuss.elastic.co/t/kafka-input-codec-json-parse-error/240726 "2020-07-16T15:16:30Z")
Hello! Telegraf send JSON to Kafka and logstash retrieve it. ES gives me parse error, but i can't understand what's wrong with my data. JSON from Telegraf: {"fields":{"created":1594396144981,"value":"{\\"id\\": 1, \\"ty… --- ## [Use Ruby to get field string length](https://discuss.elastic.co/t/use-ruby-to-get-field-string-length/241460)
**Author:** [@uklipse](https://discuss.elastic.co/u/uklipse)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 1:13pm UTC](https://discuss.elastic.co/t/use-ruby-to-get-field-string-length/241460 "2020-07-16T13:13:43Z")
I am trying to get the string length of a dns domain name and place into a new field. I've seen several posts to use ruby to achieve this but I can't seem to get it to work. This code is placed inside the filter {} brack… --- ## [Tag metrics as testing](https://discuss.elastic.co/t/tag-metrics-as-testing/241277)
**Author:** [@A\_Mightiev](https://discuss.elastic.co/u/A_Mightiev)\ **Replies:** 5\ **Last updated:** [July 16, 2020, 1:04pm UTC](https://discuss.elastic.co/t/tag-metrics-as-testing/241277 "2020-07-16T13:04:44Z")
Hello, I use Metricbeat with Logstash to process and send to my Elasticsearch cluster. I do performance testing in some equipment and I have to write down the start and end time of the test so I can go to Kibana and loo… --- ## [Logstash Performance Statistics](https://discuss.elastic.co/t/logstash-performance-statistics/241451)
**Author:** [@pulkit007](https://discuss.elastic.co/u/pulkit007)\ **Replies:** 0\ **Last updated:** [July 16, 2020, 12:23pm UTC](https://discuss.elastic.co/t/logstash-performance-statistics/241451 "2020-07-16T12:23:01Z")
Hi, I have these questions regarding Logstash : How can we find the time which Logstash takes to send logs to Elasticsearch? How can be find out the number of records present at any time in persistant queue for Log… --- ## [Logstash configs and grok pattern for java logs](https://discuss.elastic.co/t/logstash-configs-and-grok-pattern-for-java-logs/241135)
**Author:** [@Montego](https://discuss.elastic.co/u/Montego)\ **Replies:** 4\ **Last updated:** [July 16, 2020, 11:49am UTC](https://discuss.elastic.co/t/logstash-configs-and-grok-pattern-for-java-logs/241135 "2020-07-16T11:49:47Z")
Hi there! I have logs, wich looks like: \[INFO \] 28-04-2019 08:49:41.675 PM \[main\] client.Application - No active profile set, falling back to default profiles: default \[INFO \] 28-04-2019 08:49:41.735 PM \[main\] o.s.b.… --- ## [Logstash user-agent filter can't parse the user agent os with version](https://discuss.elastic.co/t/logstash-user-agent-filter-cant-parse-the-user-agent-os-with-version/240937)
**Author:** [@chairxiao](https://discuss.elastic.co/u/chairxiao)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 10:08am UTC](https://discuss.elastic.co/t/logstash-user-agent-filter-cant-parse-the-user-agent-os-with-version/240937 "2020-07-16T10:08:29Z")
Hello, My logstash's version is 7.7.1 I use the logstash user agent to parse the url,but i can't get the user agent os with version I got the parse info as blow: "name" =\> "Firefox", "os\_name… --- ## [Azure Event Hub logs not parsing](https://discuss.elastic.co/t/azure-event-hub-logs-not-parsing/241142)
**Author:** [@probson](https://discuss.elastic.co/u/probson)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 8:45am UTC](https://discuss.elastic.co/t/azure-event-hub-logs-not-parsing/241142 "2020-07-16T08:45:43Z")
Hi, i am using the logstash azure\_event\_hubs plugin in basic configuration, i am running7.7, the logs are been pulled but the message is not been parsed, any suggestions? Thanks Phil --- ## [Logstash pipeline Elasticsearch filter if conditional not working](https://discuss.elastic.co/t/logstash-pipeline-elasticsearch-filter-if-conditional-not-working/241390)
**Author:** [@tbege](https://discuss.elastic.co/u/tbege)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 7:35am UTC](https://discuss.elastic.co/t/logstash-pipeline-elasticsearch-filter-if-conditional-not-working/241390 "2020-07-16T07:35:53Z")
Hello, I am using the following configuration to sync some data into my Elasticsearch indeces: logstashPipeline: logstash.conf: | input { jdbc { jdbc\_driver\_library =\> "/usr/share/lo… --- ## [Logstash. Export data from Elasticsearch's nested field](https://discuss.elastic.co/t/logstash-export-data-from-elasticsearchs-nested-field/240354)
**Author:** [@Alex\_D](https://discuss.elastic.co/u/Alex_D)\ **Replies:** 7\ **Last updated:** [July 16, 2020, 7:18am UTC](https://discuss.elastic.co/t/logstash-export-data-from-elasticsearchs-nested-field/240354 "2020-07-16T07:18:39Z")
Hi all ! I'am trying to export data from ElasticSearch into csv-file. I installed plugins "logstash-input-elasticsearch", "logstash-output-csv" before. Part of my ES data scheme: mappings: { properties: { … --- ## [Delete all documents from specific ES index, that are not in the logstash feed](https://discuss.elastic.co/t/delete-all-documents-from-specific-es-index-that-are-not-in-the-logstash-feed/241318)
**Author:** [@Deleanu\_Alexandru](https://discuss.elastic.co/u/Deleanu_Alexandru)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 7:15am UTC](https://discuss.elastic.co/t/delete-all-documents-from-specific-es-index-that-are-not-in-the-logstash-feed/241318 "2020-07-16T07:15:48Z")
Hello, I am writing an ES ecommerce product search. I feed data to ES using filebeat and logstash like this: - every 24h I will upload a json file containing all the products in the ES index (all products that I want … --- ## [Error while parsing date with Date filter](https://discuss.elastic.co/t/error-while-parsing-date-with-date-filter/241312)
**Author:** [@Ashish\_kapoor](https://discuss.elastic.co/u/Ashish_kapoor)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 7:11am UTC](https://discuss.elastic.co/t/error-while-parsing-date-with-date-filter/241312 "2020-07-16T07:11:41Z")
Hi, I am trying to parse date "Jan 21, 2020 1:02:24 PM". But I am getting \_dateparsefailure. I am using below conf in filter date { match =\> \["DateTime", "MMM d, yyyy h:mm a"\] target … --- ## [Grok failure, Grok debug OK](https://discuss.elastic.co/t/grok-failure-grok-debug-ok/240693)
**Author:** [@Abdelhalim](https://discuss.elastic.co/u/Abdelhalim)\ **Replies:** 1\ **Last updated:** [July 11, 2020, 6:37pm UTC](https://discuss.elastic.co/t/grok-failure-grok-debug-ok/240693 "2020-07-11T18:37:13Z")
Hello, I am using grok to parse my logs, everything works in grok debugger in Kibana but when I use it in my pipeline it doesn't work!! here is an example of my log: Jul 10 12:21:08 vselk-vyos kernel: \[155099.389432\]… --- ## [Logstash-plain.log is not rolling after upgrade to 7.x](https://discuss.elastic.co/t/logstash-plain-log-is-not-rolling-after-upgrade-to-7-x/241393)
**Author:** [@rchung-hku](https://discuss.elastic.co/u/rchung-hku)\ **Replies:** 0\ **Last updated:** [July 16, 2020, 6:33am UTC](https://discuss.elastic.co/t/logstash-plain-log-is-not-rolling-after-upgrade-to-7-x/241393 "2020-07-16T06:33:16Z")
Hi all I was using logstash 6.5.x before, Logstash used to auto rolling the logstash-plain.log everyday to logstash-plain.log-YYYY-MM-DD-1.log.gz. However, after I upgrade the ES and Logstash to 7.6.2, the logstash-pl… --- ## [Date format in my sql](https://discuss.elastic.co/t/date-format-in-my-sql/240182)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 6:12am UTC](https://discuss.elastic.co/t/date-format-in-my-sql/240182 "2020-07-16T06:12:02Z")
I have tried to parse data from mysql into elk stack i have date column which has date format 2020-04-01 00:00:01 and 2020-06-25 13:36:24 but when i parse the data the date is not showing in date format its still showing… --- ## [GROK Pattern](https://discuss.elastic.co/t/grok-pattern/241291)
**Author:** [@Syed.Ubaid](https://discuss.elastic.co/u/Syed.Ubaid)\ **Replies:** 6\ **Last updated:** [July 16, 2020, 6:08am UTC](https://discuss.elastic.co/t/grok-pattern/241291 "2020-07-16T06:08:39Z")
I try to parse data from ELK to Kibana using logstash. I write some customize grok Pattern using this site https://grokdebug.herokuapp.com/ here the pattern is fine in but when i parse the data from logstash some data is… --- ## [Unable to retrive licence while trying to run logstash](https://discuss.elastic.co/t/unable-to-retrive-licence-while-trying-to-run-logstash/241386)
**Author:** [@vkg0312](https://discuss.elastic.co/u/vkg0312)\ **Replies:** 1\ **Last updated:** [July 16, 2020, 5:56am UTC](https://discuss.elastic.co/t/unable-to-retrive-licence-while-trying-to-run-logstash/241386 "2020-07-16T05:56:49Z")
Error 2: \[2020-07-09T15:58:54,431\]\[ERROR\]\[logstash.licensechecker.licensereader\] Unable to retrieve license information from license server {:message=\>"Unrecognized SSL message, plaintext connection?" --- ## ["\_grokparsefailure" for pattern that works on grok debugger](https://discuss.elastic.co/t/grokparsefailure-for-pattern-that-works-on-grok-debugger/241224)
**Author:** [@Khaled3500](https://discuss.elastic.co/u/Khaled3500)\ **Replies:** 2\ **Last updated:** [July 16, 2020, 1:51am UTC](https://discuss.elastic.co/t/grokparsefailure-for-pattern-that-works-on-grok-debugger/241224 "2020-07-16T01:51:21Z")
Hello, I created a grok pattern and it works well on grok debugger but when i add to filter inside the pipeline it pass the restart phase for the service but on kibana it give this tag \["\_grokparsefailure"\] i tried to de… --- ## [Missing geoip.location](https://discuss.elastic.co/t/missing-geoip-location/241004)
**Author:** [@brapnda23](https://discuss.elastic.co/u/brapnda23)\ **Replies:** 8\ **Last updated:** [July 15, 2020, 11:11pm UTC](https://discuss.elastic.co/t/missing-geoip-location/241004 "2020-07-15T23:11:54Z")
New to ELK...My test ELK instance is 7.8.x. I am using the below filter code for IIS Logs and it is working fine. I can see other geo fields but I don't see geoip.location as mentioned in https://www.elastic.co/blog/geo… --- ## [Logstash](https://discuss.elastic.co/t/logstash/241326)
**Author:** [@laxmikanth](https://discuss.elastic.co/u/laxmikanth)\ **Replies:** 1\ **Last updated:** [July 15, 2020, 9:25pm UTC](https://discuss.elastic.co/t/logstash/241326 "2020-07-15T21:25:57Z")
When I try to run logstash in local, noticing below error. Can some one please help me? --- ## [I wonder about logstash YYYY.MM.dd](https://discuss.elastic.co/t/i-wonder-about-logstash-yyyy-mm-dd/241073)
**Author:** [@111349](https://discuss.elastic.co/u/111349)\ **Replies:** 9\ **Last updated:** [July 15, 2020, 9:07pm UTC](https://discuss.elastic.co/t/i-wonder-about-logstash-yyyy-mm-dd/241073 "2020-07-15T21:07:22Z")
I am not sure what logstash YYYY.MM.dd is based on. I'm in asia/seoul area and I also sent a message at 1am. The 13th was 1:00 a.m. on the 14th, but there was no newly created 14-day index in elasticsearch. I searched … --- ## [Logstash Aggregate Filter output different result each time](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303)
**Author:** [@Aberry](https://discuss.elastic.co/u/Aberry)\ **Replies:** 5\ **Last updated:** [July 15, 2020, 5:19pm UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-output-different-result-each-time/239303 "2020-07-15T17:19:39Z")
I have 2 input files like below - inbound.txt: 17:03:56.662 53897 55931 17:03:56.737 53898 55932 17:03:56.814 53899 55933 17:03:56.889 53900 55934 outbound.txt: 17:03:56.887 77307 55931 17:03:56.953 77… --- ## [Logstash Jdbc Input Plugin format for timestamp in prepared statement](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-format-for-timestamp-in-prepared-statement/241307)
**Author:** [@styks90](https://discuss.elastic.co/u/styks90)\ **Replies:** 1\ **Last updated:** [July 15, 2020, 2:47pm UTC](https://discuss.elastic.co/t/logstash-jdbc-input-plugin-format-for-timestamp-in-prepared-statement/241307 "2020-07-15T14:47:27Z")
We are looking to use prepared statements with the Jdbc input plugin, where the source database is a Db2 instance. So far we've not been using prepared statements, so that the relevant part of the (working) config looks… --- ## [How to stop logstash gracefully when using kubernetes](https://discuss.elastic.co/t/how-to-stop-logstash-gracefully-when-using-kubernetes/241288)
**Author:** [@benz](https://discuss.elastic.co/u/benz)\ **Replies:** 1\ **Last updated:** [July 15, 2020, 1:54pm UTC](https://discuss.elastic.co/t/how-to-stop-logstash-gracefully-when-using-kubernetes/241288 "2020-07-15T13:54:26Z")
Hi, We are running logstash as statefulset over k8s. I am looking for a way to stop logstash only after it finish process all in-flight events. We already use: enable\_auto\_commit =\> "false" for kafka input so I guess… --- ## [Tag \["\_parsegrokfailure"\] anyone could help me?](https://discuss.elastic.co/t/tag-parsegrokfailure-anyone-could-help-me/241190)
**Author:** [@Khaled3500](https://discuss.elastic.co/u/Khaled3500)\ **Replies:** 5\ **Last updated:** [July 15, 2020, 12:51pm UTC](https://discuss.elastic.co/t/tag-parsegrokfailure-anyone-could-help-me/241190 "2020-07-15T12:51:54Z")
Hello, I created a grok pattern and it works well on grok debugger but when i add to filter inside the pipeline it pass the restart phase for the service but on kibana it give this tag \["\_grokparsefailure"\] i tried to de… --- ## [Get specific string from field in logstash](https://discuss.elastic.co/t/get-specific-string-from-field-in-logstash/241276)
**Author:** [@tungpzostar](https://discuss.elastic.co/u/tungpzostar)\ **Replies:** 4\ **Last updated:** [July 15, 2020, 12:29pm UTC](https://discuss.elastic.co/t/get-specific-string-from-field-in-logstash/241276 "2020-07-15T12:29:45Z")
Hello, I'm new to logstash and I'm having trouble with this. I configured filebeat to receive Nginx log: This is sample input: "log": { "offset": 2234788, "file": { "path": "/nginxlogs/nginx.abc.1234.a… --- ## [How can I change the display of data in kibana from logstash?](https://discuss.elastic.co/t/how-can-i-change-the-display-of-data-in-kibana-from-logstash/241286)
**Author:** [@menes](https://discuss.elastic.co/u/menes)\ **Replies:** 1\ **Last updated:** [July 15, 2020, 12:12pm UTC](https://discuss.elastic.co/t/how-can-i-change-the-display-of-data-in-kibana-from-logstash/241286 "2020-07-15T12:12:04Z")
Good afternoon. Logstash passes a numeric value in the form of 10000 to elasticsearch, but in kibana it is displayed as 10, 000 can I configure logstash to display numeric values by default without a separator in kibana… --- ## [Rss feed ingestion exception](https://discuss.elastic.co/t/rss-feed-ingestion-exception/241270)
**Author:** [@anu\_gaur](https://discuss.elastic.co/u/anu_gaur)\ **Replies:** 0\ **Last updated:** [July 15, 2020, 11:00am UTC](https://discuss.elastic.co/t/rss-feed-ingestion-exception/241270 "2020-07-15T11:00:47Z")
Hi, I am using logstash rss plugin for rss feeds ingestion in my elastic cloud. But when it try to ingest the feed, it throws exception Invalid RSS feed {:exception=\>#\} .… --- ## [Logstash not able to create index](https://discuss.elastic.co/t/logstash-not-able-to-create-index/239941)
**Author:** [@sumitsahay](https://discuss.elastic.co/u/sumitsahay)\ **Replies:** 19\ **Last updated:** [July 15, 2020, 10:35am UTC](https://discuss.elastic.co/t/logstash-not-able-to-create-index/239941 "2020-07-15T10:35:03Z")
Hi All, My logstash file is not able to create the Index pattern in Kibana. My logstash.conf file /etc/logstash/conf.d/logstash.conf \< input { beats { port =\> 5044 type =\> syslog ssl\_certificate =\> "/etc… --- ## [Ingesting High Volume of AWS Flowlogs](https://discuss.elastic.co/t/ingesting-high-volume-of-aws-flowlogs/241151)
**Author:** [@sam6](https://discuss.elastic.co/u/sam6)\ **Replies:** 1\ **Last updated:** [July 15, 2020, 10:16am UTC](https://discuss.elastic.co/t/ingesting-high-volume-of-aws-flowlogs/241151 "2020-07-15T10:16:10Z")
Hi, Im currently using the Logstash S3 input module to ingest flowlog data from an S3 bucket however it isn't pulling in enough data quick enough and so falling behind. I've tried upping the max batch size to 20000 in … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=327) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=329)