# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=127 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 128 --- ## [Single JSON document parsed as individual key pair in logstash](https://discuss.elastic.co/t/single-json-document-parsed-as-individual-key-pair-in-logstash/309787)
**Author:** [@parthmaniar](https://discuss.elastic.co/u/parthmaniar)\ **Replies:** 1\ **Last updated:** [July 18, 2022, 3:00am UTC](https://discuss.elastic.co/t/single-json-document-parsed-as-individual-key-pair-in-logstash/309787 "2022-07-18T03:00:27Z")
Hello, I hope my message find the community members and their loved ones safe and healthy. I am trying to ingest ~30,000 documents which are API responses with selective keys stored as a JSON document. \[ { … --- ## [Pipelines don't working](https://discuss.elastic.co/t/pipelines-dont-working/309793)
**Author:** [@White\_Hat](https://discuss.elastic.co/u/White_Hat)\ **Replies:** 3\ **Last updated:** [July 17, 2022, 6:35am UTC](https://discuss.elastic.co/t/pipelines-dont-working/309793 "2022-07-17T06:35:30Z")
I want to log two separate servers that each has filebeat installed. this is my pipeline: - pipeline.id: beats-server config.string: | input { beats { port =\> 5400 } } output { if \[source\] == 'src' { … --- ## [Log timestamp is not getting through date filter getting date\_time\_parse\_exception](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705)
**Author:** [@tahseen\_fatima](https://discuss.elastic.co/u/tahseen_fatima)\ **Replies:** 17\ **Last updated:** [July 16, 2022, 9:23pm UTC](https://discuss.elastic.co/t/log-timestamp-is-not-getting-through-date-filter-getting-date-time-parse-exception/309705 "2022-07-16T21:23:19Z")
Hi Team, I have following logstash configuration for where I have written the date pattern to parse my timestamp, but time is not working. I'm getting \_dateparsefailure. here is the configuration. input{.............}… --- ## [Unable to push messages to Kafka streams](https://discuss.elastic.co/t/unable-to-push-messages-to-kafka-streams/309458)
**Author:** [@Rao\_Nelakurti](https://discuss.elastic.co/u/Rao_Nelakurti)\ **Replies:** 5\ **Last updated:** [July 15, 2022, 4:39pm UTC](https://discuss.elastic.co/t/unable-to-push-messages-to-kafka-streams/309458 "2022-07-15T16:39:45Z")
Hi Team, I'm trying to send application logs (filebeat -\> logstash-\> kafka topic) to kafka stream. For some reason I'm not able to see logs in kafka. Here is my filebeat.conf filebeat.spool\_size: 2048 filebeat.idle\_ti… --- ## [Json message storing with message. extension in index](https://discuss.elastic.co/t/json-message-storing-with-message-extension-in-index/309726)
**Author:** [@harijld](https://discuss.elastic.co/u/harijld)\ **Replies:** 1\ **Last updated:** [July 15, 2022, 3:26pm UTC](https://discuss.elastic.co/t/json-message-storing-with-message-extension-in-index/309726 "2022-07-15T15:26:02Z")
Hi Team, I have small query here. I am sending json messag on Kafka queue and indexing the same in Elasticsearch. It is inserting properly but all the fields are storing with message.field extension. Is there any param… --- ## [Unable to run Logstash 8.3.1 (not even to check plugins): Errno::EACCES: Permission denied - NUL](https://discuss.elastic.co/t/unable-to-run-logstash-8-3-1-not-even-to-check-plugins-errno-permission-denied-nul/308871)
**Author:** [@stevedearl](https://discuss.elastic.co/u/stevedearl)\ **Replies:** 4\ **Last updated:** [July 15, 2022, 10:54am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-8-3-1-not-even-to-check-plugins-errno-permission-denied-nul/308871 "2022-07-15T10:54:35Z")
I'm seeing an issue after installing Logstash 8.3.1 on Windows 10 (as part of an upgrade from Logstash 8.1.2 which has been running perfectly fine). All I've done is unpack the Logstash 8.3.1 zip file into a suitable fo… --- ## [Error: getsockopt: connection refused](https://discuss.elastic.co/t/error-getsockopt-connection-refused/309564)
**Author:** [@benjamin\_brightson](https://discuss.elastic.co/u/benjamin_brightson)\ **Replies:** 1\ **Last updated:** [July 15, 2022, 9:04am UTC](https://discuss.elastic.co/t/error-getsockopt-connection-refused/309564 "2022-07-15T09:04:29Z")
Hi Team, This is to inform you that am trying to setup the multiple beat config with logstash , so initially i started with the single beat config where am facing the " \[Error: getsockopt: connection refused\]" this erro… --- ## [Issue for the multiline input](https://discuss.elastic.co/t/issue-for-the-multiline-input/309459)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 5\ **Last updated:** [July 15, 2022, 9:00am UTC](https://discuss.elastic.co/t/issue-for-the-multiline-input/309459 "2022-07-15T09:00:08Z")
Hi @leandrojmp There we will continue topic from Grok for data - #13 by leandrojmp now we need to complete process for parsing below date with the exact content: input { file { mode =\> read path =\>… --- ## [Logstash logfile manipulation](https://discuss.elastic.co/t/logstash-logfile-manipulation/309513)
**Author:** [@hiteshadabala](https://discuss.elastic.co/u/hiteshadabala)\ **Replies:** 1\ **Last updated:** [July 15, 2022, 7:09am UTC](https://discuss.elastic.co/t/logstash-logfile-manipulation/309513 "2022-07-15T07:09:16Z")
one log file is this way line1 line2 line3 When givenn as input to logstash with grok match "message":".\*$", the log file is entered into ES as "message":"line1\\nline2\\nline3" Each \\n is considered new line and can s… --- ## [Assistance with Grok and regex](https://discuss.elastic.co/t/assistance-with-grok-and-regex/309600)
**Author:** [@JeremyP](https://discuss.elastic.co/u/JeremyP)\ **Replies:** 2\ **Last updated:** [July 15, 2022, 7:00am UTC](https://discuss.elastic.co/t/assistance-with-grok-and-regex/309600 "2022-07-15T07:00:18Z")
Hello, I'm attempting to pull the name of a software package from a CPE from NIST. This is my sample data: cpe:2.3:a:libexpat\_project:libexpat:\*:\*:\*:\*:\*:\*:\*:\* With regular regex the following expression matches the st… --- ## [Multiple Stomp servers](https://discuss.elastic.co/t/multiple-stomp-servers/309686)
**Author:** [@luv4diamonds](https://discuss.elastic.co/u/luv4diamonds)\ **Replies:** 2\ **Last updated:** [July 14, 2022, 8:03pm UTC](https://discuss.elastic.co/t/multiple-stomp-servers/309686 "2022-07-14T20:03:16Z")
Hi, I am trying to configure Stomp input with 3 servers but this doesn't appear to work in ES 7.17.3. I have tried host =\> "host1", "host2", "host3" which does not work. Does anyone know how to configure this or is it… --- ## [Two strings with one grok](https://discuss.elastic.co/t/two-strings-with-one-grok/309646)
**Author:** [@PJss](https://discuss.elastic.co/u/PJss)\ **Replies:** 3\ **Last updated:** [July 14, 2022, 6:09pm UTC](https://discuss.elastic.co/t/two-strings-with-one-grok/309646 "2022-07-14T18:09:32Z")
Hello, please help to read this two strings with one grok rule first: mgmgmg : TTY=unknown ; PWD=/usr/local/gtail/basecomps/deploy/cache ; USER=root ; ENV=HEALTHCHECK=no BACKUP=no ; COMMAND=/usr/bin/dpkg --install goser… --- ## [Logstash Ruby filter - init not working](https://discuss.elastic.co/t/logstash-ruby-filter-init-not-working/309670)
**Author:** [@Hichem](https://discuss.elastic.co/u/Hichem)\ **Replies:** 1\ **Last updated:** [July 14, 2022, 6:01pm UTC](https://discuss.elastic.co/t/logstash-ruby-filter-init-not-working/309670 "2022-07-14T18:01:16Z")
I'm trying to initialize a variable in a ruby filter. It works properly with the code tag, but not when using a ruby file. Here is my code: pipeline.conf ruby { init =\> "@val = 5 " path =\> "script.rb" }… --- ## [Problem with date filter](https://discuss.elastic.co/t/problem-with-date-filter/309642)
**Author:** [@Thuunder7](https://discuss.elastic.co/u/Thuunder7)\ **Replies:** 11\ **Last updated:** [July 14, 2022, 5:26pm UTC](https://discuss.elastic.co/t/problem-with-date-filter/309642 "2022-07-14T17:26:24Z")
Hello, I am trying to parse the following message (e.g): 2022-07-14T13:06:16 Using the dissect filter correctly: dissect { mapping =\> { "\[message\]" =\> "%{\[my\]\[date\]}" … --- ## [Cloudwatch Output from logstash](https://discuss.elastic.co/t/cloudwatch-output-from-logstash/309661)
**Author:** [@jlbai](https://discuss.elastic.co/u/jlbai)\ **Replies:** 0\ **Last updated:** [July 14, 2022, 2:56pm UTC](https://discuss.elastic.co/t/cloudwatch-output-from-logstash/309661 "2022-07-14T14:56:14Z")
I am looking to out to cloudwatch from logstash. I can not use an IAM user access and secret access key to connect to cloud watch Does anyone have experience or solution so I can talk to my aws cloudwatch input { udp { … --- ## [Pipeline - Create a delta field](https://discuss.elastic.co/t/pipeline-create-a-delta-field/309618)
**Author:** [@Ely\_96](https://discuss.elastic.co/u/Ely_96)\ **Replies:** 0\ **Last updated:** [July 14, 2022, 8:29am UTC](https://discuss.elastic.co/t/pipeline-create-a-delta-field/309618 "2022-07-14T08:29:50Z")
Hi Guys, I'm trying to study how to create a delta field: I mean a field that should contains a difference between 2 fields inside an existing index. So, I have 2 examples of CSVs: ID;Month;Date;Date\_string;Name;Surna… --- ## [How can I change timezone](https://discuss.elastic.co/t/how-can-i-change-timezone/309615)
**Author:** [@msjhbhh](https://discuss.elastic.co/u/msjhbhh)\ **Replies:** 0\ **Last updated:** [July 14, 2022, 8:02am UTC](https://discuss.elastic.co/t/how-can-i-change-timezone/309615 "2022-07-14T08:02:29Z")
how can I change timezone $ date -Is -d '2022-07-14 15:25:26.867' 2022-07-14T15:25:26.867+0200 --- ## [Using tcp plugin to parse logs from multiple sources](https://discuss.elastic.co/t/using-tcp-plugin-to-parse-logs-from-multiple-sources/309565)
**Author:** [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)\ **Replies:** 3\ **Last updated:** [July 13, 2022, 9:16pm UTC](https://discuss.elastic.co/t/using-tcp-plugin-to-parse-logs-from-multiple-sources/309565 "2022-07-13T21:16:21Z")
We are using ELK 7.6.2 stack. I am trying to configure Logstash to parse inputs based on tcp plugin. My config looks like this: input { tcp { port =\> 6789 codec =\> json\_lines tags =\>… --- ## [NMON to JSON Converted Files Will Not Import](https://discuss.elastic.co/t/nmon-to-json-converted-files-will-not-import/309566)
**Author:** [@yoscar](https://discuss.elastic.co/u/yoscar)\ **Replies:** 1\ **Last updated:** [July 13, 2022, 8:14pm UTC](https://discuss.elastic.co/t/nmon-to-json-converted-files-will-not-import/309566 "2022-07-13T20:14:45Z")
We're trying to ingest nmon data that's recorded over a 24 hour timespan into Logstash. We first convert it to json usin nmon2json (I understand that njmon is an option, but we are trying to use existing nmon files for n… --- ## [Grok for data](https://discuss.elastic.co/t/grok-for-data/309093)
**Author:** [@INS](https://discuss.elastic.co/u/INS)\ **Replies:** 17\ **Last updated:** [July 13, 2022, 11:57am UTC](https://discuss.elastic.co/t/grok-for-data/309093 "2022-07-13T11:57:35Z")
Can anyone try to build grok for below data it's important that timestamp should be took from the first line of document 20220704061503 and interesting columns number: 0000080 data1:abort 0 type: onlist yes input of … --- ## [Logstash Nested Fields - Cloudtrail logs - No root field](https://discuss.elastic.co/t/logstash-nested-fields-cloudtrail-logs-no-root-field/309568)
**Author:** [@Pedro\_Cabral](https://discuss.elastic.co/u/Pedro_Cabral)\ **Replies:** 0\ **Last updated:** [July 13, 2022, 6:01pm UTC](https://discuss.elastic.co/t/logstash-nested-fields-cloudtrail-logs-no-root-field/309568 "2022-07-13T18:01:00Z")
Hello all, First of all I'm running version 7.10.2 on a docker. I have some cloudtrail logs stored as .gz that I'm reading and after parsing I'm renaming the Records fields and build the different nested fields. When I… --- ## [When field \_index in elasticsearch is generated?](https://discuss.elastic.co/t/when-field-index-in-elasticsearch-is-generated/309519)
**Author:** [@maoxuguang](https://discuss.elastic.co/u/maoxuguang)\ **Replies:** 1\ **Last updated:** [July 13, 2022, 4:19pm UTC](https://discuss.elastic.co/t/when-field-index-in-elasticsearch-is-generated/309519 "2022-07-13T16:19:12Z")
in elasticsearch, there is a field named \_index, is this field generated by logstash? In logstash we output same data to both elasticsearch and mongodb, while elasticsearch has a field named \_index, but mongodb has no s… --- ## [Elasticsearch Index mappings( version 6.5.4)](https://discuss.elastic.co/t/elasticsearch-index-mappings-version-6-5-4/309535)
**Author:** [@gyrao\_72](https://discuss.elastic.co/u/gyrao_72)\ **Replies:** 2\ **Last updated:** [July 13, 2022, 4:14pm UTC](https://discuss.elastic.co/t/elasticsearch-index-mappings-version-6-5-4/309535 "2022-07-13T16:14:36Z")
I am using filebeat to fetch logs, logstash to filter, and pushing them to elasticsearch. I have created an index on elasticsearch import-export-logger. This id mapping of import-export-logger. { "import-export-logg… --- ## [Why do we need logstash](https://discuss.elastic.co/t/why-do-we-need-logstash/309379)
**Author:** [@yugeeklab](https://discuss.elastic.co/u/yugeeklab)\ **Replies:** 2\ **Last updated:** [July 13, 2022, 1:43pm UTC](https://discuss.elastic.co/t/why-do-we-need-logstash/309379 "2022-07-13T13:43:12Z")
I found out my Logstash has no logic except for forwarding log to Elasticsearch. Filebeat has many logic instead of Logstash(filtering etc.) In my case, Still do i need Logstash?? Why?? --- ## [How to Replace agent.\* and ecs.version with older Fields(filebeat.version,beat.hostname)](https://discuss.elastic.co/t/how-to-replace-agent-and-ecs-version-with-older-fields-filebeat-version-beat-hostname/309526)
**Author:** [@krish0608](https://discuss.elastic.co/u/krish0608)\ **Replies:** 0\ **Last updated:** [July 13, 2022, 11:34am UTC](https://discuss.elastic.co/t/how-to-replace-agent-and-ecs-version-with-older-fields-filebeat-version-beat-hostname/309526 "2022-07-13T11:34:43Z")
Hi Team, I recently upgraded my filebeat from 5.6.5 to 7.17.3 and found in 7.x fields are different from 5.6.5. I want all the processor metadata should replaced with older keys but I still can't get rid of agent.ephem… --- ## [Failed to filter using following grok pattern](https://discuss.elastic.co/t/failed-to-filter-using-following-grok-pattern/309116)
**Author:** [@Rao\_Nelakurti](https://discuss.elastic.co/u/Rao_Nelakurti)\ **Replies:** 2\ **Last updated:** [July 12, 2022, 8:06pm UTC](https://discuss.elastic.co/t/failed-to-filter-using-following-grok-pattern/309116 "2022-07-12T20:06:15Z")
Hi Team, I'm trying filter following log message, \[2022-07-06T20:54:20.471-0700\] \[LogLevel:INFO\] \[ServerName:xyz.vcn.com\] \[IP:10.x.x.66\]\[AppName:IoT\] \[FlowName: updateDevice\] \[ID:2022-07-06 20:54:20.471\] \[ECID:yja\_Y1eg… --- ## [LogStash::Json::ParserError: Unexpected character ('(' (code 40))](https://discuss.elastic.co/t/logstash-unexpected-character-code-40/309438)
**Author:** [@John\_snow](https://discuss.elastic.co/u/John_snow)\ **Replies:** 9\ **Last updated:** [July 12, 2022, 7:59pm UTC](https://discuss.elastic.co/t/logstash-unexpected-character-code-40/309438 "2022-07-12T19:59:26Z")
I'm getting following error Error: :exception=\>#\ **Author:** [@Anusha\_Kusanghi](https://discuss.elastic.co/u/Anusha_Kusanghi)\ **Replies:** 3\ **Last updated:** [July 12, 2022, 1:43pm UTC](https://discuss.elastic.co/t/how-to-remove-drop-entire-logs-after-checking-a-condition-in-nested-json-fields/309322 "2022-07-12T13:43:46Z") Hie , Im trying to check a condition for nested json fields and if the condition is met I want to drop the entire data , but it is not working Source: "Data" =\> \[ \[0\] { "Scales" =\> \[ \[0\] { "TaskInfos" =\> \[ \[0\] { … --- ## [Logstash with Localstack Kinesis](https://discuss.elastic.co/t/logstash-with-localstack-kinesis/309410)
**Author:** [@frank\_2](https://discuss.elastic.co/u/frank_2)\ **Replies:** 0\ **Last updated:** [July 12, 2022, 11:15am UTC](https://discuss.elastic.co/t/logstash-with-localstack-kinesis/309410 "2022-07-12T11:15:56Z")
Hi, Has anyone successfully managed to run Logstash against a Localstack-provided Kinesis stream, or even just a plain kinesis-mock container? I would like very much to test Logstash configurations and throughput local… --- ## [Real time data not syncing from oracle to Elasticsearch index](https://discuss.elastic.co/t/real-time-data-not-syncing-from-oracle-to-elasticsearch-index/309298)
**Author:** [@suresh\_u](https://discuss.elastic.co/u/suresh_u)\ **Replies:** 2\ **Last updated:** [July 12, 2022, 11:02am UTC](https://discuss.elastic.co/t/real-time-data-not-syncing-from-oracle-to-elasticsearch-index/309298 "2022-07-12T11:02:44Z")
Hi Team, I am using Logstash to sync data from Oracle to ES. When I update in Oracle db the same changes are not reflecting in ES. Could you please help to resolve the issue? Below is the Logstash configuration that … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=126) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=128)