# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=479 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 480 --- ## [Json Transform in Logstash](https://discuss.elastic.co/t/json-transform-in-logstash/198698)
**Author:** [@Siddharth86](https://discuss.elastic.co/u/Siddharth86)\ **Replies:** 4\ **Last updated:** [September 11, 2019, 10:34am UTC](https://discuss.elastic.co/t/json-transform-in-logstash/198698 "2019-09-11T10:34:13Z")
I have a json like this { "consumer": { "created\_at": 1566912618154, "username": "dummyapp", "id": "079c3e6d-ca35-464b-8596-b2ace5e60481" }, "service": { "created\_at": 1567173649, "connect\_timeout": 60000, "pro… --- ## [Missing index issue](https://discuss.elastic.co/t/missing-index-issue/199060)
**Author:** [@msk\_76](https://discuss.elastic.co/u/msk_76)\ **Replies:** 0\ **Last updated:** [September 11, 2019, 10:32am UTC](https://discuss.elastic.co/t/missing-index-issue/199060 "2019-09-11T10:32:33Z")
I have a java based application which reads an index sys\_username\_mapping. This index gets update once every day. My steps are : initially load the index sys\_username\_mapping using filebeat+logstash for first time with… --- ## [Unable to create elasticsearch indices for json file](https://discuss.elastic.co/t/unable-to-create-elasticsearch-indices-for-json-file/199057)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 0\ **Last updated:** [September 11, 2019, 10:15am UTC](https://discuss.elastic.co/t/unable-to-create-elasticsearch-indices-for-json-file/199057 "2019-09-11T10:15:53Z")
I am having this json file for which I am using .conf file to parse it into elasticsearch indices but I am getting error. My .conf file is : input { file { codec =\> json path =\> \["/home/2017.09.04.json"\] start\_posi… --- ## [Multiline grepping](https://discuss.elastic.co/t/multiline-grepping/199027)
**Author:** [@satyam2593](https://discuss.elastic.co/u/satyam2593)\ **Replies:** 0\ **Last updated:** [September 11, 2019, 7:25am UTC](https://discuss.elastic.co/t/multiline-grepping/199027 "2019-09-11T07:25:47Z")
i want to grep the multilines from the logs. and am using multiline codec in my config file.. Below are the sample pattern and config file sample logs: -------------------------------------- 17 Jul 2019 00:15:00,403 … --- ## [Tag the first line in the log](https://discuss.elastic.co/t/tag-the-first-line-in-the-log/199016)
**Author:** [@Alisa\_Faingold](https://discuss.elastic.co/u/Alisa_Faingold)\ **Replies:** 0\ **Last updated:** [September 11, 2019, 6:45am UTC](https://discuss.elastic.co/t/tag-the-first-line-in-the-log/199016 "2019-09-11T06:45:12Z")
Hi, I try to calculate times of all kinds of device actions through the log. I would also like to calculate the time from the beginning of the log to a certain line appearance, and from a certain line appearance to the … --- ## [Problem with the code. Only parsing one grok pattern and not overwriting message field](https://discuss.elastic.co/t/problem-with-the-code-only-parsing-one-grok-pattern-and-not-overwriting-message-field/198625)
**Author:** [@rajdeep101](https://discuss.elastic.co/u/rajdeep101)\ **Replies:** 16\ **Last updated:** [September 11, 2019, 6:42am UTC](https://discuss.elastic.co/t/problem-with-the-code-only-parsing-one-grok-pattern-and-not-overwriting-message-field/198625 "2019-09-11T06:42:55Z")
Hi all, i am trying to parse certain log lines and fetch only required information. I am using if else statements to differentiate according to which my lines should parse. But all log line are parsing using first grok p… --- ## [Logstash grok pattern issue](https://discuss.elastic.co/t/logstash-grok-pattern-issue/198950)
**Author:** [@mugil1988](https://discuss.elastic.co/u/mugil1988)\ **Replies:** 2\ **Last updated:** [September 11, 2019, 5:45am UTC](https://discuss.elastic.co/t/logstash-grok-pattern-issue/198950 "2019-09-11T05:45:24Z")
Hi All, I am trying to configure grok patter to parse my application logs. Part of the logs is not getting displayed in kibana. Sample logs : 2019-09-10T23:04:21,584 | INFO | ://dedupresquest | \_route1 … --- ## [Unable To Split Json Array in Elasticsearch Query Response](https://discuss.elastic.co/t/unable-to-split-json-array-in-elasticsearch-query-response/199009)
**Author:** [@rameshkokare1](https://discuss.elastic.co/u/rameshkokare1)\ **Replies:** 0\ **Last updated:** [September 11, 2019, 5:29am UTC](https://discuss.elastic.co/t/unable-to-split-json-array-in-elasticsearch-query-response/199009 "2019-09-11T05:29:32Z")
Hi, Not able to parse brand filed from below response. Logstash taking requests on port 31315 via http input plugin json to be parsed: { "took": 2, "timed\_out": false, "\_shards": { "total": 1, "successful": 1, … --- ## [Failed to start filebeat.service: Unit filebeat.service not found](https://discuss.elastic.co/t/failed-to-start-filebeat-service-unit-filebeat-service-not-found/198877)
**Author:** [@Tje](https://discuss.elastic.co/u/Tje)\ **Replies:** 1\ **Last updated:** [September 11, 2019, 4:17am UTC](https://discuss.elastic.co/t/failed-to-start-filebeat-service-unit-filebeat-service-not-found/198877 "2019-09-11T04:17:57Z")
I'm trying to enable filebeat in logstash.below is my filebeat.yml output.elasticsearch: # Array of hosts to connect to. hosts: \["localhost:9200"\] username: "elastic" password: "elastic" setup.kibana: host: … --- ## [Unable to load CSV file into Elasticsearch](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826)
**Author:** [@Ramya\_Tanikanti](https://discuss.elastic.co/u/Ramya_Tanikanti)\ **Replies:** 5\ **Last updated:** [September 11, 2019, 4:10am UTC](https://discuss.elastic.co/t/unable-to-load-csv-file-into-elasticsearch/198826 "2019-09-11T04:10:53Z")
Hi, Logstash uploads the data into Kibana once and sometimes it doesn't even though I use the same same config file. I use the same config file to load the same data again after deleting the index it created in Kibana … --- ## [Logstash stopped processing because of an error: (SystemExit) exit](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error-systemexit-exit/195393)
**Author:** [@alexus](https://discuss.elastic.co/u/alexus)\ **Replies:** 5\ **Last updated:** [September 11, 2019, 4:07am UTC](https://discuss.elastic.co/t/logstash-stopped-processing-because-of-an-error-systemexit-exit/195393 "2019-09-11T04:07:10Z")
Hello World! I'm Running Logstash on Docker | Logstash Reference \[6.8\] | Elastic and getting following error: logstash | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.propertie… --- ## [Logstash time formate grok filter required for 5/16/2019 3:04:16 AM](https://discuss.elastic.co/t/logstash-time-formate-grok-filter-required-for-5-16-2019-316-am/198998)
**Author:** [@uday\_kumar1](https://discuss.elastic.co/u/uday_kumar1)\ **Replies:** 0\ **Last updated:** [September 11, 2019, 4:04am UTC](https://discuss.elastic.co/t/logstash-time-formate-grok-filter-required-for-5-16-2019-316-am/198998 "2019-09-11T04:04:38Z")
Hello there, In my log date time format example "5/16/2019 3:04:16 AM". I am looking for grok pattern for this timestamp can some one help me to get it. and also I tried to separate in date and time which is taking st… --- ## [Books on Logstash](https://discuss.elastic.co/t/books-on-logstash/198986)
**Author:** [@SteveP](https://discuss.elastic.co/u/SteveP)\ **Replies:** 1\ **Last updated:** [September 11, 2019, 4:04am UTC](https://discuss.elastic.co/t/books-on-logstash/198986 "2019-09-11T04:04:37Z")
Hello all, Is there any good books or articles that explain Logstash and how I can get up and running with it? Thanks, Steve --- ## [Logstash Setup with GeoIP](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231)
**Author:** [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\ **Replies:** 17\ **Last updated:** [September 11, 2019, 2:58am UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231 "2019-09-11T02:58:02Z")
I am attempting to setup geoip on Logstash 7.3.0. I have seen an old post on how to set it up, but I want to make sure I'm doing it right. In the logstash config file I add the below code. Is this correct? filter { g… --- ## [New to Logstash Filter Config file questions](https://discuss.elastic.co/t/new-to-logstash-filter-config-file-questions/198969)
**Author:** [@fastxl](https://discuss.elastic.co/u/fastxl)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 9:20pm UTC](https://discuss.elastic.co/t/new-to-logstash-filter-config-file-questions/198969 "2019-09-10T21:20:09Z")
I am new to ELK and trying to learn. I have setup a "Lab" that has my pfSense firewall sending logs to Logstash. The "how-tos" I followed for this had me create 4 different files, 01-inputs.conf, 10-syslog.conf, 11-pfsen… --- ## [Jdbc for sqlserver](https://discuss.elastic.co/t/jdbc-for-sqlserver/198058)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 8:24pm UTC](https://discuss.elastic.co/t/jdbc-for-sqlserver/198058 "2019-09-10T20:24:17Z")
I am trying to connect to microsoft sqlserver but no luck I can connect to this db using microsoft sqlserver management studio. What am I missing here? input { jdbc { jdbc\_validate\_connection =\> … --- ## [Geo location coordinates in logstash configuration](https://discuss.elastic.co/t/geo-location-coordinates-in-logstash-configuration/198958)
**Author:** [@ashok9177](https://discuss.elastic.co/u/ashok9177)\ **Replies:** 0\ **Last updated:** [September 10, 2019, 6:45pm UTC](https://discuss.elastic.co/t/geo-location-coordinates-in-logstash-configuration/198958 "2019-09-10T18:45:07Z")
Hi all, Below is the my sample log message Geo cordinates are present in log message, can some one please help me how to configure in logstash configuration 2019-09-10 13:47:45.678 \[INFO\] from application in pool-1-th… --- ## [Javapipeline unrecoverable error - Logstash](https://discuss.elastic.co/t/javapipeline-unrecoverable-error-logstash/198957)
**Author:** [@Matthew\_Graville](https://discuss.elastic.co/u/Matthew_Graville)\ **Replies:** 2\ **Last updated:** [September 10, 2019, 7:31pm UTC](https://discuss.elastic.co/t/javapipeline-unrecoverable-error-logstash/198957 "2019-09-10T19:31:18Z")
I am unsure what I am doing wrong. Any help is much appreciated: Logstash cmd: \[root@logstash bin\]# ./logstash -f /etc/logstash/conf.d/rsyslog.conf --config.reload.automatic Logstash Error1: \[ERROR\] 2019-09-10 13:48:… --- ## [Logstash Servers are runs with high CPU consumption](https://discuss.elastic.co/t/logstash-servers-are-runs-with-high-cpu-consumption/198952)
**Author:** [@vsubrama](https://discuss.elastic.co/u/vsubrama)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 7:18pm UTC](https://discuss.elastic.co/t/logstash-servers-are-runs-with-high-cpu-consumption/198952 "2019-09-10T19:18:30Z")
We are using 6.7.1 of Elastic stack. Our Logstash is running at high CPU (90+%) always. What do need need to look for to understand the high CPU consumption. We are using AWS machine M5.large for Logstash. Here is one… --- ## [CIDR filter reports IP6 addresses as invalid](https://discuss.elastic.co/t/cidr-filter-reports-ip6-addresses-as-invalid/198959)
**Author:** [@daniel.dayley](https://discuss.elastic.co/u/daniel.dayley)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 7:15pm UTC](https://discuss.elastic.co/t/cidr-filter-reports-ip6-addresses-as-invalid/198959 "2019-09-10T19:15:56Z")
Hey guys, I'm hoping this is an issue on my end rather than an issue with the CIDR filter, but I'm seeing a lot of logstash errors in my CIDR filter relating to certain IP addresses: \[WARN \]\[logstash.filters.cidr \] I… --- ## [How to keep one exec plugin waiting for another?](https://discuss.elastic.co/t/how-to-keep-one-exec-plugin-waiting-for-another/198960)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 2\ **Last updated:** [September 10, 2019, 7:09pm UTC](https://discuss.elastic.co/t/how-to-keep-one-exec-plugin-waiting-for-another/198960 "2019-09-10T19:09:59Z")
Given the following configuration: input { exec { command =\> "python3 ${LOGSTASH\_HOME}/script/firmware/backend/download\_script.py" schedule =\> "${SCHEDULER}" type =\> "Backend" } exec { command =\> "… --- ## [Recommendations for ingesting XML stream](https://discuss.elastic.co/t/recommendations-for-ingesting-xml-stream/198807)
**Author:** [@dpr](https://discuss.elastic.co/u/dpr)\ **Replies:** 3\ **Last updated:** [September 10, 2019, 6:51pm UTC](https://discuss.elastic.co/t/recommendations-for-ingesting-xml-stream/198807 "2019-09-10T18:51:12Z")
Hey guys, I'd like to ingest events that are continually streaming on a TCP port. The stream format is XML, although it's not one event per line: the opening tag and closing tag are often on different lines. Here's a … --- ## [S3 Input plugin overwrites the files written to directory](https://discuss.elastic.co/t/s3-input-plugin-overwrites-the-files-written-to-directory/197209)
**Author:** [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 6:37pm UTC](https://discuss.elastic.co/t/s3-input-plugin-overwrites-the-files-written-to-directory/197209 "2019-09-10T18:37:46Z")
When I pull data from S3, the tar file gets overwritten every time. Is there any way to save each file without having them be overwritten in S3 temporary directory? Logstash configuration: input{ s3 { acces… --- ## [Simple config does not work: file input](https://discuss.elastic.co/t/simple-config-does-not-work-file-input/198941)
**Author:** [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Replies:** 4\ **Last updated:** [September 10, 2019, 5:37pm UTC](https://discuss.elastic.co/t/simple-config-does-not-work-file-input/198941 "2019-09-10T17:37:57Z")
I apologize for a stupid question, but I got stuck: the following simple config does not parse the input log-file (LS v.7.3.1): input { file { path =\> \["C:\\temp\\solr.log.3"\] #start\_position =\> "beginning" sinced… --- ## [Logstash: access name of current pipeline within cfg?](https://discuss.elastic.co/t/logstash-access-name-of-current-pipeline-within-cfg/195786)
**Author:** [@barnetta](https://discuss.elastic.co/u/barnetta)\ **Replies:** 3\ **Last updated:** [September 10, 2019, 4:39pm UTC](https://discuss.elastic.co/t/logstash-access-name-of-current-pipeline-within-cfg/195786 "2019-09-10T16:39:37Z")
Is it possible to access the name of the current pipeline within cfg? For example, my pipelines.yml might contain: - pipeline.id: pipeline.triage path.config: "/etc/logstash/pipelines/pipeline.triage.conf" - pipeline… --- ## [User Agent Filter Not Working](https://discuss.elastic.co/t/user-agent-filter-not-working/198795)
**Author:** [@wwalker](https://discuss.elastic.co/u/wwalker)\ **Replies:** 4\ **Last updated:** [September 10, 2019, 2:44pm UTC](https://discuss.elastic.co/t/user-agent-filter-not-working/198795 "2019-09-10T14:44:46Z")
I'm not sure what happend but in my migration from 7.0 to 7.2, the user agent filter has stopped identifying properly. For example, given the following user agent mozilla/5.0 (windows nt 10.0; win64; x64; rv:69.0) gecko… --- ## [Help With Grok Pattern - Mail Log](https://discuss.elastic.co/t/help-with-grok-pattern-mail-log/198888)
**Author:** [@gerard.ramos](https://discuss.elastic.co/u/gerard.ramos)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 1:29pm UTC](https://discuss.elastic.co/t/help-with-grok-pattern-mail-log/198888 "2019-09-10T13:29:05Z")
Anyone can help with this grok pattern?? grok { match =\> \[ "message", "^%{DAY:dia} %{DATESTAMP\_EVENTLOG:data} %{NOTSPACE:email\_addresss}"\] } Example of the log file: Sat 2019-09-07 08:04:37.283: \* From: based… --- ## [Unit test logstash configuration with Java](https://discuss.elastic.co/t/unit-test-logstash-configuration-with-java/198897)
**Author:** [@tomicarsk6](https://discuss.elastic.co/u/tomicarsk6)\ **Replies:** 0\ **Last updated:** [September 10, 2019, 12:37pm UTC](https://discuss.elastic.co/t/unit-test-logstash-configuration-with-java/198897 "2019-09-10T12:37:42Z")
Does anybody have an idea, or maybe example, how I can write unit tests in Java for logstash configuration? --- ## [String filter based on a file?](https://discuss.elastic.co/t/string-filter-based-on-a-file/198882)
**Author:** [@KyFly-ESIEA](https://discuss.elastic.co/u/KyFly-ESIEA)\ **Replies:** 1\ **Last updated:** [September 10, 2019, 12:08pm UTC](https://discuss.elastic.co/t/string-filter-based-on-a-file/198882 "2019-09-10T12:08:59Z")
HI there, I'm really new ELK stack. I would like to know if it was possible to base a filter on a list, contained in a file ? Basically, instead of : if "string1" in \[log\] or "string2" in \[log\] or "string3" in \[log\] o… --- ## [Error: Could not find or load main class version](https://discuss.elastic.co/t/error-could-not-find-or-load-main-class-version/197489)
**Author:** [@pyerunka](https://discuss.elastic.co/u/pyerunka)\ **Replies:** 2\ **Last updated:** [September 10, 2019, 11:24am UTC](https://discuss.elastic.co/t/error-could-not-find-or-load-main-class-version/197489 "2019-09-10T11:24:58Z")
Hello, Recently installed latest version (7.3.1) of ES stack (Kibana, logstash and elastic search). And after that when i am running my logstash config file which is using jdbc input plugin to fetch data from database, … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=478) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=480)