# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=438 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 439 --- ## [Collecting data from instagramm using logstash](https://discuss.elastic.co/t/collecting-data-from-instagramm-using-logstash/210021)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 0\ **Last updated:** [November 30, 2019, 8:27am UTC](https://discuss.elastic.co/t/collecting-data-from-instagramm-using-logstash/210021 "2019-11-30T08:27:25Z")
hi, i san in stackoverflow that ti is possible to perfome data collecting from instagram, using this logstash conf: Instagram has an API so you can definitely attempt something with the http\_poller plugin. Something … --- ## [Parsing URL with Logstash (using ECS fields) nested!](https://discuss.elastic.co/t/parsing-url-with-logstash-using-ecs-fields-nested/209953)
**Author:** [@Vincent\_Maury](https://discuss.elastic.co/u/Vincent_Maury)\ **Replies:** 3\ **Last updated:** [November 29, 2019, 8:13pm UTC](https://discuss.elastic.co/t/parsing-url-with-logstash-using-ecs-fields-nested/209953 "2019-11-29T20:13:36Z")
Following https://discuss.elastic.co/t/parsing-url-with-logstash-using-ecs-fields/205814 Several improvements like nested fields & tld parsing # Parser a url field named url (according to rsa meta field name) filter { … --- ## [Float values are imported incorrectly from mySQL DB](https://discuss.elastic.co/t/float-values-are-imported-incorrectly-from-mysql-db/209970)
**Author:** [@vschaefer](https://discuss.elastic.co/u/vschaefer)\ **Replies:** 8\ **Last updated:** [November 29, 2019, 5:21pm UTC](https://discuss.elastic.co/t/float-values-are-imported-incorrectly-from-mysql-db/209970 "2019-11-29T17:21:11Z")
I import data from a mySQL DB via Logstash into an Elasticsearch index. The database contains float values in a field. However, these values change during import. 3378.2 from the mySQL DB becomes 3378.199951171875 in … --- ## [Elastic Common Schema for website domain](https://discuss.elastic.co/t/elastic-common-schema-for-website-domain/199383)
**Author:** [@gerard1](https://discuss.elastic.co/u/gerard1)\ **Replies:** 1\ **Last updated:** [November 29, 2019, 3:37pm UTC](https://discuss.elastic.co/t/elastic-common-schema-for-website-domain/199383 "2019-11-29T15:37:38Z")
What would be the ideal field name following the Elastic Common Schema for the domain name of a website? For example, in a setup using nginx + apache, with multiple domains, I end up with different logs per domain, and … --- ## [Run logstash as a service](https://discuss.elastic.co/t/run-logstash-as-a-service/209990)
**Author:** [@Anuwa\_A](https://discuss.elastic.co/u/Anuwa_A)\ **Replies:** 0\ **Last updated:** [November 29, 2019, 4:48pm UTC](https://discuss.elastic.co/t/run-logstash-as-a-service/209990 "2019-11-29T16:48:57Z")
Currently i'm executing logstash using command bin/logstash -f logstash-simple.conf i want to run it as a service (back ground process). how can i do that --- ## [Move E.C.S. extractions / ingest pipeline to Logstash Pipeline](https://discuss.elastic.co/t/move-e-c-s-extractions-ingest-pipeline-to-logstash-pipeline/202755)
**Author:** [@CamTheMan](https://discuss.elastic.co/u/CamTheMan)\ **Replies:** 2\ **Last updated:** [November 29, 2019, 3:51pm UTC](https://discuss.elastic.co/t/move-e-c-s-extractions-ingest-pipeline-to-logstash-pipeline/202755 "2019-11-29T15:51:12Z")
Hello, We are looking at moving from 6.4.2 to 7.x amongst the changes would be moving from our current data model to the Elastic Common Schema (ECS). I understand that the Elastic Ingest Nodes have these extractions 'b… --- ## [How do i convert my existing logstash grok to match with Elastic Common Schema ECS data type](https://discuss.elastic.co/t/how-do-i-convert-my-existing-logstash-grok-to-match-with-elastic-common-schema-ecs-data-type/192171)
**Author:** [@Shiv18](https://discuss.elastic.co/u/Shiv18)\ **Replies:** 2\ **Last updated:** [November 29, 2019, 3:47pm UTC](https://discuss.elastic.co/t/how-do-i-convert-my-existing-logstash-grok-to-match-with-elastic-common-schema-ecs-data-type/192171 "2019-11-29T15:47:26Z")
Hi Team, Im trying to change the existing grok pattern for several log formats into standard ECS format. For example, ecs url schema says for defining URL, i can use url.full and the type should be 'keyword'. When i … --- ## [Elastic Common Schema and Logstash and Ingest node processing tagging](https://discuss.elastic.co/t/elastic-common-schema-and-logstash-and-ingest-node-processing-tagging/194317)
**Author:** [@tarp](https://discuss.elastic.co/u/tarp)\ **Replies:** 2\ **Last updated:** [November 29, 2019, 3:45pm UTC](https://discuss.elastic.co/t/elastic-common-schema-and-logstash-and-ingest-node-processing-tagging/194317 "2019-11-29T15:45:41Z")
Hi, I'm working on setting up my custom sources to be more in line with ECS. In my filebeat configurations I had been adding the fields environment (ex devl,qual,cert,prod) building(ex buildingA) location(ex thirdfl… --- ## [Grok patterns to Elastic Common Schema](https://discuss.elastic.co/t/grok-patterns-to-elastic-common-schema/199391)
**Author:** [@gerard1](https://discuss.elastic.co/u/gerard1)\ **Replies:** 1\ **Last updated:** [November 29, 2019, 3:41pm UTC](https://discuss.elastic.co/t/grok-patterns-to-elastic-common-schema/199391 "2019-11-29T15:41:57Z")
I'm trying to migrate my Logstash filters into Elastic Common Schema, and I'm wondering if there is any conversion for common patters, like %{SYSLOGBASE}. --- ## [Logstash from winlogbeat to file not sorting results properly](https://discuss.elastic.co/t/logstash-from-winlogbeat-to-file-not-sorting-results-properly/209307)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 2\ **Last updated:** [November 29, 2019, 2:33pm UTC](https://discuss.elastic.co/t/logstash-from-winlogbeat-to-file-not-sorting-results-properly/209307 "2019-11-29T14:33:29Z")
Hello, I'm extracting information from my winlogbeat index and sorting them by @timestamp. Unfortunately it seems like output file results are not properly sorted by that field. Any idea why? Note: I've run the pipelin… --- ## [Logstash behaivour when elasticsearch fail to index](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904)
**Author:** [@mtudisco](https://discuss.elastic.co/u/mtudisco)\ **Replies:** 5\ **Last updated:** [November 29, 2019, 2:29pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904 "2019-11-29T14:29:43Z")
Hi, I'm running a test environment with one node only with elasticsearch and generating indexes with the patern prefix-\[type\]-yyyy.mm.dd. Information is generated in logfiles, that are sent with filebeat to logstash to… --- ## [File output - Logstash is unable to create directory and write a file due to permission denied error](https://discuss.elastic.co/t/file-output-logstash-is-unable-to-create-directory-and-write-a-file-due-to-permission-denied-error/209638)
**Author:** [@saif3r](https://discuss.elastic.co/u/saif3r)\ **Replies:** 2\ **Last updated:** [November 29, 2019, 2:28pm UTC](https://discuss.elastic.co/t/file-output-logstash-is-unable-to-create-directory-and-write-a-file-due-to-permission-denied-error/209638 "2019-11-29T14:28:45Z")
Hi, I created a pipeline that outputs the content into home folder of one of the users. When i run the pipeline manually as root, there is no issue, but then i try to do the same using pipelines.yml and logstash service… --- ## [Conditionnal filter not recognize field](https://discuss.elastic.co/t/conditionnal-filter-not-recognize-field/209974)
**Author:** [@toms130](https://discuss.elastic.co/u/toms130)\ **Replies:** 2\ **Last updated:** [November 29, 2019, 2:26pm UTC](https://discuss.elastic.co/t/conditionnal-filter-not-recognize-field/209974 "2019-11-29T14:26:51Z")
Hi all, I'm trying to use conditionnal filtering, with following config : filebeat.yml filebeat.inputs: - type: log enabled: true paths: - /var/log/pure-ftpd.log fields: log\_type: ftp\_logs output.logsta… --- ## [Have a multiple statements in a single input](https://discuss.elastic.co/t/have-a-multiple-statements-in-a-single-input/206182)
**Author:** [@Orochi](https://discuss.elastic.co/u/Orochi)\ **Replies:** 0\ **Last updated:** [November 1, 2019, 2:56pm UTC](https://discuss.elastic.co/t/have-a-multiple-statements-in-a-single-input/206182 "2019-11-01T14:56:04Z")
Hi guys, I'm having a issue here, I'd like to know if i can have a single input and multiples satements for the same input : input { jdbc\_driver\_library =\> ... jdbc\_driver\_class =\> ... jdbc\_connection\_string =\> .… --- ## [Scientific Numbers in Logstash - How to handle them? Need help](https://discuss.elastic.co/t/scientific-numbers-in-logstash-how-to-handle-them-need-help/209937)
**Author:** [@Datakids](https://discuss.elastic.co/u/Datakids)\ **Replies:** 0\ **Last updated:** [November 29, 2019, 8:12am UTC](https://discuss.elastic.co/t/scientific-numbers-in-logstash-how-to-handle-them-need-help/209937 "2019-11-29T08:12:27Z")
Hi there, I have found below mentions post which is a bit helpfull but I dont know how to split a Scientific Notation and merge them afterwards . A few examples of the numbers I want to handle or to learn how to handl… --- ## [Logstash Hang](https://discuss.elastic.co/t/logstash-hang/209934)
**Author:** [@eminonal](https://discuss.elastic.co/u/eminonal)\ **Replies:** 0\ **Last updated:** [November 29, 2019, 7:24am UTC](https://discuss.elastic.co/t/logstash-hang/209934 "2019-11-29T07:24:14Z")
I have RHEL Server 6.6 Santiago - Java 1.8.0\_25 / 64-Bit Server (build 25.25-b02) - Logstash 7.4.2 Logstash is running. Logs appear in Kibana. However, the logs do not appear after 20 to 25 minutes. When I check the se… --- ## [Using for loop](https://discuss.elastic.co/t/using-for-loop/209810)
**Author:** [@Vikash\_Singh1](https://discuss.elastic.co/u/Vikash_Singh1)\ **Replies:** 9\ **Last updated:** [November 29, 2019, 7:24am UTC](https://discuss.elastic.co/t/using-for-loop/209810 "2019-11-29T07:24:01Z")
Hey I've a file which includes ip addresses and there corresponding mac addresses. Now, is there any way I can use that file inside filter tag to match the ip address with my log file and allot them corresponding mac ad… --- ## [Can I get a date yesterday in logstash config?](https://discuss.elastic.co/t/can-i-get-a-date-yesterday-in-logstash-config/209927)
**Author:** [@deep2eye](https://discuss.elastic.co/u/deep2eye)\ **Replies:** 0\ **Last updated:** [November 29, 2019, 6:28am UTC](https://discuss.elastic.co/t/can-i-get-a-date-yesterday-in-logstash-config/209927 "2019-11-29T06:28:09Z")
Hello. Can I get a date yesterday in logstash config? How do I enter yesterday's date in the SQL statement? (???? mark) input { jdbc { jdbc\_driver\_class =\> "com.mysql.jdbc.Driver" schedule =\> "0 \*/1 \* \* \*" stateme… --- ## [Help about Kibana](https://discuss.elastic.co/t/help-about-kibana/209894)
**Author:** [@mary2019](https://discuss.elastic.co/u/mary2019)\ **Replies:** 7\ **Last updated:** [November 29, 2019, 3:35am UTC](https://discuss.elastic.co/t/help-about-kibana/209894 "2019-11-29T03:35:58Z")
Hi Professionals, I just took a course about ELK, right now I was asked to analyze log data (source\_ip, destination\_ip and messages information in the log document) the question is that Which outside IP is coming into … --- ## [Collect published data from my online websites](https://discuss.elastic.co/t/collect-published-data-from-my-online-websites/209906)
**Author:** [@kumar4](https://discuss.elastic.co/u/kumar4)\ **Replies:** 2\ **Last updated:** [November 28, 2019, 10:46pm UTC](https://discuss.elastic.co/t/collect-published-data-from-my-online-websites/209906 "2019-11-28T22:46:04Z")
hi everyone, i published my 2 web site (one for wiki and other for e-commerce with product and description) and now i want to collect data that contains some words such as 'games' and 'win' that is published on my websit… --- ## [Cant get rid of error "elasticsearch - retrying failed action with response code: 403"](https://discuss.elastic.co/t/cant-get-rid-of-error-elasticsearch-retrying-failed-action-with-response-code-403/209910)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 1\ **Last updated:** [November 28, 2019, 10:12pm UTC](https://discuss.elastic.co/t/cant-get-rid-of-error-elasticsearch-retrying-failed-action-with-response-code-403/209910 "2019-11-28T22:12:58Z")
Hi, Im triyng to get rid of the error thats appears when you have low space in the disk, I have deleted some files and now I have plenty of space, now I have to try to unlock the indexes, I have tried this two commands w… --- ## [Logstash filtering on filebeat's successfully multilined event](https://discuss.elastic.co/t/logstash-filtering-on-filebeats-successfully-multilined-event/209110)
**Author:** [@nnet](https://discuss.elastic.co/u/nnet)\ **Replies:** 3\ **Last updated:** [November 28, 2019, 3:34pm UTC](https://discuss.elastic.co/t/logstash-filtering-on-filebeats-successfully-multilined-event/209110 "2019-11-28T15:34:48Z")
I have correctly configured filebeat to deal with a multiline event generated by ISC's kea software. I want to parse that event into various fields, but none of my grok statements are matching. I think it may have to do… --- ## [Old files ignored? or indexed again with new Ids?](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868)
**Author:** [@Incauto](https://discuss.elastic.co/u/Incauto)\ **Replies:** 5\ **Last updated:** [November 28, 2019, 3:24pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868 "2019-11-28T15:24:37Z")
I have a folder with multiple files with a name date format: "foo\_2019\_11\_23" I have already created indexes with those files with the folowing command bin/logstash -f /etc/logstash/conf.d/foo.conf now there are new f… --- ## [Will Logstash be a part of elastic cloud?](https://discuss.elastic.co/t/will-logstash-be-a-part-of-elastic-cloud/209870)
**Author:** [@mviegas](https://discuss.elastic.co/u/mviegas)\ **Replies:** 0\ **Last updated:** [November 28, 2019, 2:10pm UTC](https://discuss.elastic.co/t/will-logstash-be-a-part-of-elastic-cloud/209870 "2019-11-28T14:10:26Z")
I'm on some research into migrating an on-premises ELK server to the cloud, alongside with our application. However, as far as my research went, there is no logstash provided on the elastic cloud yet, right? The topic be… --- ## [Logstash Central management datas](https://discuss.elastic.co/t/logstash-central-management-datas/209717)
**Author:** [@coute](https://discuss.elastic.co/u/coute)\ **Replies:** 1\ **Last updated:** [November 28, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-central-management-datas/209717 "2019-11-28T14:05:47Z")
Hello, Could you please tell me the name of the elasticsearch index or kibana object where datas from pipeline central management are stored (pipelines configuration) Thanks. --- ## [How to use if condition in logstash](https://discuss.elastic.co/t/how-to-use-if-condition-in-logstash/209794)
**Author:** [@Sagar\_Mandal](https://discuss.elastic.co/u/Sagar_Mandal)\ **Replies:** 1\ **Last updated:** [November 28, 2019, 1:51pm UTC](https://discuss.elastic.co/t/how-to-use-if-condition-in-logstash/209794 "2019-11-28T13:51:29Z")
Hi, I want to use if statement in my logstasg config file to get some particular data from a file. but i don't know how exactly we use if statements in its config file. any one can please share me the syntax here. Th… --- ## [Indices management from multiple logstash instances](https://discuss.elastic.co/t/indices-management-from-multiple-logstash-instances/209833)
**Author:** [@manfred](https://discuss.elastic.co/u/manfred)\ **Replies:** 0\ **Last updated:** [November 28, 2019, 10:40am UTC](https://discuss.elastic.co/t/indices-management-from-multiple-logstash-instances/209833 "2019-11-28T10:40:00Z")
Hi, I'm testing ELK currently. I have the following architecture: Logstash has three instances running in three different data centers logx01 logx02 logx03 Elasticsearch cluster with 4 nodes: es01 / data node, hot … --- ## [Loop through array and create new field values](https://discuss.elastic.co/t/loop-through-array-and-create-new-field-values/209826)
**Author:** [@karlm](https://discuss.elastic.co/u/karlm)\ **Replies:** 1\ **Last updated:** [November 28, 2019, 12:52pm UTC](https://discuss.elastic.co/t/loop-through-array-and-create-new-field-values/209826 "2019-11-28T12:52:41Z")
I have an xml filter creating two arrays from an xml sql column, there could be any amount of values in the arrays. xml { source =\> "event\_xml" remove\_namespaces =\> true store\_xml =\> fa… --- ## [Ruby exception occurred：Missing Valuefier handling](https://discuss.elastic.co/t/ruby-exception-occurred-missing-valuefier-handling/209835)
**Author:** [@wzy\_ricky](https://discuss.elastic.co/u/wzy_ricky)\ **Replies:** 0\ **Last updated:** [November 28, 2019, 10:41am UTC](https://discuss.elastic.co/t/ruby-exception-occurred-missing-valuefier-handling/209835 "2019-11-28T10:41:52Z")
ruby { code =\> "event.set('timestamp', event.get('@timestamp').time.localtime + 7\*60\*60)" } \[2019-11-28T18:23:24,424\]\[ERROR\]\[logstash.filters.ruby \] Ruby exception occurred: Missing Valuefier handling for full c… --- ## [Unable to re-read a kafka topic (after resetting the offset)](https://discuss.elastic.co/t/unable-to-re-read-a-kafka-topic-after-resetting-the-offset/209548)
**Author:** [@brunor](https://discuss.elastic.co/u/brunor)\ **Replies:** 1\ **Last updated:** [November 28, 2019, 10:28am UTC](https://discuss.elastic.co/t/unable-to-re-read-a-kafka-topic-after-resetting-the-offset/209548 "2019-11-28T10:28:15Z")
Hello everyone and thanks in advance, I have been using the kafka plugin for some time. It works great reading messages from two topics I have around, but now I added a third one (alarms) and it never had the messages t… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=437) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=439)