# Logstash is not parsing the data base on config
**URL:** https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930
**Category:** Logstash
**Created:** [June 14, 2018, 1:29pm UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930 "2018-06-14T13:29:10Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Pradana](https://discuss.elastic.co/u/Pradana)
#### Post date: [June 14, 2018, 1:29pm UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/1 "2018-06-14T13:29:10Z")
I have below config :
filter{
if [type] == "log" {
grok {
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:logdate} %{LOGLEVEL:debugtype} %{DATA:source} %{TIMESTAMP\_ISO8601:smsdate},%{WORD:sourceaddr},%{NUMBER:addrton},%{NUMBER:addrnpi},%{WORD:destaddr},%{NUMBER:sourceton},%{NUMBER:sourc
enpi},%{WORD:status}" }}
grok { match =\> ["sourceaddr", "^(?.....)"] }
translate {
field =\> "operator"
destination =\> "operator\_name"
dictionary =\> [
"62811", "Telkomsel",
"62812", "Telkomsel",
"62813", "Telkomsel",
"62821", "Telkomsel",
"62822", "Telkomsel",
"62823", "Telkomsel",
"62851", "Telkomsel",
"62852", "Telkomsel",
"62853", "Telkomsel",
"62814", "Indosat",
"62815", "Indosat",
"62816", "Indosat",
"62855", "Indosat",
"62856", "Indosat",
"62857", "Indosat",
"62858", "Indosat",
"62817", "XL",
"62818", "XL",
"62819", "XL",
"62859", "XL",
"62877", "XL",
"62878", "XL",
"62831", "XL",
"62832", "XL",
"62833", "XL",
"62838", "XL",
"62895", "Tri",
"62896", "Tri",
"62897", "Tri",
"62898", "Tri",
"62899", "Tri",
"62881", "Smartfren",
"62882", "Smartfren",
"62883", "Smartfren",
"62884", "Smartfren",
"62885", "Smartfren",
"62886", "Smartfren",
"62887", "Smartfren",
"62888", "Smartfren",
"62889", "Smartfren",
"62828", "Net1"
]
```
}
date {
match => ["smsdate", "dd/MMM/yyyy:HH:mm:ss Z"]
}
```
}}
It seems when sending the message into elasticsearch, message is not been parsing since I don't have all the field above. I only have the message field that contain everything. How to resolved this ?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 14, 2018, 1:59pm UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/2 "2018-06-14T13:59:34Z")
Show an example document that's stored in ES. You can copy/paste from Kibana's JSON tab.
---
### Author:  [@Pradana](https://discuss.elastic.co/u/Pradana)
#### Post date: [June 14, 2018, 2:23pm UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/3 "2018-06-14T14:23:44Z")
Hi,
Please find below.
{
"\_index": "filebeat-2018.06.14",
"\_type": "doc",
"\_id": "sAOt\_mMBRLh\_m9zZFTO1",
"\_version": 1,
"\_score": null,
"\_source": {
"message": "2018-06-14 10:22:24,870 DEBUG [org.mobicents.smsc.library.CdrGenerator] 2018-06-14 10:22:24.854,6287737178619,1,1,6282817000071,1,1,success\_esme,SS7\_HR,message,null,112878,0,null,null,null,null,62818445209,null,0,15,null,0,0,,,,7,"PAIF QUEwlhCIAEAAUW0","",,,",
"offset": 571042,
"source": "/opt/telestax/TelScale-smsc-jboss-7.5.1-95/jboss-5.1.0.GA/server/default/log/cdr.log",
"input": {
"type": "log"
},
"beat": {
"hostname": "localhost.localdomain",
"version": "6.3.0",
"name": "localhost.localdomain"
},
"@timestamp": "2018-06-14T14:22:25.194Z",
"prospector": {
"type": "log"
},
"host": {
"name": "localhost.localdomain"
},
"@version": "1",
"tags": [
"beats\_input\_codec\_plain\_applied"
]
},
"fields": {
"@timestamp": [
"2018-06-14T14:22:25.194Z"
]
},
"sort": [
1528986145194
]
}
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 14, 2018, 2:35pm UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/4 "2018-06-14T14:35:16Z")
I'm not sure your event are being processed by Logstash at all. What does your Filebeat configuration look like? Remove all commented lines and format the rest as preformatted text using Markdown notation or the `` toolbar button.
---
### Author:  [@Pradana](https://discuss.elastic.co/u/Pradana)
#### Post date: [June 14, 2018, 2:45pm UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/5 "2018-06-14T14:45:15Z")
Please find below :
filebeat.inputs:
- type: log
filebeat.config.modules:
path: ${path.config}/modules.d/\*.yml
reload.enabled: false
setup.template.settings:
index.number\_of\_shards: 3
setup.kibana:
output.logstash:
hosts: ["103.88.253.83:5044"]
logging.level: debug
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 14, 2018, 6:45pm UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/6 "2018-06-14T18:45:03Z")
You have
> if [type] == "log" {
in your config file but your event doesn't have a `type` field.
---
### Author:  [@Pradana](https://discuss.elastic.co/u/Pradana)
#### Post date: [June 15, 2018, 12:36am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/7 "2018-06-15T00:36:56Z")
Hi Magnus,
I agree with this. Temporary I have remove this "if" and all my field is coming. Previously, I have this in my previous version of elasticsearch where in filebeat I can configure document\_type but since this method has been remove, is there any method can be use ?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 15, 2018, 6:01am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/8 "2018-06-15T06:01:34Z")
You can set any fields you like via the `fields` option in the Filebeat configuration (you'll probably want to enable `fields_under_root` too).
---
### Author:  [@Pradana](https://discuss.elastic.co/u/Pradana)
#### Post date: [June 15, 2018, 6:30am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/9 "2018-06-15T06:30:25Z")
I have add a field "sourcelog" under the field. How I am going to use it in the logstash config ? is it like below :
if [fields.sourcelog] == "cdrlog" { ?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [June 15, 2018, 6:33am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/10 "2018-06-15T06:33:25Z")
Almost, see [https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references).
---
### Author:  [@Pradana](https://discuss.elastic.co/u/Pradana)
#### Post date: [June 15, 2018, 6:56am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/11 "2018-06-15T06:56:21Z")
According to the reference, I should use :
if [fields][sourcelog] == "cdrlog" { ?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 13, 2018, 6:56am UTC](https://discuss.elastic.co/t/logstash-is-not-parsing-the-data-base-on-config/135930/12 "2018-07-13T06:56:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.