Hi
I have two configurations, one with the input as stdin, and one using file path.
both inputs are JSON structured text files.
The filters are the same (using JSON encode, json and split), the output is the same.
When I use the stdin config and enter the json file (minified) everything works fine, but when I use the file input, logstash reads every line as text and not as JSON.
Why is there any difference?
Please show your configuration instead of describing it.
This is th one with stdin:
input {
stdin {}
}
filter {
split{
field => "changedFieldList"
}
json {
source => "message"
target => "parsedJSON"
}
}
output { stdout { codec => rubydebug }}
This is the one with the file load (via filebeat):
input {
beats {
port => 5044
}
// another config:
file{
path => path/to/file/*
}
}
filter{
split{
field => "changedFieldList"
}
json{
source => "message"
target => "parsedJson"
}
}
output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}Additional Info:
This is the result when I use stdin and insert the json file manualy:
"parsedJSON" => {
"DBSHDataChangeInfo" => "Archive Log Set archiveLogSet.108058603.108058399 Info:Thread# Sequence# FirstScn LastScn ",
"documentsList" => [
[0] {
"objectSchemaName" => "YISHAIN",
"commandSequence" => "0",
"commandType" => "UPDATE",
"commandCommitScn" => "108058600",
"changedFieldsList" => [
[ 0] {
"fieldType" => "NUMBER",
"fieldValue" => "17",
"fieldChanged" => "N",
"fieldId" => "CUSTOMER_ID"
},
[ 1] {
"fieldType" => "VARCHAR2",
"fieldValue" => "Daniel",
...
This is the result when the file is loaded using file:
{
"path" => "/path/to/folder",
"@timestamp" => 2017-11-26T12:27:00.684Z,
"parsedJSON" => nil,
"@version" => "1",
"host" => "qa8",
"message" => "",
"tags" => [
[0] "_split_type_failure"
]
}
{
"@version" => "1",
"host" => "qa8",
"path" => "path" => "/path/to/folder",
"@timestamp" => 2017-11-26T12:27:00.787Z,
"message" => " {",
"tags" => [
[0] "_split_type_failure",
[1] "_jsonparsefailure"
]
}
...
As you can see, the stdin method process the json file fine, while the file method get everything as text. Why is that happening? How can I make the file method work?
It looks like it's a multiline JSON file (i.e. the top-level JSON object it split over multiple lines). Are you really passing the exact same file via stdin?
yes I did
And what does the input file look like? I find it very hard to believe that the configuration you've shown us actually works as you describe.
{
"ChangeInfo": "Archive Log Set archiveLogSet.108058603.108058399 Info:Thread# Sequence# FirstScn LastScn ",
"documentsList": [{
"commandScn": "108058599",
"commandCommitScn": "108058600",
"commandSequence": "0",
"commandType": "UPDATE",
"commandTimestamp": "2017-08-22 14:37:53+03:000",
"objectDBName": "DEV2",
"objectSchemaName": "YISHAIN",
"objectId": "CUSTOMERS",
"changedFieldsList": [{
"fieldId": "CUSTOMER_ID",
"fieldType": "NUMBER",
"fieldValue": "17",
"fieldChanged": "N"
}, {
"fieldId": "CUSTOMER_FIRST_NAME",
"fieldType": "VARCHAR2",
"fieldValue": "Daniel",
"fieldChanged": "N"
}, {
"fieldId": "CUSTOMER_LAST_NAME",
"fieldType": "VARCHAR2",
"fieldValue": "Washington",
"fieldChanged": "N"
}, {
"fieldId": "CUSTOMER_COUNTRY",
"fieldType": "VARCHAR2",
"fieldValue": "France",
"fieldChanged": "N"
}, {
"fieldId": "CUSTOMER_CITY",
"fieldType": "VARCHAR2",
"fieldValue": "La Roche-sur-Yon",
"fieldChanged": "N"
}, {
"fieldId": "CUSTOMER_STREET",
"fieldType": "VARCHAR2",
"fieldValue": "43 Maple Pass",
"fieldChanged": "Y"
}, {
"fieldId": "CUSTOMER_ZIPCODE",
"fieldType": "VARCHAR2",
"fieldValue": "85928 CEDEX 9",
"fieldChanged": "N"
}, {
"fieldId": "PHONE_NUMBER",
"fieldType": "VARCHAR2",
"fieldValue": "33-(414)618-3273",
"fieldChanged": "N"
}, {
"fieldId": "EMAIL",
"fieldType": "VARCHAR2",
"fieldValue": "dwashingtong@tiny.cc",
"fieldChanged": "N"
}, {
"fieldId": "GENDER",
"fieldType": "VARCHAR2",
"fieldValue": "Male",
"fieldChanged": "N"
}, {
"fieldId": "DATE_OF_BIRTH",
"fieldType": "DATE",
"fieldValue": "1968-04-01 00:00:00+03:000",
"fieldChanged": "N"
}, {
"fieldId": "CREDIT_LIMIT",
"fieldType": "NUMBER",
"fieldValue": "14161",
"fieldChanged": "N"
}, {
"fieldId": "INCOME_LEVEL",
"fieldType": "NUMBER",
"fieldValue": "67035",
"fieldChanged": "N"
}, {
"fieldId": "COMMENTS",
"fieldType": "VARCHAR2",
"fieldValue": "stuff",
"fieldChanged": "N"
}, {
"fieldId": "LAST_UPDTAE",
"fieldType": "DATE",
"fieldValue": "2016-01-12 00:00:00+03:000",
"fieldChanged": "N"
}],
"conditionFieldsList": [{
"fieldId": "CUSTOMER_ID",
"fieldType": "NUMBER",
"fieldValue": "17"
}, {
"fieldId": "CUSTOMER_FIRST_NAME",
"fieldType": "VARCHAR2",
"fieldValue": "Daniel"
}, {
"fieldId": "CUSTOMER_LAST_NAME",
"fieldType": "VARCHAR2",
"fieldValue": "Washington"
}, {
"fieldId": "CUSTOMER_COUNTRY",
"fieldType": "VARCHAR2",
"fieldValue": "France"
}, {
"fieldId": "CUSTOMER_CITY",
"fieldType": "VARCHAR2",
"fieldValue": "La Roche-sur-Yon"
}, {
"fieldId": "CUSTOMER_STREET",
"fieldType": "VARCHAR2",
"fieldValue": "5 Maple Pass"
}, {
"fieldId": "CUSTOMER_ZIPCODE",
"fieldType": "VARCHAR2",
"fieldValue": "85928 CEDEX 9"
}, {
"fieldId": "PHONE_NUMBER",
"fieldType": "VARCHAR2",
"fieldValue": "33-(414)618-3273"
}, {
"fieldId": "EMAIL",
"fieldType": "VARCHAR2",
"fieldValue": "dwashingtong@tiny.cc"
}, {
"fieldId": "GENDER",
"fieldType": "VARCHAR2",
"fieldValue": "Male"
}, {
"fieldId": "DATE_OF_BIRTH",
"fieldType": "DATE",
"fieldValue": "1968-04-01 00:00:00+03:000"
}, {
"fieldId": "CREDIT_LIMIT",
"fieldType": "NUMBER",
"fieldValue": "14161"
}, {
"fieldId": "INCOME_LEVEL",
"fieldType": "NUMBER",
"fieldValue": "67035"
}, {
"fieldId": "COMMENTS",
"fieldType": "VARCHAR2",
"fieldValue": "stff"
}, {
"fieldId": "LAST_UPDTAE",
"fieldType": "DATE",
"fieldValue": "2016-01-12 00:00:00+03:000"
}]
}]
}There are three reasons why the configuration you posted won't work with either the stdin or the file inputs:
- The split filter is listed before the JSON filter so the input won't have been deserialized into fields that you can split.
- You're splitting the
changedFieldListfield but there is no such field. - The stdin input will, just like the file input, read its input line by line. If your JSON objects span over multiple lines you need to use a multiline codec.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.