Unable to have data in kibana from filebeat

Elastic Stack Beats
1

Hello,

Actually, I tried to use Filebeat to transmit my logs to elastcisearch to have data in kibana. But I receive no data in Kibana.

Here is my config file for Filebeat:

filebeat.yml:

filebeat.inputs:

  • type: log
    enabled: true
    paths:

    • C:\Borne\Log\Application\Detail\2021\1\26*.json
      index: cinemabis
      json.keys_under_root: true
      json.add_error_key: true
      json.message_key: log

  • type: filestream
    enabled: false
    paths:

    • C:\Program Files\Filebeat\logs*

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s

setup.template.settings:
index.number_of_shards: 1

setup.dashboards.enabled: true
setup.ilm.check_exists: false

setup.kibana:
host: "localhost:5601"
output.elasticsearch:
hosts: ["localhost:9200"]
username: "elastic"
password: "password"
pretty: true

processors:

  • add_host_metadata:
    when.not.contains.tags: forwarded
  • add_cloud_metadata: ~
  • add_docker_metadata: ~
  • add_kubernetes_metadata: ~
  • decode_json_fields:
    fields: ["field1", "field2", ...]
    process_array: false
    max_depth: 1
    target: ""
    overwrite_keys: false
    add_error_key: true

When I write in the console "filebeat -e -d "*"", here are responses about harvested log:

'2021-01-28T17:13:19.487+0100 INFO log/harvester.go:302 Harvester started for file: C:\Borne\Log\Application\Detail\2021\1\26\Application_20210122-151659.json
2021-01-28T17:13:19.488+0100 INFO [detect_null_bytes] debug/debug.go:95 Starting debug reader with a buffer size of 16384 and max failures of 100
2021-01-28T17:13:19.488+0100 DEBUG [harvester] log/log.go:107 End of file reached: C:\Borne\Log\Application\Detail\2021\1\26\Application_20210122-151659.json; Backoff now.'

Logs have this format:

'{
"CinemaId": "3392",
"Code": "VER001",
"Date": "2021-01-22T14:16:59Z",
"Dump": null,
"Error": null,
"Exception": {
"InnerException": null,
"Message": "Le port de la carte VERA n'a pas été trouvé.",
"StackTrace": " à LCPG.Bornes.Devices.Vera.VeraController..ctor(String portName, VeraTypes type) dans C:\OPProjects\Gaumont\VISTA\Main\Sources\LCPG.Bornes\LCPG.Bornes.Devices\Vera\VeraController.cs:ligne 168\r\n à LCPG.Bornes.Client.ViewModels.Devices.VeraDataModel..ctor(VeraTypes type) dans C:\OPProjects\Gaumont\VISTA\Main\Sources\LCPG.Bornes\LCPG.Bornes.Client\ViewModels\Devices\VeraDataModel.cs:ligne 272\r\n à LCPG.Bornes.Client.ViewModels.InitialisationViewModel.TaskVera(Object param) dans C:\OPProjects\Gaumont\VISTA\Main\Sources\LCPG.Bornes\LCPG.Bornes.Client\ViewModels\InitialisationViewModel.cs:ligne 1268"
},
"IP": "192.168.65.1",
"InnerStatement": null,
"Kiosk": "op-cedrick-j2",
"Message": null,
"Server": null,
"Version": "5.4.0.27500"
}'

In the log files, the json is only on one line, I post like that for better readibility.

In kibana, I created an index like this:

'"cinema" : {
"aliases" : { },
"mappings" : {
"properties" : {
"CinemaId" : {
"type" : "keyword"
},
"Code" : {
"type" : "text"
},
"Date" : {
"type" : "date"
},
"Dump" : {
"type" : "text"
},
"Exception" : {
"type" : "nested"
},
"IP" : {
"type" : "text"
},
"InnerStatement" : {
"type" : "text"
},
"Kiosk" : {
"type" : "text"
},
"Message" : {
"type" : "text"
},
"Server" : {
"type" : "text"
},
"Version" : {
"type" : "text"
}
}
}'

Elasticsearch is running on http://localhost:9200/
Kibana on http://localhost:5601

I wonder why I never received any data with any index included "cinema*".

Thanks for advance for your Help.

Regards

2

Hi @Jean-P, welcome to the Elastic community forums!

Would you mind editing your post and enclosing any code/config/log blocks in ``` delimiters so they appear property formatted? It makes it easy to read and also copy/paste correctly for trying to reproduce your situation.

Thanks,

Shaunak

4

Hello Shaunak,

Thanks for your reply, here are all informations well formated this time. I hope it is more readable.

Here is my config file for Filebeat:

filebeat.yml:

filebeat.inputs:

* type: log
enabled: true
paths:
  * C:\Borne\Log\Application\Detail\2021\1\26*.json
index: cinemabis
json.keys_under_root: true
json.add_error_key: true
json.message_key: log
* type: filestream
enabled: false
paths:
  * C:\Program Files\Filebeat\logs*

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s

setup.template.settings:
index.number_of_shards: 1

setup.dashboards.enabled: true
setup.ilm.check_exists: false

setup.kibana:
host: "localhost:5601"
output.elasticsearch:
hosts: ["localhost:9200"]
username: "elastic"
password: "password"
pretty: true

processors:

* add_host_metadata:
when.not.contains.tags: forwarded
* add_cloud_metadata: ~
* add_docker_metadata: ~
* add_kubernetes_metadata: ~
* decode_json_fields:
fields: ["field1", "field2", ...]
process_array: false
max_depth: 1
target: ""
overwrite_keys: false
add_error_key: true

When I write in the console "filebeat -e -d "*"", here are responses about harvested log:

'2021-01-28T17:13:19.487+0100 INFO log/harvester.go:302 Harvester started for file: C:\Borne\Log\Application\Detail\2021\1\26\Application_20210122-151659.json
2021-01-28T17:13:19.488+0100 INFO [detect_null_bytes] debug/debug.go:95 Starting debug reader with a buffer size of 16384 and max failures of 100
2021-01-28T17:13:19.488+0100 DEBUG [harvester] log/log.go:107 End of file reached: C:\Borne\Log\Application\Detail\2021\1\26\Application_20210122-151659.json; Backoff now.'

Logs have this format:

{
	"CinemaId": "3392",
	"Code": "VER001",
	"Date": "2021-01-22T14:16:59Z",
	"Dump": null,
	"Error": null,
	"Exception": {
		"InnerException": null,
		"Message": "Le port de la carte VERA n'a pas été trouvé.",
		"StackTrace": " à LCPG.Bornes.Devices.Vera.VeraController..ctor(String portName, VeraTypes type) dans C:\OPProjects\Gaumont\VISTA\Main\Sources\LCPG.Bornes\LCPG.Bornes.Devices\Vera\VeraController.cs:ligne 168\r\n à LCPG.Bornes.Client.ViewModels.Devices.VeraDataModel..ctor(VeraTypes type) dans C:\OPProjects\Gaumont\VISTA\Main\Sources\LCPG.Bornes\LCPG.Bornes.Client\ViewModels\Devices\VeraDataModel.cs:ligne 272\r\n à LCPG.Bornes.Client.ViewModels.InitialisationViewModel.TaskVera(Object param) dans C:\OPProjects\Gaumont\VISTA\Main\Sources\LCPG.Bornes\LCPG.Bornes.Client\ViewModels\InitialisationViewModel.cs:ligne 1268"
	},
	"IP": "192.168.65.1",
	"InnerStatement": null,
	"Kiosk": "op-cedrick-j2",
	"Message": null,
	"Server": null,
	"Version": "5.4.0.27500"
}

In the log files, the json is only on one line, I post like that for better readibility.

In kibana, I created an index like this:

"cinema": {
	"aliases": {},
	"mappings": {
		"properties": {
			"CinemaId": {
				"type": "keyword"
			},
			"Code": {
				"type": "text"
			},
			"Date": {
				"type": "date"
			},
			"Dump": {
				"type": "text"
			},
			"Exception": {
				"type": "nested"
			},
			"IP": {
				"type": "text"
			},
			"InnerStatement": {
				"type": "text"
			},
			"Kiosk": {
				"type": "text"
			},
			"Message": {
				"type": "text"
			},
			"Server": {
				"type": "text"
			},
			"Version": {
				"type": "text"
			}
		}
	}

Elasticsearch is running on http://localhost:9200/
Kibana on http://localhost:5601

I wonder why I never received any data with any index included "cinema*".

Thanks for advance for your Help.

Regards

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.