JSON file input to Filebeat sending data to Logstash and then to Azure Data Explorer

[omkarg81 (Omkar)]

I'm trying to send a single line JSON input file to Filebeat which should read and send in the data over to Logstash which should then forward or dump it to Azure Data Explorer cluster-table.

Somehow, when the Filebeat service is set to run and .json file is put in the input directory, nothing happens and no output is seen in the logstash VM as well. Also, no data gets dumped to the ADX table.

Any help would be greatly appreciated, as I'm stuck after having tried out more than 100 odd times, with different filebeat/logstash configs.. For another file type (xml) I could successfully read the file though, so definitely filebeat-logstash are working.

filebeat.yml

### JSON configuration

json.message_key: "ApplicationName"

json.keys_under_root: true

json.add_error_key: true

### Multiline options
No multiline option enabled

output.logstash:
hosts: ["Logstash Server IP:5044"] 

JSON Input file

{"ApplicationName": "xxxAPI","ApplicationVersion": "1.0.0","CorrelationId": "0e2a9ae5-57d8- 
4868-80a2-43523a59bf6d","Data": "SomeException","Severity": "Error","TenantID": 
"DevEnvironment","Time": "2021-02-17T18:12:37"}

Logstash yml config

input {
        beats {
             port => 5044
        }
   }
   output {
           #file { path => "C:\tmp\fileout\logstash.log" }
            kusto {
                          path => "/tmp/kusto/%{+YYYY-MM-dd-HH-mm-ss}.txt"
                          ingest_url => "https://ingest-xxx.kusto.windows.net/"
                          app_id => "application id guid value"
                          app_key => "secret"
                          app_tenant => "tenant id guid"
                          database => "logstashdb"
                          table => "jsonfilelogs"
                          json_mapping => "jsonfilelogsmapping"
             }
   }

ADX table structure-mapping

.create table jsonfilelogs (ApplicationName: string, ApplicationVersion: string, CorrelationId: guid,
Data: string, Host: string, HostPrefix: string, Message: string, Severity: string, 
TenantId: string, Timestamp: timespan)

.create table jsonfilelogs ingestion json mapping 'jsonfilelogsmapping' '[{"column":"ApplicationNa 
me","Properties":{"path":"$.ApplicationName"}}, {"column":"ApplicationVersion","Properties"{"path":"$.ApplicationVersion"}}, {"column":"CorrelationId","Properties"{"path":"$.CorrelationId"}}, {"column":"Data","Properties"{"path":"$.Data"}}, {"column":"Host","Properties"{"path":"$.Host"}}, {"column":"HostPrefix","Properties"{"path":"$.HostPrefix"}}, {"column":"Message","Properties"{"path":"$.Message"}}, {"column":"Severity","Properties"{"path":"$.Severity"}}, {"column":"TenantId","Properties"{"path":"$.TenantId"}}, {"column":"Timestamp","Properties"{"path":"$.Timestamp"}}]' with '{"format":"multijson", "ingestionMappingReference":"FlatEventMapping"}'

[jsoriano (Jaime Soriano)]

Hey @omkarg81, welcome to discuss

Has your JSON file a new line at the end? Filebeat uses the new line as delimiter of each line, it may be waiting for this new line before sending this line.

[omkarg81 (Omkar)]

Nope, it doesn't. But eventually I was able to feed in a simpler format in single line json, but the problem is when the data shows up in ADX, only empty records (6-7 in number) are shown even though there is just 1 record in input file. Also, using the mapping expression I have set the ADX to read in like {"column":"ApplicationName","Properties":{"path":"$.ApplicationName"}}, but apparently none of the fields are being read in. Still stuck up, reading every property of data from input file to be pushed to individual columns of the ADX table.

[jsoriano (Jaime Soriano)]

To discard issues with logstash or ADX, could you try to configure filebeat with the console or file outputs and see how the events it is collecting look like?

[omkarg81 (Omkar)]

Could you please provide an example of how to do that as I already have logging enabled on filebeat yml config, buy it isn't showing any errors as such.

Also the logstash has been set to read from beats input and output to file which then shows only metadata type (host, beats etc.) json structure being created in the logstash log file. No actual json is printed in logstash output.

Please suggest

[jsoriano (Jaime Soriano)]

You can see events published in the debug log if you add the -d publish flag to filebeat.

You can also configure the console or file outputs, instead of the logstash one, to see the events that are being published.

Configuring filebeat to log to console is as simple as disabling other outputs in its configuration, and adding this:

output.console.pretty: true

Then you will see the events in the filebeat output.

Similarly, to send the events to file, add a configuration like this one (change the paths if you are on Windows):

output.file:
  path: "/tmp/filebeat"
  filename: filebeat

This can help to see what filebeat is collecting and sending.

[omkarg81 (Omkar)]

Thanks a lot Jaime. I was able to separate out the outputs from filebeat alone as against the logstash one and that told me where the issue is in the input file. I'm able to successfully input the JSON input file and push all the data into Azure Data Explorer.