Json parsing problem

rusty_cole · December 29, 2021, 12:02pm

Hi Guys,
I have a problem with ingesting json data.
I am guessing that the problem is with the structure.
I have tried different configurations (with and without the json codec).
Any idea?
The json data:

[{"name": "software\\microsoft\\windows nt\\currentversion\\appcompatflags\\installedsdb", "hive": "HKEY_LOCAL_MACHINE", "exists": "False"}, {"name": "software\\microsoft\\windows nt\\currentversion\\appcompatflags\\custom", "hive": "HKEY_LOCAL_MACHINE", "exists": "True", "values": {}}]

[{"name": "software\\microsoft\\windows\\currentversion\\internet settings", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"DisableCachingOfSSLPages": "0", "IE5_UA_Backup_Flag": "5.0", "PrivacyAdvanced": "1", "SecureProtocols": "2688", "User Agent": "Mozilla/4.0 (compatible; MSIE 8.0; Win32)", "CertificateRevocation": "1", "ProxyEnable": "0", "AutoDetect": "0", "EnableAutodial": "0", "NoNetAutodial": "0", "ZonesSecurityUpgrade": "b'\\xda\\xc9\\xcdi\\x90\\xab\\xd6\\x01'", "EnableNegotiate": "1", "MigrateProxy": "1", "WarnonZoneCrossing": "0"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"ContentLimit": "330", "TotalContentLimit": "495", "AppContainerTotalContentLimit": "1000", "AppContainerContentLimit": "50", "Version": "4"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Content", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "", "CacheVersion": "1", "CacheLimit": "337920"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Cookies", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "Cookie:", "CacheVersion": "1", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\DNTException", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "DNTException:", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\INetCookies\\DNTException", "CacheRelativePath": "Microsoft\\Windows\\INetCookies\\DNTException", "CacheOptions": "768", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\DOMStore", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "DOMStore", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Internet Explorer\\DOMStore", "CacheRelativePath": "Microsoft\\Internet Explorer\\DOMStore", "CacheOptions": "8", "CacheRepair": "0", "CacheLimit": "1000"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\EmieSiteList", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "EmieSiteList:", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Internet Explorer\\EmieSiteList", "CacheRelativePath": "Microsoft\\Internet Explorer\\EmieSiteList", "CacheOptions": "768", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\EmieUserList", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "EmieUserList:", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Internet Explorer\\EmieUserList", "CacheRelativePath": "Microsoft\\Internet Explorer\\EmieUserList", "CacheOptions": "768", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\feedplat", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "feedplat:", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Feeds Cache", "CacheRelativePath": "Microsoft\\Feeds Cache", "CacheOptions": "0", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\iecompat", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "iecompat:", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\IECompatCache", "CacheRelativePath": "Microsoft\\Windows\\IECompatCache", "CacheOptions": "777", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\iecompatua", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "iecompatua:", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\IECompatUaCache", "CacheRelativePath": "Microsoft\\Windows\\IECompatUaCache", "CacheOptions": "777", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\iedownload", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "iedownload:", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\IEDownloadHistory", "CacheRelativePath": "Microsoft\\Windows\\IEDownloadHistory", "CacheOptions": "9", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\MSHist012021120620211213", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": ":2021120620211213: ", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\History\\History.IE5\\MSHist012021120620211213", "CacheRelativePath": "Microsoft\\Windows\\History\\History.IE5\\MSHist012021120620211213", "CacheOptions": "11", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\MSHist012021121320211220", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": ":2021121320211220: ", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\History\\History.IE5\\MSHist012021121320211220", "CacheRelativePath": "Microsoft\\Windows\\History\\History.IE5\\MSHist012021121320211220", "CacheOptions": "11", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\MSHist012021122020211227", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": ":2021122020211227: ", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\History\\History.IE5\\MSHist012021122020211227", "CacheRelativePath": "Microsoft\\Windows\\History\\History.IE5\\MSHist012021122020211227", "CacheOptions": "11", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\MSHist012021122720211228", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": ":2021122720211228: ", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\History\\History.IE5\\MSHist012021122720211228", "CacheRelativePath": "Microsoft\\Windows\\History\\History.IE5\\MSHist012021122720211228", "CacheOptions": "11", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\Extensible Cache\\MSHist012021122820211229", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": ":2021122820211229: ", "CachePath": "C:\\Users\\itadmin.VRNSLAB\\AppData\\Local\\Microsoft\\Windows\\History\\History.IE5\\MSHist012021122820211229", "CacheRelativePath": "Microsoft\\Windows\\History\\History.IE5\\MSHist012021122820211229", "CacheOptions": "11", "CacheRepair": "0", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\Cache\\History", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"CachePrefix": "Visited:", "CacheVersion": "1", "CacheLimit": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\LowCache", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\User Agent", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\5.0\\User Agent\\Post Platform", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Cache", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"Persistent": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Connections", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"DefaultConnectionSettings": "b'F\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'", "SavedLegacySettings": "b'F\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Http Filters", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Http Filters\\RPA", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Lockdown_Zones", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Lockdown_Zones\\0", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Computer", "PMDisplayName": "Computer [Protected Mode]", "Description": "Your computer", "Icon": "shell32.dll#0016", "LowIcon": "inetcpl.cpl#005422", "CurrentLevel": "0", "Flags": "33", "1200": "3", "1400": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Lockdown_Zones\\1", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Local intranet", "PMDisplayName": "Local intranet [Protected Mode]", "Description": "This zone contains all Web sites that are on your organization's intranet.", "Icon": "shell32.dll#0018", "LowIcon": "inetcpl.cpl#005423", "CurrentLevel": "0", "Flags": "219", "1200": "3", "1400": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Lockdown_Zones\\2", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Trusted sites", "PMDisplayName": "Trusted sites [Protected Mode]", "Description": "This zone contains Web sites that you trust not to damage your computer or data.", "Icon": "inetcpl.cpl#00004480", "LowIcon": "inetcpl.cpl#005424", "CurrentLevel": "0", "Flags": "33", "1200": "3", "1400": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Lockdown_Zones\\3", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Internet", "PMDisplayName": "Internet [Protected Mode]", "Description": "This zone contains all Web sites you haven't placed in other zones", "Icon": "inetcpl.cpl#001313", "LowIcon": "inetcpl.cpl#005425", "CurrentLevel": "0", "Flags": "33", "1200": "3", "1400": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Lockdown_Zones\\4", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Restricted sites", "PMDisplayName": "Restricted sites [Protected Mode]", "Description": "This zone contains Web sites that could potentially damage your computer or data.", "Icon": "inetcpl.cpl#00004481", "LowIcon": "inetcpl.cpl#005426", "CurrentLevel": "0", "Flags": "33", "1200": "3", "1400": "3"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\P3P", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\P3P\\History", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Passport", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"NumRegistrationRuns": "6"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Passport\\LowDAMap", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Protocols", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Protocols\\Mailto", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"UTF8Encoding": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\TemplatePolicies", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\TemplatePolicies\\High", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"1400": "3"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Wpad", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\ZoneMap", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "ProxyByPass": "1", "IntranetName": "1", "UNCAsIntranet": "1", "AutoDetect": "0"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\ZoneMap\\Domains", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": ""}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\ZoneMap\\ProtocolDefaults", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "http": "3", "https": "3", "ftp": "3", "file": "3", "@ivt": "1", "shell": "0", "knownfolder": "0"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\ZoneMap\\Ranges", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": ""}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Zones", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "SelfHealCount": "1", "SecuritySafe": "1"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Zones\\0", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Computer", "PMDisplayName": "Computer [Protected Mode]", "Description": "Your computer", "Icon": "shell32.dll#0016", "LowIcon": "inetcpl.cpl#005422", "CurrentLevel": "0", "Flags": "33", "1200": "0", "1400": "0"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Zones\\1", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Local intranet", "PMDisplayName": "Local intranet [Protected Mode]", "Description": "This zone contains all Web sites that are on your organization's intranet.", "Icon": "shell32.dll#0018", "LowIcon": "inetcpl.cpl#005423", "CurrentLevel": "66816", "Flags": "219", "1200": "0", "1400": "0", "2500": "3"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Zones\\2", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Trusted sites", "PMDisplayName": "Trusted sites [Protected Mode]", "Description": "This zone contains Web sites that you trust not to damage your computer or data.", "Icon": "inetcpl.cpl#00004480", "LowIcon": "inetcpl.cpl#005424", "CurrentLevel": "69632", "Flags": "71", "1200": "0", "1400": "0"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Zones\\3", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Internet", "PMDisplayName": "Internet [Protected Mode]", "Description": "This zone contains all Web sites you haven't placed in other zones", "Icon": "inetcpl.cpl#001313", "LowIcon": "inetcpl.cpl#005425", "CurrentLevel": "70912", "Flags": "1", "1200": "0", "1400": "0"}}, {"name": "software\\microsoft\\windows\\currentversion\\internet settings\\Zones\\4", "hive": "HKEY_CURRENT_USER", "exists": "True", "values": {"": "", "DisplayName": "Restricted sites", "PMDisplayName": "Restricted sites [Protected Mode]", "Description": "This zone contains Web sites that could potentially damage your computer or data.", "Icon": "inetcpl.cpl#00004481", "LowIcon": "inetcpl.cpl#005426", "CurrentLevel": "73728", "Flags": "3", "1200": "3", "1400": "3", "1C00": "0"}}]

Badger · December 29, 2021, 4:52pm

What is the problem? What configuration have you tried and what do you not like about the results?

rusty_cole · December 29, 2021, 4:57pm

I tried many many configuration combos.
For some reason logstash is ignoring my json files (not creating index), i am working in debug mode, no error is visable.
I have many other inputs that works fine.
Can you please suggest a configuration approach for this type of file?

rusty_cole · December 29, 2021, 6:10pm

I am posting the solution:

input
{


            file
        {
            path => "C:/Evidence/Registry/**/*.json"
            start_position => "beginning"        
            sincedb_path => "nul"
			codec => "json"
			delimiter => "§¶¶§"
			mode => "read"	
            type => "json"

        }
}




filter {
   if [type] == "json" {
    json {
    source => "[message]"
    remove_field => ["[message]"]
  }
}
}




output{

		     if "json" in [type] {
elasticsearch{
        hosts => ["http://localhost:9200/"]
        index => "json_data"
    }

	}	


}

system · January 26, 2022, 6:10pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.