Logstash convert single quoted json log to double quoted

[babuzrb (Rahman)]

Hi I have log files with multiple json lines. Each json has single quotes and some of the values do not even has any quote. Due to this single quote and absence of quote issue, I cannot index them to elasticsearch. Through logstash I need to replace all single quotes to double quotes and also add double quotes to all constant type values (i.e: True, None to "True", "None" etc.).
If you could help please? Example data:

{
   'key1': 'value1',
   'key2':{
      'k1': 'v1',
      'k2': 'v2'
   },
   'key3':[
      'test1',
      'test2'
   ],
   'key3': True,
   'key4': None
}
{
   'key11': 'value1',
   'key12':{
      'k11': 'v1',
      'k12': 'v2'
   },
   'key13':[
      'test1',
      'test2'
   ],
   'key13': True,
   'key14': None
}

FYI @Badger

Thanks

[Badger]

It might be fragile, but you could try

    mutate {
        gsub => [
            "message", "'", '"',
            "message", ": None\b", ": null",
            "message", ": True\b", ": true",
            "message", "\A", "[",
            "message", "\Z", "]",
            "message", "^}$", "},"
        ]
    }
    json { source => "message" remove_field => [ "message" ] target => "data" }

[babuzrb (Rahman)]

Hi @Badger, I will try. Need a suggestion. Though I have shared the formatted jsons in the question, actually my jsons are not formatted in the log file. There are many single line jsons in each log file. Example:

{ 'key1': 'value1', 'key2':{ 'k1': 'v1', 'k2': 'v2' }, 'key3':[ 'test1', 'test2' ], 'key3': True, 'key4': None }
{ 'key11': 'value1', 'key12':{ 'k11': 'v1', 'k12': 'v2' }, 'key13':[ 'test1', 'test2' ], 'key13': True, 'key14': None }

Which codec I shall use? json_lines or multiline or json?

Thanks

[Badger]

I wouldn't use any codec for that, just use

    mutate {
        gsub => [
            "message", "'", '"',
            "message", ": None\b", ": null",
            "message", ": True\b", ": true"
        ]
    }
    json { source => "message" remove_field => [ "message" ] }

[babuzrb (Rahman)]

Thanks @Badger . Your solution worked like a charm!