Efficient JSON Data Handling: Dynamic Parsing Tips in Golang

Crafting Seamless Golang Experiences: Explore Dynamic JSON Value Parsing Techniques for Optimal Development Success.

In the realm of Golang development, the need to parse JSON often arises. Yet, when the type of value becomes uncertain, do we have an elegant solution at our disposal?

For instance, when the JSON string is { "age": 1 }, and the corresponding struct is defined as a string, parsing would result in an error.

Besides defining a deserialization method for the struct, are there alternative solutions? Today, I will introduce another approach to address this challenge.

Mapstructure is primarily employed to facilitate the decoding of arbitrary JSON data into Go structures. It serves as a powerful tool when dealing with dynamic or uncertain types in the JSON data, offering a flexible solution beyond the constraints of rigid struct definitions.

In essence, it excels at parsing data streams with underlying structures that may not be fully understood, and mapping them into the structures we have defined.

Let’s now explore how to use mapstructure through a few examples.

# 1. Conventional Usage.

type Person struct {
    Name   string
    Age    int
    Emails []string
    Extra  map[string]string
}

func normalDecode() {
    input := map[string]interface{}{
      "name":   "Foo",
      "age":    21,
      "emails": []string{"[email protected]", "[email protected]", "[email protected]"},
      "extra": map[string]string{
         "twitter": "Foo",
      },
   }

   var result Person
   err := mapstructure.Decode(input, &result)
   if err != nil {
      panic(err)
   }
   fmt.Printf("%#v\n", result)
}

Result:

main.Person{Name:"Foo", Age:21, Emails:[]string{"[email protected]", "[email protected]", "[email protected]"}, Extra:map[string]string{"twitter":"Foo"}}

This approach is likely the most commonly used, effortlessly mapping a map[string]interface{} to our defined structure.

Here, we haven’t specified tags for each field, allowing mapstructure to automatically handle the mapping.

If our input is a JSON string, we first parse it into a map[string]interface{}, and then map it into our structure.

func jsonDecode() {
     var jsonStr = `{
         "name": "Foo",
         "age": 21,
         "gender": "male"
     }`

     type Person struct {
          Name   string
          Age    int
          Gender string
     }
     m := make(map[string]interface{})
     err := json.Unmarshal([]byte(jsonStr), &m)
     if err != nil {
          panic(err)
     }

     var result Person
     err = mapstructure.Decode(m, &result)
     if err != nil {
          panic(err.Error())
     }
     fmt.Printf("%#v\n", result)
}

Result:

main.Person{Name:"Foo", Age:21, Gender:"male"}

# 2. Embedded Structures.

mapstructure enables us to condense multiple embedded structures and handle them using the squash tag.

type School struct {
    Name string
}

type Address struct {
    City string
}

type Person struct {
    School    `mapstructure:",squash"`
    Address  `mapstructure:",squash"`
    Email      string
}

func embeddedStructDecode() {
   input := map[string]interface{}{
      "Name": "A1",
      "City":  "B1",
      "Email": "C1",
   }

   var result Person
   err := mapstructure.Decode(input, &result)
   if err != nil {
      panic(err)
   }

   fmt.Printf("%s %s, %s\n", result.Name, result.City, result.Email)
}

Result:

A1, B1, C1

In this example, Person incorporates embedded structures for School and Address, and by utilizing the squash tag, it achieves a flattened effect.

# 3. Metadata.

type Person struct {
    Name   string
    Age    int
    Gender string
}

func metadataDecode() {
   input := map[string]interface{}{
      "name":  "A1",
      "age":   1,
      "email": "B1",
   }

   var md mapstructure.Metadata
   var result Person
   config := &mapstructure.DecoderConfig{
      Metadata: &md,
      Result:   &result,
   }

   decoder, err := mapstructure.NewDecoder(config)
   if err != nil {
      panic(err)
   }

   if err = decoder.Decode(input); err != nil {
      panic(err)
   }

   fmt.Printf("value: %#v, keys: %#v, Unused keys: %#v, Unset keys: %#v\n", result, md.Keys, md.Unused, md.Unset)
}

Result:

value: main.Person{Name:"A1", Age:1, Gender:""}, keys: []string{"Name", "Age"}, Unused keys: []string{"email"}, Unset keys: []string{"Gender"}

From this example, we can observe that using Metadata allows us to track the differences between our structure and map[string]interface{}. The identical parts correctly map to the corresponding fields, while disparities are expressed using Unused and Unset.

Unused: Fields present in the map but absent in the structure.
Unset: Fields present in the structure but absent in the map.

# 4. Avoiding Mapping of Null Values.

The usage here is akin to the approach used by the built-in json package, leveraging the omitempty tag to address the mapping of null values.

type School struct {
  Name string
}

type Address struct {
  City string
}

type Person struct {
  *School   `mapstructure:",omitempty"`
  *Address `mapstructure:",omitempty"`
  Age       int
  Email     string
}

func omitemptyDecode() {
   result := &map[string]interface{}{}
   input := Person{Email: "C1"}
   err := mapstructure.Decode(input, &result)
   if err != nil {
      panic(err)
   }
  
   fmt.Printf("%+v\n", result)
}

Result:

&map[Age:0 Email:C1]

Here, we observe that both *School and *Address are tagged with omitempty, this disregarding empty values during parsing.

On the other hand, Age is not tagged with omitempty, and as there is no corresponding value in the input, the parsing utilizes the zero value of the respective type, with the zero value for int being 0.

type Person struct {
    Name  string
    Age   int
    Other map[string]interface{} `mapstructure:",remain"`
}

func remainDataDecode() {
   input := map[string]interface{}{
      "name":   "A1",
      "age":    1,
      "email":  "B1",
      "gender": "C1",
   }
  
   var result Person
   err := mapstructure.Decode(input, &result)
   if err != nil {
      panic(err)
   }
  
   fmt.Printf("%#v\n", result)
}

Result:

main.Person{Name:"A1", Age:1, Other:map[string]interface {}{"email":"B1", "gender":"C1"}}

As evident from the code, the Other field is tagged with remain, signifying that any fields in the input that do not map correctly will be placed in Other.

The output shows that email and gender have been correctly placed in Other.

# 5. Custom Tags.

type Person struct {
    Name string `mapstructure:"person_name"`
    Age  int    `mapstructure:"person_age"`
}
func tagDecode() {
   input := map[string]interface{}{
      "person_name": "A1",
      "person_age":  1,
   }

   var result Person
   err := mapstructure.Decode(input, &result)
   if err != nil {
      panic(err)
   }
  
   fmt.Printf("%#v\n", result)
}

Result:

main.Person{Name:"A1", Age:1}

In the Person structure, we map person_name and person_age to Name and Age, respectively, achieving correct parsing without altering the structure.

# 6. Weakly Typed Parsing.

type Person struct {
    Name   string
    Age    int
    Emails []string
}
func weaklyTypedInputDecode() {
   input := map[string]interface{}{
    "name":   123,  // number => string
    "age":    "11", // string => number
    "emails": map[string]interface{}{}, // empty map => empty array
   }
  
   var result Person
   config := &mapstructure.DecoderConfig{
      WeaklyTypedInput: true,
      Result:           &result,
   }
  
   decoder, err := mapstructure.NewDecoder(config)
   if err != nil {
      panic(err)
   }
  
   err = decoder.Decode(input)
   if err != nil {
      panic(err)
   }
  
   fmt.Printf("%#v\n", result)
}

Result:

main.Person{Name:"123", Age:11, Emails:[]string{}}

From the code, it’s evident that the types of name, age in the input, and Name, Age in the Person structure do not match.

The email field is particularly unconventional, being a string array in one case and a map in another.

By customizing the DecoderConfig and setting WeaklyTypedInput to true, mapstructure easily aids in resolving such weakly typed parsing issues.

However, it’s crucial to note that not all problems can be resolved, and the source code reveals certain limitations:

//   - bools to string (true = "1", false = "0")
//   - numbers to string (base 10)
//   - bools to int/uint (true = 1, false = 0)
//   - strings to int/uint (base implied by prefix)
//   - int to bool (true if value != 0)
//   - string to bool (accepts: 1, t, T, TRUE, true, True, 0, f, F,
//     FALSE, false, False. Anything else is an error)
//   - empty array = empty map and vice versa
//   - negative numbers to overflowed uint values (base 10)
//   - slice of maps to a merged map
//   - single values are converted to slices if required. Each
//     element is weakly decoded. For example: "4" can become []int{4}
//     if the target type is an int slice.

# 7. Error Handling.

Mapstructure provides exceptionally user-friendly error messages.

Let’s take a look at how it delivers prompts when encountering errors.

type Person struct {
    Name   string
    Age    int
    Emails []string
    Extra  map[string]string
}

func decodeErrorHandle() {
   input := map[string]interface{}{
      "name":   123,
      "age":    "bad value",
      "emails": []int{1, 2, 3},
   }
  
   var result Person
   err := mapstructure.Decode(input, &result)
   if err != nil {
      fmt.Println(err.Error())
   }
}

Result:

5 error(s) decoding:

* 'Age' expected type 'int', got unconvertible type 'string', value: 'bad value'
* 'Emails[0]' expected type 'string', got unconvertible type 'int', value: '1'
* 'Emails[1]' expected type 'string', got unconvertible type 'int', value: '2'
* 'Emails[2]' expected type 'string', got unconvertible type 'int', value: '3'
* 'Name' expected type 'string', got unconvertible type 'int', value: '123'

The error messages here inform us about each field and how the values within the fields should be represented. These error prompts can guide us efficiently in resolving issues.

Summary.

The examples above showcase the power of mapstructure in effectively addressing real-world problems, providing practical solutions, and saving development effort.

However, from the perspective of the source code, it’s evident that the library extensively employs reflection, which may introduce performance concerns in certain specialized scenarios.

Therefore, it’s essential for developers to thoroughly consider product logic and use cases when incorporating mapstructure into their projects.

If you like such stories and want to support me, please give me a clap.

Your support is very important to me, thank you.