Using Protocol Buffer `Any` Types in Go

Introduction

This post utilizes Protocol Buffers and GRPC in the context of creating a key value store in Go. Protocol Buffers are a great way to serialize message data for transmission and GRPC builds on Protocol Buffers to describe services and how the messages passed between them. A simplistic put request for a key value store could be described with the following protocol buffer code:

1
2
3
4

message PutRequest {
    string key = 1;
    string value = 2;
}

The PutRequest has two fields, key and a value, which are assigned numbers for encoding the Protocol Buffer wire format. Generating Go code will create a struct looking like this:

1
2
3
4

type PutRequest struct {
    Key   string 
    Value string
}

The client code can easily access the Value field. However, This PutRequest isn’t ideal in the context of the key value store because it only stores strings. Procol Buffers provide a field type called Any to allow a field to represent multiple types.

Using protobuf.Any

Using the protobuf.Any type requires changing the protocol buffer file:

• Setting the syntax to proto3
• Including any.proto
• Changing value field type to google.protobuf.Any

So now the protocol buffer file looks like this:

1
2
3
4
5
6
7

syntax = "proto3";
import "google/protobuf/any.proto";

message PutRequest {
    string key = 1;
    google.protobuf.Any value = 2;
}

That was easy. However, getting the value from the generated code is more complicated. The Value field now expects a pointer to a anypb.Any type.

1
2
3
4

type PutRequest struct {
   Key   string
   Value *anypb.Any
}

And the anypb.Any struct is defined like this:

1
2
3
4

type struct Any {
    TypeUrl string 
    Value   []byte 
}

The anypb.Any struct contains two fields. The TypeUrl field is a string containing the message type in the form of a URL: type.googleapis.com/google.protobuf.StringValue. The Value field is a slice of bytes. So the TypeUrl field indicates how to interpret the slice of bytes in Value.

Working with the anypb.Any type requires two Go packages to be installed: anypb and wrapperspb. The anypb package defines the anypb.Any struct and the methods for marshalling values through it. The wrapperspb package provides structs for wrapping simple scalar types.

Marshalling Values

Marshalling an interface{} value through an anypb.Any field requires 3 steps:

1. A type switch to determine which wrapper to use (line 3)
2. Creating the wrapper message (line 5)
3. Calling anypb.New() on the wrapper message (line 10)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

func Marshal(val interface{}) (*anypb.Any, error) {
    var m proto.Message
    switch v := val.(type) {
    case string:
        m = wrapperspb.String(v)  // wrap the value 
    case float32:
        m = wrapperspb.Float(v)
    // other cases here

    return anypb.New(m), nil     // create an Any field
}

Unmarshalling Values

Unmarshalling is also accomplished in 3 steps:

1. Calling UnmarshalNew() method on the value field (line 2)
2. Determine the type of the field in a type switch (line 5)
3. Calling UnmarshalTo() method (line 9)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

func (s *KVServer) Put(ctx context.Context, r *gen.PutRequest) (*gen.Response, error) {
    m, err := r.value.UnmarshalNew()
    // handle error

    switch m := m.(type) {
    case *wrapperspb.StringValue:
        err = r.value.UnmarshalTo(m)
        // handle error
        handleStringValues(m.GetValue())

    case *wrapperspb.Int64Value:
        err = r.value.UnmarshalTo(m)
        // handle error
        handleInt64Values(m.GetValue())

    // other cases here
    }
}

Handling Array Types

The wrapperspb package doesn’t provide any way to wrap complex or array types, but it’s straight forward to support. First, define a message with a repeated type:

message StringArray {
    repeated string value = 1;
}

Next, add a case to the type switch:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

switch m := m.(type) {
case *wrapperspb.StringValue:
    err = r.value.UnmarshalTo(m)
    // handle error
    handleStringValues(m.GetValue())

case *gen.StringArray:
    err = r.value.UnmarshalTo(m)
    // handle error
    handleStringArrayValues(m.GetValue())

// other cases here
}

Conclusion

Protocol Buffers provide a field type google.protobuf.Any which is one way model a field with multiple types. Using an Any type requires a bit more work to get values in and out of a message.

• Marshaling and unmarshal requires a wrapper messages.
  • Primitive types can use messages from wrapperspb
  • Complex or array types need wrapped in custom messages
• Use anypb.New() to create anypb.Any values
• Use anypb.Any.UnmarshalNew() and anypb.Any.UnmarshalTo to unmarshal messages.

These gists illustrate all the concepts discussed above:

• protocol buffer file
• go source