REST over gRPC with grpc-gateway for Go

https://medium.com/swlh/rest-over-grpc-with-grpc-gateway-for-go-9584bfcbb835

gRPC does great work when it comes to speed and ease of use. Indeed with generated client and server code, it all comes to as little as implement the interfaces to have a solution ready to run.

There are plenty of cases however when you still need to provide clients with REST endpoints.

With grpc-gateway, you can generate a reverse proxy that would translate REST into gRPC call via marshaling of the JSON request body into respective Go structures followed by the RPC endpoint call.

Continuing the Reminder service example I started in the previous @arkadybalaba/quick-run-to-secure-your-grpc-api-with-ssl-tls-fbd910ec8eee">posts, let’s introduce REST endpoint to schedule a new reminder with several updates to our schema and the server.

Tools

Firstly make sure you have all prerequisites installed and ready to use as described @arkadybalaba/api-with-grpc-and-golang-d6aba44345a0">here. Next, installprotoc-gen-grpc-gatewayplugin:

go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway

This would download the plugin and place it into your $GOBIN folder (for Mac OS/Linux this would probably be ~/go/bin).

Schema

Let’s start with the schema. Original service declaration is:

service ReminderService {
rpc ScheduleReminder(ScheduleReminderRequest) returns (ScheduleReminderResponse) {}
}

protoc command executed for given schema would producereminder.pb.go. The file would have two code stubs for the client and the server, and dependent go code routines.

To enable the service to support JSON to gRPC transformations and respective HTTP handlers we should:

1. import google annotations
2. update the schema to use HTTP rule
3. modify protoc command to callgrpc-gatewayplugin

Google annotations can be imported the same way as other external proto definitions — usingimportkeyword.

import "google/api/annotations.proto"

We then can use options to annotate RPC endpoints with desired rules:

rpc ScheduleReminder(ScheduleReminderRequest) returns (ScheduleReminderResponse) {
    option (google.api.http) = { put: “/v1/remidner/schedule” body: “*” };
}

HTTP rule here is going to be loaded via theannotations.proto. This would make the plugin to generate REST handlers stub for the marked method. It supports a few parameters — method (GET, POST, PUT…), path and body. In this example, we use the entire JSON value to construct GoScheduleReminderRequestrequest structure and so the body parameter value is set to *.

The resulted proto schema would look like:

syntax = “proto3”;
package demo.reminder.v1;
import "google/protobuf/timestamp.proto";
import "google/api/annotations.proto";
option go_package = "reminder";
service ReminderService { rpc ScheduleReminder(ScheduleReminderRequest) returns (ScheduleReminderResponse) {
        option (google.api.http) = { put: "/v1/remidner/schedule" body: "*" };
    }
}message ScheduleReminderRequest {
    google.protobuf.Timestamp when = 1;
}message ScheduleReminderResponse {
    string id = 1;
}

To make protoc to execute grpc-gateway plugin we should include --grpc-gateway_out argument. The final command would then look like:

protoc \
-I. \
-I $GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/ \
--go_out=plugins=grpc:. \
--grpc-gateway_out=logtostderr=true:. \
reminder.proto

-I (or —-proto_path) is used to specify folders where the compiler would be looking for imported proto files.

In this example, we also specify the project root folder as the import directory because according to protoc documentation:

proto_path must be an exact prefix of the .proto file names — protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it’s harder than you think).`

This way protoc knows where to look for reminder.proto.

GOPATH env variable here is set as described on https://github.com/golang/go/wiki/SettingGOPATH .

Given command would produce two files — reminder.pb.goand reminder.pb.gw.go, where the second one is generated by the grpc-gateway plugin. If you look inside the file, you would find several functions to build a reverse proxy. I won’t go deep into the details, but the logic there is pretty straight forward — you get a function to register HTTP handler to call underlying gRPC service endpoint(s). Thanks to the plugin we would not need to care about JSON to Go structures data transformations ourself, its all been done for us in the generated code.

Server

To finally let the server to expose REST endpoint we should:

1. Create a client connection and a router
2. Register new HTTP handler to care about the incoming requests
3. Start the HTTP server to listen for the requests

We need to proxy requests received at HTTP endpoint to a running gRPC server. Similar to how we would create a connection between gRPC client and the server, we do so with for the proxy.

conn, err := grpc.DialContext(
    context.Background(),
    "localhost:8080",
    grpc.WithTransportCredentials(clientCert),
)
if err != nil {
    log.Fatalln("Failed to dial server", err)
}

Remember to set the desired certificate so client-server handshake would success.

Next register newly created request multiplexer (mux) with the generated proxy handler, in our case it is RegisterReminderServiceHandler.

router := runtime.NewServeMux()
if err = reminder.RegisterReminderServiceHandler(context.Background(), router, conn); err != nil {
    log.Fatalln("Failed to register gateway", err)
}

This would pass any incoming request registered at the router to grpcServercreated earlier.

Lastly set HTTP server to listen for incoming requests:

http.ListenAndServeTLS(":8081", "server.crt", "server.key", router)

Once we run the server with go run main.go, we would be able to handle a PUT request received at [https://localhost:8081/v1/reminder/schedul](https://localhost:8081/v1/reminder/schedulewith)e with gRPC server.

With this setup gRPC and HTTP servers would listen on two different ports — 8080 and 8081 respectively. This may be a bit of a headache as you would have to keep different configurations for gRPC and HTTP connections on the clients.

gRPC and HTTP server on the same port

We can configure the router to handle an incoming request by either gRPC or HTTP handler and therefore we would be only running one server. As result we only would use one port for the server to listen at. To recognize the request source we can read the protocol version and Content-Type header value.

Protocol versions would be

HTTP/1.1 for HTTP
HTTP/2 for gRPC

This should be done for every incoming request and therefore can be set as default handler. A handler function to do the job would be:

func httpGrpcRouter(grpcServer *grpc.Server, httpHandler http.Handler) http.Handler {
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    if r.ProtoMajor == 2 && strings.Contains(r.Header.Get(“Content-Type”), “application/grpc”) {
        grpcServer.ServeHTTP(w, r)
    } else {
        httpHandler.ServeHTTP(w, r)
    }
  })
}

Here we check the protocol version major value and route the request to the right server.

We use grpc.ServeHTTPmethod instead of originally used grpcServer.Serve. ServeHTTPis implementation of http.Handler interface from standard Go library. Note that ServeHTTP uses Go’s HTTP/2 server implementation which is totally separate from grpc-go’s HTTP/2 server. Performance and features may vary between the two paths. ServeHTTP does not support some gRPC features available through grpc-go’s HTTP/2 server, and it is currently EXPERIMENTAL and subject to change.

You no longer need to run grpcServer in its goroutine, so that piece of code may be safely removed. gRPC requests will now be served with HTTP/2 Go server.

In order to use this handler as default, we should update the listener to:

http.ListenAndServeTLS(":8080", "../server.crt", "../server.key", httpGrpcRouter(grpcServer, router))

All done. You are now ready to call the endpoint on the same port 8080 from either HTTP or gRPC client.

You can find the full example in this GitHub repository.