# NodeJS Microservice with gRPC and TypeScript


It is very difficult for an engineering team to manage a very large application. Thus instead of building a large monolith application, companies are writing multiple microservices which can be managed by multiple small teams. These microservices are loosely coupled to each other and can be developed, deployed, and scaled independently. However, in this microservice architecture, the microservices often need to communicate with each other via a lightweight communication protocol. For this inter-process communication, gRPC is the popular choice of developers.

gRPC is an open-source framework for Remote Procedure Calls. It is developed by Google. It is a lightweight, high-performant, and type-safe framework for inter-service communication. It uses http/2 for transport and protocol buffers for serializing structured data.

In this blog, we will write microservices with nodejs and use gRPC for communication between them.

## Repo Setup

We will create a monorepo where we will keep all our microservices and protobuf files. We will have a server called `product-service` and one client to test the server.

Let's create our project repo and change the current working directory to it.

```sh
mkdir nodejs-microservices
cd nodejs-microservices
```

We will create an empty node project inside it. Here we will pass the `-y` flag to accept all the defaults.
We will use `npm workspace` for local shared dependencies resolution.

```sh
npm init -y
```

We will create 2 directories `packages`, and `services`. In the `packages` directory, we will keep all our reusable libraries like generated Typescript files from protobuf etc. And in the `services` directory, we will keep all our microservices.

```sh
mkdir packages services
```

Let's create the `protos` package first.

**NOTE: This is not an ideal way to use protobuf files, as the version is not locked in the services. So any changes we make in the protos package will be auto-reflected in services. It is better to keep protobuf files in a separate repo and publish generated Typescript files to a registry. And use that in services as an npm package with a fixed version. You can read about publishing this package to npm in [my other blog post](/blogs/publish-protobuf-typescript-to-npm)**

```sh
npm init -w packages/protos --scope=@rsbh-nodejs-microservices -y
```

In the `protos` package, we will create 2 directories `src` and `dist`. In the `src` directory, we will keep our protobuf files and in the `dist` directory, we will keep the generated Typescript files from the protobuf files.

```sh
cd packages/protos
mkdir src dist
```

After this, the project structure should look something like this.

```treeview
.
├── package.json
├── packages
│   └── protos
│       ├── dist
│       ├── package.json
│       └── src
└── services
```

Now we will create protobuf files in the `src` directory. This `protos` package will contain multiple protobuf files. So, we will create subdirectories inside the `src` directory as per the proto package name.

## Create Protobuf file

For the `product-service`, we will create `product.proto` file inside the `product` subdirectory. We will define the `Product` message here. Read more about the protobuf message <a href="https://protobuf.dev/programming-guides/proto3/" target="_blank">here</a>

`src/product/product.proto`

```protobuf
syntax = "proto3";

package product;

import "google/protobuf/timestamp.proto";

message Product {
  int32 id = 1;
  string name = 2;
  string description = 3;
  string image = 4;
  repeated string tags = 5;
  google.protobuf.Timestamp created_at = 6;
  google.protobuf.Timestamp updated_at = 7;
}
```

We will create a service called `ProductService` and define the schemas of Request and Response for the rpc methods. The `ProductService` will have 3 rpc methods `CreateProduct`, `GetProduct`, and `ListProducts`.

```protobuf
syntax = "proto3";

package product;

import "google/protobuf/timestamp.proto";

message Product {
  int32 id = 1;
  string name = 2;
  string description = 3;
  string image = 4;
  repeated string tags = 5;
  google.protobuf.Timestamp created_at = 6;
  google.protobuf.Timestamp updated_at = 7;
}

message CreateProductRequest {
  string name = 1;
  string description = 2;
  string image = 3;
  repeated string tags = 4;
}

message CreateProductResponse {
  Product product = 1;
}

message GetProductRequest {
  int32 id = 1;
}

message GetProductResponse {
  Product product = 1;
}

message ListProductsRequest {

}

message ListProductsResponse {
  repeated Product products = 1;
}

service ProductService {
  rpc CreateProduct(CreateProductRequest) returns (CreateProductResponse);
  rpc GetProduct(GetProductRequest) returns (GetProductResponse);
  rpc ListProducts(ListProductsRequest) returns (ListProductsResponse);
}
```

### Generating Typescript files from Protobuf

To Generate the TypeScript files, we need to first install the `protobuf` compiler. The `protobuf` compiler will be installed globally in the system. Here I am installing in macos via brew. It will different for other operating systems. check the <a href="https://github.com/protocolbuffers/protobuf/" target="_blank">github repo</a> for installation instructions.

```sh
brew install protobuf
```

We also need to install the `ts-proto` npm package to use it as a plugin for `protoc`. We will create a build script to generate typescript files from protbuf. In the build script, we will set `dist` as the output directory and `src` as the source directory for protobuf files.

```sh
npm i -D ts-proto typescript
touch build.sh
```

`build.sh`

```sh
#!/bin/bash
protoc --plugin=$(npm root)/.bin/protoc-gen-ts_proto \
 --ts_proto_out=dist \
 --ts_proto_opt=outputServices=grpc-js \
 --ts_proto_opt=esModuleInterop=true \
 -I=src/ src/**/*.proto
```

In the build script, we are using `$(npm root)` instead of `./node_modules` because due to the npm workspace, the binary is installed in the root `node_modules`. Read more about `npm root` <a href="https://docs.npmjs.com/cli/v8/commands/npm-root" target="_blank">here</a>.

We will make the build script executable and run it.

```sh
chmod +x build.sh
./build.sh
```

After executing the build script, the typescript files will be generated in the `dist` folder, and folder structures will look like this.

```treeview
.
├── dist
│   ├── google
│   │   └── protobuf
│   │       └── timestamp.ts
│   └── product
│       └── product.ts
├── package.json
└── src
    └── product
        └── product.proto
```

## Create gRPC Server

Let's create the `product-service`. The `product-service` will be a gRPC server that will store the product data in the postgres database. We will use `typeorm` in this service.

From the root of the repo, we will run the npm init command to scaffold the package.json file.

```sh
npm init -w services/product-service --scope=@rsbh-nodejs-microservices -y
```

Then change the current working directory to `product-service`. In the `product-service` directory, we will create `src`, and `dist` subdirectories similar to the `protos` package.

```sh
cd services/product-service
mkdir src dist
```

Let's install the dependencies and dev dependencies for the server.

```sh
npm i -D typescript ts-node
npm i @grpc/grpc-js typeorm reflect-metadata pg
```

We will create the `tsconfig.json` file inside the server directory next to the `package.json` file.

`tsconfig.json`

```json
{
  "compilerOptions": {
    "target": "es2016",
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "module": "node16",
    "moduleResolution": "node16",
    "rootDir": "./src",
    "outDir": "./dist",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}
```

Let's create a Model for the `products` table. The `id` is the primary key of the table and will be auto-generated with an auto-increment value. The createdAt & updatedAt fields will be auto-generated too and set during insert and update operations.

`src/models/product.ts`

```ts
import {
  Entity,
  PrimaryGeneratedColumn,
  Column,
  CreateDateColumn,
  UpdateDateColumn,
} from "typeorm";

@Entity()
export class Product {
  @PrimaryGeneratedColumn()
  id!: number;

  @Column()
  name!: string;

  @Column()
  description!: string;

  @Column()
  image!: string;

  @Column("text", { array: true })
  tags!: string[];

  @CreateDateColumn()
  createdAt!: Date;

  @UpdateDateColumn()
  updatedAt!: Date;
}
```

Let's configure the database connection. We will create `db/index.ts` file inside the `src` directory.
We have to import the Product model here and add it to the entities property.

`src/db/index.ts`

```ts
import { DataSource } from "typeorm";
import { Product } from "../models/product";

const dataSource = new DataSource({
  type: "postgres",
  host: process.env.POSTGRES_HOST || "localhost",
  port: Number(process.env.POSTGRES_PORT) || 5432,
  username: process.env.POSTGRES_USER || "postgres",
  password: process.env.POSTGRES_PASSWORD || "postgres",
  database: process.env.POSTGRES_DB || "postgres",
  entities: [Product],
  logging: true,
  synchronize: true,
});

export default dataSource;
```

Let's setup the gRPC server. We will create a `main.ts` file inside the `src` directory. Here we will import the `dataSource`. After initializing the `dataSource`, we will get the `db` instance. We will start the `server` only after the database is connected.

`src/main.ts`

```ts
import "reflect-metadata";
import dataSource from "./db";
import { Server, ServerCredentials } from "@grpc/grpc-js";

const server = new Server();

const HOST = process.env.HOST || "0.0.0.0";
const PORT = Number(process.env.PORT) || 50051;

const address = `${HOST}:${PORT}`;

dataSource
  .initialize()
  .then((db) => {
    server.bindAsync(
      address,
      ServerCredentials.createInsecure(),
      (error, port) => {
        if (error) {
          throw error;
        }
        console.log("server is running on", port);
        server.start();
      }
    );
  })
  .catch((error) => console.log(error));
```

Update the scripts in the `package.json` file to run and build the server.

`package.json`

```json
{
  ...
  "scripts": {
    "dev": "ts-node src/main.ts",
    "build": "tsc",
    "start": "node dist/main.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  ...
}
```

We can test the server and database connection by running it. It should create the `products` table in the database.

```sh
npm run dev

or

npm run build
npm start
```

Let's start implementing the methods for the server. First, we need to install the `protos` package. Then we create a `server.ts` file.

```sh
npm i @rsbh-nodejs-microservices/protos
touch src/server.ts
```

We will create a `getProductServer` function in the `server.ts` file which will return an object with `createProduct`, `getProduct`, and `listProducts` methods. We are using this server object instead of creating a class due to the server signature <a href="https://github.com/agreatfool/grpc_tools_node_protoc_ts/blob/master/doc/server_impl_signature.md" target="_blank">issue</a>

We will define the business logic later, right now all the methods will return the status `UNIMPLEMENTED` in the response. We are doing this to test the server.

`src/server.ts`

```ts
import { sendUnaryData, ServerUnaryCall, status } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  CreateProductResponse,
  GetProductRequest,
  GetProductResponse,
  ListProductsRequest,
  ListProductsResponse,
  ProductServiceServer,
} from "@rsbh-nodejs-microservices/protos/product/product";
import { DataSource } from "typeorm";

export function getProductServer(db: DataSource): ProductServiceServer {
  async function createProduct(
    call: ServerUnaryCall<CreateProductRequest, CreateProductResponse>,
    callback: sendUnaryData<CreateProductResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }
  async function getProduct(
    call: ServerUnaryCall<GetProductRequest, GetProductResponse>,
    callback: sendUnaryData<GetProductResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }
  async function listProducts(
    call: ServerUnaryCall<ListProductsRequest, ListProductsResponse>,
    callback: sendUnaryData<ListProductsResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }

  return {
    createProduct,
    getProduct,
    listProducts,
  };
}
```

We will import the `getProductServer` function and ProductService definition in the `main.ts` file and then add the product service to the gRPC server.

`src/main.ts`

```ts
import "reflect-metadata";
import dataSource from "./db";
import { Server, ServerCredentials } from "@grpc/grpc-js";
import { getProductServer } from "./server";
import { ProductServiceService } from "@rsbh-nodejs-microservices/protos/product/product";

const server = new Server();

const HOST = process.env.HOST || "0.0.0.0";
const PORT = Number(process.env.PORT) || 50051;

const address = `${HOST}:${PORT}`;

dataSource
  .initialize()
  .then((db) => {
    server.addService(ProductServiceService, getProductServer(db));
    server.bindAsync(
      address,
      ServerCredentials.createInsecure(),
      (error, port) => {
        if (error) {
          throw error;
        }
        console.log("server is running on", port);
        server.start();
      }
    );
  })
  .catch((error) => console.log(error));
```

After running the server, We will test the service methods with `grpcurl`. Read more about `grpcurl` <a href="https://github.com/fullstorydev/grpcurl" target="_blank">here</a>

Let's test the `CreateProduct` method, we need to pass the proto file to the `grpcurl` command. I am calling the `grpcurl` from the root of the monorepo, thus the path to the proto files are relatives to the root. After that, We need to pass the request body as a JSON string. Then pass the address to the server and service method name.

```sh
grpcurl -import-path packages/protos/src -proto product/product.proto -d '{ "name": "product 1", "description": "foo bar", "image": "", "tags": ["tag 1"]}' -plaintext localhost:50051 product.ProductService.CreateProduct

ERROR:
  Code: Unimplemented
  Message: Unknown Error
```

After running the `grpcurl` call, we should get an error from the server. Here we are getting the status code `Unimplemented` as we expected.

Let's add the `controller` for `product` model. We will create `product.controller.ts` file inside `src/controllers` directory. In this file we will write the logic for database operations. We create the `createProduct` function, the function accepts the database instance and request body as the arguments.

We are defining the interface for `createProductReq` here, We can also use `CreateProductRequest` from the `protos` package instead of this. It is just my personal preference to define the interface inside the server as we are mapping the request fields with `Product` model fields. So, If in the future there is a change in `CreateProductRequest` then we can easily transform `CreateProductRequest` to `createProductReq`

`src/controllers/product.controller.ts`

```ts
import { DataSource } from "typeorm";
import { Product } from "../models/product";

interface createProductReq {
  name: string;
  description: string;
  image: string;
  tags: string[];
}

export const createProduct = async (
  db: DataSource,
  req: createProductReq
): Promise<Product> => {
  const productRepository = db.getRepository(Product);
  const product = new Product();
  product.name = req.name;
  product.description = req.description;
  product.image = req.image;
  product.tags = req.tags;
  return productRepository.save(product);
};
```

We then import the `ProductController` in the `server.ts` file and update the `createProduct` method. We will transform the result from the database insert operation to the `Product` message by passing it to the `fromJSON` method. We will wrap this logic inside a try-catch block and return the status code `INTERNAL` in case of an error.

`src/server.ts`

```ts
import { sendUnaryData, ServerUnaryCall, status } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  CreateProductResponse,
  GetProductRequest,
  GetProductResponse,
  ListProductsRequest,
  ListProductsResponse,
  Product,
  ProductServiceServer,
} from "@rsbh-nodejs-microservices/protos/product/product";
import { DataSource } from "typeorm";
import * as ProductController from "./controllers/product.controller";

export function getProductServer(db: DataSource): ProductServiceServer {
  async function createProduct(
    call: ServerUnaryCall<CreateProductRequest, CreateProductResponse>,
    callback: sendUnaryData<CreateProductResponse>
  ) {
    try {
      const product = await ProductController.createProduct(db, call.request);
      const productPB = Product.fromJSON(product);
      const response: CreateProductResponse = {
        product: productPB,
      };
      callback(null, response);
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }
  async function getProduct(
    call: ServerUnaryCall<GetProductRequest, GetProductResponse>,
    callback: sendUnaryData<GetProductResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }
  async function listProducts(
    call: ServerUnaryCall<ListProductsRequest, ListProductsResponse>,
    callback: sendUnaryData<ListProductsResponse>
  ) {
    callback({ code: status.UNIMPLEMENTED }, null);
  }

  return {
    createProduct,
    getProduct,
    listProducts,
  };
}
```

Let's test the `CreateProduct` method again with `grpcurl`. We should get the response back from the server now instead of the status `UNIMPLEMENTED` error.

```sh
grpcurl -import-path packages/protos/src -proto product/product.proto -d '{ "name": "product 1", "description": "foo bar", "image": "", "tags": ["tag 1"]}' -plaintext localhost:50051 product.ProductService.CreateProduct

{
  "product": {
    "id": 1,
    "name": "product 1",
    "description": "foo bar",
    "tags": [
      "tag 1"
    ],
    "createdAt": "2023-03-29T00:07:23.426Z",
    "updatedAt": "2023-03-29T00:07:23.426Z"
  }
}
```

Let's add the `listProducts` and `getProduct` functions to the product controller. Both functions will accept database instance as an argument similar to the `createProduct` function

`src/controllers/product.controller.ts`

```ts
import { DataSource } from "typeorm";
import { Product } from "../models/product";

interface createProductReq {
  name: string;
  description: string;
  image: string;
  tags: string[];
}

export const createProduct = async (
  db: DataSource,
  req: createProductReq
): Promise<Product> => {
  const productRepository = db.getRepository(Product);
  const product = new Product();
  product.name = req.name;
  product.description = req.description;
  product.image = req.image;
  product.tags = req.tags;
  return productRepository.save(product);
};

export const listProducts = async (db: DataSource): Promise<Product[]> => {
  const productRepository = db.getRepository(Product);
  return productRepository.find();
};

export const getProduct = async (
  db: DataSource,
  id: number
): Promise<Product | null> => {
  const productRepository = db.getRepository(Product);
  return productRepository.findOneBy({ id: id });
};
```

We will update the `getProduct` and `listProducts` methods in the `server.ts` file.

In the `getProduct` method, we will check if the database returns the `product`, then only we return the response. Otherwise, we will return the status code `NOT_FOUND` with the message.

In the `listProducts` method, we will get an array of `product` objects from the database, we will run the `map` function to transform each of them into the `Product` message.

`src/server.ts`

```ts
import { sendUnaryData, ServerUnaryCall, status } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  CreateProductResponse,
  GetProductRequest,
  GetProductResponse,
  ListProductsRequest,
  ListProductsResponse,
  Product,
  ProductServiceServer,
} from "@rsbh-nodejs-microservices/protos/product/product";
import { DataSource } from "typeorm";
import * as ProductController from "./controllers/product.controller";

export function getProductServer(db: DataSource): ProductServiceServer {
  async function createProduct(
    call: ServerUnaryCall<CreateProductRequest, CreateProductResponse>,
    callback: sendUnaryData<CreateProductResponse>
  ) {
    try {
      const product = await ProductController.createProduct(db, call.request);
      const productPB = Product.fromJSON(product);
      const response: CreateProductResponse = {
        product: productPB,
      };
      callback(null, response);
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }
  async function getProduct(
    call: ServerUnaryCall<GetProductRequest, GetProductResponse>,
    callback: sendUnaryData<GetProductResponse>
  ) {
    try {
      const product = await ProductController.getProduct(db, call.request.id);
      if (product) {
        const productPB = Product.fromJSON(product);
        const response: GetProductResponse = {
          product: productPB,
        };
        callback(null, response);
      } else {
        callback(
          {
            code: status.NOT_FOUND,
            message: `Product ${call.request.id} not found`,
          },
          null
        );
      }
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }
  async function listProducts(
    call: ServerUnaryCall<ListProductsRequest, ListProductsResponse>,
    callback: sendUnaryData<ListProductsResponse>
  ) {
    try {
      const products = await ProductController.listProducts(db);
      const productsPB = products.map(Product.fromJSON);
      const response: ListProductsResponse = {
        products: productsPB,
      };
      callback(null, response);
    } catch (err) {
      callback({ code: status.INTERNAL }, null);
      console.error(err);
    }
  }

  return {
    createProduct,
    getProduct,
    listProducts,
  };
}
```

We will test the `ListProducts` and `GetProduct` methods with `grpcurl`. They shouldn't be returning an error with the status `UNIMPLEMENTED`.

```sh
grpcurl -import-path packages/protos/src -proto product/product.proto -plaintext localhost:50051 product.ProductService.ListProducts

{
  "products": [
    {
      "id": 1,
      "name": "product 1",
      "description": "foo bar",
      "tags": [
        "tag 1"
      ],
      "createdAt": "2023-03-29T00:07:23.426Z",
      "updatedAt": "2023-03-29T00:07:23.426Z"
    }
  ]
}

grpcurl -import-path packages/protos/src -proto product/product.proto -d '{"id": 1}' -plaintext localhost:50051 product.ProductService.GetProduct

{
  "product": {
    "id": 1,
    "name": "product 1",
    "description": "foo bar",
    "tags": [
      "tag 1"
    ],
    "createdAt": "2023-03-29T00:07:23.426Z",
    "updatedAt": "2023-03-29T00:07:23.426Z"
  }
}
```

## Create gRPC Client

Let's create a client to test the `product-service`. We will create a simple program that will connect to the server, make the call and log the response in the console.

From the root of the repo, we will run the `npm init` command to scaffold the package.json file and change the current working directory to the `test-client`. Inside the `test-client` we will create the `src` and `dist` directories.

```sh
npm init -w services/test-client --scope=@rsbh-nodejs-microservices -y
cd services/test-client/
mkdir src dist
```

We will install the dependencies and dev dependencies for the client. We will install the `@rsbh-nodejs-microservices/protos` package to get the client stub.

```sh
npm i -D typescript ts-node
npm i @grpc/grpc-js @rsbh-nodejs-microservices/protos
```

We will create the `tsconfig.json` file inside `test-client`, the `tsconfig.json` is the same as in the `product-service`

`tsconfig.json`

```json
{
  "compilerOptions": {
    "target": "es2016",
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "module": "node16",
    "moduleResolution": "node16",
    "rootDir": "./src",
    "outDir": "./dist",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}
```

We will update the `scripts` in the `package.json` file to run and build the client.

`package.json`

```json
{
  ...
  "scripts": {
    "dev": "ts-node src/main.ts",
    "build": "tsc",
    "start": "node dist/main.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  ...
}
```

Let's create the `main.ts` file inside the `src` directory. We will create a main function and test each service method separately.

First, We need to import the `credentials` object from the `grpc-js` package and `ProductServiceClient` from our `protos` package. Then we will create a gRPC client. While creating the client instance, we need to pass the server url and authentication credentials. The `product-service` doesn't have any authentication thus we are passing `credentials.createInsecure()` as authentication credentials.

After creating the client instance, we will call the service method name. Here we are calling the `createProduct` method on the client. We will create the request object for the call and then call the method with the request object and callback function.

`src/main.ts`

```ts
import { credentials } from "@grpc/grpc-js";
import {
  CreateProductRequest,
  ProductServiceClient,
} from "@rsbh-nodejs-microservices/protos/product/product";

const PRODUCT_SERVICE_URL = process.env.USER_SERVICE_URL || "0.0.0.0:50051";

function main() {
  const client = new ProductServiceClient(
    PRODUCT_SERVICE_URL,
    credentials.createInsecure()
  );

  const req: CreateProductRequest = {
    name: "test product",
    description: "foo bar",
    image: "https://example.com/image",
    tags: ["tag-1"],
  };

  client.createProduct(req, (err, resp) => {
    if (err) {
      console.error(err);
    } else {
      console.log("Response :", resp);
    }
  });
}

main();
```

We can similarly call and test the `getProduct` and `listProducts` methods. I am skipping that part in this blog.

All the source code for this blog is available on [GitHub](https://github.com/rsbh/nodejs-microservices).