OpenHPS Protocol Buffers

NPM GitHub Documentation

The module @openhps/protobuf was released today which provides a CLI tool for generating protocol buffers for data frames, objects from any OpenHPS module. In addition, the module provides serializer and deserializer functions that seamlessly work together with other communication modules to use these protocol buffers to provide lightweight messages.

By default, OpenHPS serializes internal pushed data as typed JSON data, which can easily be deserialized by other nodes in a network. However, serialization to JSON produces a lot of overhead in the data. To circumvent this, protocol buffers creates a custom protocol for managing specific data. Instead of naming fields, like is the case with JSON, it assigns a binary number for each class and field.

Protocol Message Generator

Our new module comes with an automatic protocol buffer message generator (*.proto file) that works with any class of any module implementing the @SerializableObject() decorator, including your own custom classes and extensions.

package core;
syntax = "proto3";
import "AbsolutePosition.proto";
import "RelativePosition.proto";
message DataObject {
	string displayName = 1;
	int32 createdTimestamp = 2;
	string uid = 3;
	string parentUID = 4;
	AbsolutePosition position = 5;
	repeated RelativePosition relativePositions = 6;
}

Serialization and Deserialization

First the protocol buffer serializer has to be initialized with the messages.

typescript
import { ProtobufSerializer } from '@openhps/protobuf';
// Initialize the protocol buffer serializer with the protocol files
ProtobufSerializer.initialize("/home/openhps/protobuf/");
// Or
// Initialize the protocol buffer serialization by creating protocol messages in the ./tmp directory
ProtobufSerializer.initialize();

An example protocol buffer message type can be seen below for a DataObject. The absolute and relative positions are set to an Any type.

package openhps.core;
syntax = "proto3";
import "google/protobuf/any.proto";
message DataObject {
	string displayName = 1;
	int64 createdTimestamp = 2;
	string uid = 3;
	string parentUID = 4;
	google.protobuf.Any position = 5;
	repeated google.protobuf.Any relativePositions = 6;
}
package openhps.core;
syntax = "proto3";
import "Velocity.proto";
import "Orientation.proto";
import "google/protobuf/any.proto";
message Absolute3DPosition {
	string timestamp = 1;
	Velocity velocity = 2;
	Orientation orientation = 3;
	google.protobuf.Any unit = 4;
	string referenceSpaceUID = 5;
	google.protobuf.Any accuracy = 6;
	string probability = 7;
	double x = 8;
	double y = 9;
	double z = 10;
}

Usage

@openhps/protobuf is designed with remote processing in mind. We provide two examples using socket connections and MQTT. However, due to the module providing simple serialization and deserialization functions, any future module that handles communication can be used.

@openhps/socket

ts
ModelBuilder.create()
	.addService(new SocketServer({
		srv: server,
		path: "/api/v1"
	}))
	.from(new SocketServerSource({
		uid: "source",
		// Override serializer and deserializer with protocol buffer
		serialize: (obj) => ProtobufSerializer.serialize(obj),
		deserialize: (obj) => ProtobufSerializer.deserialize(obj)
	}))
	.to()
	.build();

@openhps/mqtt

ts
ModelBuilder.create()
	.addService(new MQTTServer({
		port: 1443,
	}))
	.from(new MQTTSourceNode({
		uid: "source",
		// Override frame serializer (not the options)
		serialize: (obj, options) => ({
			frame: ProtobufSerializer.serialize(obj),
			options
		}),
		deserialize: (obj) => ProtobufSerializer.deserialize(obj.frame)
	}))
	.to()
	.build();