Building xdb

#xdb

xdb is a Go SDK that provides a consistent abstraction layer over multiple databases/storage engines. xdb is designed, for application developers, to simplify data modeling, ingestion, and querying.

Not all databases are created equal. Each database comes with its own tradeoffs. xdb allows developers to build, scale, and maintain applications without worrying about the underlying database infrastructure and operations. By providing a consistent data and APIs, xdb can be used to "layer" different databases to combine their capabilities and work around their limitations.

Inspiration

The idea of xdb data model originated while implementing an ingestion worker for Dgraph. Dgraph provides a N-Quad mutation API to insert or update data. The simplicity of the N-Quad format allowed implementation of a generic worker that could ingest any N-Quad. Adding new domain models was as simple as adding a domain-to-nquad mapping. There were no SQL migrations, no query changes, etc.

How can we borrow this simplicity and apply it all data ingestion and querying usecases?

Why worry about the underlying database management, build & maintain SQL queries, etc.?

What if we build an abstraction that allows developers to:

  • focus on just modeling their domain, ingesting data, and querying it?
  • swap out the underlying database without any changes to queries?
  • switch between row-oriented and column-oriented storage engines?
  • combine capabilities of different databases to work around their individual limitations?
  • use multiple types of databases - relational, graph, search, time-series, etc. in a single application?

N-Quads

The N-Quad is a very simple, yet powerful, way to represent attributes and relationships of most domain models. Dgraph extended the N-Quad format to support additional metadata for relationships.

Here's a reduced example of a social network domain model:

<Post:9bsv0s5ocl6002kdg0fg> <title> "Hello World" .
<Post:9bsv0s5ocl6002kdg0fg> <body> "...content..." .
<Post:9bsv0s5ocl6002kdg0fg> <author> <User:1> .
<Post:9bsv0s5ocl6002kdg0fg> <tags> <Tag:golang> .
<Post:9bsv0s5ocl6002kdg0fg> <tags> <Tag:xdb> .
<User:1> <follows> <User:2> (since="2009-11-10T23:00:00Z") .
<User:2> <follows> <User:5> (since="2010-12-10T23:00:00Z") .
<User:3> <follows> <Tag:golang> .

Data Model

XDB has a simple data model inspired by N-Quads.

Key

Key is the primary key of a record. A Key is a combination of Kind and ID.

Attribute

Attribute is a tuple of key, name and a value. It's structurally similar to N-Quads.

Edge

Edge is an attribute whose value is a Key of target record. Edges can have additional metadata about the relation.

Value

XDB supports all Go primitive types as values. Custom types can be used by implementing the encoding.BinaryMarshaler or json.Marshaler interfaces.

Schema

XDB has a strict, yet easy to use, schema. A good mental model for the schema is to think of it as a intended shape of application domain, similar to defining a Terraform resource.

The schema acts as a intermediary between the application domain model and the underlying database(s).

An example of a schema for Twitter:

schema:
  - kind: Tweet
    attributes:
      - name: text
        type: string
      - name: created_at
        type: time
    edges:
      - name: author
        type: User
      - name: mentions
        type: User
        array: true
      - name: contains
        type: Hashtag
        array: true
  - kind: User
    attributes:
      - name: name
        type: string
    edges:
      - name: follows
        type: User
        array: true
        metadata:
          - name: since
            type: time

Developer Experience

Using XDB is as simple as defining a schema, initializing a store, and writing data.

For this example, we will use BadgerDB as the underlying storage engine.

import (
  "context"
  "time"

  "github.com/xdb-dev/xdb"
  "github.com/xdb-dev/xdb/kv/badger"
)

func main() {
  schema, _ := schema.Load("schema.yaml")

  store, _ := badger.NewStore(
    badger.Dir("/tmp/xdb"),
    badger.Schema(schema),
  )

  // Create a new post
	post := xdb.NewKey("Post", "9bsv0s45jdj002vjlkt0")
	author := xdb.NewKey("User", "9bsv0s3p32i002qvnf50")
	tags := []*xdb.Key{
		xdb.NewKey("Tag", "xdb"),
		xdb.NewKey("Tag", "database"),
	}

	// Create post's attributes and edges
	attrs := []*xdb.Tuple{
		xdb.NewAttr(post, "title", xdb.String("Introduction to xdb")),
		xdb.NewAttr(post, "body", xdb.String("...")),
		xdb.NewAttr(post, "published_at", xdb.Time(time.Now())),
		xdb.NewEdge(post, "author", author),
		xdb.NewEdge(post, "tags", tags[0]),
		xdb.NewEdge(post, "tags", tags[1]),
	}

	// Save attributes
	if err := store.PutTuples(ctx, attrs...); err != nil {
		panic(err)
	}
}