Introducing XDB

XDB is a new kind of database library based on tuples.

Rather than writing database-specific schemas, queries, and migrations, XDB allows developers to model their domain once and use it with one or more databases.

XDB separates the application domain model from the underlying database(s) by using a simple yet powerful data model based on tuples. This lets developers focus on modeling, ingesting, and querying data—without worrying about the underlying database infrastructure or operations.

Why?

Not all databases are created equal. Most applications at scale use multiple types of databases:

• PostgreSQL/MySQL as the main database
• Redis for caching
• Elasticsearch for search
• Clickhouse for analytics
• BigTable for versioning

Each database solves a specific problem and comes with its own tradeoffs.

An application's domain model is often a combination of data that resides in different databases. Typically, each database has its own abstraction layer for migrations and queries. Sometimes, new microservices are spun up to manage specific use cases like search, analytics, or caching.

At the end of the day, developers must stitch together domain data from multiple databases or microservices to serve user-facing APIs. Looking at an end-to-end flow, there are several layers of data fetching, mutation, and transformation. Every time a feature adds new fields or relationships, the entire stack goes through churn.

XDB aims to separate the application domain model from database implementation and operations. What if, instead of maintaining multiple database-specific implementations, developers could model their domain once and seamlessly work with multiple databases?

Inspiration

XDB draws inspiration from two key concepts: Data Services and N-Quads.

Data services are intermediary services that sit between APIs and databases. They provide simple APIs for your domain data, while automating and abstracting away the underlying database management.

N-Quad is a well-known format used to represent attributes and relationships in graphs.

Here's an example of N-Quad format:

<Post:9bsv0s5ocl6002kdg0fg> <title> "Hello World" .
<Post:9bsv0s5ocl6002kdg0fg> <description> "..." .
<Post:9bsv0s5ocl6002kdg0fg> <author> <1> .
<Post:9bsv0s5ocl6002kdg0fg> <created_at> "2025-04-01T00:00:00Z"
<Post:9bsv0s5ocl6002kdg0fg> <tags> <golang> .
<Post:9bsv0s5ocl6002kdg0fg> <tags> <xdb> .
<User:1> <follows> <User:2> .
<User:2> <likes> <Post:9bsv0s5ocl6002kdg0fg> .

XDB was inspired by Dgraph's Mutation API, which uses N-Quads to insert or update data. What if this idea could be extended to build an abstraction usable with any database?

Data Model

XDB is built around three core types - Tuple, Edge, and Record.

Tuple

A Tuple combines a kind, id, attribute name, value, and optional metadata.

This simple yet powerful structure can represent any domain model and is easily mappable to various database formats.

Here's how to create a tuple:

tuple := xdb.NewTuple("Post", "9bsv0s5ocl6002kdg0fg", "title", "Hello World")

Edge

An Edge is a special kind of tuple whose value is a reference. Edges are unidirectional and represent relationships between records.

Record

A Record is a collection of tuples that share the same kind and id. Records are similar to objects, structs, or rows in a database.

Here's how to create a record with tuples:

record := xdb.NewRecord("Post", "9bsv0s5ocl6002kdg0fg").
	Set("title", "Hello World").
	Set("description", "...").
	Set("created_at", time.Now()).
	Set("author_id", "1").
	Set("tags", []string{"golang", "xdb"})

Using XDB As A Library

XDB can be used as a library replacing the traditional repository/database layer in Go services.

Let's first define a simple domain model using standard Go structs:

type Post struct {
	ID          string    `xdb:"id,primary_key"`
	Title       string    `xdb:"title"`
	Description string    `xdb:"description"`
	CreatedAt   time.Time `xdb:"created_at"`
	AuthorID    string    `xdb:"author_id"`
	Tags        []string  `xdb:"tags"`
}

Now, let's walk through creating, storing, and retrieving a post:

// Create a new post
post := &Post{
	ID:          "9bsv0s5ocl6002kdg0fg",
	Title:       "Hello World",
	Description: "A sample post about XDB",
	CreatedAt:   time.Now(),
	AuthorID:    "1",
	Tags:        []string{"golang", "xdb"},
}

// Convert the struct to a record
record, err := xdbstruct.ToRecord(post)
if err != nil {
	log.Fatal(err)
}

// Create a new store using any of the driver implementations
store := xdbmemory.New()

// Store the record in the database
err = store.PutRecord(ctx, record)
if err != nil {
	log.Fatal(err)
}

// Retrieve the record from the database
record, err = store.GetRecord(ctx, record.Key())
if err != nil {
	log.Fatal(err)
}

// Convert the record back to a struct
var fetchedPost Post
err = xdbstruct.FromRecord(record, &fetchedPost)
if err != nil {
	log.Fatal(err)
}

Routing Data

The real power of XDB lies in its ability to "route" the same domain model to different databases. Let's explore how to create a "routing" layer that moves around tuples, edges, and records between different databases:

type RecordRouter struct {
	Primary   xdb.RecordWriter 	// e.g. PostgreSQL
	Cache     xdb.RecordWriter 	// e.g. Redis
	Indexer   xdb.RecordIndexer // e.g. Elasticsearch
}

func (r *RecordRouter) PutRecord(ctx context.Context, record *xdb.Record) error {
	// Save complete record to primary database as source of truth.
	r.Primary.PutRecord(ctx, record)

	// Then update the cache.
	r.Cache.PutRecord(ctx, record)

	// For search, only index relevant fields.
	indexRecord := record.Keep("title", "description", "author", "tags")

	r.Indexer.IndexRecord(ctx, indexRecord)
}

func (r *RecordRouter) GetRecord(ctx context.Context, key *xdb.Key) (*xdb.Record, error) {
	// Get the record from cache.
	record, err := r.Cache.GetRecord(ctx, key)
	if err != nil {
		return nil, err
	}

	// If not found, get from primary database.
	if record == nil {
		record, err = r.Primary.GetRecord(ctx, key)
		if err != nil {
			return nil, err
		}

		// Update the cache.
		r.Cache.PutRecord(ctx, record)
	}

	return record, nil
}

This pattern allows you to distribute specific attributes of your domain model to the most appropriate databases. It also centralizes code & logic for retries, error handling, monitoring, etc.

Building Blocks

XDB APIs are designed to be simple, composable, and easy to use. Let's explore the key building blocks that make up the XDB ecosystem.

Core Types

The core types used to create tuples, edges, and records form the foundation of XDB's data model.

Encoding

The encoding APIs provide consistent methods for converting between XDB's data types and various formats. Here's how to use different encoding options:

import (
	"github.com/xdb-dev/xdb/encoding/xdbjson"
	"github.com/xdb-dev/xdb/encoding/xdbproto"
	"github.com/xdb-dev/xdb/encoding/xdbstruct"
)

var record *xdb.Record
var post Post
var pb proto.Message

// Convert struct to record
record, err = xdbstruct.ToRecord(post)
// Convert record to struct
err = xdbstruct.FromRecord(record, &post)

// Convert protobuf message to record
record, err = xdbproto.ToRecord(pb)
// Convert record to protobuf message
err = xdbproto.FromRecord(record, &pb)

var jsonBytes []byte

// Convert record to JSON
jsonBytes, err = xdbjson.FromRecord(record)
// Convert JSON to record
err = xdbjson.ToRecord(jsonBytes, &record)

Drivers

Drivers serve as the bridge between XDB's tuple-based model and specific database implementations. All drivers always implement the basic Reader and Writer capabilities:

type RecordReader interface {
	GetRecords(ctx context.Context, keys []xdb.Key) ([]*xdb.Record, []*xdb.Key, error)
}

type RecordWriter interface {
	PutRecords(ctx context.Context, records []*xdb.Record) error
	DeleteRecords(ctx context.Context, keys []xdb.Key) error
}

Advanced capabilities, like full-text search, aggregation, iteration, etc. are implemented by specific drivers based on their database features:

type RecordIndexer interface {
	IndexRecords(ctx context.Context, records []*xdb.Record) error
}

type RecordSearcher interface {
	SearchRecords(ctx context.Context, query *xdb.Query) ([]*xdb.Record, error)
}

type TupleIterator interface {
	IterateTuples(ctx context.Context, func(tuple *xdb.Tuple) error, opts ...xdb.IteratorOption) error
}

type EdgeIterator interface {
	IterateEdges(ctx context.Context, func(edge *xdb.Edge) error, opts ...xdb.IteratorOption) error
}

Stores

Stores provide higher-level APIs that combine multiple drivers to support common use-cases. Here's an example of a cached store implementation:

type RecordStore interface {
	xdb.RecordReader
	xdb.RecordWriter
}

type CachedRecordStore struct {
	Primary   RecordStore
	Cache     RecordStore
}

func (s *CachedRecordStore) GetRecords(ctx context.Context, keys []xdb.Key) ([]*xdb.Record, error) {
	// ...
}

func (s *CachedRecordStore) PutRecords(ctx context.Context, records []*xdb.Record) error {
	// ...
}

func (s *CachedRecordStore) DeleteRecords(ctx context.Context, keys []xdb.Key) error {
	// ...
}

Store implementations also satisfy the capability interfaces they implement. This allows you to use a store as a driver or to layer & compose stores & drivers for more complex use-cases.

Schema

The Schema APIs provide a database-agnostic way to define and manage your application's domain models. These APIs enable:

• Runtime type checking and constraint enforcement
• Generation of database-specific schemas
• Migration management