ef4's Dev Blog

Cardstack Architecture Notes

Edward Faulkner -

Many people have expressed an interest in contributing to Cardstack and have been clamoring for some docs to help them get started. This is the first of three posts I intend to ship in the near term to unlock some of that potential energy:

  1. Cardstack Architecture Notes. This post. A high-level overview — breath over depth — that points out many of the key concepts and their motivations.
  2. Roadmap. Where are we headed, in terms of features that don't exist yet but are needed.
  3. Working demo app.

For a more general introduction to Cardstack and the motivations behind the project, you may want to watch this talk I gave at EmberConf:

Orientation: Relationship to Ember & Glimmer

Glimmer is a fast, small UI component rendering engine for the web. It gives us declarative templates that are carefully designed to be a superset of HTML that's appropriate for rich, dynamic applications. Glimmer's internals are fascinating, though not critical to this post. (It has an optimizing compiler and two-phase bytecode virtual machine that's as fast as virtual-dom implementations on initial render while remaining faster than them on update.)

Ember is a batteries-included framework for building ambitious web applications. It is built on top of Glimmer (which was originally extracted from Ember's fourth-generation rendering engine). Ember's greatest strength is its welcoming community that champions shared solutions and stability without stagnation, which has enabled a deep ecosystem of shared code to grow and flourish. Ember is an antidote to "Javascript fatigue".

Cardstack is built on top of Ember, and so in one sense is just a family of Ember Addons. But Cardstack differs from Ember by also offering standardized server-side code. This allows Cardstack plugins to do more out-of-the-box than Ember addons can normally do.

My goal is to bridge two worlds:

  • Cardstack should be useful to people who are already writing Ember apps who want to incrementally adopt its features. They are the early-adopter audience who can get the most out of it today and who can help push the project to maturity.
  • But we are also consciously building toward the point where a new, much bigger audience can adopt Cardstack: organizations that don't have the experience, budget, or appetite for risk that's needed today to build custom in-browser applications. People who would have otherwise chosen from the earlier generation of content management systems, because those systems do more out-of-the-box.

This goal dictates parts of the high-level Cardstack architecture: it's implemented as a family of separate packages, so that it can be incrementally adopted and the individual packages can be immediately useful in the Ember ecosystem. And it's designed with the ultimate goal of letting people ship ambitious applications to production without writing a line of Javascript.

Source Code

The source for all the Cardstack packages lives in the cardstack/cardstack repo. Each package is published to npm separately, under the @cardstack organization.

Several related Ember addons that are not Cardstack-specific have been spun out as completely independent repos, such as ember-toolbars. This follows the general rule: purely client-side concerns fall within the sphere of Ember, and so they ship as plain old Ember addons. Concerns that span client and server are Cardstack plugins, the core set of which lives within the cardstack/cardstack repo.

Hub, Plugins, Features

The main server entry point is the @cardstack/hub package. The Hub is not a framework for writing server apps, it is a server app. You configure and customize the Hub by adding plugins.

Cardstack plugins are npm packages that have "cardstack-plugin" in their keywords list and a cardstack-plugin section in the package.json file. By default, their Cardstack-specific code is located in their top-level directory, but this can be customized by setting the src property in the cardstack-plugin section of package.json (this is important because many Cardstack plugins are also Ember Addons, and it's nice to be able to use a completely stock Ember Addon layout and move the Cardstack-specific bits into a subdirectory).

Each Cardstack plugin can provide one or more Features. Examples of Features include:

  • field types
  • constraints
  • writers
  • indexers
  • searchers
  • authenticators
  • middleware

Running the Hub

To start up, the Hub needs seed models. These contain the minimal set of configuration needed to access the rest of its data. For an example, see the blueprint included in @cardstack/git.

In addition to being a standalone node package, the Hub is also an Ember Addon that registers development-mode and test-mode middleware with ember-cli, so that the Hub runs automatically when you do ember serve or ember test. When run this way, it mounts itself under the URL /cardstack within ember-cli's web server, and it conventionally loads seed models from cardstack/seeds/development.js or cardstack/seeds/test.js.

In production, you can start the Hub server directly via @cardstack/hub's bin/server.js, and pass the location of the production seed models as the only argument.

Don't actually deploy the Hub to production right now, this is pre-alpha software and not all authorization features are fully implemented.

Plugin Loading

Plugins must be activated before they will take effect. Eventually this will be controlled via admin UI, but for right now you do it by adding a plugin-configs model whose module attribute contains the name of the plugin's npm package. For example, assuming you are running the Hub via ember-cli in development:

POST http://localhost:4200/cardstack/plugin-configs
Content-Type: application/vnd.api+json
{
  type: 'plugin-configs',
  attributes: {
    module: '@cardstack/mobiledoc'
  }
}

Data Sources

The Hub is not the authoritative storage location for any of your data. It has no database of its own, at least not in the way you may be accustomed to. What it has is a fast cache & search index, powered internally by Elasticsearch. It uses plugins to index, search, and write to whatever set of upstream data sources you need.

This gives us the ability to present an idiomatic, JSONAPI-compliant, performant API with uniform query semantics, even when the data comes from disparate legacy system, third-party services, or hosted databases of your choice.

Data source plugins are implemented via some combination of

  • an indexer, which ingests upstream data and emits JSONAPI documents for the Hub to cache and serve. A good example is the indexer in @cardstack/git.
  • a writer, which writes changes directly to the upstream data source (after which an indexer can pick them up -- we always go round-trip to avoid split-brain synchronization problems). A good example is the writer in @cardstack/git.
  • a searcher which can do data-source-specific deep searching of data that you don't want to fully index. A good example is the searcher in @cardstack/github-auth, which lets you choose to outsource your user models to GitHub.

Schema

The Hub needs to understand your content's schema in order to properly index it, expose endpoints for reading and writing it, validate it, etc. Within the Hub, content, schema, and configuration are all represented as JSONAPI documents.

Consumers of the Hub's web-facing JSONAPI interface don't really need to make a distinction between what is content vs schema. However, authors of data source plugins need to be aware that schema is defined as models that alter how other models will be indexed. For example, if you add a new field to a content-type model, that alters the schema (and potentially requires some content to be reindexed). Whereas if you create a new article model, that is just a content change.

The list of types that are treated as schema is located here in the source. The @cardstack/hub package also contains the bootstrap schema, without which we would have no way to getting started (what fields are allowed when POSTing a new content-type? The bootstrap schema is where that information comes from.) The bootstrap schema contains built-in types like:

  • content-types You might create content-types like "events" and "articles". Each of these can map to a DS.Model class within ember-data.
  • fields You might create "title" and "body" fields that you will want to attach to your "articles" content type. Each of these can map to DS.attr, DS.belongsTo, or DS.hasMany in ember-data.
  • constraints You might attach a "length less than 40 characters" constraint to your "title" field.
  • default-values You may want to create a "now" default value that you can attach to a "last-edited-date" field.
  • grants You may want to express a rule that users in a particular team may edit or read certain content types or fields.

There is no requirement that the schema for a content-type and the content-type itself are stored in the same data source. You can keep all your shema models in a data source like @cardstack/git, even though some of the types are stored in a different data source. Alternatively, data source plugins are free to emit schema models if they want to -- this allows dynamic discovery of upstream schema.

JSONAPI

The @cardstack/jsonapi plugin contains Hub middleware that exposes endpoints for all your content and schema models. The intent is that you shouldn't need to implement custom middleware just to hold business logic -- business logic can be done via a combination of plugins that provide authenticators, grants, constraints, default values, and writers.

Authentication

The @cardstack/authentication package provides support for clientside sessions (via an ember-simple-auth authenticator) along with a Hub middleware that provides the server-side endpoints for establishing sessions.

Activating @cardstack/authentication opts your app into Cardstack-controlled sessions, but it doesn't implement any particular strategy. For that you also need to activate a plugin that offers an authenticator feature and configure it as an authentication source. An example is @cardstack/github-auth, which uses GitHub OAuth login.

Authentication plugins shouldn't necessarily dictate anything about your user model or where it's stored. The @cardstack/authentication test suite has illustrative examples of the ways you can rewrite upstream users to link them with your own user content-types.

Authorization

Authorization is built into @cardstack/hub, since it tends to be a cross-cutting concern. We have a fairly complete implementation of authentication for all write operations at both the content-type and field level. The upcoming roadmap post will go into more detail on what's needed for read-operation grants and mappings between users and groups.

Clientside Tools

The @cardstack/tools package contains the generic clientside components for rendering and tracking Hub-managed content within an Ember app and making it editable as appropriate.

The cardstack-content component is the entry point for rendering any piece of Cardstack content. It accepts a content model and a format:

{{cardstack-content content=model format="page"}}

It then uses naming conventions to find an appropriate component template to use. In the above example, if the model is of type "article", we would look for a cardstack/article-page component.

Within that component template, you would use the cs-field component to display the fields from the content, either using their default renderers:

{{cs-field content "title"}}

Or by accessing the field's value directly and customizing how it's handled:

{{#cs-field content "avatarURL" as |url|}}
  <img src={{url}} />
{{/cs-field}}

In either case, during editing the presence of cardstack-content and cs-field lets the tools discover which parts of the page belongs to which piece of content and where its fields are rendered.

An upcoming goal is to use a compile-time template transformation to simplify {{cs-field content "foo"}} to {{foo}}, which would work because we can do schema-dependent compilation.

Models

The roadmap post will cover this in more detail, but the general idea is that we want to generate Ember Data models directly from the Hub's schema so you don't need to hand-code them. This is not implemented today, and when experimenting with Cardstack you will generally need to create your own models (and probably adapters and serializers).

Routing

The @cardstack/routing package lets you delegate a subsection of your Ember app's routes to Cardstack's conventions. It provides routes for all content types, pleasant error handling for missing content, and branch query parameter support that lets your preview different versions of your site.

It's fine to not use @cardstack/routing, particularly as you're learning and debugging. It's not especially configurable yet and you may have more luck making your own routes, peeking into @cardstack/routing for ideas. But of course the longer-term goal is to not require people to hand-write routes, so this package will be important.

Field Type Plugins

Field type plugins provide

  • Ember components for rendering and editing
  • server-side functions for validation and search indexing
  • client-side functions that control how to display placeholder content for empty states

The most comprehensive example is @cardstack/mobiledoc. It provides three components. Their names follow Cardstack conventions that allow them to be discovered by @cardstack/tools:

  • field-editors/mobiledoc-editor implements the component that appears within the Cardstack sidebar to represent a mobiledoc field.
  • inline-field-editors/mobiledoc-editor implements the component that is rendered inline on top of your field's content while editing it. This is optional, not every field type needs this.
  • field-renderers/mobiledoc-renderer provides the default rendering output for mobiledoc fields.

It also includes app/fields/mobiledoc.js, which customizes the way empty mobiledoc fields appear.

And it includes a cardstack field feature cardstack/field.js that provides the server-side methods used by the Hub to customize how mobiledoc fields are validating, indexed, and searched.

Search

The general strategy for assembling many pieces of content into a particular page is to provide a @cardstack/search package that provides:

  • components like cardstack-search that accept a query and yield content models
  • a field type implementation for query, with editor components that let end users build and tune search queries.

Branching

Cardstack uses the concept of branches to allow multiple versions of a site to coexist at once. This enables powerful workflows.

While the concept and name "branch" definitely comes from Git, it's important to note that other data sources that don't happen to be Git can also be configured to support multiple branches of your site. For example, a PostgreSQL plugin should allow you to provide different database configurations that correspond to site branches like "dev", "staging", and "production".

The intent is that code, content, and schema can all be changed on one branch and will take effect immediately when viewing that branch's version of the site.

There are necessarily some exceptions, and a given server has a "controlling branch" that is more important than the others. The controlling branch is where the Hub reads configuration that controls authorization decisions, the base set of data sources that are enabled, etc. Attempting to do all these things per-branch gets too difficult to understand and manage.