Ory Homepage

Ory & Supabase Url Shortener Part 1: Backend

Learn how to build a URL shortener using open source ready-to-use solutions.

Andrew Minkin

Ory Guest

Mar 15, 2022

Developers can save time by using ready-made solutions to build new products. Most startups concentrate on solving one problem and doing it well. We have at least three cloud providers to host our code. There are multiple choices on what to use to build a frontend for our next project. For example the VueJS and ReactJS projects with broad open source communities.

In this article, I'll show you an example of how to build an URL shortener using a modern open source technology stack.

Here's an overview of the backend architecture we are going to build including Ory Kratos, Supabase, and Ory Oathkeeper:

Supabase Kratos Architecture

You can find the source code for this project on GitHub.

What we will use

  • Ory Kratos to manage identities and users. We'll use an open source self-hosted version for this tutorial, but for production I recommend the Ory Network - you get a fully featured Ory Kratos instance deployed and ready to use for free.
  • Supabase is an open source alternative to Firebase. Supabase Database comes with a Postgres database, a free and open-source database that is considered one of the world's most stable and advanced databases. We'll use Supabase as database for our URL shortener.
  • Ory Oathkeeper would be a great example of applying Zero Trust architecture for our project. We'll use it as identity and access proxy.
  • Postgres is a powerful, open source object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, robustness, and performance.
  • golang-migrate to perform database migrations.

Backend choices

I'm a huge fan of the Go programming language, and I've been coding using this language since late 2014. I love the simplicity of the Go language design and the ecosystem around it.

  • It's good to perform static code analysis to make your backend systems more robust and stable. golangci-lint is a feature-rich linter that gives you feedback about your code. You can find a lot of linters available.
  • Strongly typed programming language with static data types. One does not need to write tests to check that your code will never mess with data types. int i makes i an integer forever.
  • It's easy to follow SOLID principles and clear architecture using Go.
  • In addition, Go is fast and well-scaled programming language

Database migrations

Database migrations should be used once you use any relational database management system (RDMBS) in your project because you need to change the database schema from time to time. It helps you track the different versions of your schema and easily perform forwards and downwards migrations. In the pythonic world, it's easy to decide what tool to use for schema migrations because it's usually comes out of the box with the selected framework. For instance, we have flask-migrate for flask and Django migrations for Django. Since Go uses the UNIX philosophy to build the architecture of your project, you need to choose:

  • a HTTP router for your endpoints.
  • an ORM or a library to work with the database.
  • a tool to perform migrations.

The Go programming language has at least two tools for schema migrations. I used both goose and migrate, and for this project, I decided to go with migrate. Migrate supports more databases, and I want to make the DB layer in this project database agnostic.

My requirements for the Go migration tool:

  1. Plain SQL migrations support. I don't want to learn an additional filetype formats. I know SQL and I know how to create tables. That's enough. Unlike Django, Go does not have any good Active Record pattern implementations. I hope that there will be more ORMs and better tooling once is Go 1.18 released - generics are on the way.
  2. Support of open source RDBMS like Postgres, MySQL (and all their forks), Oracle (I don't use it yet).
  3. Programmable API or a shorthanded way to apply migrations.
  4. Upward/Downward support.

Gin

A URL shortener is a lightweight service, and it would be ideal to have a simple enough package to build an HTTP API around it. You have several options on what to choose to solve this issue, and the most popular frameworks are:

Go-kit is an excellent framework for building a complex system with many micro-services. Even a simple net/http with HTTP router would be enough for our example. Usually, I toss a coin when I choose between Echo and Gin, so I use Gin in this project.

I'm huge fan of gRPC when I build APIs with Go and I used it a lot previously. Also, I'm huge fan of the Echo framework. I chose Gin because of it's simplicity, convenience, and feature rich support. A simple net/http would be enough to build this project with httprouter but Gin is nice to work with.

Illustration of Gopher at work

Okay. Let's start hacking, shall we?

Configuration package

Create a new folder for your project, and create a file main.go:

// Copyright © 2023 Ory Corp
// SPDX-License-Identifier: Apache-2.0

package main

import (
	"log"

	"github.com/gen1us2k/shorts/api"
	"github.com/gen1us2k/shorts/config"
)

func main() {
	c, err := config.Parse()
	if err != nil {
		log.Fatal(err)
	}
	api, err := api.New(c)
	if err != nil {
		log.Fatal(err)
	}
	api.Start()
}
// Copyright © 2023 Ory Corp
// SPDX-License-Identifier: Apache-2.0

package main

import (
	"log"

	"github.com/gen1us2k/shorts/api"
	"github.com/gen1us2k/shorts/config"
)

func main() {
	c, err := config.Parse()
	if err != nil {
		log.Fatal(err)
	}
	api, err := api.New(c)
	if err != nil {
		log.Fatal(err)
	}
	api.Start()
}

Defining database schema

The database for our URL shortener should have the following tables:

  • url table to store shortened URLs.
  • url_view table to store views. This information will be useful to build additional reports for users about top referrers, urlviews or something else.
CREATE TABLE IF NOT EXISTS url (
	id SERIAL PRIMARY KEY,
	url VARCHAR(255) NOT NULL DEFAULT '',
	hash varchar(10) NOT NULL DEFAULT '',
	created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
	expired_at TIMESTAMP WITH TIME ZONE,
	owner_id VARCHAR(36) NOT NULL DEFAULT ''
);

CREATE UNIQUE INDEX idx_url_hash ON url(hash);

CREATE TABLE IF NOT EXISTS url_view (
	id SERIAL PRIMARY KEY,
	referer VARCHAR(255) NOT NULL DEFAULT '',
	url_id INT NOT NULL,
	CONSTRAINT fk_url_view FOREIGN KEY(url_id) REFERENCES url(id)
);

Creating Supabase project and tables

  • From your Supabase dashboard , click New project.
  • Enter a Name for your Supabase project.
  • Enter a secure Database Password.
  • Select the Region you want.
  • Click Create new project.

Create Project

  • Open table editor.
  • Click on SQL editor on sidebar.
  • Insert SQL table definition from the previous step.
  • Click Run to create tables.

Create tables

Designing the database

I always use interfaces for the database layer because of the following benefits:

  • I can change a database simply by implementing a designed interface.
  • It helps me to think and design a proper layer for the database.
  • It helps to reduce dependencies between different parts of the codebase.
  • It helps to write more modular and decoupled code.
  • I can always implement a mock for my database layer by simply implementing a mock layer and use it in tests.
// Copyright © 2023 Ory Corp
// SPDX-License-Identifier: Apache-2.0

package database

import (
	"github.com/gen1us2k/shorts/config"
	"github.com/gen1us2k/shorts/model"
)

type (
	// WriteDatabase represents interface for
	// a database layer with write permissions. For Analytics you need to implement
	// Analytics interface
	WriteDatabase interface {
		// ShortifyURL makes a short version of URL.
		ShortifyURL(model.URL) (model.URL, error)
		// ListURLs returns all created urls by user
		ListURLs(string) ([]model.URL, error)
		// GetURLByHash returns an URL created by given hash
		GetURLByHash(string) (model.URL, error)
		// StoreView saves an URL view to the database
		StoreView(model.Referer) error
		// DeleteURL deletes URL
		DeleteURL(model.URL) error
	}
	// Analytics interface required for analytics
	// for Write API you need to implement WriteDatabase interface
	Analytics interface {
		Statistics(model.URL) (model.Statistics, error)
	}
)

// CreateStorage creates a storage by given configuration
func CreateStorage(c *config.ShortsConfig) (WriteDatabase, error) {
	if c.DatabaseProvider == config.ProviderPostgres {
		return NewPostgres(c)
	}
	if c.DatabaseProvider == config.ProviderSupabase {
		return NewSupabase(c)
	}
	return NewPostgres(c)
}

Implementing the Database

Supabase uses Postgres as their main RDBMS. Also, they use PostgREST, a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations. The Supabase implementation of the database layer in database/database.go:

// Copyright © 2023 Ory Corp
// SPDX-License-Identifier: Apache-2.0

package database

import (
	"errors"
	"fmt"
	"strconv"
	"time"

	"github.com/gen1us2k/shorts/config"
	"github.com/gen1us2k/shorts/model"
	"github.com/supabase/postgrest-go"
)

// Supabase struct is an implementation
// of database using Supabase Database service
type Supabase struct {
	WriteDatabase
	config *config.ShortsConfig
	conn   *postgrest.Client
}

// NewSupabase creates a new instance by given configuration
func NewSupabase(c *config.ShortsConfig) (*Supabase, error) {
	conn := postgrest.NewClient(c.SupabaseURL, "", map[string]string{
		"apikey":        c.SupabaseKey,
		"Authorization": fmt.Sprintf("Bearer %s", c.SupabaseKey),
	})
	if conn.ClientError != nil {
		return nil, conn.ClientError
	}
	return &Supabase{
		config: c,
		conn:   conn,
	}, nil
}

// ShortifyURL generates short hash code of url and stores it in the database
func (s *Supabase) ShortifyURL(url model.URL) (model.URL, error) {
	var urls []model.URL
	url.Hash = generateHash(s.config.URLLength)
	url.ExpiredAt = time.Now().AddDate(0, 1, -1) // FIXME: Make this configurable
	q := s.conn.From("url").Insert(url, false, "do-nothing", "", "")
	_, err := q.ExecuteTo(&urls)
	return urls[0], err
}

// ListURLs returns all urls created by user
func (s *Supabase) ListURLs(ownerID string) ([]model.URL, error) {
	var urls []model.URL
	q := s.conn.From("url").Select("*", "10", false).Match(map[string]string{"owner_id": ownerID}) // FIXME: Make this configurable
	_, err := q.ExecuteTo(&urls)
	return urls, err
}

// GetURLByHash returns URL from the database by given hash
func (s *Supabase) GetURLByHash(hash string) (model.URL, error) {
	var urls []model.URL
	q := s.conn.From("url").Select("*", "10", false).Match(map[string]string{"hash": hash})
	_, err := q.ExecuteTo(&urls)
	if len(urls) == 0 {
		return model.URL{}, errors.New("does not exist")
	}
	return urls[0], err
}

// StoreView stores view
func (s *Supabase) StoreView(ref model.Referer) error {
	q := s.conn.From("url_view").Insert(ref, false, "do-nothing", "", "")
	_, _, err := q.Execute()
	return err
}

// DeleteURL deletes URL
func (s *Supabase) DeleteURL(url model.URL) error {
	q := s.conn.From("url").Delete("", "").Match(map[string]string{"id": strconv.FormatInt(url.ID, 10)})
	_, _, err := q.Execute()
	return err

}

Designing the HTTP API

An URL shortener is a simple project so we need only two endpoints:

  • Shorten URLs by passing a POST request to /api/url
  • Get created URLs by passing a GET request to /api/url endpoint

Here's the implementation for our API in api/api.go:

// Copyright © 2023 Ory Corp
// SPDX-License-Identifier: Apache-2.0

package api

import (
	"database/sql"
	"log"
	"net/http"

	"github.com/davecgh/go-spew/spew"
	"github.com/gen1us2k/shorts/config"
	"github.com/gen1us2k/shorts/database"
	"github.com/gen1us2k/shorts/middleware"
	"github.com/gen1us2k/shorts/model"
	"github.com/gin-gonic/gin"
)

type (
	// Server struct implements HTTP API used for service
	Server struct {
		r      *gin.Engine
		config *config.ShortsConfig
		db     database.WriteDatabase
	}
)

// New initializes API by given config
func New(c *config.ShortsConfig) (*Server, error) {

	db, err := database.CreateStorage(c)
	if err != nil {
		return nil, err
	}
	s := &Server{
		r:      gin.Default(),
		config: c,
		db:     db,
	}
	s.initRoutes()
	return s, nil
}
func (s *Server) initRoutes() {
	authMiddleware := middleware.NewAuthenticationMiddleware(s.config)
	s.r.GET("/u/:hash", s.showURL)

	// The only kratos thing would be here
	userAPI := s.r.Group("/api/")
	userAPI.Use(authMiddleware.AuthenticationMiddleware)
	userAPI.POST("/url", s.shortifyURL)
	userAPI.GET("/url", s.listURLs)
	userAPI.DELETE("/url/:hash", s.deleteURL)

	// TODO: Implement RBAC here
	analyticsAPI := s.r.Group("/analytics")
	analyticsAPI.GET("/url", s.getURLStats)

}
func (s *Server) getURLStats(c *gin.Context) {

}
func (s *Server) deleteURL(c *gin.Context) {
	hash := c.Param("hash")
	ownerID, ok := c.Get(config.OwnerKey)
	if !ok {
		c.JSON(http.StatusUnauthorized, &model.DefaultResponse{
			Message: "Unauthorized",
		})
	}
	u, err := s.db.GetURLByHash(hash)
	if err != nil {
		c.JSON(http.StatusInternalServerError, &model.DefaultResponse{
			Message: "error querying database",
		})
		return
	}
	if u.OwnerID != ownerID {
		c.JSON(http.StatusForbidden, &model.DefaultResponse{
			Message: "you can't delete this item",
		})
		return
	}
	if err := s.db.DeleteURL(u); err != nil {
		c.JSON(http.StatusInternalServerError, &model.DefaultResponse{
			Message: "error querying database",
		})
		return
	}
	c.JSON(http.StatusOK, &model.DefaultResponse{
		Message: "deleted",
	})
}

func (s *Server) showURL(c *gin.Context) {
	spew.Dump(c.Request.Header)
	hash := c.Param("hash")
	u, err := s.db.GetURLByHash(hash)
	if err != nil {
		c.JSON(http.StatusInternalServerError, &model.DefaultResponse{
			Message: "error querying database",
		})
		return
	}
	referer := ""
	if len(c.Request.Header["Referer"]) > 0 {
		referer = c.Request.Header["Referer"][0]
	}
	ref := model.Referer{
		URLID:   u.ID,
		Referer: referer,
	}
	if err := s.db.StoreView(ref); err != nil {
		c.JSON(http.StatusInternalServerError, &model.DefaultResponse{
			Message: "error querying database",
		})
		return
	}
	c.Redirect(http.StatusMovedPermanently, u.URL)
}
func (s *Server) listURLs(c *gin.Context) {
	ownerID, ok := c.Get(config.OwnerKey)
	if !ok {
		c.JSON(http.StatusUnauthorized, &model.DefaultResponse{
			Message: "Unauthorized",
		})
	}
	urls, err := s.db.ListURLs(ownerID.(string))
	if err != nil && err != sql.ErrNoRows {

		c.JSON(http.StatusInternalServerError, &model.DefaultResponse{
			Message: "error querying database",
		})
		return
	}
	c.JSON(http.StatusOK, urls)
}

func (s *Server) shortifyURL(c *gin.Context) {
	ownerID, ok := c.Get(config.OwnerKey)
	if !ok {
		c.JSON(http.StatusUnauthorized, &model.DefaultResponse{
			Message: "Unauthorized",
		})
	}
	var url model.URL
	if err := c.ShouldBindJSON(&url); err != nil {
		c.JSON(http.StatusBadRequest, &model.DefaultResponse{
			Message: "failed parse json",
		})
		return
	}
	url.OwnerID = ownerID.(string)
	u, err := s.db.ShortifyURL(url)
	if err != nil {
		log.Println(err)
		c.JSON(http.StatusInternalServerError, &model.DefaultResponse{
			Message: "failed to create short version",
		})
		return
	}
	c.JSON(http.StatusOK, u)
}

// Start starts http server
func (s *Server) Start() error {
	return s.r.Run(s.config.BindAddr)
}

Plugging everything together

Create a file in cmd/shorts/main.go with the following content

// Copyright © 2023 Ory Corp
// SPDX-License-Identifier: Apache-2.0

package config

import "github.com/kelseyhightower/envconfig"

const (
	KratosSessionKey = "ory_kratos_session"
	OwnerKey         = "owner_id"
	ProviderSupabase = "supabase"
	ProviderPostgres = "postgres"
)

// ShortsConfig stores configuration for the application
type ShortsConfig struct {
	BindAddr         string `envconfig:"BIND_ADDR" default:":8090"`
	DSN              string `envconfig:"DSN"`
	DatabaseProvider string `envconfig:"DATABASE_PROVIDER" default:"supabase"`
	URLLength        int    `envconfig:"URL_LENGTH" default:"8"`
	KratosAPIURL     string `envconfig:"KRATOS_API_URL" default:"http://kratos:4433"`
	UIURL            string `envconfig:"UI_URL" default:"http://127.0.0.1:4455/login"`
	SupabaseKey      string `envconfig:"SUPABASE_KEY"`
	SupabaseURL      string `envconfig:"SUPABASE_URL"`
}

// Parse parses environment variables and returns
// filled struct
func Parse() (*ShortsConfig, error) {
	var c ShortsConfig
	err := envconfig.Process("", &c)
	return &c, err
}

Configuring Ory Oathkeeper

I'll use Ory Oathkeeper to implement zero trust network configuration. Ory Oathkeeper acts as a reverse proxy in this example, but it checks if the request is authenticated.

URLs that follow /u/hash pattern should be available to anonymous users, and the API should be available only for authenticated users.

Basic configuration for Ory Oathkeeper

log:
  level: debug
  format: json

serve:
  proxy:
    cors:
      enabled: true
      allowed_origins:
        - http://127.0.0.1:3001
        - http://127.0.0.1:3000
      allowed_methods:
        - POST
        - GET
        - PUT
        - PATCH
        - DELETE
      allowed_headers:
        - Authorization
        - Content-Type
      exposed_headers:
        - Content-Type
      allow_credentials: true
      debug: true

errors:
  fallback:
    - json

  handlers:
    redirect:
      enabled: true
      config:
        to: http://127.0.0.1:4455/login
        when:
          - error:
              - unauthorized
              - forbidden
            request:
              header:
                accept:
                  - text/html
    json:
      enabled: true
      config:
        verbose: true

access_rules:
  matching_strategy: glob
  repositories:
    - file:///etc/config/oathkeeper/access-rules.yml

authenticators:
  anonymous:
    enabled: true
    config:
      subject: guest

  cookie_session:
    enabled: true
    config:
      check_session_url: http://kratos:4433/sessions/whoami
      preserve_path: true
      extra_from: "@this"
      subject_from: "identity.id"
      only:
        - ory_kratos_session

  noop:
    enabled: true

authorizers:
  allow:
    enabled: true

mutators:
  noop:
    enabled: true
  header:
    enabled: true
    config:
      headers:
        X-User: "{{ print .Subject }}"

Access rules for Ory Oathkeeper

Create a folder oathkeeper to hold the Ory Oathkeeper access rules:

- id: "api:anonymous"
  upstream:
    preserve_host: true
    url: "http://shorts:8090"
  match:
    url: "http://127.0.0.1:8080/u/<**>"
    methods:
      - GET
      - POST
  authenticators:
    - handler: anonymous
  mutators:
    - handler: noop
  authorizer:
    handler: allow

- id: "api:protected"
  upstream:
    preserve_host: true
    url: "http://shorts:8090"
  match:
    url: "http://127.0.0.1:8080/api/<**>"
    methods:
      - GET
      - POST
  authenticators:
    - handler: cookie_session
  mutators:
    - handler: header
  authorizer:
    handler: allow
  errors:
    - handler: redirect
      config:
        to: http://127.0.0.1:4455/login

Authenticating users

You can follow the Ory Kratos Quickstart to add Ory Kratos to your project. You can inspect the docker-compose and the Ory Kratos configuration folder in the repository, but this basic configuration is based on the Quickstart guide.

Authentication middleware

We configured Ory Oathkeeper to act as identity and access proxy, and our configuration does two things:

  • Check authentication and who the current session belongs by calling the sessions/whoami endpoint of Ory Kratos.
  • Get the identity.id from the previous step and modify the request by adding a X-User header with the user's identity id.

It means that we do not need to take any additional steps for our Go application, and we can get the value from X-User header and use it in our application.

// Copyright © 2023 Ory Corp
// SPDX-License-Identifier: Apache-2.0

package middleware

import (
	"net/http"

	"github.com/gen1us2k/shorts/config"
	"github.com/gin-gonic/gin"
)

type (
	AuthMiddleware struct {
		config *config.ShortsConfig
	}
)

func NewAuthenticationMiddleware(c *config.ShortsConfig) *AuthMiddleware {
	return &AuthMiddleware{
		config: c,
	}
}
func (a *AuthMiddleware) AuthenticationMiddleware(c *gin.Context) {
	userID := c.Request.Header.Get("X-User")
	if userID == "" {
		c.Redirect(http.StatusMovedPermanently, a.config.UIURL)
		return
	}
	c.Set(config.OwnerKey, userID)
	c.Next()
}

Next steps

We have implemented the basic URL shortener API and added authentication with Ory Kratos and a SQL backend powered by Supabase. Feel free to fork the example and play around with it.

I hope you enjoyed this tutorial and found it helpful. If you have any questions, check out the Ory community on Slack and GitHub.

Check out Part 2 where we build the frontend!

Some ideas to improve the backend further:

  • Enable 2FA for Ory Kratos
  • Add support of AWS Lambda for backend to enable cost-effective hosting