MacKuba.eu

Running Bluesky PDS undockered

2026-02-04T19:14:20Z

A bit over a year ago, in the first week of January 2025, I migrated my main Bluesky account to my own PDS on a Netcup VPS. It’s been quite easy to set up using the official installer, and it’s been running pretty much without any problems or maintenance the whole year.

Despite that, I haven’t been 100% happy with this setup for one reason: Docker. So I decided to try to take it out of the box, and I made it run first on the same VPS installed separately, and then moved it to another machine this month with a clean install. This blog post is a guide to how I did this, if you’re interested. There are a few existing posts about this already:

But I figured it doesn’t hurt to make another one that does things slightly differently again. “There are many like this, but this one is mine”. (I mostly followed the benharri.org version.)

Note: I’m describing what I did to migrate an existing PDS from in-Docker to outside-Docker, so I already had existing data and pds.env config; if you wanted to install one from scratch this way, you’d probably need to also set up the config manually.

You might be asking: why? And that’s a good question. I mostly wouldn’t recommend this setup over the standard Docker one by default, unless you know what you’re doing. The standard installation is literally running one command and answering some questions, and then it auto-updates and manages everything.

My reason is that I’m generally pretty familiar with installing things on Linux servers manually, but I’m completely unfamiliar with Docker. I always wanted to do some modifications on the PDS, but I didn’t know how, because the Docker setup basically takes over the whole server for itself. I don’t know where it pulls code from, I don’t know where it puts it, and I don’t know when it can overwrite any changes I make. I don’t feel in control. (And to be clear, this is likely a me problem.)

So here’s what I did (this setup is for Ubuntu 24.04 Noble):

Install Nginx

The standard PDS distribution uses Caddy, but I use Nginx everywhere and I have configs built for it, so I’ve set up Nginx:

# install Nginx
sudo apt-get install --no-install-recommends nginx-light

# enable HTTP on the firewall
sudo ufw allow http/tcp
sudo ufw allow https/tcp

# if you haven't enabled ufw before:
sudo ufw limit log ssh/tcp
sudo ufw enable

Also here’s a standard thing I do on VPSes to let me install webapps in /var/www from my account:

# set up environment for webapps

sudo groupadd deploy
sudo adduser psionides deploy
sudo chown root:deploy /var/www
sudo chmod 775 /var/www

I also need Certbot for LetsEncrypt:

# install Certbot

sudo apt-get install --no-install-recommends certbot python3-certbot-nginx
sudo certbot plugins --nginx --prepare

certbot plugins --nginx --prepare does some initial setup of some config files in /etc/letsencrypt that are unrelated to specific certificates.

One thing to note is that the standard setup with Caddy uses a .well-known route to verify any handles under *.yourdomain.com, and it automatically creates HTTPS certificates for those subdomains; this wouldn’t be as simple here, but I don’t need to be able to mass create new handles under my domain. If I ever need one or two, I’ll just set them up manually.

Email

We’ll also need email – you can use an external SMTP, but I like to use local sendmail that’s configured to forward emails to my main account:

# install Postfix
sudo apt-get install --no-install-recommends postfix

  # choose: "Internet site"
  # enter domain: "lab.martianbase.net"

# set up the email forwarding
sudo nano /etc/postfix/virtual

  # add a line like this:
  # kuba@lab.martianbase.net my.real.email@domain.com

sudo nano /etc/postfix/main.cf

  # add:
  # virtual_alias_domains = lab.martianbase.net
  # virtual_alias_maps = hash:/etc/postfix/virtual

sudo postmap /etc/postfix/virtual
sudo service postfix reload

# enable SMTP on firewall
sudo ufw allow smtp/tcp

If you set things up this way, you need to set the PDS_EMAIL_SMTP_URL in pds.env to smtp:///?sendmail=true.

Note: you will probably need to set up a few things the right way in the DNS and might also need more tweaks in the /etc/postfix/main.cf for the email sending and forwarding to work. I honestly don’t understand the Postfix config well enough, I just have a setup that works and I don’t touch it, but it’s old and I don’t know how correct it is, so I won’t share it here. From my experience, it’s generally not super hard to configure self-hosted email in such way that it works for sending emails to you and only to you (like I have with my PDS and my Mastodon instance). If it goes to spam, you know where to look, and if you move it to the inbox, generally the email service should remember and whitelist the sender. (Sending emails to others is a whole different story of course.)

NodeJS

Next, NodeJS. I used asdf to install it (the version 0.15 is because in 0.16 they did a complete rewrite in Go from the original Bash version, and some things don’t work as before).

# install asdf
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.15.0
nano .bashrc

  # add:
  # source "$HOME/.asdf/asdf.sh"

# ---
# log out & back in here - this is also needed for the deploy group to take effect
# ---

# install node
asdf plugin add nodejs

NODE_VER=`asdf latest nodejs 22`
asdf install nodejs $NODE_VER
asdf global nodejs $NODE_VER

corepack enable
asdf reshim nodejs

The Docker version runs on 20.x, but that’s almost EOL, so I’ve upgraded to the latest 22.x. 24.x doesn’t work at the moment, I get some ugly errors during installation.

Install the PDS

Now the actual PDS code. I’ve decided to keep the code in /var/www, fetch it from git and use git pull for updates, and keep the data separately in /var/lib.

cd /var/www
git clone https://github.com/bluesky-social/pds
cd pds/service

pnpm install --production --frozen-lockfile

Migrate the data

Now it’s time to copy over the data from the previous setup.

On the first server I did it like this (remember to also turn off and disable the old PDS service in Docker):

sudo rsync -rlpt /pds /var/lib/
sudo chown -R psionides:psionides /var/lib/pds

For the second one, I made a temporary SSH key on the old server:

ssh-keygen -t ed25519 -f ~/.ssh/migration -N "" -C "pds migration"

Added it to authorized keys on the new one:

nano ~/.ssh/authorized_keys

Prepared an empty directory:

sudo mkdir /var/lib/pds
sudo chown psionides:psionides /var/lib/pds

And rsynced the data from the old one to the new one:

# install if missing on the target server:
sudo apt-get install rsync

rsync -rltv -e "ssh -i ~/.ssh/migration" /var/lib/pds/ newserver:/var/lib/pds/

I tried to sync it from a local machine first, but I realized it would take much longer, and between two servers on the same network it took less than a minute for many GBs.

Start the service

With the data copied, it’s time to finish the installation. First, an HTTPS certificate – for now, I made one using manual DNS registration before switching the DNS records:

sudo certbot certonly --manual --preferred-challenges dns --key-type ecdsa \
    -m $myemail --agree-tos --no-eff-email -d lab.martianbase.net

The way it works is that it gives me a kind of verification token, and I need to put it in a TXT DNS record at _acme-challenge.lab.martianbase.net before pressing continue.

Then, I added a systemd service at /etc/systemd/system/pds.service:

[Unit]
Description=Bluesky PDS
After=network.target

[Service]
Type=simple
User=psionides
WorkingDirectory=/var/www/pds/service
ExecStart=/home/psionides/.asdf/shims/node --enable-source-maps index.js
Restart=on-failure
EnvironmentFile=/var/lib/pds/pds.env
Environment="NODE_ENV=production"
TimeoutSec=15
Restart=on-failure
RestartSec=1
StandardOutput=append:/var/lib/pds/pds.log

[Install]
WantedBy=default.target

I also told the PDS what port to run on in pds.env:

PDS_PORT=3000

And then enabled it like this:

sudo systemctl daemon-reload
sudo systemctl enable --now pds

Test?

$ curl http://localhost:3000

         __                         __
        /\ \__                     /\ \__
    __  \ \ ,_\  _____   _ __   ___\ \ ,_\   ___
  /'__'\ \ \ \/ /\ '__'\/\''__\/ __'\ \ \/  / __'\
 /\ \L\.\_\ \ \_\ \ \L\ \ \ \//\ \L\ \ \ \_/\ \L\ \
 \ \__/.\_\\ \__\\ \ ,__/\ \_\\ \____/\ \__\ \____/
  \/__/\/_/ \/__/ \ \ \/  \/_/ \/___/  \/__/\/___/
                   \ \_\
                    \/_/

It’s working! Now, the Nginx config.

Nginx config for the PDS

In /etc/nginx/sites-available/pds.site:

# this is needed to proxy the relevant HTTP headers for websocket
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream pds {
  server 127.0.0.1:3000 fail_timeout=0;
}

server {
  server_name lab.martianbase.net;
  listen 80;
  listen [::]:80;  # ipv6

  # redirect any http requests to https
  location / {
    return 301 https://$host$request_uri;
  }

  # except for certbot challenges
  location /.well-known/acme-challenge/ {
    root /var/www/html;
  }
}

server {
  server_name lab.martianbase.net;
  listen 443 ssl http2;
  listen [::]:443 ssl http2;

  ssl_certificate /etc/letsencrypt/live/lab.martianbase.net/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/lab.martianbase.net/privkey.pem;

  access_log /var/log/nginx/pds-access.log combined buffer=16k flush=10s;
  error_log /var/log/nginx/pds-error.log;

  client_max_body_size 100M;

  location / {
    include sites-available/proxy.inc;
    proxy_pass http://pds;
  }
}

In /etc/nginx/sites-available/proxy.inc:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Origin "";
proxy_set_header Proxy "";

proxy_buffering off;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

tcp_nodelay on;

Then:

sudo ln -s /etc/nginx/sites-available/pds.site /etc/nginx/sites-enabled/
sudo nginx -t
sudo service nginx reload

DNS

At this point it was time to update the DNS and wait…

Once it started working for me, I also had to wait a bit more for the existing relays to notice the IP change and I needed to poke them a few times to reconnect to me again, like this:

curl https://bsky.network/xrpc/com.atproto.sync.requestCrawl \
  --json '{"hostname": "lab.martianbase.net"}'

Restarting the server with sudo service pds restart also sends a requestCrawl at least to bsky.network. What you want is to see in the pds.log that after pds has started it says request to com.atproto.sync.subscribeRepos. There was a brief moment when I could post something, but after reloading bsky.app I wouldn’t see my post, because the AppView hasn’t indexed it yet… thankfully, after the relay finally reconnected, it backfilled the missing events.

I also updated the Certbot certificate again the normal way so I wouldn’t forget about it later:

sudo certbot certonly --webroot --webroot-path /var/www/html \
    --key-type ecdsa -d lab.martianbase.net

(There’s also a more automated Nginx verification method, where it adds the required certificate lines to your Nginx configs automatically, but I don’t like it because it makes a terrible mess of the configs…)

Gatekeeper

There was one more thing I did while I was already messing with the PDS, which was to add Bailey Townsend’s Gatekeeper service, which restores support for email-based 2FA. For some reason, the email 2FA was implemented in the “entryway” PDS bsky.social and not in the main code, and as of today it still hasn’t been added to the self-hosted PDS distribution… so Bailey decided that “we can just do things” and went and added it himself. You just need to install it separately (it’s written in Rust).

First, we install the 🦀:

# install Rust

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --profile minimal
# (press enter)

source ~/.cargo/env

I used rustup, because Gatekeeper requires a fairly new version of Rust and the one I had in Ubuntu LTS repo didn’t cut it.

We also need some additional dependencies for building:

# some general compilers and stuff
sudo apt-get install --no-install-recommends build-essential

# openssl and pkg-config
sudo apt-get install --no-install-recommends libssl-dev pkg-config

Now, time to clone and build the service:

git clone https://tangled.org/baileytownsend.dev/pds-gatekeeper /var/www/gatekeeper
cd /var/www/gatekeeper/
cargo build --release
sudo cp target/release/pds_gatekeeper /usr/local/bin/

(It takes a bit of time to compile, go make yourself a coffee ☕️)

The current version is missing support for sendmail email transport, so I had to make some tweaks to make it work (see PR).

There are a couple of ENV entries we need to add to the pds.env:

PDS_BASE_URL=http://localhost:3000
GATEKEEPER_PORT=8000

And again, we need to write a systemd service at /etc/systemd/system/gatekeeper.service:

[Unit]
Description=Bluesky Gatekeeper
After=network.target

[Service]
Type=simple
User=psionides
ExecStart=/usr/local/bin/pds_gatekeeper
Restart=on-failure
Environment="PDS_ENV_LOCATION=/var/lib/pds/pds.env"
TimeoutSec=15
Restart=on-failure
RestartSec=1
StandardOutput=append:/var/lib/pds/gatekeeper.log

[Install]
WantedBy=default.target

And launch it:

sudo systemctl daemon-reload
sudo systemctl enable --now gatekeeper

Finally, a few updates to the Nginx config:

upstream gatekeeper {
  server 127.0.0.1:8000 fail_timeout=0;
}

server {
  ...

  location /xrpc/com.atproto.server.createSession {
    include sites-available/proxy.inc;
    proxy_pass http://gatekeeper;
  }

  location /xrpc/com.atproto.server.getSession {
    include sites-available/proxy.inc;
    proxy_pass http://gatekeeper;
  }

  location /xrpc/com.atproto.server.updateEmail {
    include sites-available/proxy.inc;
    proxy_pass http://gatekeeper;
  }

  location /@atproto/oauth-provider/~api/sign-in {
    include sites-available/proxy.inc;
    proxy_pass http://gatekeeper;
  }
}

And reload it again:

sudo nginx -t && sudo service nginx reload

At this point the PDS should be fully operational, including email 2FA when you try to log in 🎉

For creating accounts and other admin stuff, you can use the pdsadmin scripts from the source code dir, but you need to pass a PDS_ENV_FILE env var with the path to pds.env:

cd /var/www/pds
PDS_ENV_FILE=/var/lib/pds/pds.env bash pdsadmin/account.sh list

Backups

There’s one last thing – it would also be nice to have backups of your nearly 3 years worth of Bluesky posts…

I do it like this:

# create a backup user
sudo adduser archivist

# add a separate SSH key from local Mac
sudo mkdir /home/archivist/.ssh
echo "ssh-ed25519 ... archivist@martianbase.net" | sudo tee /home/archivist/.ssh/authorized_keys
sudo chown -R archivist:archivist /home/archivist/.ssh

sudo nano /usr/local/sbin/backup_pds:

#!/bin/bash

set -e

service pds stop
rsync -rlpt --exclude="pds.log*" /var/lib/pds/ /var/backups/pds
chmod -R g+rX /var/backups/pds
chown -R root:archivist /var/backups/pds
service pds start

sudo chmod a+x /usr/local/sbin/backup_pds

sudo nano /etc/cron.d/backup_pds:

10 10 * * *   root   /usr/local/sbin/backup_pds

And then I have a cron job on the local Mac which runs this backup script:

#!/bin/bash

SSH_COMMAND="ssh -i ~/.ssh/archivist"

rsync -rltq -e "$SSH_COMMAND" "archivist@lab.martianbase.net:/var/backups/pds/" pds

This way, at one point during the day the PDS data is copied to another folder on the server, shutting down the PDS for a moment to make sure the .sqlite files don’t end up corrupted, and then at some point later, my Mac separately rsyncs the copy of the data from that second folder, while the files aren’t being actively written to. This obviously doubles the data size requirements, so it wouldn’t be feasible for a larger PDS, but it’s totally fine on mine.

And that’s it. A bit more than curl | bash, but I now control every piece of it and I can change them as I please. The old-school way, as God intended 😎

ATProto blog posts collection

2025-11-18T18:59:04Z

I come across a lot of blog posts about the AT Protocol and Bluesky technicals – both on Bluesky official blogs and those of the team members, and by independent developers from the community. So many people are blogging now (especially now that Leaflet got popular in these circles) that I started using an RSS reader again just to keep up with everything.

These posts are usually shared widely for a day or two, and then kind of forgotten – but a lot of them contain some valuable knowledge that is still relevant much later. Even if someone remembers that something like this has been written, it’s not always easy to dig it out from the archive.

I thought it would be nice to have one place collecting those old and newer blog posts to make them easier to find. So I went through those RSS feeds, my like archives and other places, and collected everything I could find here in an organized list. I also included the documents from the “Proposals” GitHub repo, and various posts from the “Discussions” section in the ATProto repo.

This is a subjective selection – from many blogs I skipped some less relevant posts or only included a couple out of many – so if you’re interested, click through to the home page from any post and look for the other posts there.

Search posts by title:

Bluesky official sources

atproto.com

ATProto for distributed systems engineers (Sep 2024)
Atproto Ethos (Apr 2025)

bsky.social/about/blog

Composable Moderation (Apr 2023)
How to verify your Bluesky account (Apr 2023)
Federation Architecture Overview (May 2023)
Bluesky: An Open Social Web (Feb 2024)
Bluesky’s Stackable Approach to Moderation (Mar 2024)
Tips and Tricks for Bluesky Search (May 2024)
Bluesky Welcomes Mike Masnick to Board of Directors (Aug 2024)
Bluesky Announces Series A to Grow Network of 13M+ Users (Oct 2024)
2024 In Review (Dec 2024)
Bluesky’s Patent Non-Aggression Pledge (Oct 2025)
What’s Next at Bluesky (Jan 2026)

docs.bsky.app

Click to expand

github.com/bluesky-social/atproto/discussions

Click to expand

2023

brainstorm ideas for Cool Developer Tools (Feb 2023)
Intention to remove repository history (Jul 2023)
Planned Changes to DID Documents (August 2023) (Aug 2023)
DID PLC Rate Limits and Validation (Sep 2023)
Upcoming Disruptive Protocol and Infra Changes (Oct 2023)
Migrating bsky.social to Multiple PDS Instances (Nov 2023)
That which we call a “BGS”, By any other name would smell as sweet (Nov 2023)
Tightening Datetime, Record Key, and TID validation (Dec 2023)

2024

Protocol Tech Debt (Feb 2024)
Summary of Recent Changes (Feb 2024)
March 2024 Protocol Updates (Mar 2024)
What does a PDS implementation entail? (Mar 2024)
What goes in to a Bluesky or atproto SDK? (Apr 2024)
OAuth Roadmap (Jul 2024)
Service auth token iteration – method binding & nonces (Aug 2024)
Brainstorming: atproto Dev Tooling and Experience (Aug 2024)
What does an AppView implementation entail? (Nov 2024)
Relay Operational Updates (Nov 2024)
Call for Developer Projects (Nov 2024)
RFC: Lexicon Resolution (Nov 2024)
Account Lifecycle Best Practices (Dec 2024)
Account Migration Details (Dec 2024)

2025

Proposal: OAuth Scopes (Mar 2025)
Relaxing DID PLC Verification Method Constraints (Jul 2025)
OAuth Client Security in the Atmosphere (Jul 2025)
Adding internal repositories to Bluesky’s workflows (Jul 2025)
Progress on Auth Scopes Implementation (August 2025) (Aug 2025)
Draft Lexicon Style Guide (Lexinomicon) (Oct 2025)
PLC Operational Updates (Oct 2025)
Lexicon Language Corner Cases (Nov 2025)
Early Permission Sets (Dec 2025)

2025

PLC Export API Update (Jan 2026)
Adding content-disposition on getBlob HTTP responses (Feb 2026)

github.com/bluesky-social/proposals

0001: User Lists, Reply-Gating, and Thread Moderation (Jun 2023)
0002: Labeling and Moderation Controls (Jun 2023)
0003: Hashtags (Jun 2023)
0004: OAuth 2.0 for the AT Protocol (Feb 2024)
0005: Ozone Moderation History (Oct 2024)
0006: AT Protocol Sync v1.1 (Feb 2025)
0007: Moderation Report Routing (Feb 2025)
0008: User Intents for Data Reuse (Mar 2025)
0009: Moderation Report Granularity (May 2025)
0010: Client assertion backend for browser-based applications (Jun 2025)
0011: Auth Scopes for ATProto (Jun 2025)
0012: Infrastructure Abuse Notices (Nov 2025)

Bluesky team

jaygraber.medium.com (Jay Graber, CEO)

Web3 is Self-Certifying (Dec 2021)

pfrazee.com (Paul Frazee, CTO)

Why RichText facets in Bluesky (Jan 2024)
Why not RDF in the AT Protocol? (Jan 2024)
Why isn’t Bluesky a peer-to-peer network? (Jan 2024)
Guidance on Authoring Lexicons (Mar 2025)
Atmospheric Computing (Dec 2025)
Practical Decentralization (Feb 2026)

pfrazee.leaflet.pub (Paul Frazee, CTO)

We probably need to rename the AppView (Sep 2025)
Update on Protocol Moderation (Sep 2025)
Private data: developing a rubric for success (Sep 2025)
Three schemes for shared-private storage (Sep 2025)
Social platforms are not neutral (Oct 2025)
The politics of purely client-side apps (Nov 2025)

bnewbold.net (Bryan Newbold)

Progress on atproto Values and Value Proposition (Aug 2024)

whtwnd.com/bnewbold.net (Bryan Newbold)

Notes on Running a Full-Network atproto Relay (Jul 2024)
Migrating PDS Account with ‘goat’ (Oct 2024)
Reply on Bluesky and Decentralization (Nov 2024)
Registering Identity Recovery Keys via PDS, using goat (Feb 2025)
A Full-Network Relay for $34 a Month (May 2025)
AT Moderation Architecture (Oct 2025)
Creating a did:web atproto account using goat (Jan 2026)

bnewbold.leaflet.pub (Bryan Newbold)

AT Namespaces for Community Spaces (Oct 2025)
Record Versioning (Nov 2025)
Big Indexing (Dec 2025)
Community Spaces on AT Protocol (Feb 2026)

dholms.leaflet.pub (Daniel Holms)

PLC Threat-modeling & Auditability (Dec 2025)
Permissioned Data Diary 1: To Encrypt or Not to Encrypt (Feb 2026)
Permissioned Data Diary 2: Buckets (Feb 2026)

jimray-bsky.leaflet.pub (Jim Ray)

The Importance of Backfillability (Dec 2025)

mozzius.dev (Samuel)

React Native, and “the native feel” (Oct 2024)
ATProto by example part 1: Records and Views (Mar 2025)

emilyliu.me (Emily Liu, ex-Bluesky)

Using Bluesky posts as blog comments (Nov 2024)

jazco.dev (Jaz, ex-Bluesky)

Solving Thundering Herds with Request Coalescing in Go (Sep 2023)
Scaling Go to 192 Cores with Heavy I/O (Jan 2024)
Your Data Fits in Memory (GraphD Part 1) (Apr 2024)
An entire Social Network in 1.6GB (GraphD Part 2) (Apr 2024)
How HLS Works (Jul 2024)
Jetstream: Shrinking the AT Proto Firehose by >99% (Sep 2024)
When Imperfect Systems are Good, Actually: Bluesky’s Lossy Timelines (Feb 2025)
Turning Billions of Strings into Integers Every Second Without Collisions (Sep 2025)

ATProto community

whtwnd.com/futur.blue (Futur)

atproto relay any% speedrun (Mar 2025)
in and out, quick appview adventure (Jun 2025)

overreacted.io (Dan Abramov, ex-Bluesky)

Open Social (Sep 2025)
Where It’s at:// (Oct 2025)
A Social Filesystem (Jan 2026)

underreacted.leaflet.pub (Dan Abramov, ex-Bluesky)

we can just do things (Oct 2025)

bsky.bad-example.com (Phil)

consuming the jetstream firehose correctly (Feb 2025)
Can atproto scale down? (Feb 2025)

mackuba.eu 🙃

A complete guide to Bluesky (Feb 2024)
Introduction to AT Protocol (Aug 2025)
Running Bluesky PDS undockered (Feb 2026)

shreyanjain.net (Shreyan)

Nostr and ATProto (Jul 2024)

jcsalterego.leaflet.pub (Jerry Chen)

We live in a space station (Oct 2025)

da.vidbuchanan.co.uk (David Buchanan, @retr0.id)

Hijacking Bluesky Identities with a Malleable Deputy (Sep 2023)
Adversarial ATProto PDS Migration (Jul 2025)

marvins-guide.leaflet.pub (Bailey Townsend)

Host a PDS via a Cloudflare Tunnel (Jul 2025)
A blob in the bucket (Aug 2025)
What the hell is the atmosphere anyway (Sep 2025)
What the hell is a rotation key? (Nov 2025)

dame.is (Dame)

How I made an automated dynamic avatar for my Bluesky profile (Feb 2025)
Creating a decentralized bathroom (powered by the AT Protocol) (Mar 2025)
A guestbook and welcome message for my atproto PDS (May 2025)

steveklabnik.com (Steve Klabnik)

How Does BlueSky Work? (Feb 2024)

knotbin.leaflet.pub (Roscoe Rubin-Rottenberg)

Wherever you get your Podcasts (Aug 2025)

blog.smokesignal.events

Creating a did-method-web Identity for ATProtocol (Aug 2025)

graysky.app

Adding comments to this blog (Feb 2024)

Other recommended blogs

Connected Places newsletter (Laurens Hof)
Connected Places Leaflet (Laurens Hof)
How Streamplace Works (Eli Mallon)
Tangled engineering (@tangled.org)
Nick’s Blog (Nick Gerakines)
Bailey’s Weekly Retrospective (Bailey Townsend)
dame’s leaflets (Dame)
icy takes (Anirudh Oppiliappan)
Juliet’s rambles (Juliet)
mildly at-musing (Samuel)
The Smoke Signal blog (@smokesignal.events)
Matthieu’s Leaflet (Matthieu Sieben)
augment (Anuj Ahooja)
Tree For You (Tree)
A New Social (Bridgy Fed)
Kuba’s Lab Notes 😉

How I ran one Ruby app on three SQL databases for six months

2025-10-15T19:11:16Z

Since June 2023, I’ve been running a service written in Ruby (Sinatra) that provides several Bluesky custom feeds (initially built with a feed for the iOS/Mac developers community in mind, later expanded to many other feeds). If you don’t know much about Bluesky feeds, you make them by basically running a server which somehow collects and picks existing posts from Bluesky using some kind of algorithm (chronological or by popularity, based on keyword matching, personal likes, whatever you want), and then exposes a specific API endpoint. The Bluesky AppView (API server) then calls your service passing some request parameters, and your service responds with a list of URIs of posts (which the API server then turns into full post JSON and returns to the client app). This lets you share such feed with anyone on the platform, so they can add it to their app and use it like any built-in feed. (If you’re interested, check out my example feed service project.)

In order to provide such service, in practice you need to connect to the Bluesky “firehose” streaming API which sends you all posts made by anyone on the network, and then save either those which are needed for your algorithm, or save all of them and filter later. I chose the latter, since that lets me retry the matching at any time after I modify the keyword lists and see what would be added after that change (and also some of the feeds I now run require having all posts). I also use the same database/service to generate e.g. the total daily/weekly stats here.

All posts made on Bluesky is much less than all posts on Twitter, of course, but it’s still a lot of posts. At the moment (October 2025), there are around 3.5M posts made on average every day; at the last “all time high” in November 2024, it was around 7.5M per day. A post is up to 300 characters of (Unicode) text, but since I also store the other metadata that’s in the record JSON, like timestamp, reply/quote references, embeds like images and link cards, language tags etc., it adds up to a bit less than 1 KB of storage per post on average.

In addition to that, the firehose stream (if you use the original CBOR stream from a relay, not Jetstream, which is a JSON-serving proxy) includes a lot of overhead data that you don’t need in a service like that, plus all the other types of events like handle changes, likes, follows, blocks, reposts, and so on. The total input traffic is around 15 Mbit/s average right now in October 2025 (or around 5 TB per month), and it used to be around twice that for a moment last year. (Jetstream sends around an order of magnitude less, especially if you ask it to send filtered data, e.g. only the posts.)

On disk, the millions of posts per day add up to a few gigabytes per day. Since I was running this on a VPS with a 256 GB disk (Netcup, RS 1000 – reflink), I have a cron job set up to regularly prune all older posts and keep only e.g. last 40 days worth of them (since I don’t really need to keep the older posts forever), so around 200-ish gigabytes total, and around 200 millions of rows in the posts table.

And until March this year, I was keeping all this data in… SQLite 🫠

Chapter 1: You’re probably wondering how I ended up in this situation

I think I had never used SQLite in a web app before this – I normally always used MySQL, since that was commonly used in PHP first and then in Ruby webapps (Rails/Sinatra w/ ActiveRecord). I was used to it, I knew how it worked more or less, and I always thought SQLite was only meant to be used in embedded scenarios like native desktop/mobile apps. But the example feed-generator project in JS shared by Bluesky used SQLite, and since I started out by porting that to Ruby, I ended up also using SQLite, planning to switch to something else later. But you know how it is, that “later” time never comes – and if it ain’t broke, don’t fix it. SQLite worked surprisingly well for much much longer than I expected, with much more data than I expected, and it turns out it can be absolutely ok for server/webapp database purposes, at least in some scenarios. But eventually I started hitting some limitations.

The main problem is that SQLite doesn’t allow concurrent write access. This means that many processes can read posts or other data simultanously, but only one process at a time can write to the database. This could be a serious problem in most webapps, but it worked fine in this particular architecture. You see, there’s really one entry point to the system where the posts are saved: the firehose stream consumer process. All posts come from there, and nothing else saves posts, the Sinatra API server only makes queries to what was already saved from the firehose. This is why this has worked for me for as long as it did.

However… at some point I added a second parallel thread, which separately reads data from the plc.directory and saves data about some accounts' handles and assigned PDS servers. There are also cron jobs running scripts and Rake tasks, which sometimes modify data and sometimes take a bit to run (like that older posts cleaner), and I sometimes run Rake tasks manually to e.g. rebuild feeds.

And this is where I started running into a second related problem: how this concurrent writing is/was handled in ActiveRecord. I don’t know if I can explain this all correctly (see this GitHub issue and this StackOverflow comment), but the gist is, ActiveRecord used some mode of data locking in SQLite, which resulted in a flow like this:

A transaction is started which locks the data for reading.
Some records are loaded and turned into AR models (through a where/find_by call etc.).
Some updates are made to the model objects.
When I call save on the model, AR tries to change the lock mode that would allow it to also make a write.
But something else has made other writes in the meantime.
Because of how the transaction/locks were set up, SQLite decides that the data I’m trying to save might have been modified in the meantime, and aborts the whole transaction and throws an error (SQLite3::BusyException: database is locked).

Changing timeout settings doesn’t fix the problem, because the exception is thrown immediately, without waiting for the other process to finish. What worked around the problem was a mix of:

trying to avoid doing multiple writing operations in parallel at all

rearranging the code a bit artificially so it opens a transaction and then first does some kind of write, even a completely pointless one, before doing the reads, which makes it create the right kind of lock from the beginning:

def rescan_feed_items(feed)
  ActiveRecord::Base.transaction do
    # lulz
    FeedPost.where(feed_id: -1).update_all(feed_id: -1)

    feed_posts = FeedPost.where(feed_id: feed.feed_id).includes(:post).order('time')
    feed_posts.each do |fp|
      ...
    end
  end
end

or so that it does the write without touching the original model, e.g.:

# instead of:
@post.update(thread_id: root.id)

# do:
Post.where(id: @post.id).update_all(thread_id: root.id)

This has mostly let me avoid the problem for a long time, but this meant it kept popping back up sometimes, and I had to write some code sometimes in a way that didn’t logically make sense and was only like that to avoid the exceptions. In particular, with the plc.directory thread, I had to make it pass any changes back to the main thread through a queue so they can be saved to the database from there.

(Ironically, the ActiveRecord folks finally fixed this whole problem in the 8.0 release, just as I finished migrating away from SQLite…)

A second problem was that I started having some performance issues that I couldn’t find a good solution for – e.g. post write operations were occasionally randomly taking e.g. 1 or 2 seconds to finish instead of milliseconds; and I wanted to optimize the app to be able to potentially save as many posts per second as possible, to prepare for larger traffic in the future.

When I was asking for advice, everyone was telling me “dude, just switch to Postgres” 😛 But I haven’t really worked with Postgres before other than briefly, and I knew some things were different there, and I wasn’t sure if I want to switch to something unknown rather than what I knew (MySQL).

And since I have way too much time, no life, and probably a good bit of neurodivergence, I chose the most obvious solution: set up the app on both MySQL and on Postgres on two separate & identical VPSes, and compare how it works in both versions… 🫣

Chapter 2: Getting it to run

Turns out, SQL databases are a bit different from each other in a lot of aspects, and migrating a webapp from one to the other is a bit more work than just editing the Gemfile and database.yml – who would’ve guessed…

Beyond the obvious, here are some things I had to change on the MySQL branch:

Some column type changes:

integer column sizes – in SQLite, all numbers are just integers of any size, so here I changed some to smallint and some to bigint
similarly, some string columns were changed to text
for datetime columns, I’ve set the decimal precision explicitly to 6 digits (this is now the default since AR 7.0, I started on 6.x for some reason)

In queries:

I removed some index hacks like .where("+thread_id IS NULL”), which tell the SQLite query optimizer to use/not use a given index
some date operations had to be rewritten to use different functions, e.g. DATETIME('now', '-7 days') to SUBDATE(CURRENT_TIMESTAMP, 7)
some queries had to be rewritten or updated because they were just throwing SQL syntax errors – e.g. I had to explicitly list table names on some fields in SELECT; or there was this thing in MySQL where I had to nest a subquery for DELETE one level deeper
ActiveRecord::Base.connection.execute returns rows as arrays instead of hashes for some reason, indexed by [0] not by ['field']

For Postgres, in addition to most of the above:

I had to replace 0 / 1 used as false/true in boolean columns with an explicit FALSE / TRUE
date functions in queries were slightly different again, e.g. DATE_SUBTRACT(CURRENT_TIMESTAMP, INTERVAL '7 days')
strings in queries had to be changed to all be in single quotes, not in double quotes, which in Pg are reserved for field names like "post_id" (what SQLite and MySQL use the backticks for: `post_id`)

I could also finally remove all the code hacks added to work around the SQLite concurrency issues – start normally saving handles in the PLC importer thread, rewrite some transaction blocks to a more logical form, and so on.

Chapter 3: Migrating the data

For migration to MySQL, I used the sqlite3mysql tool, written in Python:

pip install sqlite3-to-mysql

sqlite3mysql -K -E -f bluesky.sqlite3 -d bluefeeds_production -u kuba -i DEFAULT -t feed_posts handles post_stats subscriptions unknown_records ...

For the posts table (which is the vast majority of the database size), I used the -c (--chunk) option to import posts in batches:

sqlite3mysql -K -E -f bluesky.sqlite3 -d bluefeeds_production -u kuba -i DEFAULT -t posts -c 1000000

For Postgres, I used pgloader. Unlike sqlite3mysql, it isn’t configured through command-line flags, but instead you need to write “command” files with a special DSL and then pass the filename in the argument. So my command files looked something like this:

load database
from sqlite://./db/bluesky.sqlite3
into postgresql:///bluefeeds_production

with data only, truncate
including only table names like 'feed_posts', 'handles', 'post_stats', 'subscriptions', 'unknown_records';

I’ve split the tables into several command files, because I wanted to do it a bit more step by step since some of the imports were failing.

For the posts table, I’ve similarly set a “prefetch rows” flag to do it in batches:

load database
from sqlite://./db/bluesky.sqlite3
into postgresql:///bluefeeds_production

with data only, truncate, prefetch rows = 10000
including only table names like 'posts';

Using the two importers are two very different experiences: sqlite3mysql takes quite a lot of time to import, but shows very nice progress bars and remaining time estimates; pgloader gives you basically no updates until it’s finished, and even if some tables fail to import, it’s not immediately clear from the summary what happened – however, it does the job in as much as two orders of magnitude less time 😅 I don’t know how much this is because of the tool or the database though (and there are probably ways to speed up the import in MySQL, e.g. by dropping indexes first).

One surprise realization I had during the import: in SQLite, when you define a column as string with limit 50, it doesn’t actually enforce that limit! Apparently I had a whole bunch of records in some tables where the values were much longer than the expected max length… because I was missing Ruby-side AR validations (validates_length_of) in some models – and those records were being rejected by both new databases. So I had to add all those missing length validations and clean up the invalid data first.

An additional problem cropped up in MySQL, which has different text collation rules depending on accents and unicode normalization than SQLite & Postgres. I have a “hashtags” table listing all hashtags that appeared anywhere in the posts, with a unique index on the hashtag name – but the import to MySQL was failing, because some hashtags were considered by MySQL as having the same text as some others, while SQLite had considered them different… I tried to pick a different collation for the table (utf8mb4_0900_as_cs, i.e. both accent-sensitive and case-sensitive), but that only partially helped with some name pairs (“pokemon” vs. “pokémon”), but not with others (different normalization, or invisible control characters, and there are *countless* different types of those, as I have learned…). I eventually gave up and ended up just dropping the unique index for now.

In Postgres, in turn, the problem was that it apparently doesn’t support strings that contain null bytes, and there are some occasional posts that somehow end up with a \0 in the post text… So I had to just filter out such posts as invalid.

if text.include?("\u0000")
  return
end

Finally, something that somehow caused an issue in both versions was a thing that every programmer loves – timezones… When I made a query to count posts added in the last 5 minutes, and it always returned 0, but “last 65 minutes” didn’t, I immediately knew what was going on 🫠

In Postgres, the solution was to tell ActiveRecord to use the timestamptz data type for timestamp columns instead of the default timestamp:

ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.datetime_type = :timestamptz

(and then migrate the existing columns to use that type).

In MySQL, you need to either tell AR to use the timezone the database is using, if it’s not set to UTC:

ActiveRecord.default_timezone = :local

Or tell the database to use UTC:

[mysqld]
default-time-zone = '+00:00'

(but not both…)

Chapter 4: Optimizing

To be able to test both databases with real production traffic in a way that would let me compare them in a fair competition, I’ve set up a kind of “database A/B test” 🙃

The feed was configured to load data from my original SQLite server as before, but in the request handler on that server, instead of calling the feed class locally, the code picked one of the two other servers based on either the “DID” (user ID) of the caller or current timestamp, proxied the call to that server, took the response, and returned it back to the caller:

get '/xrpc/app.bsky.feed.getFeedSkeleton' do
  feed_key = params[:feed].split('/').last

  if ['hashtag', 'follows-replies'].include?(feed_key)
    server = Time.now.hour.odd? ? SERVER_A : SERVER_B
  elsif ['replies'].include?(feed_key)
    server = Time.now.hour.even? ? SERVER_A : SERVER_B
  else
    did = parse_did_from_token
    server = did.nil? || did =~ /[a-m0-4]$/ ? SERVER_A : SERVER_B
  end

  url = "https://#{server}/xrpc/app.bsky.feed.getFeedSkeleton?" + request.query_string
  headers = env['HTTP_AUTHORIZATION'] ? { 'Authorization' => env['HTTP_AUTHORIZATION'] } : {}
  response = Net::HTTP.get_response(URI(url), headers)

  content_type :json
  [response.code.to_i, response.body]
end

This meant that each feed load took a bit longer, but it wasn’t very noticeable in practice, and I had the real traffic split into two more or less equal parts, going to the two servers. (Coincidentally, it’s exactly one year today since I deployed that change – I’ve been procrastinating way too long on this blog post 🫠)

And then started my months-long work of optimizing the databases and queries…

Some problems showed up immediately: a query was returning data immediately in SQLite, and in MySQL or Postgres it just hangs. So in those cases, I often had to add some missing index, or modify the index somehow (e.g. add an additional field, or switch a field in a composite index to DESC), or rearrange a query to make it use the intended index, add some limiting condition, or occasionally (in MySQL) add FORCE INDEX. Those issues were generally fairly easy to fix, and the fix either clearly worked or not. I think some queries I had were just logically not fully thought through, but they had been working fine before because SQLite has some things organized differently on disk and some access patterns work better, hiding the issue with the query.

The bigger problem and one I’ve spent a ton of time on (in the Postgres version) was one specific query in a set of “replies feeds”. I mostly wrote everything about in on my Journal blog on micro.blog back in January, and I remembered much more about it then than I do now, so I’ll just link to that old blog post here.

The TLDR is that I have a set of three feeds: Follows & Replies, Only Replies and Only Posts, which share the same code, just with slightly different filters; these are personalized feeds (i.e. having different content depending on who’s loading them), which for a given user fetch the list of the accounts that user is following, and then make a query asking the database for “most recent N posts from any of the users that this account follows” – so basically a reimplementation of the standard “Following” feed, with some changes.

This query was much slower on the Postgres server than on the other two databases, and Postgres insisted on using the posts index on (time) (scanning possibly millions of rows to find the right ones) instead of using the one on (user, time) some number of times and merging the results (and apparently asking “how to do FORCE INDEX in Postgres” is a terrible heresy 😛). I spent a lot of time on this and it took me a lot of trial and error, asking more experienced people on Bluesky, reading docs and articles, chatting with ChatGPT, and so on. I went through: bumping up the STATISTICS target and/or hardcoding n_distinct (and rolling those back), tweaking some configuration variables, rearranging the query/index in various ways – and what I finally settled on was:

changing the (user, time) index to also include the id primary key as the third field, i.e. (user, time DESC, id), to let it do “Index Only Scans” on it – I haven’t realized that indexes in Postgres don’t reference the primary key, so they can’t be used this way unless the id is explicitly included there!
and setting up a cron job to do very frequent manual VACUUM (4× a day, with forced index_cleanup), because otherwise it has to re-check some of the ids fetched from the index to verify if the rows haven’t been deleted

After those changes, it finally started working really nicely on Postgres, with the mean response time from this query going below 20 ms, while the MySQL version was doing around 50 ms, and the initial Postgres version before the index changes had slowed down to as much as 200-300 ms mean time. (It still occasionally picks the wrong index, for accounts that are somewhere in the middle follows range, over 1000 followed accounts – for those with many thousands, the (time) index is almost always better – but I think that’s somewhat unavoidable.)

Epilogue: There can be only one

In the end, after all the tweaks and optimizations, both servers on both databases were working quite fine, and I think I would probably be ok with either of them. But I had to pick one.

I ended up picking…

…

Postgres! 🏆

In those few months, I managed to read sooo many pages of the documentation, articles about various specific settings, spent so much time in the psql console, reading output from the analyzer, that I got much more comfortable with it than I was at the beginning… So ironically, the fact that I had to spend more time tweaking it to get it to work all smoothly made me prefer it in the end. I felt like specifically because there were so many different dials and switches, I felt more “in control” with Postgres than with MySQL – the tutorials for tuning Postgres mentioned 5-10 different settings at least, and the ones for MySQL basically said “ah, just set innodb_buffer_pool_size to half the RAM and you’re done”. And if you’ve already set that and you’d like to optimize things further? Well… ¯\_(ツ)_/¯

Postgres’s query analyzer output is also more readable and helps you more with figuring out how it’s actually handling the query, and generally various debugging commands seem to provide more readable info about current parameters of the system – while MySQL mostly has a SHOW ENGINE INNODB STATUS command, which just pukes several pages of text output at you.

I’ve been doing various manual benchmarks of how different parts of the system work, what the average response times and query times are and so on, on both versions, and keeping the results in tables, but I can’t really get any general conclusion from this on which database is better at what. It was often things like: one does single row deletes faster and the other does multi-row deletes faster, or one is faster at inserts and the other at counts… but generally it was changing a bit too much over time, and I wasn’t doing it in a super scientifically controlled way.

One thing that I could see on Munin charts in the end was that the Postgres server had higher numbers on the disk read IO/throughput charts, while the MySQL server had noticeably higher numbers on the disk write IO/throughput charts. Not sure if this is a good assumption, but my guess was that with higher traffic in the future, it would generally be easier to scale the read load in Postgres (with various caches, replicas etc.) than to scale the write load in MySQL. Also, in the end after all the optimizations, the key query in the “replies” feeds was working noticeably better in the Postgres version, and in the test “how quickly it can possibly process events when catching up at max speed” (where post record inserts are generally the bottleneck), the Postgres version also ended up with a slightly higher processing speed.

So since March, the app has been running on a new 512 GB VPS on a Postgres database, and it’s been working fine since then.

Introduction to AT Protocol

2025-08-20T17:32:45Z

(Last update: 22 Apr 2026.)

Some time ago I wrote a long blog post I called “Complete guide to Bluesky”, which explains how all the user-facing features of Bluesky work and various tips and tricks. This one is meant to be a bit like a developer version of that – I want to explain in hopefully understandable language what all the pieces of the network architecture are and how they all fit together. I hope this will let you understand better how Bluesky and the underlying protocol works, and how it differs from e.g. the Fediverse. This should also be a good starting point if you want to start building some apps or tools on ATProto.

This post is a first part of a series – next I want to look at some comparisons with the Fediverse and some common misconceptions that people have, and look at the state of decentralization of this network, but that was way too much for one post; so this one focuses on the “ATProto intro tutorial” part.

What is “Bluesky”? Which “Bluesky” are we talking about?
Records & blobs
Lexicons
Identity
AT URIs
User repositories
XRPC
Rich text / facets
PDS
Relay
AppView (app server)
Labellers
Feed generators
Client apps
DMs
How it all fits together
Non-Bluesky Atmosphere apps
Where to go next

What is “Bluesky”? Which “Bluesky” are we talking about?

Before we start, a little philosophical aside. Discussions about Bluesky sometimes get a little confusing because… “Bluesky” could mean a few different things – language is hard.

(I’m not sure if this section clears things up or just the opposite, so feel free to skip it.)

First, we have Bluesky the company, the organization. Usually, when people want to clarify that they’re talking about the group of people or the company, they say “Bluesky PBC” (PBC = Public Benefit Corporation), or “Bluesky team”. (If you want to read a bit about where Bluesky came from and what’s the current state of the company, read these two sections in the Bluesky Guide blog post.)

But Bluesky is also the thing they built, the social network, and that one is much more nebulous and hard to define – because unlike the centralized social networks like Twitter or Facebook, Bluesky has built theirs on a distributed protocol they’ve called the Authenticated Transfer Protocol, or AT Protocol (or even shorter, ATProto). More and more parts of what people generally experience as “Bluesky” are actually built, run and controlled by people from outside, independently of the company.

To complicate things further, Bluesky even in this wider meaning is not all that the AT Protocol is designed to achieve, far from it. The protocol is meant to be a system for building many different social networking apps (and other kinds of apps), which share the underlying mechanism of storing and sharing data, the way accounts and servers are organized and so on, but could be dealing with very different kinds of data and content for very different use cases. The commonly accepted term for the whole shared “multiverse” of all such ATProto apps (like “The Fediverse” for the ActivityPub network) is “The Atmosphere”, or “ATmosphere” with a capital T (the name was coined by someone from the community, but was accepted by the team and is now mentioned on the official atproto site).

How do we define which parts of the Atmosphere or ATProto are “Bluesky”, and which aren’t?

The boundary is generally drawn by the names of the “lexicons”, the identifiers of the data types and API endpoints. I talk about it in a later section, but these identifiers used for record types and endpoints follow the “reverse domain name” format, so naturally different people/teams building on the protocol use identifiers under different domains that they control. And so: com.atproto signifies generic, base parts of the protocol meant to be used by everyone; app.bsky are the parts related to Bluesky apps and everything compatible with them; and people building completely separate apps will use other, different prefixes for their identifiers. So “Bluesky” would be primarily everything identified with an app.bsky.* lexicon – the data types like Bluesky posts and follows, the APIs for handling them and for accessing other Bluesky-specific features, the rules according to which they all work together, and the whole “social layer” that is created out of all of this, the virtual “place” – the thing that people have in mind when they say “this website”, even when it’s accessed through a mobile app. Other apps use the same protocol primitives, but have their separate data types, APIs, different rules and UIs.

And how do we call these different “neighborhoods” in the Atmosphere, these sets of “data types + rules + custom servers + client apps” that define different use cases of the network? Well… that’s a good question that I don’t have a good answer for. Bluesky team generally calls them just “apps” and that tends to be the name used in most cases, but especially in case of app.bsky, I find this term more confusing than helpful. This is because “app” kind of implies a client app, or maybe a webapp, so just the front part of the whole thing. There are multiple mobile apps and independent webapps made by other developers from the community, which are still built for making and displaying Bluesky posts and threads, so I consider them also a part of the Bluesky… app? See, that just doesn’t work here. I sometimes use the term “service”, though that in turn kind of implies a specific backend. Daniel from the Bluesky team has recently been calling it a “social modality” in his blog posts about adding private data to the protocol, but that’s probably too technical. So I guess we’re stuck with some combination of the above.

Now, let’s start with defining the various building pieces of the protocol:

Records & blobs

The most basic piece of the ATProto world is a record. Records are basically JSON objects representing the data about a specific entity like a post or profile, organized in a specific way. A post/reply, repost, like, follow, block, list, entry on a list, user profile info – each of these is one record. Most public actions you take on Bluesky, like following someone or liking a post, are performed by creating a record of an appropriate type (or editing/deleting one created before).

For example, this is a post record. This is one of the likes of that post.

Records are stored on disk and transferred between servers in a binary format called CBOR, although in most API endpoints they’re returned in a JSON form (they are equivalent, just different encodings of the same data).

The key thing about records, which has very real consequences for user-facing features, is that you can only create and modify your own records, not those owned by others (and there are no “shared” records at the moment, each record is owned by a specific account). This means that e.g. when you follow someone, you create a follow record on your account, and that other person can’t delete your record, which is why there’s currently no “soft-blocking” feature, i.e. you can’t make someone stop following you (though you can block them). There are workarounds though, as I’ll explain later in the AppView section.

This also means that there’s often an unexpected asymmetry between seemingly similar actions: for example, getting a list of people followed by person X is very simple (they’re all X’s records, so they’re all in one place), but getting a list of all followers of X is much harder (each record is in a different place!). This is something that the AppView helps with too, as we’ll see later.

A second, complimentary way of storing user data is blobs. Blobs are basically binary files, meant mostly for storing media like images and video. For example, here is a direct link to an image blob showing a photo of when I started writing this blog post. Blobs are stored on the same server as records, but somewhat separate from them, since it’s a different type of data.

Lexicons

Each record belongs to a specific “record type” and stores its data organized in a specific structure, which defines what kinds of fields it can have with what types, what they mean, which are required, and so on – kind of like XML/JSON Schema. This schema definition which describes a given record type is called a lexicon in ATProto. (If you’re curious why make a new standard, see threads e.g. here, here, or here, or this blog post).

A lexicon needs to have an identifier (called NSID, Namespace Identifier), which uses the reverse domain name format, e.g. app.bsky.feed.post. All lexicons that are used to store the data of a specific app are usually grouped under the same prefix, e.g. Bluesky lexicons all start with app.bsky.

The structure of a given lexicon’s records is defined in a special JSON file – for example, this file defines the app.bsky.feed.post lexicon. As you can see, this is the place which for example specifies that a post’s text can have at most 300 characters (more specifically, Unicode graphemes). This also means that you can’t create a different server which would make posts longer than 300 characters that would be Bluesky-compatible and displayed on bsky.app – such posts would not pass the validation against the post record schema, and would be rejected by any server or client which performs such validation. Essentially, whover designs and controls the given lexicon, decides what kinds of data it can hold and any constraints on it. In order to store a different, incompatible type of data, you need to create a new lexicon (although you can add additional fields to a record that aren’t defined in its lexicon; many third party apps are doing that, like e.g. Bridgy Fed).

Lexicon name prefixes generally define boundaries between “apps” as in “services”, and between the “territory” that’s owned by different parties. The lexicons and endpoints defined by Bluesky are defined either under app.bsky.* – these are things specific to Bluesky the microblogging service – or under com.atproto.*, which are things meant to be used by all ATProto apps and services regardless of the use case. There are also a couple of other minor namespaces like chat.bsky.* for the (centralized) DM service, and tools.ozone.* for the open source Ozone moderation tool.

The lexicon prefix is generally (in most cases) a good way to tell if a piece of the protocol is something Bluesky-specific (specific to the Bluesky service), or something general for all ATProto. There are no record types defined in com.atproto, so things like post, profile, follow are all Bluesky-specific and under app.bsky, as are APIs for e.g. searching users, getting timelines, custom feeds and so on. Meanwhile, com.atproto APIs deal more with things like: info about a repository, fetching a repository, signing up for a new account, refreshing an access token, downloading a blob, etc.

Third party developers and teams building apps on ATProto/Bluesky, which either extend Bluesky’s features or make something completely separate, use their own namespaces for new lexicons, like blue.flashes, dev.npmx, events.smokesignal, pub.leaflet, sh.tangled, and so on. (There is a lot of nuance to whether you should use your own lexicons or reuse or extend existing ones when building things, and there have been a lot of discussions about it on Bluesky, and even conference talks. A good starting point is this blog post by Paul Frazee.)

Identity

Each user is uniquely identified in the network with their Decentralized Identifier (DID). DIDs are a W3C standard, but (as I understand) this standard mostly just defines a framework, and there can be many different “methods” of storing and resolving the identifiers, and each system that uses it can pick or create different types of those DIDs.

The format of a DID is: did:<type>:<…>, where the last part depends on the method. ATProto supports two types of DIDs, but in practice, almost everyone uses one of them, the “plc”. Each DID has a “DID document”, a JSON file (see mine) which describes the account – in ATProto at least, the document includes things such as: the assigned handles, the PDS server hosting the account, and some cryptographic keys.

An important thing to note is that DIDs are permanent; it’s the only thing that is permanent about your account, because something has to be. There needs to be some unique ID that all databases everywhere can use to identify you, which doesn’t change, and the DID is that ID. This means that you can’t change a DID of one type into another type later.

The main DID method is did:plc, where IIRC “plc” originally stood for “placeholder” (I think it was meant to be temporary until something better is designed), and was later kind of retconned to mean “Public Ledger of Credentials” 🙃 The DIDs of this type are identified by a random string of characters, which looks like this: did:plc:vc7f4oafdgxsihk4cry2xpze. The DID documents of each DID are stored in a centralized service hosted at plc.directory, which basically keeps a key-value store mapping a DID to a JSON file. It also keeps an “audit log” of the previous versions of the document (this means that, for example, the whole history of your old handles is available and you can’t erase it!). There’s also some cryptographic stuff there which, as I understand it, lets anyone verify that everything in the database checks out (don’t ask me how).

The PLC directory is probably the single most centralized and not easily replaceable element of the protocol, which has drawn some fair criticism, but the Bluesky team has some ideas for how to make it less of a problem, which they’ve been working on for some time now. In September 2025, they announced they were working on transferring the control of the directory to an independent non-profit organization located in Switzerland that would not be controlled by them; in March 2026 at the AtmosphereConf they announced the PLC organization was successfully founded and had selected initial board members. They’ve also shared some tooling for making independent replicas of the directory.

The other, rarely used DID method is did:web. Those DIDs look like this: did:web:witchcraft.systems, and the DID document is stored in a specific .well-known path on the given hostname, in this case witchcraft.systems (yes, that’s an actual TLD ;). It does not store an audit log/history like plc does.

The reason why it’s rarely used and not recommended, is because, first, it’s more complicated to create one (though that’s a solvable problem of course, see e.g. this guide); but second and more importantly, since DIDs are permanent, this means that your account is permanently bound to that domain. You need to keep it accessible and not let it expire, or you lose the account – you can’t migrate it to did:web:another.site at some point later. It gives you more independence, but at the cost of being tied to that domain you have, and this isn’t a tradeoff that most people are likely to want, and definitely not people who don’t understand what they’re getting into.

If you’re fine with that choice, you can create a did:web account and almost everything in Bluesky and ATProto should work exactly the same. “Almost”, because some services forget to implement that second code path, since it’s so rarely used 😉 but in that case, politely nudging the developer to fix the issue should help in most cases :>

Handles

What DIDs enable is that since they act as the unique identifier, your handle doesn’t have to, like it does on Mastodon. I can be @mackuba.bsky.social one day, @mackuba.eu the next day, and @mackuba.martianbase.net the week after. All existing connections – follows & followers, my posts, likes, blocks, lists I’m on, mentions in posts, etc. all work as before, because they all reference the DID, not the handle. With mentions specifically it works kinda funny, because they use what’s called a “facets” system (see later section), where the link target is specified separately from the displayed text. So you can have an old post saying “hey @mackuba.bsky.social”, where the handle in it links to my profile which is now named “@mackuba.eu”. The link still works, because it really links to the DID behind the scenes.

Unlike on Mastodon, the format of handles is just a hostname, not username + hostname. You assign a whole hostname to a specific account, and if you own any domain name, that can be your username (and if you own a well known domain name, it’s strongly recommended that you do, as a form of self-verification!).

The handle to DID assignment is a two-way link – a DID needs to claim a given handle, and the owner of the domain needs to verify that they own that DID. On the DID side, this happens in the alsoKnownAs field of the DID document (see here in mine). On the domain side, there are two ways of verifying a handle, depending on what’s more convenient to you: either a DNS TXT entry, or a file on a .well-known path.

You might be wondering how handles like *.bsky.social work – in this case, each such handle is its own domain name, and you can actually enter a domain like aoc.bsky.social into a browser and it will redirect to a Bluesky profile on bsky.app. Behind the scenes, this is normally handled by having a wildcard domain pointing to one service, which responds to HTTP requests on that .well-known path by returning different DIDs, depending on the domain. That’s not only a bsky.social thing – e.g. there’s now an open Blacksky PDS server which hands out blacksky.social handles, and there are even “handle services” which only give out handles – e.g. you can be yourname.swifties.social if you want ;)

One place where handle changes break things is (some) post URLs on bsky.app. The official web client uses handles by default in permalinks, which means that if you link to a Bluesky post e.g. from a blog post and you change your handle later, that link will no longer work. You can however replace the handle after /profile/ with the user’s DID, and the router accepts such links just fine, they just aren’t used by default. So the form you’d want to use when putting links in a blog post or article (like the one you’re reading) would be something like: https://bsky.app/profile/did:plc:ragtjsm2j2vknwkz3zp4oxrd/post/3llwrsdcdvc2s.

AT URIs

Each record can be uniquely addressed with a specific URI with the at:// scheme. The format of the URI is:

at://<user_DID>/<lexicon_NSID>/<rkey>

Rkey (record key) is an identifier of a specific record instance – a usually short alphanumeric string, e.g. Bluesky post rkeys look something like 3larljiybf22v. So a complete post URI might look like this: at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.feed.post/3larljiybf22v. You can look up at:// URIs in some record browser tools, e.g. PDSls.

AT URIs are used for all references between records – quotes, replies, likes, mute list entries, and so on. If you look at this like record, for example, its subject.uri points to at://did:plc:vwzwgnygau7ed7b7wt5ux7y2/app.bsky.feed.post/3lv2b3f5nys2n, which is the URI of a post record you can see here. Since the URIs use DIDs in the first part, handle changes don’t affect such links.

There is a great guide to AT URIs and DIDs and how to handle them written by Dan Abramov, you may want to check it out here.

User repositories

All user data (records and blobs) is stored in a repository (or “repo”). The repository is identified by user’s DID, and stores:

records, grouped by lexicon into so-called collections
blobs (stored separately from records)
authentication data like access tokens, signing keys, hashed passwords etc.

Internally, an important part of how the repo stores user records is a data structure called “Merkle Search Tree” – but this isn’t something that you need to understand when using the protocol, unless you’re working on a PDS/relay implementation (I haven’t needed to get into it so far).

You can download the records part of your (or anyone else’s!) repo as a bundle called a CAR file, a Content Addressed Archive (fun fact: the icon for the button in the Bluesky app which downloads a repo backup is the shape of a car 🚘).

The cool part is that a repository stores all data of the given user, from *all* lexicons. Including third party developer lexicons. This means that if someone has their account hosted on Bluesky servers, but uses third party ATProto apps like Tangled or Grain, Bluesky lets them store these apps’ records like Grain photos or Tangled pull requests on the same server where it keeps their Bluesky posts. (And yes, of course someone made a lexicon/tool for storing arbitrary files on your Bluesky PDS… and did it in Bash, because why not 🙃)

XRPC

XRPC is the convention used for APIs in the ATProto network. The API endpoints use the same naming convention as lexicon NSIDs, and they have URLs with paths in the format of /xrpc/<nsid>, e.g. /xrpc/app.bsky.feed.getPosts. There are similar lexicon definition files which specify what parameters are accepted/required by an endpoint and what types of data are returned in the JSON response. PDSes, AppViews, labellers and feed generators all implement the same kind of API, although with different subsets of specific endpoints. Third party apps don’t have to use the same convention, but it’s generally a good idea, since it integrates better with the rest of the ecosystem.

Rich text / facets

This one is kinda Bluesky-specific, but it’s pretty important to understand, and I think you can reuse it for non-Bluesky apps too.

The “facets” system is something used for links and possibly rich text in future in Bluesky posts. It’s perhaps a little bit unintuitive at first, but it’s pretty neat and allows for a lot of flexibility.

The way you handle links, mentions, or hashtags, is that they aren’t highlighted automatically, but you need to specifically mark some range of text as a link using the facets. A facet is a marking of some range of the post text (from-to) with a specific kind of link. If you look e.g. at this post here, you can see that it has a facet marking the byte range 60-67 of the post text as a hashtag “ahoy25”. If there was no facet there, it would just render as normal unlinked text “#ahoy25” in the post (when you see that, it’s an easy tell that a post was made using some custom tool that’s in early stages of development). It works the same way for mention links and normal URL links.

(If you’re curious why they implemented it this way, check out this blog post.)

Facets are also used for URL shortening – if you just put a long URL in the text of a post made through the API, it will be neither shortened nor highlighted. You need to manually mark it with a facet, and manually shorten the displayed part to whatever length you want.

Note that the displayed text in the marked fragment doesn’t have to match what the facet links to; this means that you can have links that just use some shorter text for the link instead of a part of the URL, in order to fit more text in one post (although in the official app, clicking such link triggers a warning popup first). E.g. some Hacker News bots commonly use this format, see this post. The Bsky app doesn’t let you create such posts directly, but some other clients like Skeetdeck do.

Likely the most tricky part is that the from-to index numbers you need to use for the ranges are counted on a UTF-8 representation of the text string, but they’re counted in bytes and not in unicode scalars, which is kinda annoying in languages like Ruby or Python which index strings by unicode scalars. This is somewhat of an unfortunate tech debt thing as I understand, and it was made this way mostly because of JavaScript, which doesn’t operate on UTF-8 natively, only in the form of byte arrays; so this means you need to be extra careful with the indexes.

Ok, now that we got through the basic pieces, let’s talk about servers:

PDS

The original copy of all user data is stored on a server called PDS, Personal Data Server. This is the “source of truth”. A PDS stores one or more user accounts and repos, handles user authentication, and often serves as an “entry point” to the network when connecting from client apps. At least in the current implementation of Bluesky, most network requests from the client are sent to your PDS, and then some of them are handled directly by the PDS, and others are proxied e.g. to the AppView server.

Each PDS has an XRPC API with some number of endpoints for things like listing repositories, listing contents of each, looking up a specific record or blob, account authentication and management, and so on. It also has a websocket API called a “firehose” (the subscribeRepos endpoint). The firehose streams all changes happening on a given PDS (from all repos) as a stream of “events”, where each event is an addition, edit, or deletion of a record in one of the repos, or some change related to an account, like handle change or deactivation.

One of the most important features of ATProto is that an account is not permanently assigned to a PDS. Unlike in ActivityPub, where if you sign up on mastodon.social, your identifier is bound to that hostname forever and it can never change, because everything uses it as the unique ID, here the unique ID is the DID. The PDS host is assigned to a user in the DID document JSON (e.g. on plc.directory), but you can migrate to a different PDS at any point, and at the moment there are even some fairly user-friendly tools available for doing that, like PDS MOOver or EU-Haul. In theory, you should even be able to migrate to a different PDS if your old PDS is dead or goes rogue, if you have prepared in advance (this is a bit more technical). If everything goes well, nobody even notices that anything has changed (on bsky.app you can’t even easily see what PDS someone is on, although there are external tools for that, like internect.info).

Initially, during the limited beta in 2023, Bluesky only had one PDS, bsky.social. In November 2023, several additional PDSes were created (also under Bluesky PBC control), and existing users were quietly all spread to a random one of those. At that point, the network was already “technically federated”, operating in the target architecture, although with access restricted to only Bluesky-run servers. This restriction was lifted in February 2024 with the public federation launch.

Since then, ATProto enthusiasts started setting setting up PDS servers for themselves, either creating alt/test accounts there, or moving their main accounts. As of April 2026, there around 2800 ~~third party~~ independent PDS servers, although most of them are very small – usually hosting one person’s main and/or test accounts, and maybe those of a couple of their friends. I have a list of them on my website, and there’s also a more complete list here (mine excludes inactive PDSes and empty accounts). There are around 70k active accounts on them in total at the moment (depending on how you count), and around 10k of them are posting at least once every week (25k including Bridgy). This is of course a tiny fraction of the total Bluesky userbase, but the fraction is steadily growing.

As you can see on my list, there’s one big PDS for Bridgy Fed, the Bluesky-Mastodon bridge service, hosting around 40k bridged accounts from the Fediverse, Threads, Nostr, Flipboard, or the web (blogs); then some number of medium-sized community PDSes that have sprung up over the course of 2025/26 (Eurosky, Blacksky, selfhosted.social), and “companion” PDSes run by various ATProto services, meant to make it easier to get started if you don’t have a Bluesky/ATProto account yet (Tangled’s tngl.sh, Spark’s pds.sprk.so, or npmx’s npmx.social); and a very long tail of servers with single-digit number of accounts. Things are changing pretty quickly on this front, so this might look different again by the end of 2026.

The vast majority of PDSes at the moment use the reference implementation from Bluesky (written in TypeScript), but there are a few alternative implementations at various levels of maturity (Tranquil written in Rust, cocoon in Go, Pegasus in OCaml, or millipds in Python, and many others), with at least a few of these being used in production with real accounts. The Bluesky’s version is very easy to set up and very cheap to run – it’s bundled in Docker, and there’s basically one script you need to run and answer a few questions.

As for the Bluesky-hosted PDSes, there’s currently around 90 of them, and each of them hosts a few hundred thousands of accounts (!). And what’s more, they keep the record data in SQLite databases, one per account. And it works really well, go figure. The Bluesky PDSes are all given names of different kinds of mushrooms (like amanita, boletus or shiitake), hence they are often called “mushroom servers”; you can see the full list e.g. here. bsky.social was left as a so-called “entryway server”, which handles shared authentication for all Bluesky-hosted PDSes (it’s a private piece of Bluesky PBC infrastructure that’s not open source and not needed for independent PDS hosters). No accounts are actually hosted on a PDS bsky.social anymore – don’t let the handles confuse you.

Relay

A relay is probably the piece of the ATProto architecture that’s most commonly misunderstood by people familiar with other networks like the Fediverse. It doesn’t help that both the Fediverse and Nostr also include servers called “relays”, but they serve a different purpose in each of them:

a relay in Nostr is a core piece of the architecture: your posts are uploaded to one or more relays that you have configured and are hosted there, where other users can fetch them from
a relay in the Fediverse is an optional helper service that redistributes posts from some number of instances who have opted in to others, in order to make content more discoverable e.g. on hashtag feeds

In ATProto, a relay is a server which combines the firehose streams from all PDSes it knows about into one massive stream that includes every change happening anywhere on the network. Such full-network firehose is then used as the input for many other services, like AppViews, labellers, or feed generators. It serves as a convenient streaming API to get e.g. all posts on the network to process them somehow, or all changes to accounts, or all content in general, from a single place.

Initially, the relay also had to keep a complete archive of all the data on the network, from all repos, from the beginning of time. This requirement was later removed in the updates in late 2024, at least partially triggered by the drastic increase in traffic in November 2024, which overwhelmed Bluesky’s and third party servers for at least a few days. Currently, Bluesky’s and other relays are generally “non-archival”, meaning that they live-stream current events (+ a buffer of e.g. last 24 or 36 hours), but they don’t keep a full archive of all repos (this change has massively lowered the resource requirements / cost of running a relay, making it much more accessible). There are no currently operating archival relays AFAIK, but one can always be set up – Bluesky has recently given a grant to one member of the community to let them build one.

Bluesky operates one main relay at bsky.network, which is used as a data source for their AppView and most other services in the ATProto ecosystem at the moment. The relay code is implemented in Go, and isn’t very hard to get up and running (especially the recent 1.1 update improved things quite a lot). Some people from the dev community have been running alternative relay services for some time, some even using custom implementations in Rust or Zig – I’m tracking a list of known public ones with some statistics on a website called pulsar.feeds.blue. I’m also running my own small relay myself, which is feeding content only from independent, non-Bluesky PDSes (so a very small part of the network traffic).

Jetstream

There is also a variant of a relay called Jetstream – it’s a service that reads from a real CBOR relay and outputs a stream that’s JSON based, better organized, and much more lightweight (the full relay includes a lot of additional data that’s mostly used for cryptographic operations and other low-level stuff). For many simpler tools and services, it might make more sense to stream data from that one instead, if only to save bandwidth. Bluesky runs a couple of instances listed there in the readme, but you can also run your own, and I’m also tracking community instances on Pulsar.

AppView (app server)

The terribly named AppView is the second most important piece of the network after the PDS. (There’s been some talk about renaming it, but nothing really conclusive, and so far most people refer to it by that name.)

The AppView is basically an API server that serves processed data to client apps. It’s an equivalent of an API backend (with the databases behind it) that you’d find on a classic social media site like Twitter. AppView streams all new data written on the network from the relay, and saves a copy of it locally in a processed, aggregated and optimized form. For example, an AppView backed by an SQL database could have a posts table with a text column, a likes table storing all likes with a foreign key post_id, probably also an integer likes_count column in posts for optimization, and so on.

The AppView is designed to be able to easily give information such as:

the latest posts from this user
all the replies in a given thread organized in a tree
most recent posts on the network with the hashtag #rubylang or mentioning “iOS 26”
how many likes/reposts has a given post received, and who made them
how many follows/followers does a given user have, and who are they
is user A allowed to view or reply to a post from user B

All this data originates from users’ PDSes and has its original copy stored there, but the “raw” record don’t always allow you to access all information easily. For example, to find out how many likes a post has, you need to know all app.bsky.feed.like records referencing it from other users, and each of those like records is stored in the liking user’s repo on that user’s PDS. Same with followers, as I mentioned earlier in the section on records, or with building threads (again, different replies in one thread are hosted in different repos), or for basically any kind of search. So having this kind of API with processed data from the entire network is essential for client apps and various tools and services built around Bluesky by other people.

AppView also applies some additional rules to the data, sometimes overriding what people post into their PDSes, since anyone can technically post anything into their PDS. For example, the AppView prevents you from looking at the profiles of people who have blocked you, at least when you’re logged in. It also hides them from your followers list, even if they have a follow record referencing you, making it seem like they don’t; and if they try to make an app.bsky.feed.post replying to you (they can create such record on their PDS!), it excludes such reply from feeds and threads, as if it never happened. Same goes for “thread gates” which lock access to threads, and so on.

The AppView is one of the few components which aren’t completely open source. Initially, the AppView used Postgres as its data store; that version is still in the public repository. In late 2023, Bluesky has migrated to a “v2” version, which uses the NoSQL database ScyllaDB instead, to be able to handle the massive read traffic from many millions of concurrent users. The upper layer with the “business logic” is kept in the public repository, while the so called “dataplane” layer that interacts directly with Scylla is not. The reason is mostly that it’s built for a specific hardware setup they have and wouldn’t be directly usable by others, while it would add some unnecesary work for the team to publish it. It’s still possible to run the AppView with the old Postgres-based data layer (and I think the team uses that internally for development), it just can’t handle as much traffic as the current live version.

This is the piece that’s hardest to run yourself, and one that requires the most resources. That said, a private AppView should be possible to run right now for under $200/month – the biggest requirement is the disk space, something on the order of 20 TB at the moment. The truly costly part is not collecting and storing all this data, but serving it to a huge number of users who would use it as a backend for the client app in daily use. An alternative full-network Bluesky AppView that is used by a few thousands of users would be fairly doable and not prohibitevely costly, but to be able to serve millions, you’ll need a lot of hardware and something more custom than the Postgres-based version.

At the moment, there is one public independent full-network Bluesky AppView, run by the Blacksky community (launched in early 2026). It uses the Bluesky AppView implementation, with some custom indexing code, and Blacksky’s fork of the Bluesky webapp at blacksky.community is configured to use their AppView as the data source. Other teams are reportedly working on their own AppView instances too. There have also been some attempts at alternative implementations, e.g. Parakeet written in Rust, or AppViewLite built in C#, which goes to great lengths to minimize the resource use.

CDN

A part of the AppView (at least the Bluesky one) is also a CDN for serving images & videos. The API responses from e.g. getTimeline or getPostThread generally include links to any media on the Bluesky CDN hostname, not directly on the PDS, even though you can fetch every blob from the PDS, since that’s the “source of truth” (although IIRC the Bluesky PDS implementation doesn’t set the CORS headers there). It’s recommended to access any media this way in order to not use too much bandwidth from the PDS.

Labellers

(Or “labelers” officially, but I like the British spelling more here, sue me ¯\_(ツ)_/¯)

We’re now getting to more Bluesky specific things (i.e. specific for the Bluesky-service, although some parts of it are ATProto-general and mentioned on the atproto.com site).

A labeller is a moderation service for Bluesky (or other ATProto app), which can be run by third parties. Labellers emit labels, which are assigned to an account or a record (like a post). Each labeller defines its own set of labels, depending on what it’s focusing on; then, users can “subscribe” to a labeller and choose how they want to handle the labels it assigns: you can hide the labelled posts/users, mark them with a warning badge, or ignore given label.

Labellers were initially designed to just do community moderation of unwanted content, e.g. you can have a service focused on fighting racism, transphobia, or right-wing extremism, and that service helps protect its users from some kinds of bad actors; or you can have one marking e.g. posts with political content, users who follow 20k accounts, or who post way too many hashtags. In practice, many existing labellers are meant for self-labelling instead, letting you assign e.g. a country flag or some fun things like a D&D character class to yourself.

The way it works technically is:

a labeller either runs a firehose client pulling posts from the relay, or relies on reports from users and/or its operating team (usually using the Ozone tool for that)
labels, which are lightweight objects (not ATProto records) are emitted from labeller’s special firehose stream (the subscribeLabels endpoint)
the AppView listens to the label firehoses of all labellers it knows about, in addition to the relay stream, and records all received labels in its database
when a logged in user pulls data like threads or timelines from the AppView, it adds relevant label info to the responses depending on which labellers the user follows
the specific list of labellers whose labels should be applied is passed explicitly in API requests in the atproto-accept-labelers header (there is a “soft” limit of 20 labellers you can pass at a time, which is why the official app won’t let you subscribe to more)
in the official app, Bluesky’s official moderation service (which is “just” another labeller) is hardcoded as one of those 20 and you can’t turn it off; when connecting from your own app or tool, you’re free to ignore it if you want

(Read more about labellers here.)

Feed generators

Custom feeds are one of the coolest features of Bluesky. They let you create any kind of feed using any algorithm and let everyone on the platform use it (even as the default feed, if they want to).

The way this system works is that you need to run a “feed generator” service on your server. In that service, you expose an API that the AppView can call, which returns a list of post at:// URIs selected by you however you want in response to a given request.

A minimal feed service can be pretty simple – the API is just three endpoints, two of which are static, and the third returns the post URIs. One “small” problem is that in order to return the post URIs, you need to have some info about posts stored up front, which in practice means that you almost always need to connect to a relay’s firehose stream and store some post data (of selected or all posts, depending on your use case).

The flow is like this:

a feed record is uploaded to your repo, including metadata and location of the feed generator service, which lets other users find your feed
when the user opens that feed in the app, the AppView makes a request to your service on their behalf
your service looks at the request params and headers, and returns a list of posts it selected in the form of at:// URIs
the AppView takes those URIs and maps them to full posts (so-called “hydration”), which it returns to the user’s app

How exactly those posts are selected to be returned in the given request is completely up to you, the only requirement is that these are posts that the AppView will have in its database, since you only send URIs, not actual post data. In most cases, feeds use some kind of keyword/regexp matching and chronological ordering, but you can even build very complex, AI-driven algorithmic “For You” style personalized feeds.

You don’t necessarily have to code a feed service yourself and host it in order to have a custom feed – there are a few feed hosting services that don’t require technical knowledge to use, like SkyFeed or Graze.

Client apps

Ok, that’s technically not a server, but stay with me…

The final piece that you need to fully enjoy Bluesky is the client app – a mobile/desktop one or a web frontend. Unlike on Fedi, where an instance software like Mastodon usually includes a built-in web frontend that is your main interface for accessing the service, the PDS doesn’t include anything like that, just a database and an API (which also means it’s much more lightweight and needs less resources). All browsing is done through a separate client, and the client always does everything through the public API – kind of like when you run a custom web client for Mastodon like Elk or Phanpy, you connect it to your instance, and you view your timeline on elk.zone.

So when you go to bsky.app, that’s what you’re seeing – a web client that connects to your PDS (Bluesky-hosted or self-hosted) through the public API, no more, no less. The official app is built for both mobile platforms and for the web from a single React Native codebase (apparently React Native on the web and normal web React is not the same thing 🧐). This has allowed the still very small frontend team (and IIRC at first it was literally just Paul) to build the app for three platforms in any reasonable amount of time and maintain it going forward. The downside is that it’s kinda neither a great webapp nor a great mobile app… But the team is doing what they can to improve it, and it’s already much better than it used to be, and tbh more than good enough for me.

There aren’t nearly as many alternative clients as there are for Mastodon, but there are several options available now – see the apps part of my Bluesky Guide blog post for links, and there is another more complete list maintained here. Apart from fully independent client implementations, there are also several forks of the Bluesky webapp project aka social-app, adding or changing various features in it, the most popular ones being blacksky.community and witchsky.app.

DMs

Notice that I haven’t mentioned DMs anywhere – that’s because they aren’t a part of the protocol at the moment. The Bluesky team wants to eventually add some properly implemented, end-to-end encrypted, secure DMs using some open standard, but they won’t be able to finish that in the short term, and a lot of people were asking for at least some simple version of DMs in the app. So they’ve decided as an interim solution to implement them as a fully centralized, closed source service. It is accessible to third-party Bluesky clients through the API (the chat.bsky.* namespace), but it’s not something you can run yourself. The team is very open about the fact that it’s not a proper replacement for something like Signal, and that for sensitive communication, you should ideally just use it for swapping contacts on Signal on iMessage and move the conversation there. They also kinda don’t want to spend too much time adding features there, because it’s considered a temporary solution, so it’s pretty basic in terms of available features.

There are also a few other closed-source helper services, like the “cardyb” they use for generating link card details, or the video service for preprocessing videos, but they’re all specific to some Bluesky use cases only and not strictly necessary to use.

How it all fits together

So the flow and hierarchy is like this:

the client app you use creates new records as a result of actions you take (new posts, likes, follows), and saves them into your PDS
your PDS emits events on its firehose with the record details
Bluesky relay and other relays are connected to the firehoses of each PDS they know about, and they pass those events to their output firehose (the PDS has a setting PDS_CRAWLERS that tells it which relays to ask to connect to it, but since that’s generally set to just bsky.network, relays can and do also connect by themselves to PDSes or other relays)
the Bluesky AppView (and other AppViews) listen to the firehose of their selected relay; it could also be multiple relays, or it could even just stream directly from PDSes, but in practice this will normally be one trusted relay
the AppView receives events that include your records, and if they are relevant, saves the data to its internal database in some appropriate representations
when other users browse Bluesky in their client apps, they load timelines, feeds and threads from the AppView, which returns info about your post from that database it saved it to; the choice of which AppView that is depends on both the PDS and the client app – the client should send an atproto-proxy header specifying that, but the PDS also has a hardcoded default in the config file

Additionally:

feed generators run by third party feed operators also stream data from Bluesky’s or some other relay and save it locally, so they can respond to feed requests from the AppView
labellers also stream data from Bluesky’s or some other relay, and emit labels on their firehoses, which get sent to the AppView (note: there is no official “labeller relay” sitting between labellers and the AppView, although one third party dev wrote one)

Note:

PDSes do not connect to each other directly, and they don’t store posts of users from other PDSes, only their own
although right now the vast majority of users use the Bluesky relay and AppView, anyone can set up their own alternative relays and AppViews, which feed from all or any subset of known PDSes
it’s absolutely possible and expected that two users using different PDSes, which use separate AppViews feeding from separate relays will be able to talk to each other and see each other’s responses on their own AppView, as long as the users aren’t banned on the other user’s infrastructure; the metaphor that’s often used to describe these relationships is that PDSes are like websites which publish some blog posts, and relays & AppViews are like search engines which crawl and index the web and then let you look up results in them – in most cases, a website should be indexed and visible in all/most available search engines

Non-Bluesky Atmosphere apps

Most of this blog post was naturally focused mainly on Bluesky the microblogging service, since it’s kind of the supermassive ~~black~~ blue hole at the center of the ATProto galaxy. But the long-term plans for the AT Protocol are much bigger than just a decentralized Twitter clone – the Atmosphere is meant to be a network of many different apps operating on many different kinds of content, interoperating in various messy and creative ways. And there is a very enthusiastic, friendly, and crazy (complimentary) community of developers and builders, who are working on building that network of apps at this very moment.

Some popular apps like this include:

blogging platforms like Leaflet or Pckt (don’t try to pronounce it “packet”)
stream.place, a streaming video site
Tangled, a GitHub alternative
Grain, a photo sharing site

So how does all of the above apply to apps like these?

each such app (if it’s not only operating on Bluesky records) needs to have some number of custom lexicons defining its records – e.g. here’s a lexicon for a Grain photo; some apps are also cooperating on creating shared lexicons reused between apps, e.g. a few of the ATProto blogging platforms have agreed on a shared blog post lexicon called standard.site
the app lets users log in using their “Atmosphere account”; these days it’s expected for new apps to authenticate using OAuth with proper granular permissions, requiring the user to only enter their handle and looking up the assigned PDS automatically (with support for both non-Bluesky PDSes and did:web), and as a nice touch, a lot of apps autocomplete the handle from the AppView as you’re typing, showing you suggestions with Bluesky avatars
in the app, users create records of a given custom lexicon, which are saved to their own PDS, whatever that is – e.g. here’s the raw record of one of my blog posts on Leaflet
an Atmosphere app is expected to work with any PDS, whether it’s Bluesky-run or self-hosted, so the app doesn’t technically need to have its own, but at least some apps choose to also run a PDS as part of their service, so when someone comes to their website or downloads their app who doesn’t yet know what “ATProto” or “Atmosphere” is and doesn’t even have a Bluesky account, they can click a “sign up” button and create an account on that PDS, without having to know what a PDS is; apps like Tangled, Spark, Pckt or npmx have their own PDSes like this. Of course, you can host any ATProto records on these PDSes, including Bluesky posts and content from other apps.
Bluesky’s PDSes are totally fine with hosting records & blobs coming from these independent apps (subject to some rate limits and abuse policy), even if the Bluesky AppView doesn’t understand them; similarly, Bluesky’s relay and other relays pass through the events about these records just fine, as long as the events are properly formatted. If you’re reading data from such relay e.g. for the purposes of a feed service, you will also receive all of these records too.
on the other hand, the Bluesky AppView (and other Bluesky-compatible AppViews, like the Blacksky one) will completely ignore records of unknown types – remember, an AppView is basically a service with an SQL database with tables like “posts” and “likes”; it would have no place to store a “Leaflet blog post” or a “Tangled pull request”, since these are not Bluesky post-shaped. So an app with its own custom lexicon needs to have its own custom AppView that’s built to index that kind of content.
at least at this stage, these apps tend to not have full XRPC APIs exposed and multiple client app options using those APIs, but rather have the client and the AppView fused together as a website with custom backend; you log in to the website using OAuth, the backend stores an access token to your account, and then the backend creates records and sends them to your PDS, while also saving them to its local database (and ideally also listening on the relay for any possible records that could have been created in some other way)
labellers and labels could be used to some degree on these apps, but in practice I haven’t really seen that happen yet
feed generators are rather Bluesky-specific at the moment

This part of the Atmosphere is much more fresh than the Bluesky-focused one, but it’s growing rapidly, so much so that’s it hard to keep up with everything!

Where to go next

And that’s about it – I think with the above, you should have a pretty good grasp of the big picture of ATProto architecture and all the specific parts of it. Now, if you want to start playing with the protocol and building some things on it, a lot will depend on what specifically you want to build and using what languages/technologies:

SDKs:

Two languages are officially supported by Bluesky:

JavaScript/TypeScript, in which most of their code is written (see the packages folder in the atproto repo)
Go, which is used in some backend pieces like the relay, or the goat command line tool used e.g. for PDS migrations (see the indigo repo)

For other languages, some community-run projects are listed on the SDKs page on atproto.com, and I have a website called sdk.blue, which lists all libraries and SDKs I know about, grouped by language. As you can see, there is something there for most major languages; there are pretty solid options for e.g. Python, Rust, Swift, C#, or Dart, and I’ve built and maintain a group of Ruby gems, which you can see on ruby.sdk.blue.

If you want to use a language that doesn’t have any libraries yet, it’s really not that hard to make one from scratch – for most things you just need an HTTP client and a JSON parser, and maybe a websocket client (implementing OAuth from scratch will definitely be a hard part though, if you need that).

Docs:

There is quite a lot of official documentation, although it’s a bit spread out and sometimes not easy to find.

The places to look in are:

atproto.com – the official AT Protocol website; a bit more formal documentation about the elements of the protocol, kind of like what I did here, but with much more info and detailed specifications of each thing. Recently (early 2026) fully redesigned, with some more practical content now too.
docs.bsky.app – more practical documentation with guides and examples of specific use cases in TS & Python (roll down the sections in the sidebar); it shows examples of how to make a post, upload a video, how to connect to the firehose, how to make a custom feed, etc.
atproto.com/blog – developer blog with updates about protocol changes
HTTP reference – a reference of all the API endpoints
something that I also find useful is to have the atproto repo checked out locally and opened in the editor, and look things up in the JSON files from the /lexicons folder

And a few other articles that might work better for you:

“ATProto for distributed systems engineers”, Bluesky’s technical overview of the server and data flow architecture
“ATProto Ethos”, also on the Bluesky blog, based on a conference talk
“What the hell is the atmosphere anyway: A slightly less technical intro to the technical side of Bluesky”, by Bailey Townsend (Sep 2025)
“Open Social” – a high-level overview of how ATProto brings back the openness of the original pre Web 2.0 web, by Dan Abramov (Sep 2025)
“Where It’s at://” – a well written guide to AT URIs and DIDs, by Dan Abramov (Oct 2025)
the “Statusphere” app example on atproto.com

Community:

Someone said recently that “bsky replies are the only real documentation for ATProto”, and honestly, they’re not wrong. We have a great community of third party developers now, building their own tools, apps, libraries, services, even organizing conferences. If you’re starting out and you have any questions, just ask and someone will probably help, and some of the Bluesky team developers are also very active in Bluesky threads, answering questions and clarifying things. So a lot of such knowledge that’s not necessarily found in the official docs can be found somewhere on Bluesky (even if you first need to find the head it’s stored in).

The two places I recommend looking at are:

the “ATProto Touchers” Discord chat – ping me or some other developer for an invite :)
my ATProto feed on Bluesky, which tries to catch any ATProto development discussions – it should include posts with any mention of “ATProto” or things like “AppView” or various API names and technical terms, or you can use #atproto or #atdev hashtag to be sure

Also, there’s a fantastic newsletter called Connected Places (formerly Fediverse Report) by Laurens Hof, who chronicles and comments on the events happening in the Bluesky/ATProto world and on the Fediverse (and *a lot* of things are happening).

Ideas:

Some easy ways to start tinkering:

use one of the existing libraries for your favorite language and make a website or command-line tool which loads some data from the AppView or PDS: load and print timelines, calculate statistics, browse contents of PDSes and repos, etc.
make a bot that posts something (not spammy!)
make a simple custom feed service using one of the available templates
connect to the relay firehose and print or record some specific types of data

Tools:

And a couple of tools which will certainly be useful in development:

internect.info – look up an account by handle/DID and see details like assigned PDS or handle history
PDSls – PDS and repository browser, lets you look up repos by account DID or records by at:// URI (there are a few others, but this one is most popular)

Changelog:

22 Apr 2026:

rewritten/reorganized the initial “What is Bluesky” section
added link to a conference talk about using Bluesky lexicons

20 Apr 2026:

mentioned some new Atmosphere apps in several places, and tools like EU-Haul
updated section about independent PDSes with more up to date info about new communities (e.g. Eurosky, selfhosted.social, npmx) and new independent PDS implementations
updated section about relays with info about community relays, independent implementations, and the Pulsar site which tracks them
updated section about AppViews with info about the Blacksky AppView
added info about the independent PLC organization
added link to the blog post tracking a list of client apps, and links to Blacksky webapp and Witchsky
rearranged some things in the “How it all fits together” section
added a whole new section “Non-Bluesky Atmosphere apps”
updated section about available libraries and SDKs
a lot of small tweaks and corrections everywhere

15 Oct 2025:

added link to Dan’s AT URI guide

26 Sep 2025:

you can now migrate back to Bluesky’s PDSes
added links to Bailey Townsend’s and Dan Abramov’s blog posts
some small corrections

Social media update 2025

2025-06-30T14:43:43Z

So here we are, halfway through 2025, a bit over 2.5 years after the Eloncalypse… For better or worse, the Twitter as we knew it in the 2010s and the communities we had there are mostly gone. But it doesn’t feel like we’ve all settled on anything comparable.

If you’re a software developer who was active on Twitter before, by now you’ve almost certainly tried at least one of the alternatives – Mastodon, Bluesky, and Threads, and you’re probably posting actively on at least one of these, but probably not on all of them. The problem is that nobody has enough mental space to be active on 3-4 similar social networks, so we’ve split into different camps which only partially overlap. You’re probably still missing some friends from Twitter and some interesting content. It’s all a bit in flux and a bit of a mess.

Myself, I’ve basically left Twitter; I haven’t spent much time on Threads (among other reasons, it was unavailable in Europe for a long time); and I’m mostly hanging out on Bluesky and somewhat on Mastodon.

So where do we go from here?

Obviously everyone has their own take on that, this is just mine. But I really think we should all try to make an effort to focus on the widely understood “open social”, or what Laurens Hof from Fediverse Report now calls “Connected places”. That means Bluesky and Mastodon/Fediverse (with emphasis on “and”), and to some degree maybe also Threads, although that depends on how their integration with ActivityPub progresses (and it’s looking more and more like they aren’t very serious about it).

My advice:

→ If you’re currently cross-posting or bridging between Mastodon and Bluesky: awesome! ❤️

→ If you’re active on Mastodon, but currently ignoring or forgot about Bluesky: please reconsider it. I know that these two communities have a lot of differences between them, and we love to hate each other (I fully admit I’m guilty of that myself). It’s likely you prefer one or the other of these for various reasons, and you might not be a fan of the other one. But I think it’s clear at this point that none of them will disappear in the near-term at least or replace the other for everyone.

It would be great if we all made some effort to connect to the other side, for those who like it more there. What are the options? Depending on what’s more convenient to you:

there are some native apps which let you post to two or more services in parallel, e.g. Croissant, Openvibe or SoraSNS
most social media management services like Buffer now support both Fediverse and Bluesky, so you can use that to post to both, including scheduling etc. There are several others like this, and they usually have some free plans.
- e.g. Fedica was one of the ones that had support for Bluesky from very early on
- also my friend from my first job, Peter Solnica (known from some Ruby libraries like DataMapper/ROM, dry-rb, Hanami, and now some Elixir libs too) is building his own called JustCrossPost
I use a little tool in Ruby I wrote for myself named tootify, which lets me selectively cross-post relevant posts to Mastodon, almost effortlessly: I post on Bluesky and then “like” my own post if I want it to be copied to Mastodon (which clears the like). So this way I can cross-post e.g. cat photos or iOS related posts, but skip Bluesky-specific content.
There are probably similar tools going in the other direction, although it’s a bit more complicated this way because of the post length limits (usually 500 on Mastodon vs. 300 on Bluesky).
I know some people have also written some iOS Shortcuts, scripts, browser extensions etc. (I don’t have any links at hand)
and last but not least, there’s Bridgy Fed: basically enable it once by following the bridge account, and you can forget about it. It creates a “mirror” account of yours on the other side, but when people interact with it there, the likes/reposts/comments go back to you (as long as that other person also has the bridge enabled).
there are probably other options too, let me know in the comments :)

→ If you’re active on Bluesky: wonderful! ☺️ But again, all of the above applies. Don’t forget about the friends and strangers who prefer the elephant site 🦣. At the very least, enable the Bridgy bridge.

→ If you do have both Mastodon and Bluesky accounts, but you’ve decided that you want to use them for different kind of content (e.g. tech vs. non-tech)… please reconsider, in light of what I wrote above. Most of the people who know you online would probably prefer to follow you in one or the other place they like more, not to have to follow you everywhere in parallel. (Though nobody says you can’t have e.g. two different accounts which are both bridged.)

→ If you had a bridged account, but turned off the bridge because you prefer to be posting there directly, but you don’t really post there directly (you know who you are 😛) – I am begging you, please either figure out some cross-posting solution (see above), or enable the bridge 🙏🏻

→ If you’re mostly active on Threads (is that a thing?…) – turn on the fediverse sharing option (it might not be available in the EU though, according to the support article?), and follow the Bridgy account to enable bridging to Bluesky (there aren’t many accounts connected like this, but it should work, let me know if you have any problems).

→ If you’re still mostly active on Twitter (I see you 👀 and I am kinda judging you)… please rethink it. I mean… you really don’t mind helping Elon and his buddies? I know, there’s more content there, more engagement (maybe), big accounts are still there, news is still there, more people talking there about startups, AI, indie dev or whatever. But wouldn’t you want to change that? Wouldn’t you prefer to use and build on a network with an open API that isn’t controlled by one rich American far-right guy1) who thinks he’s the president of the world?… Where you don’t see a full screen ad every few posts, and don’t need to pay a fuckton of dollars to build some fun tool on it? We need to make an effort to move away from there. Someone’s gotta start, and then maybe others will come.

→ If you’re posting both on Twitter and on Mastodon/Bluesky, then, well… it’s ok I guess 🙃 I think the most important thing is to let the people who do want to move away from Twitter completely, have the content they want to follow in the new place. That’s the first step. A natural second step is then to start decreasing the amount of content that there is on Twitter, so people have more incentive to look for it elsewhere, but I understand if not everyone is ready for that step yet.

→ If you tried Mastodon and/or Bluesky before, but you felt like it was too empty there, you didn’t have enough content to read or you felt like you were shouting into the void – please give it another try, and please give it some time. Twitter also wasn’t immediately the place you remember from the first day you signed up, was it? You need to put in some effort, like you did everywhere else. Some tips:

on Bluesky: find some good custom feeds and/or starter packs to follow
on Mastodon: follow some hashtags, and use hashtags when posting to get more reach (but within reason!). On a private instance, this gets a bit more tricky due to the ActivityPub architecture, so it might make sense to start on a larger one at first.
on both: try to find some people you recognize from Twitter, and then look who they follow or repost, and recursively look through their profiles too. Also, interact with people, give likes/faves, good replies, and so on.

→ If you’re on Nostr, there are some kind of Nostr ↔ Fedi bridges or gateways, and you can use those + Bridgy to reach Bluesky too (e.g. here’s a Nostr account “double-bridged” to Bluesky).

→ If you’ve moved to something like Micro.blog, or just blogging – that’s also cool! But I hope you’re somehow posting the links to Bluesky & Fedi too :)

→ If you’ve just given up on microblogging or social media in general… I understand. It’s probably not a bad choice in the current times. I hope we somehow meet again someday, digitally or in person 🩵

So let’s connect, let’s build bridges. But also, let’s finally help the X-shaped zombie die 🧟‍♂️

(You can find me on Bluesky at @mackuba.eu (I have my private PDS at lab.martianbase.net), which is my main account these days, and on Mastodon at mackuba@martianbase.net (also a private instance), where I’m cross-posting a good chunk of the posts from Bluesky using Tootify. If you want to see everything including some shitposting, politics and ATProto-specific posts, the Bluesky account is also bridged as mackuba.eu@bsky.brid.gy. Twitter account @kuba_suder is left as an archive, because I don’t like deleting useful content from the Internet, but I don’t use it anymore. Lately I’ve been also blogging somewhat more regularly than here on my new “journal” Micro.blog.)

1) If you now want to write a comment mentioning someone with the initials “J.D.”, don’t even think about it! 🫠

Micro.blog journal

2025-02-04T16:00:05Z

Update 17.11.2025: I’ve migrated this journal blog now from Micro.blog to Leaflet, a new blogging service built on top of Bluesky’s ATProto. I wrote about this here, the new URL is https://lab.mackuba.eu (I’ll add a redirect from ‘journal’ later).

Just a quick update, if you’re following this blog via RSS: I’ve started a separate “journal” blog on micro.blog: journal.mackuba.eu.

Micro.blog is an interesting service: it’s a one-man indie business that’s sort of a hybrid between a blogging platform and a microblogging social network. You can write anything between full-size blog posts and tweet-sized single messages, and you can cross-post them to Bluesky, Mastodon etc. You can also follow people from the community that’s formed there and reply to them, all in the form of those mini-blogposts (there are no likes or retweets though). The idea, as I understand, is to use a network of blogs to build a social network that uses the web itself as the foundation.

I’m not really planning to use it in this social network mode, since I’m pretty happy now posting on Bluesky and to limited degree on Mastodon (I’ve completely stopped posting on Twitter at this point, since last autumn, when Elon started openly supporting Trump). I’m also not completely sold on this “web as a social network” idea. And I don’t intend it to replace this blog here either – I will still be (very) occasionally posting those super long articles here like the one about NSButtons or the guide to Bluesky.

But I’ve felt the need for a while to have a place to post something in between those – not full blog, and not a micro blog, but a “mediumblog” so to say (not to be confused with a Medium blog) – like this post, for example. Something where I can sometimes post my thoughts more easily, when I want to write something that doesn’t really fit in a few skeets/toots, with less effort required to start and finish it. This seems like it could work for that.

It’s also nice that it’s supposed to sync replies from Bluesky/Mastodon under the posted link back to the blog page as comments below (I’ll try to implement the same thing here). I also have it configured with my own domain, so I can possibly migrate it to something self-hosted like Jekyll or Hugo at some point, keeping all the links and content.

For now, I’ve posted two updates about what I’ve been working on recently:

about tuning a Postgres database to which I’m trying to migrate my Bluesky feeds service
and a review of all I’ve done in 2024

I don’t know how often I will end up posting there, I don’t want to pressure myself, just to have a place to post when I have a need. So if you’re curious, follow me there via RSS (or on Bluesky or Mastodon).

March 2024 projects update

2024-03-27T01:14:35Z

I’ve been still pretty busy with various Bluesky- and social-related projects recently, so here’s a small update on what I’ve been working on since my November post, if you’re interested:

Skythread – quote & hashtag search

I was missing one useful feature that’s still not available on Bluesky: being able to see the number of quote posts a post has received and looking up the list of those quote posts. The Bluesky AppView doesn’t currently collect and expose this info, so it’s not a simple matter of calling the API. But since everything’s open, anyone can build a service that does this, they just need to collect the data themselves.

Since I’m already recording all recent posts in a database for the purposes of feeds and other tools, I figured I could just add an indexed quote_id column and set it to reference the source post on all incoming posts that are quotes, and later look up the quotes using that field.

Skythread, my thread reading tool, seemed like a good place to add a UI for this. When you look up a thread there, it now makes a call to a private endpoint on my server which returns the number of quotes of the root post, and if there are any, it shows an appropriate link below the post. The link leads you to another page that lists the quotes in a reverse-chronological order, like this (it doesn’t currently do pagination though). You can open that page directly by appending the bsky.app URL of a post after the quotes= parameter here: https://blue.mackuba.eu/skythread/?quotes=.

In the same way, I also indexed posts including hashtags, since hashtags were being written into post records since the autumn, but it wasn’t possible to search for them in the app. However, this has now been added to the Bluesky app and search service, so you don’t need to use Skythread for that. I hope that the quote search also won’t be needed for much longer :)

Handles directory

One very cool feature of Bluesky is that you can verify the authenticity of your account by yourself, by proving that you own the domain name that you’ve used as your handle. So for official accounts like The New York Times, The Washington Post, or Alexandria Ocasio-Cortez, it’s enough if they just set their handle to their main website domain (or a subdomain of house.gov in AOC’s case) to prove they’re legit – they don’t need to apply anywhere to get a blue or gold tick on their profile.

I was thinking one day that it would be nice to see how many e.g. .gov handles there are and notice easily when new ones show up. So I grabbed a list of all custom handes from the plc.directory and started recording new and updated ones from the firehose.

In the end, I decided to build a whole catalog of all custom handles, grouped by TLD, and show which TLDs are the most popular. At first I only included the “traditional” main TLDs and country domains, but a lot of people liked it and I got a lot of requests to also include domains like .art, .blue, .xyz and so on, so in the next update I’ve added all other domains too. (I gave it an old-school tables-based design as a homage to the old “web directory” websites like Yahoo Directory 😉)

Apart from handles, the website now also tracks third party PDS servers that started to show up after the federation launch in February.

Bluesky activity charts

I’ve also made a page that shows some charts tracking Bluesky user activity – the number of daily posts and unique users that have posted in a given day. The activity has been gradually falling since October until February, then there was a huge spike when Bluesky opened up for registrations without an invite (when Japan suddenly took over), and then it’s been falling down again since then (currently around the level of the October top).

You can also see some other interesting stats on Jaz’s page and Eric Davis’s Munin charts, especially the one tracking daily/weekly/monthly active user count. (I also have a few more ideas for what to add to my charts.)

DIDKit

In the last few weeks, I’ve been updating the code that tracks custom handles again to adapt to some protocol changes. The #handle event in the firehose, which included handle info on every handle change, is now deprecated and being replaced with a new #identity event, which only tells you to go fetch the account info from the source again (source being usually plc.directory).

At the same time, I also implemented validation of custom handles – clients and services that display handles are supposed to verify the handle reference in both directions themselves, because some accounts may have handles in the PLC registry assigned to domains that they don’t actually own or which don’t exist (the Bluesky official app shows such accounts with an “⚠ Invalid Handle” label, which you’ve probably seen before). For example, the handles directory page initially listed an amongus.gov account under .gov TLD, which was loaded from plc.directory, but is not in fact a real domain.

This should ideally be done by not relying on Bluesky servers, and instead checking the DNS TXT entry and the .well-known URL of a given domain manually. There’s a bunch of pretty generic logic there that will be needed in most projects that need to convert between DIDs and handles, so I extracted it to another Ruby gem named DIDKit, which lets you do things like:

get the DID of an account with a given handle
load the DID JSON document, which includes info like assigned handle(s) or hosting PDS server
check if any of the assigned handles from the document resolve back to the same DID
fetch all updates to all DIDs in batches from the PLC directory

mackuba ∕ didkit

A library for handling DID identifiers used in Bluesky AT Protocol

Skyfall

I’ve also been making some minor updates to my Skyfall library for streaming data from the Bluesky relay firehose.

One thing I’ve been trying to fix is a rare but annoying issue with the websocket connection getting stuck. From time to time, it manages to get into a state where no data is coming, but the connection doesn’t time out and just waits for new packets for hours, until I notice it and restart it. It isn’t only happening to me, others have mentioned it too (and not only in Ruby code); but it happens rarely enough that it’s really hard to debug.

My proposed fix is adding a “heartbeat” timer, which runs with some interval like every 30 seconds, and checks if there have been any new packets in some period of time; if there haven’t been any in a while, then it will forcefully restart the connection. (This isn’t included in the latest release yet, I’m waiting for it to get triggered a few times first.)

Another thing I’ve added is being able to connect to a new kind of firehose exposed by “labellers” a.k.a. moderation services. Bluesky has released this new important piece of the federated architecture earlier this month – third party developers and communities can now set up independent moderation services, which manually or automatically add various “labels” to accounts or specific posts, flagging them e.g. as “racism” or “disinformation”. Anyone can subscribe to any labellers they choose, and they’ll see the labels from those selected services shown in the app. The new firehose (the subscribeLabels endpoint) allows you to connect to a specific labeller and stream all new labels that it’s adding.

I’m also tracking all new registered labeller services and keeping a list here (it’s not curated, just a dump from a database table, so it also includes various test servers etc.).

mackuba ∕ skyfall

A Ruby gem for streaming data from the Bluesky/AtProto firehose

The “Bluesky guide”

Last month I wrote a long blog post titled “Complete guide to Bluesky”, where I included various info and tips for beginners about Bluesky history, available apps, handles, custom feeds, privacy, or currently missing features. I’m now updating it every time Bluesky releases another big feature :) Check it out if you’ve missed it.

I have ideas for a few more Bluesky introduction posts with a developer focus – a general intro to the protocol and architecture, and about working with the XRPC API and the firehose. I hope I’ll be able to find time for that in the next few months.

Tootify – cross-posting to Mastodon

I still want to finish my Mac app for cross-posting to Twitter, Mastodon and Bluesky one day, but it’s a lot of work and I’ve got too many different things in progress at the same time, so it’s moving at a glacial pace… In the meantime, I started thinking if I could maybe quickly build something much simpler that also does the job. I’ve been mainly hanging out on Bluesky in recent months and posting on Mastodon only occasionally, because having to copy-paste things from one tab to another is annoying, especially if images and alt text are involved. But the folks I know from Twitter still mostly follow me on Twitter and Mastodon only and aren’t coming to Bluesky.

I also didn’t want to simply copy every single post from here to there, because a lot of things I post on Bluesky are specifically about Bluesky stuff, so it doesn’t always make sense to post them to Mastodon – I only want some selected ones to be copied. But at the same time, I wanted to minimize the amount of friction this would add.

So the idea I had one night was that I could mark the Bluesky posts to be copied to Mastodon by simply “liking” my own posts that I want copied; a service or a cron job would then periodically look at the list of my recent likes, and when it notices one made on my own post, it would copy that post to Mastodon (and remove the like).

I managed to build it in about a day and a half, complete with image support with alt text and copying of quote-posts as posts with plain links. It’s now running happily on a Raspberry Pi on my local network 😎

The code is published here, if you’re interested – but it’s a bit of a proof of concept at the moment, just enough to make it work for myself, so it’s probably not very user-friendly. But maybe I’ll build it up into something bigger if people find it useful. (Just to clarify, this is meant to be a one-way sync by design – syncing in the other direction would be harder for various reasons, e.g. because of the complex “facets” system that Bluesky uses for post record data, and because Mastodon’s post length limit is higher than on Bluesky.)

mackuba ∕ tootify

Toot toooooooot

And One More Thing ;)

Paul Frazee, Bluesky’s lead dev, has a lovely cat named Kit and often posts photos of her. I’m a big fan of Kit, so I made a feed named the Kit Feed, which only includes posts with these photos 🙂 Like and subscribe! 🐱

A complete guide to Bluesky 🦋

2024-02-21T18:05:12Z

(Last update: 18 Nov 2025.)

For the past year and a half, I’ve been a pretty active user of Bluesky. I enjoy it a lot, and I’ve managed to learn a lot about how it works, what works well and what doesn’t, and also what’s likely coming next.

I’ve decided to write down some of the tips & tricks that I often give to friends when I invite them there, or the advice and answers that I sometimes give to people that I find in some feed asking about things.

This of course got much longer than I planned 😅 so if only have a moment, here’s a TLDR:

there are official iOS and Android apps, but you can also use bsky.app in the browser, or try e.g. Skeets (iOS), Skywalker (Android), or deck.blue (multi-column webapp)
tweak the algorithm in your “Discover” feed by opening the “…” context menu on a post in that feed and selecting “Show more like this” or “Show less like this”; or, if you prefer a classic chronological feed, you can set the Following feed as your main feed and remove Discover
if your timeline feels empty, you can find some people to follow through “starter packs”, or you can go to the “Feeds” tab, scroll down to the “Discover New Feeds” section and look for some feeds on the topics that interest you (on the top list or in the search); follow these feeds, and then if you find some interesting people posting in those feeds, follow them too. You can also search for feeds on goodfeeds.co.
don’t be afraid to interact with people, repost good posts, like good comments, comment in threads and so on – that’s how you make friends! (but be nice :)
if you see too much NSFW stuff, look for “Content filters” settings in the Moderation tab in Settings
everything you post here is very public, so don’t share anything too private 😏
if you own some cool domain name like “taylorswift.com”, you can set it as your handle
bookmarks, trends, thread composer, pinned posts, videos, GIFs and DMs (simple version) are now available
2FA and more DM stuff are coming, post editing is planned; some version of private posts shared to limited audience is probably coming at some point, but not in near future
Jack Dorsey has nothing to do with Bluesky anymore ;)

And now the long version:

What is Bluesky?
Apps
Feeds & algorithms
Safety & moderation
Privacy of your data
Account security
Handles & IDs
How are things called here?
What is this federation thing?
Search
Hashtags
Video & GIFs
Starter packs
Trending topics
Bookmarks
Settings
Missing features
Other tools

What is Bluesky?

(A bit of history, skip if you’re not interested :)

Bluesky is a project started originally by Twitter (now an independent company), whose goal is to create a decentralized Twitter-like social network, or more generally, a platform for building various decentralized social networks.

The project was started in Dec 2019 by Jack Dorsey, former Twitter CEO. The basic idea was to design a protocol on which you could build something that would work like Twitter, but which would not be under the control of a single company that makes unilateral decisions about everything on it. It would be more like web and email, which are open standards that anyone can build on – any company can set up an email service or write an email app, and anyone can sign up for an account and start sending emails. There’s no one central authority on the Internet that can ban you from email altogether.

Such network would consist of many servers owned by different companies and people connecting together, and the idea was that eventually, Twitter itself could become a part of that network, as just one of its elements.

(If this all sounds a lot like the Mastodon social network, or “the Fediverse”, then you’re right – there are a lot of similarities between these two. However, Bluesky is built on a completely different system they’ve designed from scratch, called the AT Protocol or ATProto. They’re hoping that this will let them build some things better than they are done in Mastodon, making the network less confusing, more useful and more user-friendly. Bluesky does not directly connect with Mastodon servers and apps, although there are some unofficial ways to connect the two worlds.)

After an initial research phase, in 2021 a team was chosen to build the platform and the Bluesky company was formally created. A woman named Jay Graber, formerly a developer at Zcash, was chosen as the CEO. Thankfully, Jay had the foresight at that point to insist that they’d set it up as an independent company, which was funded, but not controlled by Twitter. Had they not, it almost certainly would have been shut down last year after Elon’s Twitter takeover.

The team has been working on designing and building the pieces of the system throughout 2022, and in February 2023 they’ve launched a very early and rough beta and started slowly letting in some users who wanted to try it out and have signed up on a waitlist. However, the whole Elon thing happened in the meantime and that was a moment when everyone was looking for a Twitter alternative, so the interest has wildly exceeded expectations and they weren’t ready yet to take in everyone.

Since then, the user base has been gradually growing, with people being let in from the waitlist and existing users inviting their friends using invite codes. Meanwhile, the team had to speed some things up to adapt to the new situation and has been working hard on adding the most important features, and building up the backend to allow for more and more traffic. Finally, almost a year later, in February 2024 Bluesky has opened up for registrations from everyone.

The Bluesky company

Bluesky is still a fairly small team at the moment. The dev team is probably something like a dozen people altogether, and that’s for the frontend, backend, protocol, servers and so on. So they just can’t add new features as fast as they’d like to, but they’re doing what they can.

The team members interact with people on the platform all the time, answering questions and just having fun in general. They’re also building almost everything in public – the source code of the app and servers is available on GitHub, so we can track in real time what they’re working on next, report bugs and sometimes submit code with some new features they can merge in.

The company is set up as a “public benefit corporation”, which basically means (in my non-US layman understanding) that it is a business and it’s meant to make profit, but that profit is not it’s only and main goal. It can and should have other, more noble goals that benefit the public, as the term implies, in this case: creating a protocol for decentralized social apps that everyone can build on.

Bluesky isn’t really making any money at the moment (other than a small domain reselling service). The general plan is that, rather than the standard route of eventually putting ads everywhere and selling user data, they will resist “enshittifying” the platform and will instead add some optional premium plans and services, but in such a way that wouldn’t change the social dynamics of the platform:

Bluesky will always be free to use — we believe that information and conversation should be easily accessible, not locked down. We won’t uprank accounts simply because they’re subscribing to a paid tier.

The first planned step is a “Discord-like” subscription, which was planned for the end of 2024 / beginning of 2025, but was delayed because of the huge influx of new users that happened in November. More long term, they were also talking about coming up with some ways of letting creators and devs on the platform earn some money from their followers, with Bluesky taking some commision from the payments.

In any case, they’re explicitly building the network to be resilient even in the unlikely scenario that they themselves “turn evil” in the future – the network is meant to be “billionaire-proof”, impossible to completely take over by one guy with too much money. To quote the lead dev:

“Our culture doc includes the phrase “The company is a future adversary” to remind us that we won’t always be at the helm – or at our best – and that we should always give people a safe exit from our company. It’s weird at times to frame our priorities as protecting users from us, but that’s exactly what we’re trying to do.

The Bluesky team is made up of users. None of us come from big tech companies. We all came together because we were frustrated by the experience of feeling helpless about how our online communities were being run. We don’t want to give that same feeling to other people now that we’re the builders.

When we build an open protocol, we’re giving out the building blocks. We want to start from the premise that we’re not always right or best, that when we are right or best then it might not last, and that communities should be empowered to build away from us. Sometimes this can all feel very intangible and abstract, and for the average user the goal is to just feel like a good & usable network. But this is one big reason why we put all the Fancy Technology under the hood.

Also, contrary to what you might have heard, this isn’t “Jack Dorsey’s company”. Yes, he started the whole thing, but he isn’t running the company – Jay Graber is. Jack was very little involved in the project after they started building; he basically gathered a team, gave them a lot of money and let them do their thing.

In fact, he deleted his account on the platform last summer, after he was booed off it by the users, and now he’s mostly hanging out on Nostr instead. In early May 2024, it was announced that Jack has left the board of directors too – he’s been replaced by Mike Masnick later. Dorsey also doesn’t own any shares of Bluesky.

Apps

Bluesky has an official mobile app for iOS and Android. It’s written in React Native, so it doesn’t feel fully native in all places and some system integration features take a while to be added – the reason is that they’ve started with a very small team at first (I think initially just one guy did all the frontend), so the only way they could do it was to build for both platforms & the web from one codebase.

At this point, after various improvements made later, the mobile app is mostly ok and gets better with every update, it’s also obviously the most feature-complete one. One issue is that it doesn’t support iPad yet.

There is of course also a web interface, at bsky.app, which is pretty good – this is the main UI that I use Bluesky with (also works on the iPad). At the moment it’s mostly meant to be used while logged in, although some pages like posts/threads and profile views can be viewed unauthenticated (the search doesn’t currently work when logged out for performance reasons).

But there is also a small but enthusiastic group of third party developers building various apps, tools and experimenting with the protocol. They’ve built several independent apps, libraries to access the API in various languages, and they often manage to build various new features in their apps before the Bluesky team gets around to doing that in official apps.

Here’s a few of these apps (in various stages of development):

Graysky – a mobile app, first full-featured third party app, although its author was hired by Bluesky later, so it hasn’t been updated much lately
deck.blue – a web-based “Tweetdeck” column UI, written in Flutter (pending rewrite in React)
Skeetdeck – another web-based, more lightweight “Tweetdeck”
Tokimeki – a web app (+ PWA) with a column UI from a Japanese developer
Skeets – native iOS app with iPad support and some interesting features
Skywalker – a native app for Android
Catbird – new iOS client in beta
SkyFeed – a web app with a column UI that also lets you build custom feeds
SoraSNS – a multi-network client for iOS
Openvibe – another multi-network client with support for cross-posting and a unified timeline
Dhaaga – a multi-network client for Android

The Ivory team (Tapbots) is also working on their own client called Phoenix, but at the moment it’s still work in progress.

More recently, a new wave of client apps and services built on Bluesky/ATProto have also started appearing that are focused on either photos or videos, hoping to create a more open alternative for Instagram and TikTok:

Bluescreen, Skylight, Spark, Skyswipe, and Twilight for video
Pinksky, Flashes, and Grain for photos

Feeds & algorithms

Social networks generally include two kinds of feeds: a chronological one, showing all posts from the people you follow in order, and an algorithmic one, showing what the service thinks you will like (which often means though: what they think will bring them more profit). Centralized platforms generally push you towards an algorithmic feed and often make the chronological feed harder to reach (if at all). Mastodon on the other hand only includes chronological feeds and no algorithms.

Bluesky has both – and much, much more.

When you sign up, you start with two default feeds. “Discover” is Bluesky’s main algorithmic feed – it mixes some posts from the people you follow with some other posts that you might like. You can pick the option “Show more like this” or “Show less like this” from the menu on each post to give it some hints on what you like or don’t (note: if you tried it before and didn’t see any effect from these hints, there have been some fixes there very recently, June/July 2025, so try again). The devs are constantly tweaking it and asking for feedback, so it should be getting better over time.

The second one, “Following”, is a classic chronological feed. Initially it had a sort of unique twist, in that it could show you replies from the people you follow made to anyone, regardless if you follow that other person, but this option was removed in August 2025. If you miss that old mode, which was more noisy, but allowed you to meet friends of friends more easily, I made two custom feeds that let you peek at those replies: Follows & Replies and Only Replies.

But that’s just the tip of the iceberg. Bluesky has built a system where anyone with a server and some knowledge of coding can implement their own algorithmic feeds that they can share with everyone else. There are currently about 40 thousands custom feeds (as of Feb 2024) made by the Bluesky community that you can add to your app. And more importantly, the “Following” and “Discover” feeds are just what you start with by default – you can set any of the thousands of other feeds as your main feed, and you can even remove the two built-in feeds if you don’t like them, and leave e.g. only the “Cat Pics” custom feed as your only feed tab. Nothing here is forced on you.

The way a feed works is that it basically reads all new posts on Bluesky from a giant stream and decides which of them to keep and how to arrange them (this can be a shared feed, same for everyone, or a personalized feed that looks different to each user). Most feeds match posts by keywords – these are feeds on some specific topics like Linux, food, gardening, climate change, astronomy, and so on. They usually define some sets of words, phrases, hashtags, sometimes emojis, and include all posts that contain any of these, chronologically.

There are also various “top posts of the week / all time” feeds, posts using AI models to match some specific kinds of photos like pictures of cats, frogs, or moss, or personal algorithmic feeds that show you posts selected for you according to some specific idea: posts from your mutual follows, from people who follow you, posts with photos only, posts from those of your friends who post less than others, and so on. These are all feeds built by third party developers who just had an idea and implemented it, without having to register or apply anywhere or ask anyone for permission.

Some people have also built web-based user friendly UIs for building feeds, which let you build feeds using a web form and have them hosted for you, without having to write code or host it yourself, which allowed a lot of people without programming knowledge to build their own feeds. The first such tool was SkyFeed, built by one German developer, which at some point ran the vast majority of all feeds on Bluesky; another newer one is Graze, a larger service with a bit friendlier interface, which recently partnered with Bluesky to run a trending feed for the New York mayor election night.

There’s a nice guide to feeds that describes them in more detail on the Goodfeeds site (archived), there is also a blog post about feeds on Bluesky’s blog.

Useful feeds

There is a feed search engine integrated in the official app, in the Feeds tab (the “Discover new feeds” section). You can also search for interesting feeds on these two external sites: goodfeeds and SkyFeed Builder Feed Stats.

Some general feeds that you might find useful:

What’s Hot, Catch Up and Week Peak Feed – general feeds with most popular posts
Quiet Posters – posts from people you follow who post less than others
Latest From Follows or Latest From Mutuals – just one most recent post from each person you’re following
Popular With Friends – recent posts liked by the people you follow
Follows & Replies and Only Replies – my feeds that include all replies from your follows to anyone
Only Posts – picks only top-level posts from Following without replies or reposts
The ‘Gram – only posts with images from the people you follow

And some “utility feeds”:

All-Time Bangers – posts with the highest number of likes on the whole platform
My Bangers – your own posts with the most likes
Quotes – all quotes of your posts
Mentions – all replies to you or mentions of you
My Pins – your “bookmarks” made by commenting with the 📌 emoji

Safety & moderation

Like on most other networks, you can mute someone if you find them annoying, or you can block them if you find them really annoying. It’s also possible to mute words, phrases or hashtags. You can now also mute words for a specific period of time (there’s no way to mute an account for a specific time yet).

(Note, the blocking mechanism here is pretty aggressive, in that it also hides all previous interactions between the two users *for everyone*; so don’t be surprised if someone blocks you and your reply or quote “disappears” – it wasn’t deleted, just hidden.)

There’s also a number of features for controlling interactions with your posts or threads:

you can lock replies in a thread to only your followers, only the people you follow yourself, or some other combination, or disable them completely (also some time later after replies appear)
you can choose to “hide” some specific negative replies, like on Twitter (they’re moved to a “hidden replies” section at the bottom of the thread)
you can disable the option of quoting your post
you can “detach” existing quote posts of your post – the quoting post stays visible, but without the embed showing your post
and finally, you can temporarily “deactivate” your whole account – this makes it appear deleted to others so people won’t bother you if you need to take a break from social media

You can also create “moderation lists” for muting or blocking some groups of people all at once, which can be shared with others. This is meant to let various communities on Bluesky build their own “defences”, by collecting lists of people who are unpleasant or annoying in some way and letting others mute or block them in advance before they come across them.

If you get added to a user list (the kind meant for following) or a starter pack that you don’t want to be on, and the author refuses to take you off it despite your request, you can solve the problem by blocking the author – you will “disappear” from their list then. For obvious reasons, this doesn’t apply to mute/block lists, since usually no one wants to be on those, but you can now report lists that are clearly hateful or made in bad faith, and they are taking those down. If you don’t know what lists you’re on, you can look that up on Clearsky.

Bluesky’s own moderation seems to generally work ok, though there are some occasional problems, controversies, and periods when they get a bit overwhelmed with new waves of users; but obvious trolls, spams and scams are usually got rid of pretty quickly. They claim to have around 100 people in the moderation team now, as of January 2025.

The general high-level plan for moderation at Bluesky and on the AT Protocol is something they call “composable moderation” (old blog post) or “stackable moderation” (recent post). It’s an idea that moderation will have many layers – from server operators including the Bluesky company, through various tools and services provided by other companies, organizations, and communities, ending with some ways to privately personalize your experience according to your personal needs.

Labellers

A core part of this is a feature called “labellers” that they’ve released in March 2024, which are basically third-party moderation services. They work by assigning a set of labels/tags (manually or automatically) to accounts and posts – this can be because one of the labeller’s moderators has come across an offending post, or because it was detected automatically using some custom software, or because the post or account was reported to the service by a user (users can send moderation reports to any set of these services).

All users on the platform can “subscribe” to one or more of these services and configure how they want these labels to affect their experience: for any label type, they can choose if the user/post marked with such label should be hidden from their view, just marked with a label, or if this kind of label should be ignored. (A labeller doesn’t have the full power of a built-in platform moderation in that it can’t just ban someone from the site and delete their account – but they can make someone effectively disappear for those users who trust and agree with the given service.)

Labellers will usually be specialized in some area: they could be protecting their users from things such as racism, antisemitism, or homophobia; they could be automatically detecting some unwanted behaviors like following a huge number of people quickly; marking some specific types of accounts like new accounts without an avatar, or accounts from a different network; fighting disinformation or political extremism; or they could be serving a community using a specific language or from a specific country.

A simple labeller can be run by one person, but bigger ones can be managed by a whole group of people that collaborate on processing the reports (Bluesky has built an open source tool for this called Ozone). This system allows different communities to handle moderation in their own way independently, to make their members feel safer and have a better experience in the aspects that are important for them. And most importantly, different communities could often have somewhat conflicting or even completely opposing views on some things – and Bluesky as a company doesn’t have to try to satisfy everyone (which is impossible) or always pick a side. They also don’t necessarily have to specialize in every country, language and culture on Earth. Of course they reserve the right to take down some accounts completely, because some things and some people have to removed from the platform for everyone (e.g. things that are just illegal), but in less serious or less clear cases, they can just use labels or defer to other labellers (the Bluesky built-in moderation is now “just” another labeller among many others, using the same API, and with only some special powers). They also have a few country-specific labellers now, so they can comply with government takedown orders by hiding given posts/accounts in a given country, but leaving it online for everyone else.

You can read more about labellers in the guide titled “Bluesky Crash Course: Labelers” written by Kairi, who used to run one of the most popular labellers called Aegis (now defunct).

There’s no easy way to search for labellers in the app yet, but I’m keeping a rough list myself on this page. I also made a “Label Scanner” tool where you can find all labels assigned to a given account from any labeller.

By the way, it’s fascinating how the open architecture of Bluesky allows its features to be used sometimes in completely unexpected ways. Labellers were initially designed to be used mostly for negative, unwanted things, but quite a lot of them ended up being built to let users label themselves with some badges they want to show to others: there are labellers to let you assign a country flag, pronouns, RPG class, Zodiac sign, Hogwarts house and other things.

Some other useful labellers:

XBlock, which lets you hide screenshots of posts from other platforms like Twitter
Profile Labeller, which marks e.g. accounts created recently, without an avatar, ones that changed handle recently etc.
No GIFs Please, which is exactly what it sounds like ;)
Asuka’s Anti-Transphobia Field

Privacy of your data

One important thing from the privacy aspect that may not be obvious at first, which you need to be aware of: the underlying protocol on which Bluesky runs is extremely open. Anyone who knows how to code can write an app or tool that can read practically any data about anyone, without having to ask anyone for permission (since there’s no central authority that can require registration and payment to get API keys like on Twitter). This is by design, because all the different pieces of the network that make it work, apps, tools and services, need to be able to access the data to provide their functionality, and we want everyone to be able to build those to keep the network decentralized and not controlled by one corporation.

This has both advantages and disadvantages. For a developer, it means that the only limit is your time and imagination (and maybe API rate limits). You can build feeds that show all posts with cat photos, a bot that responds to the text “/honk” with a random photo of a goose, implement some new features before the Bluesky team gets around to that, count the statistics of how the percentage of languages used in posts has changed over time, and whatever else you can think of. You don’t have any monthly quotas or paid plans.

For a user though, this means that everything you do is very public – kind of like it was on Twitter, just more so, because there are fewer restrictions.

Specifically, all of these are publicly accessible (even if they aren’t all displayed in the official app):

your posts
your likes
the photos you’ve attached to posts
all the handles you’ve previously used (you can’t delete those)
the list of people you follow
the list of people you block (!)
the user lists and moderation lists (mute/block lists) that you’ve created
which moderation lists (yours or others’) you are blocking people with

And these things are private and known only to you (and the apps and tools that you’ve explicitly granted access to your account):

the people you’re muting (individually or through lists)
the words, phrases, hashtags etc. that you’re muting
your “saved posts” / bookmarks (the new built-in ones, not the 📌 style)
your selected languages and other preferences
your email address, birthday and phone number
who invited you and who you have invited
the moderation services you’re subscribed to
the accounts you’re subscribed to for post update notifications
the custom feeds that you’ve saved or pinned
- one caveat though: the provider of the feed knows when you are opening it

DMs are private between you and the person/people you’re chatting with, but they’re currently not end-to-end encrypted, so the Bluesky team can theoretically access them, and will access them if ordered to. A fully encrypted version will come some time later. For sensitive conversations, Bluesky devs recommend that you use the DMs to just exchange e.g. Signal usernames and then continue there.

The difference between muting and blocking is because muting is simply a filter applied only for you – nobody else needs to know that you’ve asked your app to hide some of the posts. On the other hand, blocking is inherently a two-way thing – that other person, their app/server and any other pieces of the network that process their posts need to be aware of the block, so that they can prevent them from interacting with you. So the fact that the block exists needs to be publicly known.

Now, what all of this can mean in practice:

anyone can download anyone’s posts and do various targeted or global analysis on it, track your likes and contact graph and so on
if you are posting personal photos, especially NSFW photos or photos that can be geolocated, anyone can be downloading all of them automatically (though the official apps strip metadata from photos)
some (any) companies can potentially train some kind of AI models on the data (speaking purely about technical possibility, not legality of course; there aren’t any secret clauses in the ToS that let Bluesky sell your data to AI companies)
blocking someone only adds friction, but it can’t completely prevent them from seeing your posts (same as it always was on Twitter, since you could always open a post in an “incognito” window or use an alt account)
there is no way to add a feature that would let you “lock” a profile for followers only, hiding your content from others, because all post data has to be public for the network to work
copies of any posts you delete might still remain on some third party servers (most services do delete their copies when you delete a post on Bluesky, but you can’t prove it or enforce it)

Some of this might sound scary, but most of this is or was always the case on other social networks too, those that have APIs at least – if you’re posting something publicly anywhere, you need to realize that anyone can record it forever. The main difference is that it’s easier to do it here, because the API has fewer restrictions than on centralized platforms, and that it’s currently not possible to do any semi-private content that’s visible to some people but not to others.

Account security

For logging in to third party apps and tools, Bluesky has initially added a temporary system of “app passwords”. Recently, they’ve also implemented more convenient OAuth authorization, where you authorize a service to use your account like on Twitter and other sites. However, it’s pretty complex to implement, so not all third party apps and tools support it yet, and you might still need to use the app passwords for some of those.

An app password is a special one-time password that you can generate in the official app (Settings / App Passwords), which look something like this: abcd-ef56-vxyz-qq34. You can generate a separate password for every tool and app that you log into. Such password grants almost the same privileges as the main password, but without a few critical ones, and you can revoke it at any time from the Settings screen if you’re not using that app anymore, which disables access to your account for that app. You can also specify if the app password should give the app access to your DMs or not.

For apps and tools that use OAuth, it works just like when authorizing apps to use your Twitter or Facebook account – you get redirected to your Bluesky server, you log in to your account, grant access for the app, and are redirected back there. You can see and manage your list of currently authorized apps on the account page (if your account is not hosted on a Bluesky server, you need to open that on your server’s domain instead).

For additional security, there is for now a simple email-based Two-Factor Authentication (2FA) feature (note: not currently working if you’re on a third-party server). A more complete 2FA system is coming.

Handles & IDs

Bluesky has a really cool system of handles. They couldn’t have just used single-element handles like “@donaldtusk”, because that wouldn’t really make sense in a decentralized system. They also didn’t want to bind your account permanently to the name of the server you’re on like in Mastodon, where you’re e.g. “ivory@tapbots.social” and you can’t change that unless you make a new account.

So here’s what they’ve come up with: internally, your account is identified by a unique identifier called “DID” (Decentralized Identifier) – an ugly string of random letters (mine is did:plc:oio4hkxaop4ao4wz2pp3f4cr). To that DID you assign a handle, which you can switch at any moment to a different one, and any contacts, references and connections will (mostly) stay intact; and the handle is actually just any domain name, usually displayed with an “@“ at the beginning, but without any additional username before it.

By default, when you first join Bluesky (the current official server) you’re given a handle which is a subdomain of bsky.social, e.g. “georgetakei.bsky.social”. This is a real domain name, you can type it into the address bar of the browser and it will redirect you to the profile on Bluesky.

But at any moment you can switch to a different handle by assigning any domain name that you own. It can a very short name like retr0.id, or something long with one of those quirky new TLDs like .horse or .computer, or it can even be a very official sounding domain like washingtonpost.com.

There are two ways to assign a domain, either via HTTP by putting a file in a specific place on the website hosted on the domain, or via DNS by putting a new entry in your domain configuration – the complete instructions are here. (BTW, Bluesky also runs a service that resells and automatically configures domains through a partnership with Namecheap.)

This also means that you don’t need to rush to “reserve your handle” at ***.bsky.social – because custom handles are cooler anyway 😎 (if you change your handle from *.bsky.social to a custom domain, that old handle is now reserved and unavailable for others, that didn’t work like this at first). This system also makes it much easier to move your account to a different server – you can take your content and connections with you and keep your existing identity, because your DID identifier never changes.

There’s even a number of “handle services” now that let anyone use a subdomain of their domain as the handle – e.g. go to swifties.social if you want your handle to be “yourname.swifties.social” 👩🏻‍🎤

Verification

The system of domain names in handles serves as a primary way of verifying accounts on Bluesky. If someone has a handle that matches the domain of e.g. some newspaper, organization or government branch that you recognize, you can assume that the account is operated by someone who was authorized by that entity. So e.g. @wyden.senate.gov is definitely US Senator Wyden (or at least his staff), and @cnn.com is definitely CNN – no one has to manually verify their documents. We also have here e.g. the European Commision @ec.europa.eu, government of Brazil @brasil.gov.br, Interpol @interpol.int, or European Space Agency @esa.int 🚀

If you’re setting up an account for some well known organization, it’s highly recommended to switch to your domain as the handle from the start to make it clear that it’s an official account.

As an additional verification factor, e.g. for publicly known people who don’t have a well known domain or can’t use one, Bluesky has now added classic “blue check” verification badges. They’re not really meant as a common thing like they are on Twitter now, and they have some important limitations, e.g. they expire if you modify your handle or display name in any way, but they’re useful for making sure that, for example, yes, it’s the real Barack Obama.

The checkmark verification system is designed in such a way that Bluesky can grant other organizations the right to verify the people they vouch for themselves; so you can see that e.g. a Financial Times journalist has a blue checkmark, but if you click on it, you see that he’s been verified by Financial Times, not by the Bluesky team. That way, Bluesky folks don’t need to verify each account themselves manually.

How are things called here?

The app generally uses “neutral” terms like “post”, “repost”, “feed”, “timeline” and so on. That said, pretty early on someone came up with with the word “skeet” for posts, for “sky + tweet”, and it stuck, despite (or likely because of) the team’s protests. So the term is used pretty commonly, though maybe a bit ironically, despite being a bit controversial because of its existing, other slang meaning (see Urban Dictionary)… By analogy, you can also “reskeet”, “subskeet” and so on.

The timeline/home feed is also sometimes called a “skyline”.

New users that have joined Bluesky recently are often called “newskies” – there is even a Newskies feed, which includes every new user’s first post (and only their first post). On the opposite end, some folks sometimes jokingly call people with a long experience on the platform “Bluesky elders” (a reference to one post that was widely made fun of). There is a labeller that gives you an elder label if you had an account before summer 2023.

A “hellthread” is something that existed for some time in the spring of last year – the initial implementation of notifications notified you of any reply somewhere below your post or comment, to an unlimited depth. People have started creating extremely long and nested threads, which notified everyone involved of any reply, and there was no way to opt out of it. A certain community has formed around these hellthreads, of people who hang out together and sometimes waged wars in them. Eventually, the notifications were fixed, but due to protests, the devs have kept an option to still create hellthreads in one place, hardcoded in the app code. This was finally removed a few months later.

A “nuclear block”, alternately “apocablock”, is a name for the feature where when one user blocks another, it hides all previous replies between the two from everyone – it basically “nukes” the whole conversation to discourage other people from joining in (though you can still find the posts though if you’re determined, e.g. on the users' profile page feeds or in Skythread).

A “contraption” was an informal name for set of popular mutelists/blocklists maintained by a user named Kairi in 2023 (which were later turned into a widely used labeller called Aegis, which eventually shut down a few months later) – the trolls and haters who have crossed a line were told “okay, you’re going into the contraption”. The term is still being used as a generic name for any set of blocklists.

What is this federation thing?

The goal of Bluesky is to be a network that consists of many, many servers run by a lot of different companies, organizations and people that connect with each other – that’s roughly what federation means. Initially, all the critical pieces were controlled by Bluesky PBC; however, this is gradually changing. They’ve taken the first big step towards federation in February 2024, by letting people migrate their accounts to self-hosted servers. This is still a technically complex process, and there aren’t really any public non-Bluesky servers which allow open signup, but there are currently a couple thousands of mostly personal “PDS” servers, where people keep their accounts and data. I’m currently posting from my own PDS named lab.martianbase.net.

If you’re worried that things will get more complicated, maybe you have some bad experiences from Mastodon – then don’t worry. They’ve specifically designed everything to be less confusing. The Bluesky-managed accounts have actually already been spread out internally on a number of separate servers – they’ve been migrated sometime in November 2023, and few people have noticed, because everything just kept working. Now, you have an option to move your account to a server controlled by someone else, if you want to – but you can just ignore the whole thing if you don’t care about it. You can’t even easily see who is on which server, unless you check it with a third party tool like internect.info.

(Fun fact: the Bluesky PDS servers are all named after different kinds of mushrooms, e.g. my original PDS was “amanita” 🍄 – so if you see someone talking about “mushroom servers”, they’re probably talking about those :)

Bridgy Fed

Bluesky opening to federation does not mean that it will connect with Mastodon servers though, since they use different, incompatible protocols. There is however a “bridge” service called Bridgy, built and operated by a third party developer, but supported by the Bluesky team.

The way it works is that it “mirrors” Mastodon accounts and their posts to Bluesky or Bluesky accounts to Mastodon (for accounts that have enabled the bridge). It’s possible to create whole threads where some replies are made by Mastodon accounts and some by Bluesky accounts, all of them being visible in both places. See e.g. here the bridged profile of Eugen Roshko (Gargron), creator of Mastodon, as seen on Bluesky.

If you want your Bluesky profile to be visible on the Fediverse, you need to “opt in” by following the official Bridgy account on Bluesky, @ap.brid.gy. Your profile will be seen on the Fedi side as @your.handle@atproto.brid.gy.

To follow Mastodon accounts that are already bridged (like Gargron’s) from the Bluesky side, just look them up in the search and follow them as normally. If someone you want to follow is not bridged yet, you need to either ask them yourself to enable Bridgy (by following the Mastodon equivalent of the Bridgy account, @atproto.brid.gy@atproto.brid.gy), or you can DM the @ap.brid.gy account on Bluesky and tell the bot the Mastodon handle of the user you want to follow, and it will ask them on your behalf.

Note that there are some obvious limitations because of the differences in supported features, e.g.:

posts from Mastodon longer than 300 characters are truncated, and a link is added to the full original post
Bluesky doesn’t support post editing yet, so edits made after you create a post aren’t visible on Bluesky

The Atmosphere

The “Atmosphere”, also often spelled “ATmosphere”, is a name given to the wider network of apps and services that are being built by third party developers on the Bluesky platform and the AT Protocol. It includes not only Bluesky-specific clients and tools, but also somewhat separate apps/sites like:

blogging platforms, e.g. WhiteWind or Leaflet
photo sharing sites like Grain
Tangled, a GitHub-like code hosting & collaboration site
the video apps mentioned earlier

These are not directly a part of Bluesky, aren’t run by Bluesky PBC and aren’t always closely integrated with Bluesky, but they are built on the same system, and depending on the specific tool, they can potentially integrate and share data with Bluesky and with each other. In each of those, you can usually sign in using your existing Bluesky account and use the same handle/avatar/name that you have configured on Bluesky.

Search

Search works pretty well now on Bluesky. You can search for words, phrases and hashtags, and sort by popular posts or by latest. This is a global search like on Twitter, so it finds posts from everyone everywhere, not like the limited full text search on Mastodon.

You can also add “from:some.handle” to find only posts from a given user, or “from:me” to search within your own posts. There are several more filters available, like filtering by date or by language, although there’s no easy UI for everything yet – but the Bluesky team has posted a tutorial “Tips and Tricks for Bluesky Search” on their blog recently.

Hashtags

Support for hashtags was added in February 2024. When you click on a hashtag you have an option to search for all posts with this hashtag, only posts from the given person, or to mute that hashtag.

Note that for various reasons, hashtags aren’t as integral part of the platform and aren’t as commonly used here as on the Fediverse; people also are wary of using them because they have a bit of a bad reputation from Twitter or Instagram, where spammers often abuse them. Try not to overdo hashtags, e.g. don’t use more than 1-2 in a post – ideally just take note of how other people are using them.

One thing that’s defined in the protocol but not implemented in the official app yet is that hashtags were designed to have two forms: inline tags like on Twitter, and external (outline) tags which are shown below the post, which I think is the way they work on Tumblr (Mastodon now also shows trailing hashtags at the end of a post in a similar way). You will be able to use either or both in a post according to your preference, and they will be interchangeable, with both being returned in the same search. These tags are currently supported in some third party apps like Skeetdeck.

Video & GIFs

Videos are available on Bluesky since September 2024. They’re currently limited to 3 minutes.

For GIFs, there is a built-in support for adding Tenor GIFs in the post compose dialog, by pressing the GIF button and picking something from the search results, and you can also upload custom GIFs – now also on mobile.

You can also embed e.g. YouTube videos in posts, which will play inline inside a post when clicked. (This whole feature was actually implemented by a third party developer.)

Starter packs

Starter packs are a unique Bluesky feature added in June ‘24, which lets you create a list of accounts and feeds that you want to recommend to others. You can make e.g. a list of AT Protocol developers, climate scientists and reporters, or astronomers, and then people can use it to find some interesting new accounts to follow, or just follow them all at once with one click.

You can also share a link to a starter pack with people who don’t have a Bluesky account yet, and they can use it to sign up with to have some initial set of accounts to follow.

There’s no list/search of starter packs built-in in the app, but you can browse and search for them e.g. on the third party Bluesky Directory website.

Bookmarks

Native bookmarks (called “saved posts”) were added in early September. You can save a post using the “bookmark” icon below it, and you can access saved posts through the “Saved” section in the sidebar. Bookmarks are private – a counter of number of saves is shown below a post, but you can’t see who saved it.

An earlier, popular workaround was that you could reply to posts with a comment “📌”, and there’s a special bookmark feed which shows you all your comments with that emoji. This option is still available, although since these bookmarks are essentially normal comments, they are all public.

There is also a tool available to import all your previous “pins” to the new “saved posts” system.

Settings

Some things that you may want to look at in the settings, going from the top:

Switch account – you can log in to multiple accounts, and you can switch between them here in the Settings, by clicking your avatar in the top-left corner of the page on the web, or by pressing and holding your avatar in the tab bar on mobile
Account » Export my data – lets you download a backup of your data like posts and follows (at the moment it doesn’t include media like images)
Account » Deactivate account – lets you temporarily lock your account (it appears deleted to others)
Privacy and Security – enable Two-Factor Authentication and create “app passwords” for third party apps; you can also specify if you want others to be able to get notifications whenever you make a new post
Moderation – manage muted words & tags, muted and blocked accounts, and your moderation lists; you can also choose to hide “blue check” badges if they annoy you
Moderation » Content Filters – here you can set what kind of potentially objectionable content you may want to show or hide – generally various NSFW things. I’m not sure what the defaults are currently, but it’s worth checking and tweaking to your preferences. (Watch out, there can be quite a lot of somewhat NSFW content there that you can randomly come across in some feeds, although it’s generally hidden behind a content warning.)
In the “Advanced” section below, you can adjust which of the moderation labels from each specific labeller you want to apply in your feeds – this includes the built-in “Bluesky Moderation Service” labeller.
Notifications – since July 2025, you can configure here precisely what kind of notifications you want to get: you can e.g. turn off all notifications about likes, reposts or follows, or leave them on only for the people you follow
Content & Media » Thread Preferences – you can choose to have threads sorted by newest, oldest or most liked comments; there is also an experimental nested (tree-like) thread view that I highly recommend enabling (although for longer threads you may want to use my tool Skythread instead :]
Content & Media » Following Feed Preferences – choose if you want to see replies, quotes, reposts etc. in the home feed. (Some earlier options from this section were removed last year, see the Feeds section for more info & workaround.)
Content & Media » Autoplay videos and GIFs – turn off to prevent videos and gifs from automatically playing in your feeds; instead you get a play button on each that you have to press first to see it
Content & Media » Enable trending topics – turn off if doomscrolling isn’t your thing
Appearance – you can make the fonts smaller or larger, or switch light/dark mode
Accessibility » Require alt text before posting – you can turn this on to always be reminded to set an alt text on photos (it’s generally considered nice to add the alt text to images whenever possible, for people who use tools like screen readers or VoiceOver; as a bonus, it also makes your posts easier to find in search and feeds)
Languages – turn on all the languages that you understand or want to see; Bluesky generally hides posts in languages that you don’t have enabled from most places like feeds and search results, except your Following timeline. For your own posts, you set a post’s language yourself in the compose post window – you can switch between a few languages, and if you have the wrong one set when writing, a popup with a warning should appear.

Also in the “Chat” section of the app, there is a separate Chat Settings section under the “cog wheel” icon, where you can specify who you want to be able to contact you via DMs: everyone, no one, or only people that you follow yourself (default is people you follow); conversations you’ve started before are accessible even if you change the setting.

Missing features

There are a few things missing on Bluesky that are available on some other networks. Some of these are limitations of the protocol, and some are just a matter of too much work and too few hands to do it – the team is still pretty small and they have a ton of things to build to catch up with more mature platforms (Mastodon) or those with more people and funds (Threads), and they need to prioritize and some things are always put off.

If you’re a developer and want to help, the app and most of the related services are open source, and the team is happy to accept outside contributions at least for bug fixes and smaller tweaks.

Some things that are still missing:

Private profiles / circles

Like I’ve mentioned in the Privacy section, the AT Protocol as built currently requires all data apart from your private settings to be completely public. There is no way to make something that you only share with some people but not others. There may very well be something like this in the future, because a lot of people are asking for this and the team wants to look into it – but they will have to first invent some other, separate way to share content in the protocol, which will take a lot of time and thinking. So it’s likely to come at some point, but not anytime soon, because it’s much harder than it may look.

DMs on the protocol

For the same reason, sharing messages with only one or a few people using the AT Protocol is currently not possible. The team wants to eventually add DMs to the protocol, but this is something that will require a lot of research first.

But since *a lot* of people wanted to have some way of talking privately with friends, even if it’s an imperfect one, the team has recently added a simple implementation of DMs that isn’t currently a part of the protocol. The DMs are currently using a single centralized service hosted by Bluesky (although third-party apps can access this API), and are not end-to-end encrypted – so they basically work like on Twitter. The first version also doesn’t support group chats and images, only 1-to-1 text chat – but more features are coming soon.

Eventually, the team wants to figure out and add a more full-featured, decentralized and private version of DMs (which may involve integrating with some existing private messages standard). This is however pretty far down the list at the moment, so something not likely to come this year.

Post editing

This will definitely be added at some point, but it’s also a relatively complex feature, because of the need to store the previous versions of an edited post that you should be able to access somehow. They want to do it right and that will require some thinking on how to solve the problem in the most general and elegant way.

Soft-blocking / removing followers

There is an issue currently that there is no way to remove someone from your followers except by having them blocked. If you block-and-unblock them, what some people call “soft blocking”, they stay on the followers list. This is a current limitation of how follows are designed in the protocol – when someone follows you, they do it by adding a “follow record” to their account, and only they can update or delete their own records, you can’t do that from your side.

I think this is likely to get fixed at some point with some kind of workaround, but it’s not trivial to add and not high priority at the moment – but they’re thinking about it.

Polls

This might be a bit tricky, because we probably don’t want to have everyone’s poll choices public to everyone, and right now everything is public… But this is also one of those things that people ask about regularly, so I hope they’ll figure something out.

Two-factor authentication

See the “Account security” section.

Longer posts

This seems to have been a conscious decision that the team wanted to create a medium that’s more like Twitter than like Mastodon when it comes to post length. This isn’t a simple matter of a field length, because this affects the way people communicate, and allowing longer posts has advantages and disadvantages, it’s just a different text form. (See a comment from lead dev about this.)

So it looks like this won’t be changing – although there might later be other “apps” implemented on the AT Protocol that aren’t a part of Bluesky that will allow longer posts, even complete articles, and they might be somehow integrated into Bluesky in the future (e.g. posts bridged from Mastodon by Bridgy, which can be up to 500 characters long, include the full original content in the record data and apps may choose to display the full text inline).

Disabling reposts per user

It’s not a hard technical problem, but it seems to be blocked by some other things right now – but it should be added eventually, since everyone is asking about it.

Links on the profile, scheduling, …

I haven’t heard much about those, but I’m assuming it’s all coming at some point once they get through the hard stuff they’re busy with now. (Scheduling can currently be done using some third party clients like deck.blue, or cross-posting services like Buffer.)

Other tools

Finally, here are a few tools written by third party devs that you might find useful:

Firesky – a site that shows you a live feed of every single new post made on Bluesky – feels like watching the Matrix screen
Clearsky – lets you look up the list of all people that are blocking you (or someone else), or the mute/block lists, user lists, and starter packs that you’ve been added to
Listifications sends notifications when you’re added to a list or starter pack, or when someone blocks you
wolfgang.raios.xyz – also shows your blockers, block statistics and can generate “interaction circle” images that show who you mostly keep in touch with
cred.blue – calculates your “Bluesky score”
Jaz’s Profile Cleaner – lets you delete old data (old posts etc.) from your account
internect.info – lets you look up the DID of an account, its server name and how the assigned handle has changed in the past
Jaz’s post stats (total count and daily posters), my stats (daily/weekly stats), and bskycharts (charts of firehose activity and daily/monthly active users)
Skythread – a thread reader by yours truly, mentioned earlier :]
Label Scanner, for checking if there are any labels assigned to an account or post
Bluesky quiet posters – shows the list of people you follow ordered by how long ago they’ve posted anything
Clean follow – lets you clean up your follows list of blocked and suspended accounts
Gentle Unfollow – lets you browse your follows one by one, showing each person’s recent posts, and lets you decide if you want to keep them or unfollow
Bluesky network analyzer – analyzes your 2nd degree connections (follows of your follows), and shows you suggestions of people that a lot of your friends are following
nws-bot.us tools – a set of tools for managing lists and starter packs

And some other guides (some mentioned earlier in the blog post):

Bluesky Guide (Amanda Wyatt Visconti, Aug 2023 + updates)
Bluesky Social New User Guide (Kairi, Nov 2023 – archived)
How to get started on Bluesky (Emily Hunt, Nov 2023)
Bluesky user FAQ (Bluesky, May 2023)
The Newskies' Guide to Safety and Privacy on Bluesky (eepy, Oct 2023)
How to set your domain as your handle (Bluesky, Apr 2023)
Tips and Tricks for Bluesky Search (Bluesky, May 2024)
The Guide to Feeds (Jerry Chen – archived)
Algorithmic Choice with Custom Feeds (Bluesky, Jul 2023)
How to use Bluesky to grow your brand (Dame, Jun 2025)
What the hell is the atmosphere anyway: A slightly less technical intro to the technical side of Bluesky (Bailey Townsend, Sep 2025)
Open Social – a high-level overview of how the AT Protocol brings back the openness of the original pre Web 2.0 web (Dan Abramov, Sep 2025)
Bluesky Crash Course: Labelers (Kairi, Apr 2024 – archived)

Thanks to Mozzius, Shreyan and Marshal for the feedback on the first draft :)

Changelog:

18 Nov 2025: added links to Tokimeki, Catbird, Dhaaga, and Graze, removed link to Skychat

26 Sep 2025: added links to Bailey Townsend’s and Dan Abramov’s blog posts

8 Sep 2025:

added native bookmarks
custom GIF upload now works on mobile

29 Jul 2025: added link to guide by Amanda Wyatt Visconti

9 Jul 2025:

added new sections about trending topics, verification badges, and the Atmosphere
videos have a 3-minute limit now
mentioned several new photo and video apps, and Tapbots' plans for their Bluesky client
added links to a few new sites and tools, removed US Politics Labeller, Skybridge and PLC Handle Tracker
reordered & expanded settings page descriptions
mentioned Bluesky’s subscription plans
mentioned recent fixes in Discover
mentioned that subscriptions for account new post notifications are private
mentioned reporting malicious lists
mentioned Ozone and country-specific moderation
added info that .bsky.social handles are now kept reserved after change
added some tips about hashtags use
added info about OAuth account management page
added recommendation of Signal for sensitive DM conversations
added info about GIF uploads

12 Nov 2024: added info about threads composer and the plans for disabling reposts

8 Nov 2024:

added info or mentions of new features: videos, starter packs, pinned posts, listing quotes, timed word muting, new safety features, and OAuth
added a list of popular labellers
added a mention of handle services
expanded the “how are things called” section a bit
updated section about feeds, added a list of recommended feeds
updated section about federation, added info about Bridgy Fed
updated info about Jack Dorsey’s involvement

24 Jun 2024: removed link to Aegis (rip)

19 Jun 2024:

added info about built-in Tenor GIFs and DMs
added new section about labellers
Jack Dorsey is no longer on the board
updated some mentions about the Mastodon bridge, which is now live
some changes to feeds section – defaults for Following have changed, Discover is now much better, and you can remove the default feeds

26 Mar 2024: added mention about handle history in the Privacy section

2 Mar 2024: hashtags and word muting are now available, updated the part about longer posts

23 Feb 2024: federation is live!

21 Feb 2024: search now returns results when you search for a hashtag.

2023: Year of social media coding

2023-11-09T14:46:07Z

I had different plans for this year… then, Elon Musk happened.

Elon took over Twitter in October last year, which set many different processes in motion. A lot of people I liked and followed started leaving the platform. Mastodon and the broader Fediverse, which has been slowly growing for many years but never got anything close to being mainstream, suddenly blew up with activity. A lot of those people I was following ended up there.

Then, Twitter started getting progressively worse under the new management. Elon’s antics, the whole blue checks / verification clusterfuck, killing off third party apps and effectively shutting down the API, locking the site behind a login wall, finally renaming the app and changing the logo – each step made some of the users lose interest in the platform, making it gradually less interesting and harder to use.

Changes, so many changes… and things changing meant that I had to change my workflows, change some plans, build a whole bunch of new tools, change plans a few times again, and so on. My GitHub looks like this right now, which is way above the average of previous years:

As usual, I ended up writing way more Ruby and JavaScript than Swift, which goes a bit against my general career plans – but I’ve built so much stuff this year and I had a ton of fun doing it. So in this blog post, I wanted to share some of the things I’ve been working on lately.

The Dead Bird Site 🦤

I had a bunch of private tools written for the Twitter API. For example, I had a script that downloaded all tweets from my timeline and some lists to a local database. I was also running various statistics on tweets, e.g. which people contribute how much to the timeline and list feeds, and automatically extracted links from tweets from some selected lists.

And then Elon shut off access to the API (unless you can afford $100 per month for a “hobbyist” plan), which meant I had to try to find other ways to get that data.

I quickly got the idea that I could somehow intercept the JSON responses that the Twitter webapp (I refuse to call it the new name, sue me) is loading from the JavaScript code. The JSON responses are very complicated with a lot of extra content, but they do contain everything I need. The problem is how to get them; I wanted to get data from my personal timelines, so I couldn’t do anonymous requests, and I didn’t want to make authenticated requests for my account from some hacked-together scripts, for fear of triggering some bot detection tripwire that would lock my account.

So the approach I settled on was to passively collect the requests in the browser, using Safari’s Web Inspector, and export them to a HAR file that can be parsed and processed like the data from the public API. (It would be even better to have a browser extension that intercepts XHR calls on twitter.com automatically, but as far I can tell, there is no way for request monitoring extensions to look at the content of responses, unless you inject scripts to the site.)

I initially tried to implement it as a Mac app, which gave me a chance to start experimenting with Core Data a bit. But in the end, I rewrote it in Ruby and released it as gem I called “BadPigeon” – named after the friends who visit my balcony every day 🐦

The gem is designed to output extracted data in the same form as the Twitter API, in a way that can be plugged into the popular twitter gem, so I could use all existing tools I had written with very little changes. The obvious downside is that it needs some manual help with the recording first, but I can live with that. I’ve been using this setup since June and it works pretty well for me so far.

I had one more Twitter-related project that I sadly had to shut down though – the Rails Bot which has been running non-stop since 2013, mostly unattended, picking and retweeting tweets from some developers in the Ruby community. It requires access to the API to fetch its home timeline periodically from crontab, so I couldn’t make it work this way.

mackuba ∕ bad_pigeon

A tool for extracting tweet data from GraphQL requests made by the Twitter website 🐦

Mastodon’t 🦣

As the migration of developer communities out of Twitter started, I was initially skeptical; looking back, I guess I just had to go through the “five stages of grief” at my own pace… I also didn’t initially see the change as that bad as some others did, and to be honest I still don’t – to me, Twitter still isn’t literal hell on Earth, it’s just that month after month, it got progressively less useful, less interesting and more annoying.

So I finally started looking at Mastodon with interest. The idea of the “Fediverse”, a distributed system of many independent servers with a completely open API, where I don’t need to pay absurd prices for an access key, don’t have monthly download limits and which can’t be taken over and locked down, was appealing to me.

You see, I’m a bit crazy about data hoarding and processing information – for many years I’ve been having various ideas about tools I could write to somehow automate finding more relevant content in the noise of social media, to let me waste less time on it while still finding what’s important (the Rails Bot was a very early example of that). So I thought that maybe in this new open world, where the only limit is my imagination, I could build any tools I ever wanted and share them with others.

Well, turns out it’s not that simple…

It’s true that the Mastodon APIs are completely open and generally permissionless; for example, you can easily download any account’s complete history of “toots”, going as far as a few years back, anonymously. The problem is that there is a certain culture of the existing community of the Fediverse that was there way before the great migration, which is extremely against any kind of data collection, archiving and indexing. Making information searchable – information which is broadcasted in the open to the world – is seen as a threat to safety, and anyone who attempts that is labelled a “tech bro”, derided and attacked.

Sometime in winter I went down the rabbit hole of many, many threads discussing several of such tools, with the authors being attacked and told that they shouldn’t have built them. Just by mentioning in one of the threads that I’m thinking about building a Mac app that allows you to search the history of your home timeline, I got called out on “#fediblock” (Fediverse’s popular channel for warning about bad actors) as someone worthy of blocking.

All of this has very quickly cured me of any ideas to build pretty much any public tool for the Mastodon API. I just don’t have the energy and mental strength to deal with people attacking me this way for simply building tools on an open API that they don’t like.

What I ended up doing though was setting up my own personal Mastodon instance, martianbase.net. I joked that I’m probably the only person who hates Mastodon and also has their own Mastodon instance… But the first instance I signed up on last year was shut down unexpectedly, giving me no chance to migrate the account. That’s another thing I dislike about the ActivityPub system – your account identity and data is bound to the domain of your instance, and there is no easy way out if your admin misbehaves or disappears, or just has a different view on which other servers you should be able to talk to. So at that point I decided not to trust another instance admin, but to set up my own place, so that I can have full control over it.

Blue skies ahead 🌤

And then, just as Twitter was slowly going down and Mastodon has disappointed me – I started hearing about Bluesky.

Started as an idea of Jack Dorsey from Twitter back in 2019, with a goal of building a “decentralized Twitter” that Twitter itself could possibly one day be a part of, the project has been going on for a few years, and just as the whole Twitter chaos started, Bluesky got to the point where it could be presented to the world.

(Important note here, since media has widely promoted Bluesky as “Jack’s social network” and his name puts a lot of people off: it’s not in fact Jack’s social network. He’s not the CEO (a woman named Jay Graber is), he does not manage or control the company, and AFAIK he’s actually been very little involved in it recently, having mostly switched his interest to Nostr – to the point that he has even deleted his Bluesky profile.)

The attention and interest that Bluesky has received after lauching an invite-only beta has widely exceeded the team’s expectations, but this was both a blessing and a curse. They weren’t really prepared to run a real Twitter competitor that could accept the “refugees” escaping Elon’s playground. The thing is, they were mostly focused on the underlying protocol before, and the site itself has been launched as a bit of a demo. A lot of things that people consider pretty essential in a social networking site weren’t ready. But the team – which at that point was less than 10 developers in total, AFAIK – started adapting to the new reality, working as hard as they could to make the site usable for much larger crowds that they had been planning to.

I got access to Bluesky in late April. I don’t want to get into too much detail here about what it’s like, how it’s evolved since then and so on – I’m going to write a few more blog posts about Bluesky specifically. But long story short, I was completely hooked from day one.

Yes, it’s invite-only, has a much smaller userbase than the Fediverse, it’s an early beta, it doesn’t have videos, gifs or even hashtags. The iOS/Swift developers there are as rare as a unicorn. But it has a really nice community of users, third party developers who hack on various tools and help each other, and team members who interact with us on the site all the time. It looks and feels more like Twitter than Mastodon does, and it somehow just feels more fun to be on.

But the thing that excites me the most is the AT Protocol it’s built on and its potential. It’s a completely open federated protocol, just like ActivityPub that Mastodon uses, and it’s intended to eventually create another “fediverse” of distributed social apps (though the federation part is not live yet, but coming soon). It’s designed to take some lessons from what doesn’t work well in ActivityPub (and would be hard to change) and design the architecture better. For example, it uses “Decentralized IDs” (DID) independent of the hosting server to identify accounts, which makes it easy to migrate accounts between servers (and your handle can be any domain name you own, like @mackuba.eu).

The code it’s running on is open source, the APIs are completely open, and it all just invites you to write some tools and libraries for it – to be the first person to write a Ruby library, a Swift SDK, a command line client, a website with statistics. To be the first to plant a flag where others will come later.

I’ve been spending most of the time since April working on one Bluesky-related project after another, sometimes switching between a few in parallel. In the rest of this blog post, I wanted to show you some of the things I’ve been busy building:

Minisky

On the first day after I got in, I already started digging in the API and I wrote a small Ruby script for archiving my timeline and likes (of course I did…). This eventually evolved into a Ruby gem I called Minisky, which provides a minimalistic API client that handles logging in, refreshing access tokens, making GET and POST requests to the API and returning parsed JSON responses.

It doesn’t include any higher-level features like “get posts”, you have to know the name of the endpoint, what params to pass and what fields it returns, but it handles all the basic boilerplate for you. I use it as a base for some internal scripts, and for manually getting or sending some data to the API in the Ruby console. If you want to start playing with the Bluesky API or build some more specific tool that uses it, you can give this library a try (see the example folder for some ideas). It has no dependencies apart from Ruby stdlib.

mackuba ∕ minisky

A minimal client of Bluesky/AtProto API

Custom feeds on Bluesky

Bluesky has a really cool feature that I think is pretty unique among all the social networks. On social sites, you normally have either a reverse-chronological timeline of posts from the people you follow, or some kind of algorithmic “home” feed that mixes them up with other suggested posts, in a way that you usually don’t fully understand and may not like (or both of these feeds).

Bluesky has both of these, but it also lets anyone build a custom feed that selects and orders posts however you like, and most importantly, lets you make this feed available to everyone else. Custom feeds are a core feature of the app; it lets you browse popular feeds from other people, feeds are listed in a separate tab on the feed author’s profile, and you can “pin” the feeds you use often, which puts them in the top bar in the mobile app, as if it was another built-in timeline. People build all kinds of feeds – thematic feeds like various scientific or art or NSFW feeds, feeds for specific communities like “Blacksky”, general “top posts this week” feeds, or different variations of an algorithmic “home feed” using various approaches.

The way the feeds work is that you need to provide an HTTP service on your server which implements a couple of endpoints. The Bluesky server then makes a request to your service on user’s behalf when they want to view the feed, and your service should respond with a JSON that includes a list of post URIs. Bluesky then takes these URIs and turns them into full post JSONs that it returns to the client.

When the team launched this feature back in May, they included a sample feed service project implemented in TypeScript. But I’m not a big fan of JS/TS and Node, so of course I had to reimplement it all in Ruby :]

I’ve spent quite a lot of time working on the feeds and related code this summer, and the result of this is three separate Ruby projects that I’ve open sourced on GitHub (in addition to my main project which is private).

BlueFactory

The first part is an implementation of the feed service itself. I based it on Sinatra, and it implements the three API endpoints required from a feed service. You need to provide some configuration (hostname, DID of the owner etc.) and your custom class to call back to in order to get the list of post URIs and the feed metadata. If you want, you can further customize the server using the Sinatra API, e.g. adding some custom routes with HTML content.

Feeds can generally be divided into two categories: general and thematic feeds that return the same content for everyone, and personalized feeds that show the feed from a specific user’s perspective. The latter are usually much more complicated to build, since you will often need much more data of different kinds to generate the response, depending on your algorithm. If you want to build a personalized feed, the request includes a JWT token that you can use to get the requesting user’s DID, and the gem can pass that as a param to your class (although note that at the moment it does not verify the token, so it can be easily faked).

mackuba ∕ blue_factory

A simple Ruby server using Sinatra that serves Bluesky custom feeds

Skyfall

To return the post URIs from the feed service, first you need to get the posts from somewhere. You could possibly get them from the API, but realistically, a much better option is to connect to a so-called “firehose” web service and stream and save them as they are created, keeping a copy in a local database.

The firehose streams every single thing happening on the network, live – every new and deleted post, follow, like, block, and so on. Depending on your specific feed idea, you will usually only need to keep a small fraction of this data, e.g. only posts and only those that match some regexps – but you need to parse it all first to know what to keep. What further complicates things is that the firehose data does not come in a JSON form, but instead uses a bunch of binary protocols originated from IPLD/IPFS.

The second Ruby gem is meant to simplify this for you. It uses an existing CBOR library to do some of the binary protocol parsing and faye-websocket for the websocket connection. It connects to the firehose websocket on a given hostname and returns parsed message objects with the info about specific add/remove operations and relevant JSON records.

The firehose (and the Skyfall gem) isn’t only useful for creating feed services – you could possibly use it for any other project that needs to track some kind of records from the network in real time, whether it’s follows (to create a connection graph of the whole network, or to track when a follower unfollows you), or blocks (to find out who is blocking you), or to monitor when you or your company or project are mentioned by anyone anywhere. I’ve also included an examples folder with some sample scripts in the repo.

mackuba ∕ skyfall

A Ruby gem for streaming data from the Bluesky/AtProto firehose

Bluesky feeds template

This project puts the previous two together and combines them into an example of a complete Bluesky feed service, which reads posts from the firehose, saves them to an SQLite database and serves them on a required endpoint – basically a reimplementation of the official TypeScript example in Ruby.

This is a “template” repo, which means it’s not meant to be used as-is, but instead forked and modified in your own copy. The reason is there are simply too many things that you may want to do differently – deployment method, chosen database, specific data to keep etc., and making this all configurable would be an impossible task. Instead, I’ve extracted the “input” and “output” parts as separate gems that can be used directly, and you build the parts in the middle – but you can use this template project as a good starting point.

My own feed service project is a private repo, but I’m keeping it in a similar structure to this template and I’m manually backporting some fixes and new features from time to time.

mackuba ∕ bluesky-feeds-rb

Template of a custom feed generator service for the Bluesky network in Ruby

My feeds

And now we get to the part that all of this was for – building my own custom feeds.

I mentioned earlier that I often think about and experiment with various ways to find most relevant content to me on social media. So when I heard about the custom feeds feature, I immediately had an idea to build a feed for Mac/iOS developers that filters only posts on this topic, using a long list of keywords and regexps (I’ve actually reused a lot of work I’ve done a while ago for an unfinished thing I played with on Twitter).

It took me a couple of months to build all the pieces of the “feed generator”, but I’ve launched the Apple Dev feed in July. It isn’t very busy so far, to put it mildly, because there still aren’t that many iOS devs on Bluesky 😅 But as of today, it has 35 likes – only 50 likes less than the other Swift feed :]

Apart from the iOS dev feed, I’ve also made a more general macOS users feed and a couple of other feeds that were mostly a proof of concept / playground while building the service, but a lot of people seem to find them useful anyway, so I’ve left them running:

Skythread

The last one of my Bluesky-related projects, also with a “sky” in the name (most of the third party projects so far have either the word “blue” or “sky” as part of the name 😄). And this one is written in JavaScript for a change.

If you use Twitter and/or Mastodon a lot, you probably have the experience of reading some complicated thread and getting lost, not knowing who replies to whom or if you haven’t missed a whole part of the discussion. These two display branching out threads a bit differently – Twitter hides some of the branches, while Mastodon shows all direct and indirect replies in one flat list. In both cases, it’s not a perfect solution for reading some heated “hellthreads” that branch out endlessly. For me, a UI more like the one on Reddit would be ideal. (Bluesky has recently a thread view with limited nesting, as an experimental feature.)

So that’s what I’ve built, as a web tool. You enter a URL of the root of the thread on bsky.app, and it renders the whole thread as a tree. You can use the +/– buttons to collapse and expand parts of the tree, just like on Reddit, and if you log in, you can also click the heart icons below a comment to like it:

The initial version of Skythread required logging in first to see anything, but I’ve recently switched it to a different API that allows me to load whole threads without an access token. Note that the official Bluesky web app currently does not allow viewing any content unauthenticated – just like Twitter after the recent changes – so tools like Skythread, and other similar ones (e.g. Skyview) are the only way right now to share links to posts and threads with people who don’t have an account; but this is a temporary situation and Bluesky should be open to the world (for reading at least) in near future.

mackuba ∕ skythread

Thread viewer for Bluesky

One app to rule them all

So where can you find me now on social media? As you might have guessed from the earlier sections, I’m spending most of the time on Bluesky now; which may be a bit strange, because that’s not where most of my friends and follows from Twitter ended up. A large part of the iOS/Mac/Swift programming community has moved to Mastodon and stayed there, with some stubbornly sticking to Twitter or posting to both. Possibly also to Threads, which I don’t even have access to.

But there’s something about Bluesky and the AT Protocol that really draws me to it… I think it’s some combination of a nicer UI/UX, tech/architecture that I like more, a new community that is only just forming, and having this feeling like I’m blazing the trail, being able to build all the tooling that doesn’t exist yet. I like being part of something that’s being created around me, flying on that plane that’s being built in the air, watching the devs build it live and feeling like I’m part of it all. I enjoy being there, I really want it to succeed, and I want to help with that as much as I can.

So I have friends on all three platforms, and even though I spend most time on Bluesky, I check all three everyday, for slightly different content – Twitter for news, Mastodon for Swift programming, Bluesky for… dopamine? And since some people only follow me here and some only there, I end up manually cross-posting a lot of things to 2 or 3 websites.

Wouldn’t it be nice to have a tool, kind of like Buffer, that can let you post to Twitter, Mastodon and/or Bluesky in parallel? There doesn’t seem to be, so I’ve decided to build one myself :] This one isn’t available yet and it still needs a lot of work before I can call it an “MVP”, but it’s going to look something like this:

In the meantime, you can follow me here on any of these platforms – listed in the order of preference :)

Social media update - Elon's Twitter and Mastodon

2022-12-22T16:21:01Z

Update 01.03.2023: Updated Mastodon address - my previous instance has been unexpectedly shut down and I had to make a new account. I’ve decided to set up my own server to make sure it won’t happen again.

Update 09.11.2023: I made a follow-up post which talks about the social media related projects I’ve been working on this year, and about Bluesky, where I’m spending most of the time now.

This is just a small update about Twitter and Mastodon, since things have been… very unstable and chaotic in the last few weeks, as you’ve surely noticed if you log in to these even occassionally.

Twitter has been my internet home for over 13 years now. I started using it when my colleagues from Lunar Logic showed it to me, and especially in the recent years it’s been my main source of information and news. It’s where I went to keep track of what was happening in the Apple/Swift world, find useful tips about UIKit, SwiftUI or Xcode, follow the news, rumors and dramas on the Crypto Twitter, and find out every day what important thing was happening in the world, including following the Covid pandemic and the Russian invasion of Ukraine this year.

Like most of the people I’m following, I’m not very happy about Elon’s takeover and his actions, how he randomly makes changes to the rules based on his current mood, blocks journalists who write about him and how he fired or scared away most of the people who kept the site working. I’m worried about how the future looks for the platform, if Twitter will even exist in this form a year or two from now. I wish this all hadn’t happened, and I’m angry at the people who made it happen for their own gain.

But so far, Twitter is still working and is still a great place to get the news, tips and information about so many things. I’m not ready to give up on this site as long as the feed is loading and there are some tweets left to read.

I’ve been trying out Mastodon like everyone else and I slowly get more comfortable there, but it still feels a bit alien to me. It feels like when you move in to a new apartment and everything is different there than you’re used to, some things are missing, some things are better, some things are worse than in your old place, but a lot of your subconscious habits and muscle memory stop working. I had some ways of using Twitter that worked for me and a bunch of private tools I wrote for myself to help me automate some things - I will have to figure this all out again now.

So I am on Mastodon, if only because I don’t want to miss out on things, but I am still on Twitter and I’m planning to stay and keep posting there, as long as it stays usable. I will probably be posting more on Twitter than Mastodon, because I feel more comfortable there. I hope some of my friends and people I follow stay on the platform, or at least check in from time to time.

So here’s where you can find and follow me (updated 09.11.2023):

The SwiftUI cookbook for navigation

2022-07-03T20:54:35Z

This year SwiftUI adds new APIs for handling navigation which scale well from simple to complex UIs, include support for programmatic navigation and deep linking

The existing API

The existing API is based on links (NavigationLink) embedded inside NavigationView

You have a list of navigation link buttons, each specifying the view it links to, and when the button is tapped, the specified view is pushed onto the stack

This works great for basic navigation, and you can continue using this pattern with the new API

To programmatically perform navigation, you could use the NavigationView initializer that binds the state of the navigation link to a variable (using isActive or selection)

However, this only worked well in simple cases and didn't let you easily do things like deep-linking deep into the navigation hierarchy straight from the root, or moving back to the root from there

It also required you to keep separate bindings for each link, or at least each level of the hierarchy

The new navigation APIs

The new API lets you bind the state of the whole navigation stack to a single array managed by the root container

Navigation links push additional values into that array when they're selected

This allows you to quickly move through the hierarchy by modifying that array binding, e.g. to deep-link to a specific view by assigning a collection of values to it, or to pop the stack to the root view immediately by removing all values

Navigation containers:

The new API consists of a couple of new container views and a new version of the navigation link

The first container is NavigationStack:

NavigationStack(path: $path) {
    RecipeDetail()
}

NavigationStack is used for simple push-pop navigation happening in a single container, like in iPhone apps

The second container is NavigationSplitView:

NavigationSplitView {
    RecipeCategories()
} detail: {
    RecipeGrid()
}

NavigationSplitView is perfect for multi-column apps like Mail or Notes on the Mac or iPad

It automatically adapts to a single-column view in contexts like iPhone, Apple TV or slide-over on the iPad

To create a three-column layout, use another version of the initializer:

NavigationSplitView {
    RecipeCategories()
} content: {
    RecipeList()
} detail: {
    RecipeDetail()
}

NavigationSplitView has plenty of options that let you customize things like column widths or sidebar presentation, or even programmatically show or hide columns

You can hear more about configuring a layout like this in the talk SwiftUI on iPad: Organize your interface

Navigation links:

Previously, a navigation link included a title and a destination view to be presented:

NavigationLink("Show detail") {
    DetailView()
}

NavigationLink("Show detail", destination: DetailView())

NavigationLink(destination: DetailView()) {
    Text(title)
}

The new version of NavigationLink still includes a title, but instead of a view to present, it provides a value associated with the link:

NavigationLink("Apple Pie", value: applePieRecipe)

Clicking the link pushes that value into its container's path array

However, the exact view that's pushed when the link is clicked is determined by the navigation stack that the link is contained in

Navigation patterns

Let's now look at the ways you can use these views to build navigation in your apps:

1. Basic stack:

The first pattern is a basic navigation stack like in the Settings app on the iPhone or Apple Watch

The main view presents a list of items, and when an item is selected, it pushes a detail view on the stack which fills the whole screen

This approach works on all platforms, but it's best suited for iPhone, Apple TV and Apple Watch

In the basic form, you can just replace the NavigationView with NavigationStack and use NavigationLinks like before, with the destination view inline:

var body: some View {
    NavigationStack {
        List(Category.allCases) { category in
            Section(category.localizedName) {
                ForEach(dataModel.recipes(in: category)) { recipe in
                    NavigationLink(recipe.name) {
                        RecipeDetail(recipe: recipe)
                    }
                }
            }
        }
        .navigationTitle("Categories")
    }
}

To be able to perform navigation programmatically, assign a value to the navigation link instead:

ForEach(dataModel.recipes(in: category)) { recipe in
    NavigationLink(recipe.name, value: recipe)
}

And the destination view definition is then pulled out of the link block and moved to a new .navigationDestination modifier defined on the NavigationStack container:

.navigationDestination(for: Recipe.self) { recipe in
    RecipeDetail(recipe: recipe)
}

The modifier describes what view should be pushed onto the stack when a given value is pushed into container's path by any of the navigation links anywhere in the view hierarchy (or programmatically), for all values of the given type

The navigation stack's path starts out initially as an empty array []

When the navigation link is clicked:

1. It appends the assigned value of Recipe type into the path
2. The navigation stack then checks the navigationDestination configuration which stores a mapping of Recipe -> some View to get an appropriate view to present
3. The new view is pushed onto the view stack

When you press the "Back" button, the last item is removed from the path array and the top view on the stack is popped

You also need to create a binding of the navigation stack's path to a local state variable:

@State private var path: [Recipe] = []

→ you can use an array of a specific type if you only use values of one type in the path; if you want to build mixed paths of multiple types, use the new NavigationPath wrapper type instead

Pass a binding to NavigationStack's initializer:

var body: some View {
    NavigationStack(path: $path) {
        ...
    }
}

You can now modify the view stack by assigning to the state variable:

func showRecipeOfTheDay() {
    path = [dataModel.recipeOfTheDay]
}

func popToRoot() {
    path.removeAll()
}

You can see the NavigationStack in action in the talk Build a productivity app for Apple Watch

2. Multi-column presentation without stacks:

This interface is similar to the Mail app on the Mac and iPad

The layout consists of:

a sidebar, initially hidden, which shows a list of categories
the second column that shows a list of recipes
the main area that shows the recipe details

This interface is great on larger devices like the iPad and Mac

It allows you to see more information at the same time on the screen

In the first column (sidebar), we add a list of navigation links for each category of recipes:

@State private var selectedCategory: Category?

var body: some View {
    NavigationSplitView {
        List(Category.allCases, selection: $selectedCategory) { category in
            NavigationLink(category.localizedName, value: category)
        }
        .navigationTitle("Categories")
    } content: {
        // ... second column ...
    } detail: {
        // ... detail view ...
    }
}

Instead of binding a property to the navigation view's path like before, here we bind it instead to the selected item in the list, on the root level only

When a NavigationLink is contained within a list with the same selection type as the navigation link's value, the link will automatically update the selection of the list when clicked

→ see more about this in the SwiftUI on iPad: Organize your interface talk

In the second column (content), we show the same kind of list, listing recipes in the selected category, and again, we use NavigationLinks inside with assigned values of the same type as the list's selection binding

When a recipe is selected in the second column, the recipe details will be shown in the main area:

@State private var selectedCategory: Category?
@State private var selectedRecipe: Recipe?

var body: some View {
    NavigationSplitView {
        // ...
    } content: {
        List(dataModel.recipes(in: selectedCategory), selection: $selectedRecipe) { recipe in
            NavigationLink(recipe.name, value: recipe)
        }
        .navigationTitle(selectedCategory?.localizedName ?? "Recipes")
    } detail: {
        RecipeDetail(recipe: selectedRecipe)
    }
}

Note: unlike NavigationStack, NavigationSplitView *does not* keep a path property with a list of selected items that you can bind to – the navigation on this level is managed only through the selection bindings of the two lists:

func showRecipeOfTheDay() {
    let recipe = dataModel.recipeOfTheDay

    selectedCategory = recipe.category
    selectedRecipe = recipe
}

NavigationSplitView also does not use the .navigationDestination modifier

In environments that don't support multi-pane views (iOS, tvOS and iPad apps in split view or slide-over), NavigationSplitView renders as a single-pane NavigationStack which pushes the selected views onto its stack

3. Putting the two modes together – a multi-column view with a stack:

Here, we're building a two-column layout like in the Photos app on the iPad and Mac:

the left column (sidebar) shows a list of categories
the main area will show recipe photos in a grid
when a recipe is selected, a recipe details view will be pushed on the navigation stack within the detail area

To build this kind of layout, we use NavigationSplitView and NavigationStack together:

1. The first column displays a list like in the three-pane version
2. The detail area contains a navigation stack like in the first example

We track the state of the navigation using two variables: a list selection binding for the split view's first column, and a path of values for the stack view inside the second column:

@State private var selectedCategory: Category?
@State private var path: [Recipe] = []

struct ContentView: {
    var body: some View {
        NavigationSplitView {
            List(Category.allCases, selection: $selectedCategory) { category in
                NavigationLink(category.localizedName, value: category)
            }
            .navigationTitle("Categories")
        } detail: {
            NavigationStack(path: $path) {
                RecipeGrid(category: selectedCategory)
            }
        }
    }
}

The detail view displays a grid of navigation buttons wrapping image tiles; when a tile is pressed, the detail screen for a recipe is pushed onto the stack within the detail pane, which is defined using NavigationLinks and the .navigationDestination modifier:

struct RecipeGrid: View {
    var category: Category?

    var body: some View {
        if let category = category {
            ScrollView {
                LazyVGrid(columns: columns) {
                    ForEach(dataModel.recipes(in: category)) { recipe in
                        NavigationLink(value: recipe) {
                            RecipeTile(recipe: recipe)
                        }
                    }
                }
            }
            .navigationTitle(category.name)
            .navigationDestination(for: Recipe.self) { recipe in
                RecipeDetail(recipe: recipe)
            }
        } else {
            Text("Select a category")
        }
    }
}

To update navigation programmatically, assign the two bound variables to modify selection in the two panes:

func showRecipeOfTheDay() {
    let recipe = dataModel.recipeOfTheDay

    selectedCategory = recipe.category
    path = [recipe]
}

Persisting navigation state

Navigation state can be persisted between launches with the use of Codable and @SceneStorage properties

We'll wrap the navigation state properties into a single model type, so that they can be persisted together to keep the state consistent

This model type will conform to Codable to let us save & restore it

@SceneStorage will be used to automatically persist it together with the scene

Looking at the last example (with both NavigationSplitView and NavigationStack), here's how we update it to bind the UI to the properties in the new model type:

@StateObject private var navModel = NavigationModel()

var body: some View {
    NavigationSplitView {
        List(Category.allCases, selection: $navModel.selectedCategory) { category in
            NavigationLink(category.localizedName, value: category)
        }
        .navigationTitle("Categories")
    } detail: {
        NavigationStack(path: $navModel.path) {
            RecipeGrid(category: navModel.selectedCategory)
        }
    }
}

One thing to note about the NavigationModel: the values used for navigation & list selection will often be your model data types, i.e. whole structs like Recipe here, but we don't want to store the whole structs with all their properties as a part of the navigation path – it would be redundant, because we probably have all the recipe data stored somewhere else in some kind of database.

To store the navigation state, we only need some kind of references to them, like their IDs – so we're going to customize the Codable conformance here to save the recipes in the path as their ID values and restore them back from that on load:

class NavigationModel: ObservableObject, Codable {
    @Published var selectedCategory: Category?
    @Published var path: [Recipe] = []

    enum CodingKeys: String, CodingKey {
        case selectedCategory
        case recipePathIds
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        try container.encodeIfPresent(selectedCategory, forKey: .selectedCategory)
        try container.encode(path.map(\.id), forKey: .recipePathIds)
    }

    required init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        self.selectedCategory = try container.decodeIfPresent(Category.self,
                                        forKey: selectedCategory)

        let recipePathIds = try container.decode([Recipe.ID].self, forKey: .recipePathIds)

        // some recipes might have been deleted in the meantime, so skip those
        self.path = recipePathIds.compactMap { DataModel.shared[$0] }
    }

    // helper to serialize the model to/from JSON Data
    var jsonData: Data? {
        get { ... }
        set { ... }
    }

    // async sequence of updates (from the sample code project)
    var objectWillChangeSequence: AsyncPublisher<Publishers.Buffer<ObservableObjectPublisher>> {
        objectWillChange
            .buffer(size: 1, prefetch: .byRequest, whenFull: .dropOldest)
            .values
    }
}

The one missing piece is persisting the model using @SceneStorage

Since SceneStorage can only hold values of simple types like URL, String or Data, and not complex object types, we need to manually restore the model from the persisted form and make sure it's kept in sync after every change; SceneStorage will handle the persistence of the encoded value for us, but we need to handle the conversion to/from it

We do this by providing a .task that will run when the view is displayed, which loads the model from the data

The task then subscribes to a sequence of changes from the model and encodes the model into storage every time it changes:

@StateObject private var navModel = NavigationModel()
@SceneStorage("navigation") private var data: Data?

var body: some View {
    NavigationSplitView {
        // ...
    }
    .task {
        if let data {
            navModel.jsonData = data
        }
        for await _ in navModel.objectWillChangeSequence {
            data = navModel.jsonData
        }
    }
}

For more details and examples about the new APIs, check out the "Migrating to new navigation types" documentation article

See also "Bring multiple windows to your SwiftUI app" for info about opening new windows and scenes

What's new in AppKit

2022-06-08T23:57:53Z

Stage Manager

New UI workflow that cleans up inactive windows in your workspace, while your active window takes center stage, letting you put the focus on one task at a time

You can also pull windows into sets that are swapped in and out as a group

This has an impact on how your app's windows present themselves:

newly presented windows replace those on the stage to keep the workspace tidy
auxiliary windows like panels, popovers, settings windows cohabitate with primary windows

This should mostly work as expected using existing NSWindow APIs:

Stage Manager won't swap out a window if it's a floating panel, a modal window, or a window with toolbarStyle set to .preference
it also obeys the window's collectionBehavior flags (an old API from OS X 10.5-10.7 which mostly defines how a window behaves within desktop spaces, Exposé and full screen mode): a window will not displace an active window in center stage if it includes the .auxiliary, .moveToActiveSpace, .stationary or .transient collection behavior flags

Preferences

The System Preferences app now has a whole new look, with a different navigation and new design

To align with Settings apps on other platforms, it's been renamed to System Settings

If you're shipping a custom preference pane bundle, it will continue to work in the new Settings app

Your custom pane will appear in the Settings app sidebar and will be loaded on first access, like in Monterey and earlier versions

In-app settings:

To match the newly renamed System Settings app, your app's Preferences menu entry in the app's main menu is now renamed to "Settings…"

The menu item's label is automatically updated if you build the app with the latest SDK

Check your app for any other uses of the word "Preferences" in the UI referring to the preferences window and update them accordingly

New form design:

There is a new interface style for control-rich forms like the views commonly used in settings windows

It's designed to present an interface that displays a lot of controls in a clear, organized way

In this kind of UI, the form provides a lot of visual structure, so many system controls (like popup buttons) automatically adapt to this context by drawing normally with a lower visual weight and only revealing a heavier border or background on hover

You can build a form like this using the SwiftUI Form element with an .insetGrouped style:

Form {
    TextField("Computer Name", text: $name)
    Toggle("Screen Sharing", isOn: $screenSharing)
    Toggle("File Sharing", isOn: $fileSharing)
    Picker("AirDrop", selection: $airdrop) {
        ForEach(AirDropVisibility.allCases) {
            Text($0.label).tag($0)
        }
    }
}
.formStyle(.insetGrouped)

This renders Picker as a borderless popup button, TextField as a borderless inline text field, and Toggle as tiny switches

SwiftUI automatically handles the visual style, layout and the scrolling behavior of such form

If you aren't using SwiftUI yet, then… well… this is a good moment to start 😅

Control updates

NSComboButton:

A combo button is a new control – a button which provides an immediate action and a menu for additional options

It combines a standard button and a pull-down menu into a single control

This design is commonly used for use cases like a "move to folder" button in Mail, which moves an email to the pre-selected folder if you click the main left part of the button (the primary action), but also lets you pick an alternative location from the menu if you click the arrow on the right side

There are two styles of the combo button which change its appearance and behavior:

split style (the default), with a separator between the main button part and the arrow part
unified style – looks like a normal button, performs the primary action on click, and only presents the menu if you click and hold

NSColorWell:

NSColorWell has a new default design, with a white border and rounded corners

There are also two optional new styles you can pick:

1) A "minimal" style:

no borders, filled completely with color
a disclosure arrow appears on hover
it shows a color selection UI in a popover – by default a standard system grid of colors, but you can customize what's shown there

2) An "expanded" style:

used in iWork apps previously
combines both interaction models: a borderless minimal style on the left, with a disclosure arrow and a popover, and a button part which shows the full color picker on the right

Select the style using the colorWellStyle property

The behavior of the popover in "minimal" and "expanded" modes is configured through a new target-action pair: pulldownTarget + pulldownAction (if nil, then a standard system popover is used)

→ you can use this to present a custom popover, or a completely different UI like a menu

Toolbars:

There are two new delegate methods of NSToolbar that let you customize the toolbar behavior:

1) toolbarImmovableItemIdentifiers(_:)

defines a list of toolbar items that the user shouldn't be able to move or remove (they also don't animate when you enter the toolbar editing view)
used e.g. in Mail app for a filter button that should always appear above the messages list

2) toolbar(_:, itemIdentifier:, canBeInsertedAt:)

allows you to manually approve any reordering, insertion or removal of a toolbar item
use this to implement a custom set of toolbar customization rules, e.g. an item that has to stay within one toolbar section

Centered toolbar section:

a new property centeredItemIdentifiers allows you to specify multiple centered items, which have to stay within the toolbar's center section
previously there could be only one centered item, specified using centeredItemIdentifier

Preventing item resizing:

if a toolbar item changes its label depending on the state (e.g. "Mute / Unmute" in Mail app), it may have to change its size and cause other nearby buttons to shift back and forth when switching
to prevent this, you can now list all possible labels in the possibleLabels property of an NSToolbarItem; the toolbar will automatically position items according to their longest possible label so that they don't have to be resized

Alerts design:

There is a slight update to the design of alerts

Alerts in macOS Big Sur and above have a new, compact layout with centered text, which is optimized for small amounts of text accompanied by a few clear choices

In general, this is how an alert should look: alerts work best with shorter text, which your users are more likely to read before dismissing the dialog

However, sometimes you really can't make the description shorter, especially when the alert is about something very important like deleting a disk volume

For these cases, there is now a new expanded look of alerts which is more optimal for longer text

The expanded style uses a wider window, horizontally arranged buttons, and left-aligned text

There is no option to enable this – expanded style is used automatically when:

the description text is too long to fit comfortably in the compact size
or if the alert has an accessory view that's too large to fit in a compact alert

Note: the layout variant is chosen at the time when the alert is presented, so the alert is not resized if the text is changed after it's displayed

You should still aim to reduce the length of the alert text whenever possible, but this should improve the UI for those cases when you can't do that

NSTableView performance updates:

NSTableView is designed to efficiently handle a very large number of rows

It does this by lazily populating and reusing views as the table is scrolled

This becomes a challenge when the views have different heights, because it needs to calculate the total height of the table and the position of each row

Previously, NSTableView did this be pre-calculating the size of each row in the table, which impacts the initial load time

In macOS Ventura NSTableView now calculates the row heights lazily:

row heights are calculated for rows which are in or near the scrolling viewport
for the rest of the rows, NSTableView estimates the height based on the row heights that it has already measured
as the table is scrolled, the table view requests more row heights as needed and replaces the estimates with real measurements

This optimization significantly improves load times for very large tables

It's automatically used for all apps on macOS Ventura and doesn't require any changes

Note that this might affect the timing of delegate callbacks like tableView(_:, heightOfRow:)

SF Symbols

SF Symbols 4 adds more than 450 new symbols, including things like new currency symbols, household items or sports objects

SF Symbols can be used with one of 4 styles: monochrome, hierarchical, palette and multicolor

In macOS Ventura, symbols may now specify a "preferred rendering mode" – one of these styles that is preferred for this specific symbol and is used automatically if not configured otherwise

You can still use NSImageSymbolConfiguration to override the preferred style

Variable symbols:

There is a new type of symbol which represents some value or quantity, like WiFi signal strength or volume

These symbols have several versions depending on the value, with a smaller or larger part of the symbol filled or colored

You configure these symbols by providing a floating point value between 0 and 1:

class NSImage {
    public init?(symbolName: String,
                 variableValue: Double,
                 accessibilityDescription: String?)

    public init?(systemSymbolName: String,
                 variableValue: Double,
                 accessibilityDescription: String?)
}

If the symbol doesn't define any thresholds depending on the value, the value is ignored

Sharing

macOS Ventura includes a new redesigned share sheet, which now looks similar to the one on iOS

It includes a list of suggested people to share with, like the iOS one

It also adds new APIs which apps can use to let the user invite their contacts to collaborate on a document

The share sheet is accessed using the existing API NSSharingServicePicker

You can still filter the list of services and add custom ones, use existing delegate callbacks etc.

If you share a file URL, the sheet will automatically display the file name, type, size and icon in the header

If you're sharing a custom item, you can manually provide details to be displayed in the header using a new protocol NSPreviewRepresentableActivityItem:

protocol NSPreviewRepresentableActivityItem: AnyObject {
    /* The item to be shared */
    var item: Any { get }

    /* A localized string representing the item's name or title */
    optional var title: String? { get }

    /* A provider for a full-sized image that represents the item */
    optional var imageProvider: NSItemProvider? { get }

    /* A provider for a thumbnail-sized icon that represents the item */
    optional var iconProvider: NSItemProvider? { get }
}

or the helper class NSPreviewRepresentingActivityItem which implements the protocol:

class NSPreviewRepresentingActivityItem: NSPreviewRepresentableActivityItem {
    init(item: Any, title: String?, image: NSImage?, icon: NSImage?)
    init(item: Any, title: String?, imageProvider: NSItemProvider?, iconProvider: NSItemProvider?)
}

Use the preview class if you already have all the data you need, and the protocol with NSItemProviders if it's too performance-intensive to generate it up front

If you want to launch the share sheet from a menu, like the main menu bar or a context menu when clicking a collection view item, the share sheet API now provides a standard "Share…" menu item that you can put in any menu to display the sheet:

class NSSharingServicePicker {
    var standardShareMenuItem: NSMenuItem
}

In case of context menus, the displayed popover will be anchored to the same view on which the context menu was shown

There's a lot more new features for managing collaboration

See more in separate talks: "Enhance collaboration experiences with Messages" and "Integrate your custom collaboration app with Messages"

Add rich graphics to your SwiftUI app

2022-06-04T22:50:55Z

Safe area

By default SwiftUI positions your content within the safe area, so that it doesn't get obscured by things at the top and the bottom of the screen, like the navigation bar and the home bar on Face ID phones

If you do want the view to extend to the whole screen, edge to edge, below the top/bottom bars (e.g. on screens that show some kind of graphics), you can opt out of the safe area using .ignoresSafeArea:

ContentView()
    .ignoresSafeArea()

You can also choose on which sides you want to extend the view into the safe area:

ContentView()
    .ignoresSafeArea(edges: .bottom)

You don't need to do this on most screens, since most content should be kept within the safe area

The keyboard, if visible, is also outside the safe area – it actually has its own inner safe area that's positioned inside the main "container safe area"

By default your view is positioned within the keyboard's safe area, so that it doesn't get covered by the keyboard – to opt out just of the keyboard safe area (but stay within container safe area), add the additional .keyboard parameter:

ContentView()
    .ignoresSafeArea(.keyboard)

Backgrounds

There are now new versions of the .background modifier where if you don't pass any specific color or style, it uses the default screen background (white or black, depending on light/dark mode):

HStack {
    ...
}
.background()

These backgrounds (and backgrounds like .background(Color.green) now automatically extend below the safe area, unless you customize it

You can also specify that the background should only be used within a defined shape around the view, like a rounded rectangle:

HStack {
    ...
}
.background(.cyan, in: RoundedRectangle(cornerRadius: 12))

In this case, the view and its entire background area both stay within the safe area

Materials and vibrancy

Materials are a new kind of backgrounds – the kind of translucent blur background used in various system areas:

.background(.regularMaterial)
.background(.thickMaterial, in: RoundedRectangle(cornerRadius: 12))

Materials are great if you want to show some of the colorful content positioned below some kind of panel bleeding into its background

There's a set of materials: .ultraThin, .thin, .regular, .thick, .ultraThick – the "thinner" the material, the more of the color from behind comes through

Some of the content displayed over a material background (using foreground colors of "secondary" and below) uses an effect called "vibrancy" – it blends the color of the text with the material background behind it in a specific way that makes it more readable

To apply the vibrancy effect, use the new .foregroundStyle modifier instead of .foregroundColor:

HStack {
    Text("\(stops.count) colors")
        .foregroundStyle(.secondary)
}
.background(.regularMaterial)

You can use .foregroundStyle with hierarchical styles like .secondary and with actual colors (or even gradients), and you can even use both together, in which case SwiftUI applies some alpha effect to the selected color:

VStack {
    Text("Primary").foregroundStyle(.primary)
    Text("Secondary").foregroundStyle(.secondary)
    Text("Tertiary").foregroundStyle(.tertiary)
    Text("Quaternary").foregroundStyle(.quaternary)
}
.foregroundStyle(.purple)

Use .foregroundColor if you want to opt out of vibrancy for a specific view and use an unmodified color

What's more, a Text can even present an attributed string that has multiple different colors applied to different ranges and use a vibrancy effect for the parts that don't have a specific color set (it can only have one single foregroundStyle though)

You can embed one Text inside another to construct a Text with multiple colors in different ranges:

Text("\(stops.count) \(Text("colors").foregroundColor(.red))")
    .foregroundStyle(.secondary)

It's ok to use .foregroundStyle on non-material backgrounds – the vibrancy effect is automatically disabled unless there is an appropriate background behind the text

Text views also automatically disable vibrancy for any embedded emoji

You can read more about vibrancy and materials in my old blog post about the introduction of dark mode in macOS Mojave – "Dark Side of the Mac: Appearance & Materials"

Safe area inset

There is a new view modifier .safeAreaInset that allows you to position a view on top of a ScrollView in a way that adjusts the scroll view's top/bottom content insets so that all the content can be reached and the beginning/end isn't obscured by the overlay

This is especially useful for views using the blur material background, since you want them to be z-stacked with some content on a layer below to show a hint of the colors behind it, but you want all the content in the view below to be accessible by scrolling it above the overlay

To position a view this way, put the whole overlay view inside the .safeAreaInset closure:

List {
    ...
}
.safeAreaInset(edge: .bottom) {
    HStack {
        Text("New gradient")
        Spacer()
        Text("\(stops.count) colors")
            .foregroundStyle(.secondary)    
    }
    .padding()
    .background(.thinMaterial)
}

Canvas

When drawing a large number of graphical elements in a shared area, we can use a ZStack and the .drawingGroup modifier:

var body: some View {
    GeometryReader { proxy in
        ZStack {
            ForEach(state.entries.indices, id: \.self) { index in
                let entry = state.entries[index]
                entry.symbol
                    .scaleEffect(...)
                    .position(...)
                    .onTapGesture {
                        withAnimation { ... }
                    }
            }
        }
    }
    .drawingGroup()
}

A .drawingGroup tells SwiftUI to combine all the views it contains on a single layer to improve performance

This can be used for graphical elements like images, but not for UI controls like text fields or lists

Using .drawingGroup still allows you to do things like setting accessibility properties or applying gestures to each element individually

However, keeping each view's separate identity adds a bit of overhead

When you draw a really large amount of elements and you need all the performance you can get, you can now use the new Canvas element

→ the tradeoff is that you can't attach gestures to individual drawings anymore

A Canvas is given a closure that's run every time the canvas is redrawn

This is a normal, imperative code closure, not a view builder

It works similarly to drawRect in AppKit or UIKit – the canvas gives you a context object that you can run some draw commands on:

Canvas { context, size in
    let image = Image(systemName: "sparkle")

    for i in 0..<10 {
        context.draw(image, at: CGPoint(
            x: 0.5 * size.width + Double(i) * 10,
            y: 0.5 * size.height
        ))
    }
}

When drawing an image, the context needs to "resolve" it to get the actual data it can draw, so if you use the same image multiple times in the canvas, you can resolve it manually up front just once

This also lets you access information like the image size:

Canvas { context, size in
    let image = context.resolve(Image(systemName: "sparkle"))
    let imageSize = image.size

    for i in 0..<10 {
        context.draw(image, at: CGPoint(
            x: 0.5 * size.width + Double(i) * imageSize.width,
            y: 0.5 * size.height
        ))
    }
}

To draw shapes in the canvas, use context.fill()

You can draw e.g. bezier curves or paths derived from standard SwiftUI shapes

You can also use standard SwiftUI color objects:

Canvas { context, size in
    let image = context.resolve(Image(systemName: "sparkle"))
    let imageSize = image.size

    for i in 0..<10 {
        let frame = CGRect(
            x: 0.5 * size.width + Double(i) * imageSize.width,
            y: 0.5 * size.height,
            width: imageSize.width,
            height: imageSize.height
        )

        context.fill(
            Ellipse().path(in: frame),
            with: .color(.cyan)
        )
        context.draw(image, in: frame)
    }
}

To change some properties globally on the context and revert them later, you can create a copy of the context on which you modify the settings, draw some elements using the new context, and the original context will not be affected:

Canvas { context, size in
    let image = context.resolve(Image(systemName: "sparkle"))
    let imageSize = image.size

    for i in 0..<10 {
        let frame = ...

        var innerContext = context
        innerContext.opacity = 0.5
        innerContext.fill(Ellipse().path(in: frame), with: .color(.cyan))

        // drawn with the original opacity
        context.draw(image, in: frame)
    }
}

You can set the color with which a symbol is drawn by setting the shading on the resolved image:

let image = context.resolve(Image(systemName: "sparkle"))
image.shading = .color(.blue)

The Canvas is supported on all platforms – it works the same way on iOS, watchOS, tvOS and macOS

Timeline view

Timeline view is a new tool that allows you to create animating views by describing exactly how the view should look at a given point in time

The timeline view requires a "schedule" – a description of how often it should change

For drawing an animated canvas, we'll use the .animation schedule, which redraws the view as often as possible

TimelineView gives you a timeline object from which you can read the current time using the date property:

TimelineView(.animation) { timeline in
    Canvas { context, size in
        let now = timeline.date.timeIntervalSinceReferenceDate

        let angle = Angle.degrees(now.remainder(dividingBy: 3) * 120)
        let x = cos(angle.radians)

        var image = context.resolve(Image(systemName: "sparkle"))
        let imageSize = image.size

        for i in 0..<count {
            let frame = CGRect(
                x: 0.5 * size.width
                    + Double(i) * imageSize.width * x,
                y: 0.5 * size.height,
                width: imageSize.width,
                height: imageSize.height
            )

            context.draw(image, in: frame)
        }
    }
}

Since there aren't any individual SwiftUI elements inside the Canvas, you can't specify accessibility information for each element, only for the whole canvas

If you want to include a more detailed description with separate elements, you can use the new .accessibilityChildren modifier to provide a completely separate SwiftUI view hierarchy used only for accessibility purposes

See more in "SwiftUI accessibility: Beyond the basics"

What's new in Foundation

2022-06-04T21:54:43Z

Attributed strings

An attributed string allows you to associate attributes in the form of key-value pairs with specific ranges of the string, in order to make a part of the text use a different style

A part of a string can have multiple attributes applied to it and the attribute ranges can overlap

Attributed strings are usually used with APIs that support rich text, like UITextView/NSTextView

String attributes are defined in the SDK, but you can also add your own

Attributed strings in Foundation previously used an ObjC-based reference type called NSAttributedString

This year, there is a completely new Swift-only struct type called AttributedString, which takes full advantage of Swift features

The Swift AttributedString is a value type, compatible with String, fully localizable and with a more type-safe API

ℹ️ I've replaced some of the code samples in this section to better demonstrate the use of the API in this form

Assigning attributes to strings:

To assign attributes to the whole string, simply assign an appropriate value (which is type-checked at compile time) to a property like font or foregroundColor on the attributed string object:

var thanks = AttributedString("Thank you!")
thanks.font = .body.bold()

var website = AttributedString("Please visit our website.")
website.font = .body.italic()
website.link = URL(string: "http://www.example.com")

To assign attributes to a range, use slicing:

let msg = AttributedString()
let start = msg.startIndex
let after5 = msg.index(start, offsetByCharacters: 5)

msg[start ..< after5].foregroundColor = .blue

You can also find a range by looking for a substring in the text:

var website = AttributedString("Please visit our website.")

if let range = message.range(of: "visit") {
    message[range].font = .body.italic().bold()
    message.characters.replaceSubrange(range, with: "surf")
}

It may also be useful to hold a set of attributes separately from the string itself and then apply it to one or more strings at some point – for that you can use an AttributeContainer:

var container = AttributeContainer()
container.foregroundColor = .red
container.underlineColor = .primary

website.mergeAttributes(container)

Iterating over an attributed string:

To iterate over the string's contents, you can use one of the provided "views" into the string:

characters, which provides a collection of separate characters
runs, which lists ranges of text, each having a single uniform set of attributes

These views are Swift collections, so you can use any standard collection methods on them

Here, we iterate over the collection of characters in a string, modifying the attributes of some of them:

let characters = message.characters

for i in characters.indices where characters[i].isPunctuation {
    message[i ..< characters.index(after: i)].foregroundColor = .orange
}

Now, assuming we have an attributed string like this:

var line1 = AttributedString("Thank you! ")
line1.font = .boldSystemFont(ofSize: 10)

var line2 = AttributedString("Please visit our website")

// make the word "website" a link
let range = line2.range(of: "website")!
line2[range].link = URL(string: "http://example.com")

let message = line1 + line2

If we iterate over the string's runs collection like this, we get 3 items, because there are three distinct sections in the string:

let parts = message.runs.map { run in
    String(message.characters[run.range])
}

// => ["Thank you! ", "Please visit our ", "website"]

However, we can also divide the string into runs only by looking at one specific attribute, e.g. .link – in this case, there will be only two separate runs:

let parts = message.runs[\.link].map { (value, range) in
    String(message.characters[range])
}

// => ["Thank you! Please visit our ", "website"]

The value provided for each one will be of URL? type in this case – the first run will have a value of nil, and the second one will have the URL we've assigned earlier:

for (value, range) in message.runs[\.link] {
    if let url = value {
        print(url.host!)    // => "example.com"
    }
}

Attributed string localization:

Attributed strings are fully localizable

The old ObjC-based NSAttributedString now also has localization support added

Attributed string localizations are listed in .strings files, just like for normal strings

Both plain strings and attributed strings can now be localized in Swift code using String interpolation, in a similar way to how it's done in SwiftUI:

func prompt(for document: String) -> AttributedString {
    AttributedString(localized: "Would you like to save the document '\(document)'?")
}

Xcode can now collect localizable strings for the .strings files from such initializers in your code using the Swift compiler

You can turn this on with the build setting: "Localization > Use Compiler to Extract Swift Strings"

Markdown support:

Attributed strings now automatically parse Markdown content and generate attributes from tags like **bold**:

struct ReceiptView: View {
    var body: some View {
        VStack {
            Text("**Thank you!**")
            Text("_Please visit [our website](https://example.com)._")
        }
    }
}

Conversion and archiving:

The new struct-based attributed strings can be easily converted to and from the old ObjC-based reference types:

let nsString = NSAttributedString(message)
let swiftString = AttributedString(nsString)

AttributedString conforms to Codable, so you can encode a string with all its attributes into whatever form that Codable supports

Custom attributes:

Apart from the built-in string attributes provided by AppKit, UIKit, SwiftUI and other system frameworks, you can also define your own attributes

A string attribute consists of a key and a value

The key is a type the conforms to the AttributedStringKey protocol

It defines the type used for the value (through the associated type Value) and the name used for archiving (static var name)

The key can also customize encoding and decoding by conforming to some other protocols

Here, we define a "rainbow" attribute which colors the range of the text with rainbow colors, whose value is one of the three enum cases that set the intensity:

enum RainbowAttribute: AttributedStringKey {
    enum Value: String {
        case plain
        case fun
        case extreme
    }

    public static var name = "rainbow"
}

If we want the attribute to be encoded & decoded with the string, it needs to be Codable:

enum RainbowAttribute: CodableAttributedStringKey {
    enum Value: String, Codable {
        case plain
        case fun
        case extreme
    }

    public static var name = "rainbow"
}

You can also make the custom attribute available to the Markdown parser, using a special custom attribute syntax

To use an attribute in Markdown, it needs to also conform to MarkdownDecodableAttributedStringKey

The attribute is added to a range of text using a similar syntax like images and URLs:

This text contains ^[an attribute](rainbow: 'extreme').

This text contains ^[two attributes](rainbow: 'extreme', otherValue: 42).

This text contains ^[an attribute with 2 properties](someStuff: {key: true, key2: false}).

The part in the square brackets is the marked text, and the part in parentheses are the attributes

The attribute values are written as a JSON5 object

Support for JSON5 parsing was also added to existing JSON parsing APIs like JSONSerialization and JSONDecoder (you need to enable a proper option first):

let json5text = "{ foo: 'bar' }"
let json5data = json5text.data(using: .utf8)!

let decoded = JSONSerialization.jsonObject(with: json5data, options: .json5Allowed)

The names of attributes from the attributes JSON object are looked up in "attribute scopes"

An attribute scope defines a list of attributes from one domain, e.g. SwiftUI or your app

When creating an AttributedString from Markdown, you need to specify one single attribute scope from which attributes will be looked up

However, attribute scopes can be nested in one another, so you can include e.g. a scope of all SwiftUI attributes inside your scope (which in turn includes Foundation attributes)

An attribute scope is defined as a type conforming to AttributeScope, nested inside the AttributeScopes namespace:

extension AttributeScopes {

    // attribute scope for our "Caffe" app

    struct CaffeAppAttributes: AttributeScope {
        // this is our "rainbow" attribute
        let rainbow: RainbowAttribute

        // here, we include standard SwiftUI attributes
        let swiftUI: SwiftUIAttributes
    }

    // expose our attribute scope on a keypath in AttributeScopes
    var caffeApp: CaffeAppAttributes.Type { CaffeAppAttributes.self }
}

// now, pass the keypath to our scope when creating a string:

let header = AttributedString(
    localized: "^[Fast & Delicious](rainbow: 'extreme') Food",
    including: \.caffeApp
)

To actually render a text marked with custom attributes in a specific way, you need to create a custom view that takes an AttributedString with those attributes and displays it – see the example RainbowText SwiftUI view in the "Building a Localized Food-Ordering App sample code project

New formatter APIs

Formatters are used for converting values like dates or numbers into localized and user-presentable strings

Current formatters (NSFormatter) are quite heavy objects, so it's a common pattern to create them once, cache them and reuse them, often across different parts of the app, which isn't always convenient

This year, there is a completely new API for formatters in Swift, which is aimed to improve both their usability and efficiency

The formatting is now done by calling a format method on the formatted value directly

Instead of formatting a date with a date formatter:

let dateFormatter = DateFormatter()
dateFormatter.dateStyle = .medium
dateFormatter.timeStyle = .medium

...

dateLabel.text = dateFormatter.string(from: quakeTime)

We ask the date for a formatted representation:

dateLabel.text = quakeTime.formatted(date: .abbreviated, time: .standard)

Formatting floating point numbers, before:

magnitudeLabel.text = String(format: "%.1f", magnitude)

after:

magnitudeLabel.text = magnitude.formatted(.number.precision(.fractionLength(1)))

This is not only more readable, but also safer – the old version requires using string format specifiers that have to be written in a specific way, aren't checked by the compiler and can't be easily remembered, and if the value is not a floating point number – you will get a wrong output instead of a compiler error

You also now get autocompletion in Xcode for the formatting options

There are new formatting API equivalents for all existing formatters in Foundation – for lists, date components and intervals, measurements, data sizes etc.

They're designed to help you avoid some common pitfalls when using string-based format specifiers, like using a wrong date format field that only fails in some specific cases

Formatting dates:

Simplest version:

let formatted = date.formatted()
// => "6/7/2021, 9:42 AM"

Configure time and date styles:

let onlyDate = date.formatted(date: .numeric, time: .omitted)
// => "6/7/2021"

let onlyTime = date.formatted(date: .omitted, time: .shortened)
// => "9:42 AM"

List specific fields to include:

let formatted = date.formatted(.dateTime.year().day().month())
// => "Jun 7, 2021"

Customize field styles:

let formattedWide = date.formatted(.dateTime.year().day().month(.wide))
// => "June 7, 2021"

let weekday = date.formatted(.dateTime.weekday(.wide))
// => "Monday"

Standardized formats:

let logFormat = date.formatted(.iso8601)
// => "20210607T164200Z"

let fileNameFormat =
    date.formatted(.iso8601.year().month().day().dateSeparator(.dash))
// => "2021-06-07"

Setting a specific locale:

let formatted = date.formatted(.dateTime.locale(myLocale))

The general pattern is:

start with the value to be formatted
call one of the formatted methods
pass a style in the argument (e.g. for dates it's .dateTime or .iso8601)
call methods on the format to customize it

The order of the fields in the chained call doesn't matter – it just lists the fields that should be included somewhere in the final output, and the formatter decides on the order based on the locale

Formatting date ranges:

let range = (now..<later).formatted()
// => "6/7/21, 9:42 - 11:05 AM"

let noDate = (now..<later).formatted(date: .omitted, time: .complete)
// => "9:42:00 AM PDT - 11:05:20 AM PDT"

let timeDuration = (now..<later).formatted(.timeDuration)
// => "1:23:20"

let components = (now..<later).formatted(.components(style: .wide))
// => "1 hour, 23 minutes, 20 seconds"

let relative = later.formatted(.relative(presentation: .named, unitsStyle: .wide))
// => "in 1 hour"

Formatting values as attributed strings:

You can ask a formatter to create an attributed string instead by calling .attributed()

The generated attributed string does not have any visual styles applied to it, but it marks all ranges of text that contain specific fields like month or year with formatter-specific properties

You can then analyze the returned AttributedString and use these format properties to mark different tokens within the generated string with different colors or text styles

var dateString: AttributedString {
    var str = date.formatted(.dateTime
                .minute()
                .hour()
                .weekday()
                .locale(locale)
                .attributed())

    // alternative way to create an AttributeContainer
    let weekday = AttributeContainer.dateField(.weekday)
    let color = AttributeContainer.foregroundColor(.orange)

    // we can ask an AttributedString to replace the attributes
    // in one container set with those in another container

    // here, the text range marked with a "weekday" attribute
    // will have an orange text color applied to it
    str.replaceAttributes(weekday, with: color)

    return str
}

Parsing dates from strings:

To convert the values the other way, you can use a new Date initializer that takes a "strategy" argument

The strategy is an object that tells the parser what kind of fields to expect in the input

You can use one of the format objects shown earlier as a parsing strategy:

let format = Date.FormatStyle().year().day().month()
let formatted = date.formatted(format)
// formatted is "Jun 7, 2021"

if let date = try? Date(formatted, strategy: format) {
  // date is 2021-06-07 07:00:00 +0000
}

You can also use a more specialized strategy object, like this one for parsing dates sent from the server in a specific strict format:

let strategy = Date.ParseStrategy(
    format: "\(year: .defaultDigits)-\(month: .twoDigits)-\(day: .twoDigits)",
    timeZone: TimeZone.current
)

if let date = Date("2021-06-07", strategy: strategy) {
    // date is 2021-06-07 07:00:00 +0000
}

Instead of using magic string format specifiers, ParseStrategy lets you specify the fields using string interpolation – checking the format at compile time, and helping you in Xcode with autocomplete and inline documentation

"No more guessing how many Y characters you should use to parse a year"

Formatting numbers:

let value = 12345

var formatted = value.formatted()
// => "12,345"

let percent = 25
let percentFormatted = percent.formatted(.percent)
// => "25%"

let scientific = 42e9
let sciFormatted = scientific.formatted(.number.notation(.scientific))
// => "4.2E10"

let price = 29
let priceFormatted = price.formatted(.currency(code: "usd"))
// => "$29.00"

Formatting lists:

To format a list of things, call .formatted() on an array and specify which of the formats should be used for each member:

let list = [25, 50, 75].formatted(.list(memberStyle: .percent, type: .or))
// => "25%, 50%, or 75%"

Formatting text field content in SwiftUI:

In SwiftUI, you can assign a format to a text field

The text field will automatically parse and reformat the value entered by the user, and if a correct value of a given type can be parsed from the input, it will assign it through the binding to the connected state property as e.g. a number or a date value:

struct ReceiptTipView: View {
    @State var tip = 0.15

    var body: some View {
        HStack {
            Text("Tip")
            Spacer()
            TextField("Amount",
                value: $tip,
                format: .percent)
        }
    }
}

Automatic grammar agreement

In some languages, like in Spanish, there needs to be a grammatical agreement between the words in a sentence, e.g. in a phrase "two small salads" the number, the noun and the adjective need to have not only the same pluralization, but also same grammatical gender

This makes it often extremely complex to provide full and correct localization for all combinations of UI strings, and either makes the localization process more time-consuming, or makes the translation less correct or less natural

E.g. instead of just translating "small", "large", "juice" and "salad" you need each combination of "small juice", "large salad" etc. (and then add another dimension for pluralizations)

Foundation in iOS 15 adds a new feature called "automatic grammar agreement" that handles a lot of these problems automatically (available in English and Spanish at the moment)

You can now write a string like this:

Text("Add ^[\(quantity) \(size) \(food)](inflect: true) to your order")

The custom attribute Markdown syntax for Attributed Strings is used to mark a part of the text that needs to be automatically inflected with the "inflect" attribute

This will be exported to an English strings file like this:

"Add ^[%lld %@ %@](inflect: true) to your order" =
    "Add ^ [%lld %@ %@](inflect: true) to your order";

"Pizza" = "Pizza";
"Juice" = "Juice";
"Salad" = "Salad";

"Small" = "Small";
"Large" = "Large";

And to a Latin American Spanish strings file like this:

/* in Spanish the order of the noun and adjective is reversed */

"Add ^[%lld %@ %@](inflect: true) to your order" =
    "Añadir ^[%1$lld %3$@ %2$@](inflect: true) a tu pedido";

"Pizza" = "Pizza";
"Juice" = "Jugo";
"Salad" = "Ensalada";

"Small" = "Pequeño";
"Large" = "Grande";

The key change is that you don't need to provide separate translations for e.g. "pequeño" (masculine) and "pequeña" (feminine) – the grammar agreement engine handles this automatically

User's term of address:

In some languages some of the localized text also needs to change depending on the person reading it, because it needs to be adjusted for the person's term of address (depending on the gender)

Now, users using supported languages (currently Spanish) can specify their term of address in the iOS Settings app, under "Language & Region", and can choose to share it with apps

This is done the same way as in the previous example:

^[Bienvenido](inflect: true) a Notas

Such text can be displayed as "Bienvenido…" or "Bienvenida…" depending on the user's gender

You can also provide a default text to be used if this information is not available:

^[Bienvenido](inflect: true, inflectionAlternative: "Te damos la bienvenida") a Notas

Discover concurrency in SwiftUI

2022-06-01T21:24:26Z

The SwiftUI run loop and Observable Objects

The SwiftUI engine runs a loop that receives some events, lets you update your models, checks them for changes, recreates necessary view objects and re-renders the parts of the UI that should be updated

The run loop runs in "ticks" happening in regular intervals, and all of it executes on the main thread / main actor

The events sent from ObservableObject's @Published also need to be sent on the main actor for this to work correctly

Let's see exactly why:

With such update method:

class Photos: ObservableObject {
    @Published private(set) var items: [SpacePhoto] = []

    func updateItems() {
        let fetched = fetchPhotos()
        items = fetched
    }

    func fetchPhotos() -> [SpacePhoto] { ... }
}

the update goes like this:

when we assign the items property, which is @Published, the Published wrapper automatically sends an objectWillChange event that is observed by SwiftUI, before the changes happen
at that point SwiftUI records a snapshot of the data before it changes
then, the new value is written to the property
at the next "tick" of the run loop – which might happen immediately afterwards, or a bit later – SwiftUI compares the current value of the property to the saved snapshot, and this tells it what exactly was changed
since some of the data was changed, the appropriate parts of the UI are re-rendered

If all of the above happens on the main thread, these steps are guaranteed to happen in order

However, if the code takes some time to run, this may cause the run loop to miss the next tick, because the main thread will be busy calculating the data – which will result in dropped frames and a possibly degraded user experience

So you may want to run the calculations on a background queue:

func updateItems() {
    DispatchQueue.global().async {
        let fetched = fetchPhotos()
        self.items = fetched
    }
}

This way, the main thread isn't blocked, the run loop is free to run the next ticks there until the data is ready

However, if the assignment (and the resulting notification) happens outside of the main thread, this could mess up the order of operations and could result in the view not being updated

For example:

the long-running operation finishes running on the background thread
it sends the objectWillChange event
SwiftUI takes a snapshot of the data
*but* because this is all happening on the background thread, the run loop is free to perform another tick on the main thread in the meantime, which it might possibly do at this very moment, right after objectWillChange
in this case, SwiftUI compares the snapshot to the current data, but the data hasn't changed yet, so there's nothing to update
now, the value of the property is updated on the background thread
but SwiftUI has already compared the snapshots, so it has forgotten about the previous state and will not compare the data again…

To make sure the order of operations is always right:

1. objectWillChange,
2. the state changes,
3. and the run loop runs a tick,

all of these need to be done on the same thread – the main thread

Previously, the solution would have been to jump back to the main thread before saving the data:

func updateItems() {
    DispatchQueue.global().async {
        let fetched = fetchPhotos()
        DispatchQueue.main.async {
            self.items = fetched
        }
    }
}

With Swift 5.5, a better approach is to instead make the fetching an asynchronous operation and use await to wait for the result

By using await we "yield" the current (main) thread instead of blocking it and it's free to perform next run loop ticks in the meantime, and when the response is received, the method continues execution – on the same thread where it started, automatically:

func updateItems() async {
    let fetched = await fetchPhotos()
    items = fetched
}

To make sure that all of the updates to properties in the Photos class happen on the main thread and that we don't make a mistake somewhere, we can mark the class with the global actor @MainActor:

@MainActor
class Photos: ObservableObject {
    @Published private(set) var items: [SpacePhoto] = []

    ...
}

This way the Swift compiler guarantees that the code in this class runs on the main thread, at compile time

Associating async tasks with views

SwiftUI provides a new .task modifier that can be used to associate an asynchronous task with a given view

The task is run at the beginning of the view's lifetime, like .onAppear, but the closure your provide to it is asynchronous, which makes it easier to make asynchronous calls with await inside

In this app, we can use the .task modifier to run the method which prepares the photo data in the model that we've written above:

struct CatalogView: View {
    @StateObject private var photos = Photos()

    var body: some View {
        NavigationView {
            List {
                ...
            }
        }
        .task {
            await photos.updateItems()
        }
    }
}

What's more, the task's lifetime is tied to the view's lifetime, so the task is also automatically cancelled when the view is removed from the UI

You can use this to e.g. create a task that continuously reads data from an async sequence for the whole duration of the view's lifetime

Async images

SwiftUI now includes an AsyncImage() view that automatically loads the contents of the image for you from a given URL:

AsyncImage(url: photo.url)

You can customize both the look of the resulting image and the placeholder that is displayed while the image is loading:

AsyncImage(url: photo.url) { image in
    image
        .resizable()
        .aspectRatio(contentMode: .fill)
} placeholder: {
    ProgressView()
}

You can also customize the error handling – for that, check out the AsyncImage(url:scale:transaction:content:) initializer of this view

Running async code from SwiftUI action handlers

To call asynchronous code from inside a button handler closure, which is synchronous, use a Task block to execute a piece of asynchronous code in synchronous context:

struct SavePhotoButton: View {
    var photo: SpacePhoto
    @State private var isSaving = false

    var body: some View {
        Button {
            Task {
                isSaving = true
                await photo.save()
                isSaving = false
            }
        } label: {
            Text("Save")
                .opacity(isSaving ? 0 : 1)
                .overlay {
                    if isSaving {
                        ProgressView()
                    }
                }
        }
        .disabled(isSaving)
        .buttonStyle(.bordered)
    }
}

💡 Tip: to show a spinner inside a button like this, use .opacity() to hide its label while the spinner is displayed – since the label is still there, just with 0 opacity, it makes sure that the button stays at the same size in the loading state

ℹ️ In the talk, the asynchronous code was instead wrapped in a call to an async {} function, which was later replaced with the Task initializer

Reloading the data using pull-to-refresh

SwiftUI adds a new .refreshable modifier this year which automatically implements a pull-to-refresh behavior in your view, where scrolling the contents of the view beyond the top edge triggers a reload of the data

Inside the closure passed to .refreshable, provide the reloading code that should be run on refresh – the code can be asynchronous, so it can use await:

List {
    ForEach(photos.items) { item in
        ...
    }
}
.refreshable {
    await photos.updateItems()
}

Customize and resize sheets in UIKit

2022-05-28T21:42:00Z

iOS 13 introduced a refined appearance for sheets, which don't cover the full screen anymore on the iPhone, but instead show a curved top edge that can be used to dismiss the sheet by pulling it down

iOS 15 expands the sheets API adding a lot of new customizations:

you can make vertically resizable sheets that only cover half of the screen
you can remove the dimming view, creating non-modal UIs where the user can interact with the content behind the sheet
you can display non-full-screen sheets on iPhones in landscape position

Creating sheets

Sheet appearance is managed through a new presentation controller class, named UISheetPresentationController

You don't create a sheet controller instance yourself, but instead get it from the view controller managing the modal screen that you want to present:

if let sheet = viewController.sheetPresentationController {
    // customize the sheet...
}

present(viewController, animated: true)

The sheetPresentationController property will always return a non-nil value as long as the view controller's modalPresentationStyle is formSheet or pageSheet (which it is by default unless you override it)

Detents

A sheet can be configured with a list of so-called "detents"

Detents are positions at which the end of the sheet can rest after it's opened or resized

There are currently two system-defined detents available:

large, which is the normal full size of a sheet, as in previous versions of iOS (the sheet going up almost to the top of the screen on the iPhone)
medium, which is about half the standard height (the sheet filling bottom half of the screen on the iPhone)

You can set the detents property of a sheet to either or both of these:

// standard full-height sheet - the default value
sheet.detents = [.large()]

// a sheet that can be switched between half-height and full-height
// the order matters - the first detent is the initial position
sheet.detents = [.medium(), .large()]

// a sheet that is medium height and cannot be made larger
sheet.detents = [.medium()]

This can even be used on system modal views that normally expect to be presented in a full-height sheet, like the PHPhotoPicker:

func showImagePicker()
    let picker = PHPickerViewController()
    picker.delegate = self

    if let sheet = picker.sheetPresentationController {
        sheet.detents = [.medium(), .large()]
    }

    present(picker, animated: true)
}

func picker(_ picker: PHPickerViewController,
    didFinishPicking results: [PHPickerResult]) {

    // assign result to imageView.image
    // do not dismiss the picker
}

In this example:

calling the first function shows a system photo picker in a half-height sheet, in the bottom half of the screen
the top half of the screen shows the parent screen behind it, which displays the previously selected photo
when a photo is selected, we display it in the parent view, but we do not dismiss the picker sheet, allowing the user to quickly test different photos
the sheet can be closed by pressing the Cancel button inside the sheet or by dragging the sheet down
the sheet can also be resized by the user by dragging it up or down, switching between half height and full height

The sheet can also be resized by scrolling the scrollable contents inside the sheet, e.g. the grid of photos in the picker in this case

This is normally useful, but it might not be what you want

To disable this behavior, turn off this option in the sheet:

sheet.prefersScrollingExpandsWhenScrolledToEdge = false

In the example above, if we do not dismiss the dialog when the user makes a selection, it will work fine in "medium" half-height mode, but in the "large" full-height mode it might seem like the app is broken, because the user will not see the view behind the dialog

In such cases, it makes sense to programmatically adjust the detent position

You can do that by setting selectedDetentIdentifier:

func picker(_ picker: PHPickerViewController,
    didFinishPicking results: [PHPickerResult]) {

    // assign result to imageView.image

    if let sheet = picker.sheetPresentationController {
        sheet.animateChanges {
            sheet.selectedDetentIdentifier = .medium
        }
    }
}

The sheet.animateChanges does exactly what you'd expect – if you want to change the detent position instantly, without animation, then just set the property without using an animateChanges block

Sheets also automatically adjust when the software keyboard is shown or hidden – if the sheet is in half-height mode and the keyboard slides up, it will automatically move to the top of the screen, and back to the medium position when the keyboard goes away

Removing dimming overlay:

You can choose to hide the overlay covering the view behind the sheet

This makes the back view more visible, and also makes it possible to interact with it while the sheet is open, making the sheet non-modal

To hide the overlay, you set the largest detent level at which the dimming overlay should be hidden; at detent levels higher than this, the overlay will be visible

sheet.largestUndimmedDetentIdentifier = .medium

So:

setting it to .medium means that .medium is the largest detent with no overlay, and .large will still show an overlay
setting it to .large means that it will show no overlay in either .medium or .large
leaving it at the default value (nil) means the overlay will be shown in all positions

ℹ️ In the talk, the property was called smallestUndimmedDetentIdentifier, which didn't really make sense – the name was changed in one of the betas

Sheets in horizontal mode

When the iPhone is in horizontal orientation, presented view controllers are normally displayed in full screen mode, without the cut-off rounded edge at the top and covering full width of the device

You can now request to show a modal view as a sheet also in horizontal position by setting:

sheet.prefersEdgeAttachedInCompactHeight = true

The sheet will by default use the whole available width within in the safe area

If you want the sheet to have a custom width, you can additionally set:

sheet.widthFollowsPreferredContentSizeWhenEdgeAttached

You can customize the preferred sheet width using preferredContentSize, just like on the iPad

Additional customizations

You can show a "grabber" bar at the top edge of the sheet, like on the bottom sheet in the Maps app, in order to make it more obvious that a sheet can be dragged by moving the top edge:

sheet.prefersGrabberVisible = true

You can also customize the corner radius of the top corners of the sheet:

sheet.preferredCornerRadius = 20.0

Note that when the screen behind the sheet is shown in a stacked sheet, it will also use the same corner radius for consistency

Popovers adapting into sheets

It's possible to present the same view as a popover on the iPad and as a sheet on the iPhone and in partial-width modes on the iPad (automatically adjusting between the two as you resize the window)

To do that, customize the popover presentation controller and use adaptiveSheetPresentationController to configure its sheet presentation:

func showImagePicker(_ sender: UIBarButtonItem) {
    let picker = PHPickerViewController()
    picker.delegate = self

    // show the view as a popover by default
    picker.modalPresentationStyle = .popover

    // picker.sheetPresentationController is nil
    if let popover = picker.popoverPresentationController {
        popover.barButtonItem = sender

        // access the sheet controller of the popover
        let sheet = popover.adaptiveSheetPresentationController

        sheet.detents = [.medium(), .large()]
        sheet.smallestUndimmedDetentIdentifier = .medium
    }

    present(picker, animated: true)
}

Note that if modalPresentationStyle is set to .popover, the sheetPresentationController of the presented view will always be nil, so you need to access it through popoverPresentationController.adaptiveSheetPresentationController

Updating your app

How you can update your app for iOS 15:

review your app for areas that would benefit from medium-height or non-modal sheets
if you have any custom built half-height sheet views, replace them with built-in sheets

Improve access to Photos in your app

2022-05-28T16:10:23Z

Improvements to the Photo Picker

Privacy:

When an app doesn't ask for a Photo Library access and only presents the Photo Picker – which runs out of process and automatically has access to the whole library, from which the user can pick some photos for the app – it was possible for people to misinterpret this workflow and assume that the app itself had a full access to the whole photo library, even if it didn't

So now, in the Settings app, on the Photos Privacy screen, there is a new section for apps that only use the Photos Picker, titled "Apps with one-time photo selection"

Ordered selection:

In some apps, users may need to be able to select a few photos in a specific order (e.g. when adding them to a social media post)

In iOS 15, your app can configure the picker to show selection order as badges ① ② ③ etc.

To do that, set the selection property in the picker configuration to .ordered:

var configuration = PHPickerConfiguration()
configuration.selectionLimit = 0
configuration.selection = .ordered

Selection adjustment:

You can now preselect some photos in the picker to display user's previous selection that they can edit, adding additional photos or removing some previously selected ones

To preselect photos, pass an array of asset identifiers to the picker configuration:

let assetIdentifiers: [String] = previousSelection

var configuration = PHPickerConfiguration(photoLibrary: .shared)
configuration.selectionLimit = 0
configuration.preselectedAssetIdentifiers = assetIdentifiers

⚠️ One caveat: when the picker returns updated results, all photos that have been previously selected (and included in the preselection config you provide) will not include the item providers

So you need to keep the original list of photos with their data until after the updated picker results are returned

If the preselected picker dialog is cancelled, it will return the preselected assets you've provided with all item providers empty

In the delegate callback, you need to merge together the old results with the new results, taking the value from the old results for any photo that was selected previously (since there won't be an item provider in the new one):

func picker(
    _ picker: PHPickerViewController,
    didFinishPicking newResults: [PHPickerResult]
) {
    dismiss(animated: true)

    let existingSelection: [String: PHPickerResult] = self.lastSelection
    var newSelection = [String: PHPickerResult]()

    for result in results {
        let identifier = result.assetIdentifier!
        newSelection[identifier] = existingSelection[identifier] ?? result
    }

    self.lastSelection = newSelection

    // do something with selected assets
}

Reporting progress:

Some assets that the user selects in the picker may not be immediately available and need to be downloaded from the cloud first, which may take a moment (especially for videos) – this can happen if they're selecting something from their iCloud Photos and the "Optimize Storage" option is turned on

In that case, previously your app could only show a spinner indicator, since it didn't have access to information about the download progress

Now, the asset's item provider can give you a Progress object that you can use to track download progress and show a more appropriate loading UI:

let result: PHPickerResult = ...
let provider: NSItemProvider = result.itemProvider
let identifier: String = UTType.movie.identifier

if provider.hasItemConformingToTypeIdentifier(identifier) {
    let progress: Progress = provider.loadFileRepresentation(
        forTypeIdentifier: identifier
    ) { url, error in
        // Do something with the video, or handle the error
    }

    // Show progress in the meantime
}

New cloud identifier APIs

There are some categories of apps that require full or partial direct access to the user's photo library, e.g. photo editing apps, camera apps or apps whose purpose is to display the photo library in some specific way

Those apps use the PhotoKit APIs to access and modify the photo library

When using those APIs, assets are returned with unique identifiers that your app can save for later and then use again to retrieve the same assets on the next launch

These identifiers are specific to each device, even if some of the assets are synced between devices using iCloud Photos

If your app syncs some user-generated data that references user's photos between devices, and you want to access the same photos on multiple devices, you can use the new cloud identifiers that identify the same photo globally

There is a mapping between local identifiers and cloud identifiers and you can use a cloud identifier to look up a local copy of the photo

Cloud identifiers work even if the device is not signed into iCloud Photos (or even never was)

The new identifiers are represented by PHCloudIdentifier objects

How to use cloud identifiers:

1) Get some local photos from a device's library using CloudKit

2) Map local identifiers to cloud identifiers:

let cloudMappings = PHPhotoLibrary.shared()
        .cloudIdentifierMappings(forLocalIdentifiers: localIdentifiers)

for (localIdentifier, cloudMapping) in cloudMappings {
    if let cloudIdentifier = cloudMapping.cloudIdentifier {
        // save the cloudIdentifier for later
        resolved[localIdentifier] = cloudIdentifier
    } else {
        // handle the cloudMapping error
    }
}

3) Transfer the cloud identifiers to other devices using whatever communication method you want to use (iCloud / CloudKit etc.)

→ use stringValue to encode the identifier to a string

4) Look up local identifiers on each device based on the cloud identifiers in the same way:

let localMappings = PHPhotoLibrary.shared()
        .localIdentifierMappings(for: cloudIdentifiers)

for (cloudIdentifier, localMapping) in localMappings {
    if let localIdentifier = localMapping.localIdentifier {
        // add the localIdentifier to our resolved assets
        resolved[cloudIdentifier] = localIdentifier
    } else {
        // handle the localMapping error
    }
}

5) Use the local identifiers to fetch the assets and display them

Error handling:

The mapping in both directions may return an error instead of the identifier, so you need to be able to handle that case

There are two possible kinds of errors to take into account:

1) Identifier Not Found – if the app isn't able to find or access the relevant record:

let error = localMapping.error! as NSError

if error.code == PHPhotosError.identifierNotFound.rawValue {
    // couldn't find this photo, add it to the missing photos list
    missingPhotos.append(cloudIdentifier)
}

2) Multiple Identifiers Found – this can happen if the cloud state isn't completely in sync, and the app tries to find the image using content match and finds multiple copies; in this case, the error info will contain the list of matching identifiers under PHLocalIdentifiersErrorKey

let error = localMapping.error! as NSError

if error.code == PHPhotosError.multipleIdentifiersFound.rawValue {
    // found multiple matches, prompt the user to pick one
    let matches = error.userInfo[PHLocalIdentifiersErrorKey] as! [String]
    multipleMatches[cloudIdentifier] = matches
}

Note: looking up cloud identifiers takes some work, so use local identifiers for normal app interactions and map them to cloud identifiers only for syncing with other devices

Updates to the limited library

The limited library is when you access the user's photo library using PhotoKit after the user has requested to only share selected photos with your app

This is designed to work transparently for your app – all PhotoKit APIs work fine, but they work as if the photo library only contained those selected photos and nothing else

See last year's talk Handle the Limited Photos Library in Your App for more info

In iOS 15, apps can now create, fetch and update their own photo albums within the user's photo library when running in the limited library mode

Also, when you call photoLibrary.presentLimitedLibraryPicker() to let the user adjust a previous selection of photos shared with your app, you can now provide a callback which will be given a list of photos that have just been added to the selection:

let library = PHPhotoLibrary.shared()
library.presentLimitedLibraryPicker(from: controller) { addedIdentifiers in
   // fetch the newly added photos and use them in your app 
}

If your app still uses the old Assets Library framework that was deprecated in iOS 9 (ALAssets*) – please switch to PhotoKit and the Photos Picker, the old API will be removed in a future SDK

Craft search experiences in SwiftUI

2022-05-17T22:39:39Z

This year, SwiftUI adds a .searchable view modifier which adds a search UI appropriate for the current platform

Here's how it's used in the new Weather app written in SwiftUI:

NavigationView {
    WeatherList(text: $text) {
        ForEach(data) { item in
            WeatherCell(item)
        }
    }
}
.searchable(text: $text)

What happens behind the scenes is that the .searchable modifier puts the search field configuration into the Environment, and the views within the hierarchy look for the search field configuration there and apply it in the best way for the given platform

In this case, the NavigationView knows how to display the search field and will show it e.g. at the top of one of its panes on iOS, or on the right side of the window toolbar on macOS

If no view knows how to display the search field, a default rendering is used which puts it in the toolbar

A search field isn't very useful without another piece of UI that displays the search results, which is something that you need to implement yourself

The Weather app does it this way:

struct WeatherList: View {
    @Binding var text: String

    @Environment(\.isSearching) private var isSearching: Bool

    var body: some View {
        WeatherCitiesList()
        .overlay {
            if isSearching && !text.isEmpty {
                WeatherSearchResults()
            }
        }
    }
}

the .searchable modifier puts an \.isSearching key in the Environment, which you can check to see if a search field is focused somewhere on the screen, and show a different UI with search results if it is
the view checks the text property (which you need to pass yourself) and if it's not empty, displays a list of matching locations
the results list is displayed as an overlay on top of the standard UI – this is done so that when the user leaves the search field and the results list is hidden, the main UI below remains unchanged (unless they user has picked something from the results)

The second example is the "Colors" app:

NavigationView {
    Sidebar()
    DetailView()
}
.searchable(text: $text)

Here's how we implement search here:

when you apply .searchable to a two-pane NavigationView, it will add the search field to its first pane (the master view); if you want it to be added to the detail view, add the modifier to the detail view directly
on iOS and iPad OS, the Sidebar view uses the \.isSearching Environment key to display search results as an overlay over the list sidebar
on macOS, NavigationView puts the search field in the right side of the sidebar, automatically collapsing it into a button if the window is too small
on the Mac we display the search results instead in the main window pane, on top of the DetailView
on tvOS, we use a slightly different layout structure, adding a tab bar with one tab used for the standard navigation and the other for search, and putting the search UI in the second tab above a results list:

NavigationView {
    TabView {
        Sidebar()
        ColorsSearch()
            .searchable(text: $text)
    }
}

In most cases, the .searchable modifier applied to NavigationView will automatically render the search field in the appropriate place of the hierarchy

However, you always need to decide how and where to display the search results in a way appropriate for your app (which may vary between platforms)

In some cases you may want to use a different layout between platforms, as we do here in case of tvOS

Search suggestions

You can provide search suggestions for the search UI, which are examples of search queries that may be autocompleted into the field when the user picks them; suggestions give the user an idea of the types of things they can search for

Search suggestions also render in a platform-appropriate way:

as a menu popover below the field on macOS
as a complete search results screen on iOS
as a button that opens the list on watchOS

Suggestions are provided as an optional view builder closure that returns a set of buttons:

.searchable(text: $text) {
    ForEach(suggestions) { suggestion in
        Button {
            text = suggestion.text
        } label: {
            ColorsSuggestionLabel(suggestion)
        }
    }
}

There is also a shorthand version available, using the .searchCompletion modifier

The modifier should be applied to a non-interactive element like a Label, and it transforms it into a button which:

updates the search text, like the button above
dismisses the suggestions view

.searchable(text: $text) {
    ForEach(suggestions) { suggestion in
        ColorsSuggestionLabel(suggestion)
            .searchCompletion(suggestion.text)
    }
}

If you want to run a search query only when the user submits the search field, not live while they're typing (e.g. because it requires a request to a server), you can use the new .onSubmit modifier:

.searchable(text: $text) {
    ForEach ...
}
.onSubmit(of: .search) {
    fetchResults()
}

Direct and reflect focus in SwiftUI

2022-05-17T22:23:54Z

SwiftUI in most cases manages focus automatically on your behalf based on given platform's conventions

In more complex cases, where SwiftUI can't figure this out by itself, there are APIs that let you customize the focus behavior

Example situations:

focusing the text editor of a new note when the "New" button is pressed in Notes
moving focus from a vertical list of buttons in the sidebar to a horizontal list of items in the main area on tvOS
programmatically dismissing the keyboard on iOS

The @FocusState API

A view property marked @FocusState is a special type of state that changes depending on which of the view's subviews is currently focused:

enum Field: Hashable {
    case email
    case password
}

struct ContentView: View {
    @FocusState private var focusedField: Field?

    var body: some View {
        VStack {
            TextField("Email", text: $email)
                .focused($focusedField, equals: .email)

            SecureField("Password", text: $password)
                .focused($focusedField, equals: .password)
        }
    }
}

In this case, the Field type is a custom enum type, but you can also use strings, integers or any other Hashable type

Notice that the property is an optional, since it's set to nil if none of the fields is focused – in general, a @FocusState should always be optional

The binding is two-way – the value changes when the focus changes, but you can also move the focus by modifying the value

For example, you can move the focus back to the email field if the user tries to submit the form, but the email is invalid:

VStack {
    ...
}
.onSubmit {
    if !isEmailValid {
        focusedField = .email
    }
}

You can also dismiss the keyboard when the form is submitted by setting the value to nil:

VStack {
    ...
}
.onSubmit {
    if !isEmailValid {
        focusedField = .email
    } else {
        focusedField = nil
        logIn()
    }
}

Creating navigation targets (tvOS)

On tvOS, all navigation is done by moving focus and then selecting the focused element

You may sometimes want to be able to move focus between two parts of a screen with different content, and the default focus navigation will not work automatically if the two elements in two sections are not directly adjacent to each other

You can fix this by extending the effective focusable area of some elements like buttons to a larger area that is not normally focusable by itself

This is done using the new .focusSection API:

HStack {
    VStack {
        TextField("Email", text: $email)
        SecureField("Password", text: $password)
        SignInWithAppleButton(...)
    }
    .onSubmit { ... }
    .focusSection()

    VStack {
        Image(photoName)
        BrowsePhotosButton()
    }
    .focusSection()
}

When the .focusSection() view modifier is applied to a view like the VStack here, the view becomes capable of accepting focus as long as it contains any focusable subviews (the browse button)

Now, the user is able to move from the "Browse photos" button to the login sidebar on the left, even though the buttons in the sidebar aren't directly to the left of the button, and to move right from the email/password fields to the browse button, even though the button isn't directly to the right

What's new in SwiftUI

2022-05-15T00:48:11Z

Async images

Built-in support for loading images asynchronously:

AsyncImage(url: photo.url)

AsyncImage provides a default placeholder, but it can also be customized:

AsyncImage(url: photo.url) { image in
    image
        .resizable()
        .aspectRatio(contentMode: .fill)
} placeholder: {
    Color.blue
}

Custom animations and error handling:

AsyncImage(
    url: photo.url,
    transaction: .init(animation: .spring())
) { phase in
    switch phase {
        case .empty: ...
        case .success(let image): ...
        case .failure(let error): ...
    }
}

Improvements to lists

Pull to refresh:

Support for pull-to-refresh in lists on iOS:

List {
    ...
}
.refreshable {
    await items.reload()
}

Tasks:

Asynchronous task associated with a view:

.task {
    await items.load()
}

The task is run when the view appears and is automatically cancelled when the view is removed

This can be also used to load data from an async sequence that produces items continuously for the whole duration of the view being visible

See more about new concurrency features in "Discover concurrency in SwiftUI"

Bindings to list items:

A List can now accept a binding to an array property and will pass a binding to each specific item to the closure

This lets you use controls that require a binding (like TextField) within list items:

List($items) { $item in
    Text(item.title)
    TextField("Item", text: $item.text)
}

The previous solution for cases like this was to iterate over list indexes and bind to e.g. $list[index].text, but this causes SwiftUI to reload the list after every change anywhere

This is actually a Swift language change, so it works in all previous versions of SwiftUI too

Also works in ForEach and some other places

Customizing lists:

.listRowSeparatorTint(Color.blue) – customizing the color of separators

→ can be defined on the item to give each item a different separator

.listRowSeparator(.hidden) – hide separators completely

Same for customizing separators between sections: .listSectionSeparator, .listSectionSeparatorTint

Swipe actions:

You can now define custom swipe actions on list items:

CharacterProfile(character)
.swipeActions {
    Button {
        character.isPinned.toggle()
    } label: {
        Label("Pin", systemImage: "pin")
    }
    .tint(.yellow)
}

By default the action is on the right edge – use .swipeActions(edge: .leading) to put the action on the left edge

Add multiple .swipeActions modifiers to include swipe actions on both sides

Improvements on the Mac:

List style with alternating backgrounds, like in standard NSTableView:

.listStyle(.inset(alternatesRowBackgrounds: true))

New control for full-featured multi-column tables:

Table(characters) {
    TableColumn("<>") { CharacterIcon($0) }
      .width(20)
    TableColumn("Villain") { Text($0.isVillain ? "Villain" : "Hero") }
      .width(40)
    TableColumn("Name", value: \.name)
    TableColumn("Powers", value: \.powers)
}

Like a List, a Table presents a list of rows rendered from the given content, except a Table has a defined list of columns

Table supports single- and multi-row selection:

@State private var singleSelection: StoryCharacter.ID?
@State private var multiSelection = Set<StoryCharacter.ID>()

Table(characters, selection: $multiSelection) { ... }

You can sort the table by selected column – to support sorting in a column, provide a key-path to a model value which should be used for sorting:

@State private var sortOrder = [KeyPathComparator(\StoryCharacter.name)]

Table(characters, selection: $selection, sortOrder: $sortOrder) {
    TableColumn("Name", value: \.name)
    ...
}

Tables also support various visual styles and allow you to customize the appearance of each column

Integration with Core Data fetch requests:

Fetch requests now provide a binding to their sort descriptors, which lets you build a sortable multi-column table backed by a Core Data request:

@FetchRequest(sortDescriptors: [SortDescriptor(\.name)])
private var characters: FetchedResults<StoryCharacter>

Table(characters, selection: $selection, sortOrder: $characters.sortDescriptors) {
    ...
}

New @SectionedFetchRequest which lets you build a list divided into sections from a Core Data request:

@SectionedFetchRequest(
    sectionIdentifier: \.isPinned,
    sortDescriptors: [
        SortDescriptor(\.isPinned, order: .reverse),
        SortDescriptor(\.lastModified)
    ],
    animation: .default
)
private var characterSections: SectionedFetchResults<...>

The value of the property is a list of sections, each containing a list of items:

List {
    ForEach(characterSections) { section in
        Section(section.id)
        ForEach(section) { character in
            CharacterRowView(character)
        }
    }
}

Support for search in lists:

.searchable(text: $characters.filterText)

SwiftUI automatically adds a search field in the appropriate place depending on the platform, and may automatically provide suggestions based on the context

The modifier takes a binding to the entered filter text, which you should use to filter the results

See more in "Craft search experiences in SwiftUI"

Sharing data with other apps

Support for customized previews during drag & drop operation:

CharacterIcon(character)
.onDrag {
    character.itemProvider
} preview: {
    CharacterDragPreview(character)
}

New ways to import and export data to/from your app using Item Providers

Importing items from outside:

.importsItemProviders(StoryCharacter.imageAttachmentTypes) { itemProviders in
    guard let firstItem = itemProviders.first else { return false }

    Task {
        selectedCharacter.imageAttachment = await StoryCharacter.loadImage(from: firstItem)
    }

    return true
}

This lets you e.g. import an image taken with an iPhone's camera to a Mac app using the Continuity Camera feature

The new ImportFromDevicesCommands() helper lets you add the necessary menu entries (File > Import from iPhone or iPad):

WindowGroup { ... }
    .commands {
        ImportFromDevicesCommands()
    }
}

Exporting items to another app, e.g. to Shortcuts:

.exportsItemProviders(StoryCharacter.contentTypes) {
    if let selectedCharacter = selectedCharacter {
        return [selectedCharacter.itemProvider]
    } else {
        return []
    }
}

Graphics

SF Symbols:

Many new symbols

Two new rendering modes:

hierarchical – uses one color like monochrome, but adds multiple levels of opacity to emphasize key elements of the symbol
palette – gives you fine grained control over which parts of the symbol should be filled with what color

New SwiftUI colors: mint, teal, cyan, indigo, brown

Symbol variants like "filled" are now automatically chosen for you depending on the context

So if e.g. filled symbols should be used in tab bars, you can just use "person.circle" instead of "person.circle.fill" and the framework will choose the right variant automatically

This makes your code more reusable – e.g. you can use the same tab bar on macOS and it will use the outline versions of the same symbols there

More in "What's new in SF Symbols" and "SF Symbols in SwiftUI"

Canvas view:

A new view that gives you a rectangular canvas for free drawing, like custom NSViews/UIViews with drawRect: in AppKit/UIKit

You can use this to draw a specific custom element, or to draw multiple elements like a grid of small icons that don't need individual tracking and invalidation

Canvas { context, size in
    for symbol in symbols {
        let image = context.resolve(symbol.image)
        context.draw(image, in: symbol.rect)
    }
}

To provide accessibility to a Canvas element, you can use the new .accessibilityChildren modifier:

Canvas {
    ...
}
.accessibilityChildren {
    List(symbols) { Text($0.name) }
}

This provides a completely separate view hierarchy to accessibility interfaces which is more readable than the actual view

Timeline view:

TimelineView is a new view that generates different content based on the time of rendering

Can be used to build e.g. an animated screen saver for tvOS

TimelineView(.animation) {
    let time = $0.date.timeIntervalSince1970

    Canvas { context, size in
        // draw items differently based on the time parameter
    }
}

The timeline takes a TimelineSchedule parameter, which specifies when or how often it should be updated

Timeline view can also be used to provide a view for a Watch app which automatically updates when the app is in dimmed mode on always-on displays

Also in the dimmed mode on watchOS you can use the new .privacySensitive modifier to blur out elements which can include private information and should be hidden when the Watch is not being used:

VStack(alignment: .center) {
    Text("Favorite Symbol")
    Image(systemName: favoriteSymbol)
        .privacySensitive(true)
}

The .privacySensitive modifier can also be used in widgets – in this case some content may be hidden (blurred) when the widget is shown on the lock screen while the device is locked

More info in "What's new in watchOS 8" and "Principles of great widgets"

Materials:

You can now create views with material backgrounds (the ones with a translucent blur effect):

VStack {
    Text("Symbol Browser")
        .font(.largeTitle)
    Text("\(symbols.count) symbols")
        .foregroundStyle(.secondary)
        .font(.title2)
}
.padding()
.background(.ultraThinMaterial,
  in: RoundedRectangle(cornerRadius: 16.0))

When you use semantic styles like .foregroundStyle(.secondary) for the text, the content of a container with material background uses the appropriate "vibrancy" effect

The effect is applied to text, with the exception of emoji, which are excluded because this would render them an in incorrect way

On the Mac, system controls like sidebars and popups use the material background and now also apply the vibrancy effect to their content

More about canvas and materials in the "Add rich graphics to your SwiftUI app" talk

You can also read more about vibrancy and materials in my old blog post about the introduction of dark mode in macOS Mojave – "Dark Side of the Mac: Appearance & Materials"

Safe area inset:

There is a new view modifier .safeAreaInset that allows you to position a view on top of a ScrollView in a way that adjusts the scroll view's top/bottom insets so that all the content can be reached and the beginning/end isn't obscured by the overlay:

ScrollView {
    ...
}
.safeAreaInset(edge: .bottom, spacing: 0) {
    VStack {
        Text("\(symbols.count) symbols selected")
    }
    .padding()
    .background(.regularMaterial)
}

See more in the "Add rich graphics to your SwiftUI app" talk

Preview enhancements:

New .previewInterfaceOrientation modifier for choosing portrait/landscape orientation:

static var previews: some View {
    ColorList()
        .previewInterfaceOrientation(.vertical)

    ColorList()
        .previewInterfaceOrientation(.horizontal)
}

The Attributes Inspector in the preview now includes a section for accessibility attributes, and there is a new inspector tab that shows the accessibility properties of all view elements at a glance

Text and keyboard

Markdown support:

Text now automatically interprets Markdown formatting:

Text("**Hello**, [WWDC](https://developer.apple.com/wwdc21)!")
Text("`print(helloText)`")

This is built on top of the new Swift-native AttributedString in Foundation

It has a new rich, type-safe API for adding attributes, and even allows defining custom attributes that can be applied to text through the Markdown syntax

See more in "What's new in Foundation"

Localization:

Xcode 13 can now automatically extract strings for localization using the Swift compiler (see option in build settings)

See more in "Localize your SwiftUI app"

Dynamic Type:

You can now restrict some content to only a specified range of Dynamic Type text sizes – so that when the user picks one of the extra large sizes which would make your text too large to fit, it stays at the maximum allowed size:

Header("Today's Activities")
    .dynamicTypeSize(.large .. .extraExtraLarge)

Text selection:

You can enable selection of non-editable text:

Text(activity.description)
    .textSelection(.enabled)

This allows the user to select and copy any part of the text on macOS

On iOS this only allows copying the whole text though

New formatter APIs:

Foundation now includes new .formatted() APIs for various values like dates that allow you to specify the format in a type-safe way:

Text(date.formatted())

Text(date.formatted(date: .omitted, time: .shortened))

Text(date.formatted(
    .dateTime.weekday(.wide).day().month().hour().minute()))

There's even a list formatter API like ListFormatter:

people.map(\.nameComponents).formatted(
    .list(memberStyle: .name(style: .short), type: .and))

// => "Matt, Jacob, and Taylor"

Format styles can also be specified in TextField:

@State private var newAttendee = PersonNameComponents()

var body: some View {
    ...
    TextField("New Person", value: $newAttendee, format: .name(style: .medium))
    ...
}

The entered text is automatically validated and reformatted according to the specified format, and then parsed into a value of a correct type if possible

See more about the new format styles in "What's new in Foundation"

Text field prompts:

TextField now includes a prompt: parameter, which allows you to specify a placeholder value (an example text to enter) that is different from the label (which normally should specify the name or meaning of the field)

Text fields inside a Form on macOS now display with labels outside on the left, right-aligned to the text field, with the prompt (if any) shown as the placeholder

→ previously they were rendered like on iOS, with the label used as the placeholder inside the field

Text field submission:

New .onSubmit modifier allows you to specify an action to execute when a text field is submitted by pressing a physical or virtual Return key:

TextField(
    "New Person",
    value: $newAttendee,
    format: .name(style: .medium)
)
.onSubmit {
    people.append(Person(newAttendee))
}

This can also be applied to the entire form:

Form {
    TextField("Username", text: $viewModel.userName)
    SecureField("Password", text: $viewModel.password)
}
.onSubmit(of: .text) {
    guard viewModel.validate() else { return }
    viewModel.login()
}

You can now also configure the title of the Return key on the iOS keyboard (which gives it a blue background):

TextField(...)
.onSubmit {
    ...
}
.submitLabel(.done)

Keyboard accessory views:

Adding accessory views to the iOS keyboard – configure toolbar items with the placement of .keyboard:

Form {
    ...
}
.toolbar {
    ToolbarItemGroup(placement: .keyboard) {
        Button(action: selectPreviousField) {
            Label("Previous", systemImage: "chevron.up")
        }
        .disabled(!hasPreviousField)

        Button(action: selectNextField) {
            Label("Next", systemImage: "chevron.down")
        }
        .disabled(!hasNextField)
    }
}

→ on macOS the buttons will be shown in the Touch Bar instead

Focus state:

New property wrapper @FocusState that reflects the state of focus and provides precise control over it

A @FocusState property is bound to a field's focus using .focused

The value of the property will then show if and which view is currently focused, and you can move the focus by modifying the value

Simple form: declare a boolean property, and it will be set to true when the view is focused:

@FocusState private var newPersonIsFocused: Bool

var body: some View {
    TextField("New Person", value: $newAttendee,
        format: .name(style: .medium)
    )
    .focused($newPersonIsFocused)

    Button {
        newPersonIsFocused = true
    } label: {
        Label("Add Attendee", systemImage: "plus")
    }
}

In a more advanced version, the focus property can use any Hashable as its type, e.g. an enum, and be used to show which of the possible fields is currently focused:

private enum Field: Int, Hashable {
    case name, location, date, addAttendee
}

@FocusState private var focusedField: Field?

var body: some View {
    TextField("New Person", value: $newAttendee,
        format: .name(style: .medium)
    )
    .focused(focusedField, equals: .addAttendee)

    Button {
        focusedField = .addAttendee
    } label: {
        Label("Add Attendee", systemImage: "plus")
    }
}

This allows you to build any kind of complex screen with multiple focusable fields that you may want to move between programmatically

You can also use .focused to dismiss the keyboard on iOS by unfocusing a focused field:

@FocusState private var newPersonIsFocused: Bool

var body: some View {
    TextField(...).focused($newPersonIsFocused)
}

func endEditing() {
    newPersonIsFocused = false
}

See more about the new focus API in "Direct and reflect focus in SwiftUI"

Buttons

Support for bordered buttons on iOS:

Button("Add") { ... }
    .buttonStyle(.bordered)

Like all modifiers, this can be added to a larger container and applies to all buttons inside

Choose a color for a bordered button by specifying a tint:

Button("Buy") { ... }
    .buttonStyle(.bordered)
    .tint(.green)

Make buttons smaller or larger by using the .controlSize modifier, previously used only on macOS:

ForEach(entry.tag) { tag in
    Button(tag.name) { ... }
        .tint(tag.color)
}
.controlSize(.small)

→ this only seems to work for buttons at the moment

You can also make "prominent" bordered buttons which use a style that stands out more (stronger background):

Button("Submit", action: submitForm)
    .buttonStyle(.borderedProminent)

ℹ️ In the video, this is shown as .controlProminence(.increased). The .controlProminence modifier was replaced in a later beta with a separate .borderedProminent button style.

Buttons with a .large size can be used to build a menu of action buttons at the bottom of a screen/form, with the default button marked as prominent:

VStack {
    Button(action: addtoJar) {
        Text("Add to Jar").frame(maxWidth: 300)
    }
    .buttonStyle(.borderedProminent)
    .keyboardShortcut(.defaultAction)

    Button(action: addToWatchlist) {
        Text("Add to Watchlist").frame(maxWidth: 300)
    }
    .tint(.accentColor)
    .buttonStyle(.bordered)
}
.controlSize(.large)

Note: do not add .borderedProminent to every single button on the screen – this should be only used for single primary actions

→ this is the equivalent of a blue-colored default button in a dialog on macOS, one that is bound to the Return key shortcut, of which by definition there can be only one within a dialog

Buttons using the new styles like .controlSize, .borderedProminent and .tint automatically provide appropriate pressed states and automatically adapt to dark mode, Dynamic Type font sizes etc.

You can now specify a "role" for a button, currently .cancel or .destructive:

Button("Delete...", role: .destructive) { ... }

Marking a button as destructive adds a red foreground and/or background depending on the context, to emphasize that the action is potentially dangerous

Cancel and destructive buttons are often used in confirmation dialogs, which now have a new view modifier made specifically for them:

RowCell(entry)
.contextMenu {
    Button("Delete...", role: .destructive) {
        showConfirmation = true
    }
}
.confirmationDialog("Are you sure you want to delete \(title)?",
    isPresented: $showConfirmation
) {
    Button("Delete", role: .destructive) {
        deleteEntry(entry)
    }
} message:
    Text("Deleting \(title) will remove it from all of your jars.")
}

A confirmation dialog is shown as an action sheet on iOS, as a popover on the iPad, and as an alert on macOS

Another type of button is a menu button (available before):

Menu("Add") {
    ForEach(jarStore.allJars) { jar in
        Button("Add to \(jar.name)") {
            jarStore.add(buttonEntry, to: jar)
        }
    }
}
.menuStyle(.borderedButton)

ℹ️ In the video, this is shown as .menuStyle(.button). There is however no such menu style as .button – use .borderedButton on macOS and .borderlessButton elsewhere (or just keep the default).

A bordered button menu is rendered as a button with a down arrow on the right on macOS (a pull-down menu button)

A new option for button menus (both macOS and iOS) is to provide a "primary action":

Menu("Add") {
    ForEach(jarStore.allJars) { jar in
        Button("Add to \(jar.name)") {
            jarStore.add(buttonEntry, to: jar)
        }
    }
} primaryAction: {
    jarStore.add(buttonEntry)
}

This makes the button act as both a normal button and a menu:

on iOS, a normal tap runs the primary action, and a long press shows the menu
on macOS, the button uses a slightly different style and lets you either click the main button part for the primary action, or click the arrow on the right to show the menu (without a primary action, clicking either of the parts shows the menu)

You can also choose to hide the arrow indicator of a menu button on macOS using .menuIndicator:

Menu("Add") {
    ...
}
.menuStyle(.borderedButton)
.menuIndicator(.hidden)

Such menu button looks like a completely plain button, but still acts as a menu button:

without a primary action, it always shows the menu when pressed
with a primary action, it runs the primary action when pressed, and to show the menu you need to do a long-press like on iOS

You can use such buttons with a hidden indicator to decrease the visual prominence of the button, e.g. if there are a lot of them in the same view

A Toggle can now also be displayed as a toggle button:

Toggle(isOn: $showOnlyNewFilter) {
    Label("Show Only New", systemImage: "sparkles")
}
.toggleStyle(.button)

This works like a toggle button on macOS (NSButton.ButtonType.toggle) – one that cycles between an "on" state (with a blue background) and an "off" state when pressed; it works the same way on iOS – in the "on" state it's rendered as a bordered button with a blue background

There is also a new control that groups buttons, mainly toolbar buttons – ControlGroup:

ControlGroup {
    Button(action: archive) {
        Label("Archive", systemImage: "archiveBox")
    }
    Button(action: delete) {
        Label("Delete", systemImage: "trash")
    }
}

Toolbar buttons in a ControlGroup are displayed grouped together visually, as a kind of segmented control – as opposed to buttons in ToolbarItemGroup, which are only arranged side by side, but not joined visually in any way

ControlGroup with menu buttons can be used to create a pair of standard back/forward navigation buttons:

ControlGroup {
    Menu {
        ForEach(history) { ... }
    } label: {
        Label("Back", systemImage: "chevron.backward")
    } primaryAction: {
        goBack(to: history[0])
    }
    .disabled(history.isEmpty)

    Menu {
        ForEach(forwardHistory) { ... }
    } label: {
        Label("Forward", systemImage: "chevron.forward")
    } primaryAction: {
        goForward(to: forwardHistory[0])
    }
    .disabled(forwardHistory.isEmpty)
}

App essentials in SwiftUI

2022-05-09T22:47:08Z

You can now build the entire app using just SwiftUI

New APIs for defining apps and their scenes and views

Defining apps & scenes

An app consists of one or more "scenes"

→ see "Introducing Multiple Windows on iPad" from 2019 for an introduction to the UIKit scenes API

A scene roughly maps to a window:

on iOS, watchOS and tvOS the app can present one scene at a time
on iPadOS you can see multiple scenes from the same or different apps side by side
on macOS, each scene is displayed in a different window or inside tabs within a window

Each scene contains a view hierarchy and/or a hierarchy of child scenes

An app declaration looks similar to a view declaration:

it's also a lightweight struct
it implements the App protocol similar to View
the struct can include properties defining data dependencies, marked with wrappers like @State
the protocol requires a single body property which returns some Scene
inside the body definition, a DSL similar to the one for views is used to define the scene hierarchy:

@main
struct BookClubApp: App {
  @StateObject private var store = ReadingListStore()

  var body: some Scene {
    WindowGroup {
      ReadingListView(store: store)
    }
  }
}

The @main attribute is a new feature of Swift 5.3 – it allows a type to serve as the program entry point

This replaces the @NSApplicationMain/@UIApplicationMain or main.swift file

The code above is a complete declaration of the app structure

Even though it's just a few lines of code, this automatically provides quite a lot of functionality for free

Window Group

The most common type of scene is a WindowGroup, which defines the primary interface (main view) of your app

A WindowGroup displays its view in the expected way or each platform:

on iPhone, tvOS and watchOS it's shown as a single full-screen window
on the iPad it supports the new multi-window scene system
on the Mac it creates multiple Mac windows

The lifecycle of scenes depends on the platform

On iPadOS, the app switcher shows the app name and an optional scene title above each scene/window – this can be the document title, title of the open web page etc.

In latest SwiftUI, you can set that scene title using the new .navigationTitle("Name") view modifier, which replaces .navigationBarTitle (it sets both the title seen in the navigation bar and the scene name in the app switcher)

On the Mac, WindowGroup automatically provides a "File > New Window" command that opens a new window and a Window menu that lists all windows

The scene title is used as the window title in the title bar and the Window menu

Scene windows can also me merged into a single window with tabs by choosing Window > Merge All Windows

Scene storage

@SceneStorage is a new property wrapper that helps you manage persistence and restoration of view state in scenes

Provide a unique key under which the given value should be saved, and it will be saved and restored at appropriate times automatically

struct ReadingListViewer: View {
  @SceneStorage("selectedItem") var selectedItem: String?

  var body: some View {
    NavigationView {
      ReadingList(selectedItem: $selectedItem)
    }
  }
}

Document Group

Another basic kind of scene in the app definition is DocumentGroup, which defines documents in a document-based app:

@main
struct ShapeEditApp: App {
  var body: some Scene {
    DocumentGroup(newDocument: SketchDocument()) { file in
      DocumentView(file.$document)
    }
  }
}

See more in "Build document-based apps in SwiftUI"

Preferences windows

The last scene type, Settings, is Mac-only and defines standard Mac Preferences windows:

var body: some Scene {
  WindowGroup {
    ...
  }

  Settings {
    BooksSettingsView()
  }
}

The Preferences window is automatically linked to from the main app menu with the standard Cmd+, keyboard shortcut

Menu commands

The new .commands view modifier allows you to define menus and menu items with shortcuts for the macOS app menu:

WindowGroup {
  ...
}
.commands {
  BookCommands()
}

struct BookCommands: Commands {
  @FocusedBinding(\.selectedBook) private var selectedBook: Book?

  var body: some Commands {
    CommandMenu("Book") {
      Section {
        Button("Update Progress...", action: updateProgress)
          .keyboardShortcut("u")
        Button("Mark Completed", action: markCompleted)
          .keyboardShortcut("C")
      }
      .disabled(selectedBook == nil)
    }
  }

  private func updateProgress() { selectedBook?.recordNewProgress() }
  private func markCompleted() { selectedBook?.markCompleted() }
}

The commands are also used on the iPad and are displayed in the keyboard shortcuts help popup

The @FocusedBinding property wrapper used in this commands block allows you to selectively enable/disable some groups of commands based on your focus, similar to how the responder chain is used in AppKit or UIKit

See more about commands in the reference documentation

New edition of the "Guide to NSButton styles"

2021-12-30T14:28:59Z

Note (Oct 2023): The names of the buttons have been changed again in the SDK in macOS Sonoma - I will update the blog post again once I have Sonoma on one of my Macs :)

Back in October 2014 I wrote a post about different styles of NSButtons.

That was in the era of OS X Yosemite and Xcode 6. I started researching what each kind of button available in Interface Builder was for, because I couldn’t figure that out from Xcode and the built-in documentation - I dug a bit into the Human Interface Guidelines, some older documentation archives and into Apple apps themselves. I collected everything into a long post that went through all the button styles and described what I could find about each one.

It seems that a lot of people also had the same problem, because the post turned out to be extremely popular. It’s around #3 in total page views on this blog, and 7 years and 7 major macOS versions later it still usually comes out #2 in monthly or yearly stats and still gets a couple hundred visits a month. Even with greatly improved documentation in Xcode and much expanded content in the modern HIG, there’s clearly demand for this kind of information collected in one place.

However, the post was kind of asking to be updated for a long time now… The original screenshots were made in 1x quality, since I didn’t get a Mac with a Retina screen until the end of 2016. Big Sur was released in the summer of 2020, significantly changing the design of the OS, and making Catalina suddenly look outdated (to the point that I’ve seen some people already call the Yosemite-Catalina era design “classic macOS”!). Some new button variants were added, some older buttons were no longer used in system apps the way I presented them, and the button styles available in Xcode were no longer shown and described as shown on screenshots from Xcode 6.

The Big Sur launch seemed like a great moment to give that post a refresh, and I started working on it at the end of last year, but then 2021 came and this year turned out to be kind of rough - surprisingly more so than 2020… as it was for a lot of people, I suppose. I only managed to get back to this project this month.

I thought it would take maybe a week or two… it took around three in total 😬 That included setting up new Mavericks and Yosemite installations in VirtualBox to get updated Retina screenshots from there, building a number of versions of a sample app full of all kinds of buttons that I took a ton of screenshots of on several macOS versions, cutting every screenshot pixel-perfect to size a few times, merging different versions of the same information from a few different sources, including versions of HIG going as far back as 2006, looking through Apple apps searching for buttons, and view-debugging some of them with SIP turned off to check what controls were used there… whew 😅

I’m really happy with the result though. This is now by far the longest post on the blog, with around 11k words total (although around 1/3 of that is just quotes) and around a hundred images. I expanded a lot of the content, adding some things I hadn’t thought about last time and clarifying some that I hadn’t fully understood after I found some new information - and each button now has three different screenshots, sometimes even four:

Of course I’ve also learned a few new things myself and organized the information better in my head again, which is always my primary motivation when writing this kind of blog posts :) After Big Sur I don’t expect a next massive redesign for another few years, so I probably won’t need to repeat this anytime soon - but if macOS 13 or 14 changes some minor things here and there, I’ll try to keep the post up to date.

Read the updated blog post here - if you want to see the old version for some reason, you can find it in the Web Archive.

What's new in AppKit

2021-06-11T12:41:10Z

Design & control updates

There are design updates for some system controls:

popovers appear with an animation
sliders smoothly glide into position when clicked
smaller things like increased spacing between table sections or slightly wider toolbar buttons

Control tinting:

Individual controls like buttons, segmented controls or sliders can have a custom tint

Properties: NSButton.bezelColor, NSSegmentedControl.selectedSegmentBezelColor, NSSlider.trackFillColor

These properties have been introduced in macOS Sierra for Touch Bar controls; in macOS Monterey they’re functional also for normal app controls

This is useful for specific controls that need to have some kind of semantically meaningful color

→ e.g. a green “Accept Call” and a red “End Call” buttons in a video call app

Avoid confusion with the default button if there is one in the same view, since it will also be colorful

Make sure to indicate purpose with more than just the color (using a clear label or an icon), since some of your users may not be able to distinguish buttons by color

Push buttons:

Push buttons no longer highlight using the accent color on click – they behave just like e.g. segmented controls in macOS Big Sur

Don’t make assumptions about how the highlight state looks (e.g. drawing white text over a button that should be blue when pressed, but will now be light gray)

Instead, check the interior background style NSButtonCell.interiorBackgroundStyle:

.normal = colorless state
.emphasized = colorful state

The old “regular square button” aka “bevel button” has now been refreshed as “Flexible push style button” and can be used as a variable height push button

It supports the same kind of configuration as a regular push button, so it can serve as a default button and can be tinted

Its corner radius and padding now match other button styles

It can contain larger icons or multi-line text

The vast majority of buttons should still use the standard fixed height push button – the variable height button is meant for special cases

Localizing keyboard shortcuts

Some keyboard shortcuts should be localized for different keyboard layouts, because in some layouts they may be hard or impossible to type, or it may make sense to adapt them for right-to-left languages

E.g. Cmd + \ is not possible to type on the Japanese keyboard, which doesn’t have a backslash key

AppKit can now handle this for you

In macOS Monterey, the system automatically remaps such shortcuts to different ones that are more natural on the given keyboard layout

Shortcuts like Cmd + [ and Cmd + ] to go back and forward will be swapped in right-to-left languages

→ this applies to brackets, braces, parentheses and arrow keys

You can opt out using:

NSMenuItem.allowsAutomaticKeyEquivalentMirroring – for directional keys like brackets
NSMenuItem.allowsAutomaticKeyEquivalentLocalization – turns off all key localization, including mirroring
if you really don’t want to use this feature at all, you can also disable it completely in your app by implementing the NSApplicationDelegate method: applicationShouldAutomaticallyLocalizeKeyEquivalents(_:)

Update to SF Symbols

New version – SF Symbols 3

Expands capabilities of the SF Symbols app

Some symbols now have multiple layers that can be individually colored

Updated format for custom symbols – allows you to annotate distinct layers within an image

Big Sur had two coloring modes for symbols:

traditional monochrome template style, drawing the whole symbol using one accent color
a multicolor style that uses multiple colors that are predefined in the symbol itself

SF Symbols 3 in macOS Monterey adds two new rendering modes:

"hierarchical" – uses a single tint color, but draws different layers of the image in an emphasized or deemphasized way (lighter or darker than the base color)
"palette" – lets you assign each layer any custom color independently

APIs for the new rendering modes:

NSImage.SymbolConfiguration(hierarchicalColor: .red)
NSImage.SymbolConfiguration(paletteColors: […])
NSImage.SymbolConfiguration.preferringMulticolor()

Symbol variants:

There are also new APIs for mapping between symbol variants, e.g. outline heart symbol ⭤ filled heart symbol, or variants with circles etc.

Useful e.g. when you’re building a picker control that uses outline icons for unselected states and filled variants of the same icons for the selected item

To convert between variants, call e.g.: baseImage.image(with: .fill)

There are constants for each kind of symbol variant, and you can combine multiple variants together (e.g. circle + fill)

See “Design and build SF Symbols” for more info

TextKit 2

Huge update to the text system

TextKit is a great text engine with a long track record, used across all Apple systems

However, TextKit is a linear text layout engine, which means it typesets a block of text from the beginning to the end

There are a lot of use cases where a non-linear layout engine is more useful

TextKit 2 always uses a non-linear layout system

This means it can perform layout on a more granular level, which allows it to avoid some unnecessary work

For example, when you’re looking at a middle fragment of a long document, a linear layout system needs to process all text from the beginning up to the given fragment in order to render it; a non-linear system can start at the nearest start of a paragraph

The non-linear layout system also makes it easier to mix text with non-text elements, and improves performance for large documents

TextKit 2 provides a lot of customization points, which allow you to extend its behavior

The new version coexists with TextKit 1, you can choose which engine to use for each view

TextKit 2 has actually already been used in some system apps and controls in Big Sur

See “Meet TextKit 2” for more info

New Swift features

Swift 5.5 introduces some important new features for managing concurrency: async/await and actors

In AppKit, many asynchronous methods that return value through a completion handler now have variants that work with async/await:

@IBAction func pickColor(_ sender: Any?) {
  async {
    guard let color = await NSColorSampler().sample() else { return }
    textField.textColor = color
  }
}

The actor model is a great fit for a UI framework like AppKit where most APIs should be called on a single main thread

The macOS SDK now has a @MainActor property wrapper that marks all types that have to be accessed from the main thread

Classes such as NSView, NSView/WindowController, NSApplication, NSCell, NSDocument etc. are now marked with @MainActor

Code running in the main actor can freely call methods on other main actor types

However, code that isn’t running on the main thread needs to use async/await to run code on a @MainActor type

This is enforced at the compiler level, which lets you avoid common errors that happen when mixing concurrency with UI code

See “Meet async/await in Swift” and “Protect mutable state with Swift actors” for more info

AttributedString:

Swift 5.5 also adds a new value type AttributedString

It has type-safe attributes and a more swifty API for reading & writing attributes

You can easily convert between AttributedString and NSAttributedString

See “What’s new in Foundation” for more info

Updating NSViews:

There is a new Swift property wrapper which should reduce boilerplate around view properties

Let’s say we have a custom view class like this:

class BadgeView: NSView {
  var fillColor: NSColor
  var shadow: NSShadow
  var scaling: NSImageScaling
  …
}

These properties will usually need to have a didSet which updates properties like needsDisplay or needsLayout when they’re modified:

var fillColor: NSColor {
  didSet { needsDisplay = true }
}

The new @Invalidating attribute in NSView lets you easily specify which other view properties should be updated when the given property is modified:

@Invalidating(.display) var fillColor: NSColor

Properties that can be invalidated include: display, layout, constraints, intrinsic content size, restorable state

The marked property needs to be Equatable, since AppKit checks if the value was actually changed before triggering a view update

You can extend the invalidation system by conforming to NSViewInvalidating protocol

Shortcuts

iOS Shortcuts are now available on the Mac

Shortcuts appear in all the places where you can access services today – if your app supports services, it will also support Shortcuts

AppKit decides which shortcuts are available at the given place by checking the responder chain

It asks each responder whether it can provide or receive the type of data used by each shortcut

The types of data are represented by NSPasteboard.PasteboardType (usually a UTI)

To support shortcuts in a given responder object, implement the method:

func validRequestor(forSendType sendType: NSPasteboard.PasteboardType?,
                              returnType: NSPasteboard.PasteboardType?) -> Any

In that method return an instance of a type implementing NSServicesMenuRequestor (usually the same object):

protocol NSServicesMenuRequestor {
  func writeSelection(to pasteboard: NSPasteboard,
                              types: [NSPasteboard.PasteboardType]) -> Bool

  func readSelection(from pasteboard: NSPasteboard) -> Bool
}

Siri Intents

You can now use Siri Intents in a Mac app by adding an Intents Extension

You can also return an intents handler from the application delegate:

protocol NSApplicationDelegate {
  optional func application(_ application: NSApplication,
                        handlerFor intent: INIntent) -> Any?
}

The returned object should conform to an appropriate intent handler protocol, depending on the intent type

CloudKit Best Practices

2020-11-14T15:47:57Z

Short CloudKit overview

Apple uses CloudKit in their applications, so you can be confident that it scales, because for Apple it scales to hundreds of millions of users

CloudKit lets you focus on building your applications and not worry about building backend services for them

It provides your users automatic authentication – if the user is logged in to iCloud on their device, they don’t need to log in separately in your app

A CloudKit container now includes 3 databases:

public database for data visible to everyone
private database for a given user’s private data
new this year: shared database for user data that they decided to share with others

Zones:

public database has 1 default zone
private database has a default zone and it can have one or more custom zones
shared database includes some number of shared zones

A record always exists in a specific zone

Building an app with a sync feature

A common use case (e.g. Notes app):

user creates some data/records/documents on one of their devices
later, they open another device and they expect to see these documents there and be able to read/edit them

The way this is implemented is that CloudKit needs to be the source of truth, and the devices should maintain a local cache of all the app data and synchronize it using CloudKit

The recommended workflow:

1. On app launch, fetch changes from the server
2. Subscribe to any future changes
3. Fetch changes when you receive a push

Subscriptions:

Subscriptions let you ask the server to notify you whenever a change happens in the specified set of data. Previously you could subscribe to a specific query to a record type or to all changes in a zone.

New in iOS 10 – CKDatabaseSubscription – lets you subscribe to all changes in the whole database (private or shared).

Types of subscription notifications:

1. Silent push:

let notificationInfo = CKNotificationInfo()

// we only set this, but none of the UI related keys
notificationInfo.shouldSendContentAvailable = true

// do this once. no need to ask the user for push notifications permission,
// since we won't show any visible notifications
application.registerForRemoteNotifications(…)

2. Visual notification:

let notificationInfo = CKNotificationInfo()

// set any of these
notificationInfo.shouldBadge = true
notificationInfo.alertBody = "alertBody"
notificationInfo.soundName = "default"

// we need to prompt the user for push notification access:
application.registerUserNotificationSettings(…)
application.registerForRemoteNotifications(…)

Remember that push notifications can be coalesced, so you may only get one out of a series. Push notifications tell you that *something* has changed, but not necessarily every single thing that has changed.

Creating a subscription:

This only needs to be done the first time you launch an app – so we set a flag when we create a subscription and the next time we skip this part.

if subscriptionIsLocallyCached { return }

let subscription = CKDatabaseSubscription(subscriptionID: "shared-changes")

let notificationInfo = CKNotificationInfo()
notificationInfo.shouldSendContentAvailable = true
subscription.notificationInfo = notificationInfo

let operation = CKModifySubscriptionsOperation(
    subscriptionsToSave: [subscription],
    subscriptionIDsToDelete: []
)

operation.modifySubscriptionsCompletionBlock = { …
    if error != nil {
        …
    } else {
        self.subscriptionIsLocallyCached = true
    }
}

operation.qualityOfService = .utility
self.sharedDB.add(operation)

Listening for pushes:

turn on “Remote notifications” and “Background fetch” capabilities

func application(_ application: UIApplication,
    didReceiveRemoteNotification userInfo: [NSObject: AnyObject],
    fetchCompletionHandler completionHandler: (UIBackgroundFetchResult) -> Void) {

    let dict = userInfo as! [String: NSObject]
    let notification = CKNotification(fromRemoteNotificationDictionary: dict)

    if notification.subscriptionID == "shared-changes" {
        fetchSharedChanges {
              completionHandler(.newData)
        }
    }
}

Fetching the changes:

Steps:

ask in which zones something was changed (in shared db – because there may be new zones added when a new user shares some content)
ask which records have changed in each relevant zone

The server will not send you pushes about the changes you’re doing on this device, but you may receive those changes you’ve done on the list when fetching a delta download

fetchAllChanges: previously, in some operations you had to manually check for a flag that says there are more results waiting for you that you need to manually request (i.e. another page)

Now, CloudKit does the paging automatically for you if fetchAllChanges = true (which is the default)

func fetchSharedChanges(_ callback: () -> Void) {
    let changesOperation = CKFetchDatabaseChangesOperation(
        previousServerChangeToken: sharedDBChangeToken  // cached between runs
    )

    // this gives you IDs of changed zones
    changesOperation.recordZoneWithIDChangedBlock = { … }

    // this gives you IDs of deleted zones
    changesOperation.recordZoneWithIDWasDeletedBlock = { … }

    // this gives you the current change token which you need to save
    // may be called multiple times if the operation fetches multiple pages of content
    // save the token each time, so in case of an error you don"t repeat all work
    changesOperation.changeTokenUpdatedBlock = { … }

    changesOperation.fetchDatabaseChangesCompletionBlock = {
        (newToken: CKServerChangeToken?, more: Bool, error: NSError?) -> Void in

        self.sharedDBChangeToken = newToken
        self.fetchZoneChanges(callback)
    }

    self.sharedDB.add(operation)
}

fetchZoneChanges looks very similar, but fetches changes for a specific zone using CKFetchRecordZoneChangesOperation (you pass it a list of zones)

CloudKit best practices:

Automatic authentication:

CloudKit allows you to authenticate users (if they’re logged in to iCloud) without requiring any private information

You use the CloudKit user record for authentication

The user record is unique per container and never changes for that user

container.fetchUserRecordID(completionHandler: (CKRecordID?, NSError?) -> Void)

CKOperation API:

The convenience API works on single items and it’s simpler to use

Every convenience API call has a CKOperation counterpart that lets you perform an operation on a batch of records

The CKOperation also has other advantages – for example, it lets you:

set up dependencies between operations
specify quality of service and queue priorities
cancel operations that have started executing
specify if you want the operation to work over cellular network
limit the number of records or set of fetched keys
report progress
… and everything that NSOperation provides

(*) watch the Advanced NSOperations talk from 2015 to learn more about NSOperation

Quality of service:

QoS: select a quality of service (.userInteractive / .userInitiated / .utility / .background) depending on the task priority

default is .utility
.utility and below enable discretionary networking

Discretionary networking means that:

the system decides when is the best moment to run your request, so it may take longer than you expect
however, all network failures will be automatically retried for you
the request gets a timeout period of 7 days by default

Long lived operations:

If you have some operations that you want to continue/retry if they don’t manage to complete by the time your app is terminated, iOS 9.3 adds “CloudKit long lived operations”

Once you run such operation, the system will finish it even if the app is killed by the system or the user

The request is executed even if your app isn’t running, the result is cached and is returned to you once the app restarts

Results are kept by the OS for at least 24 hours

To use this API:

set isLongLived = true on CKOperation
save the operation’s operationID
use CKContainer.fetchLongLivedOperation(withId:) to get the operation object back
set completion blocks and run it again just like a new one

CKContainer.default().fetchLongLivedOperation(withID: myOpID) {
    (operation: CKOperation?, error: NSError?) in

    let fetchRecords = operation as! CKFetchRecordsOperation
    fetchRecords.fetchRecordsCompletionBlock = { … }

    CKContainer.default().privateCloudDatabase.add(fetchRecords)
}

Parent references:

A new type of reference added this year to help you better model data, especially with sharing in mind

If your app supports sharing, it’s recommended that you set the parent reference to create a hierarchy between records

Example: Album ⭢ list of photos

let photoRecord = CKRecord(recordType: "photo")
photoRecord.setParent(albumRecordID)

What this gives you: when the user shares the album record, the whole record hierarchy under this album (photos and other data) will also be shared

Types of errors:

1) Fatal error (bad request)

Error codes like:

.internalError
.serverRejectedRequest
.invalidArguments
.permissionFailure

→ in this case, you should show an alert to the user and tell them this can’t be executed

2) Connection/server error

Error codes like:

.zoneBusy
.serviceUnavailable
.requestRateLimited

→ in this case, check for CKErrorRetryAfterKey and retry after specified time

3) Errors that are returned before connection is even made

.networkUnavailable

you should monitor network reachability (SCNetworkReachability) and retry when the device is connected again

.notAuthenticated

when the user is not logged in and can’t access their private database
you should register at startup for CKAccountChangedNotification, and when it fires, recheck account status and update the UI

CloudKit Tips and Tricks

2020-11-11T18:39:56Z

Error Handling

Accounts:

To check the account status of the current user:

container.accountStatusWithCompletionHandler { status, error in … }

All APIs that fail because they require an authenticated user return CKErrorNotAuthenticated

You can now subscribe for CKAccountChangedNotification to be notified when account status changes

You should avoid showing alerts to the user about a missing account – simply disable parts of the UI that require an account, and reenable them when you get a notification that an account is now available

Network errors:

Network connection errors that you may sometimes get: CKErrorNetworkFailure, CKErrorServiceUnavailable, CKErrorZoneBusy, CKErrorRequestRateLimited

These errors include a key CKErrorRetryAfterKey in their user info dictionary that tells you how long you should wait before retrying

Handling conflicts:

If you try to save a record that has been modified in the meantime on the server (meaning: the record change tag you’re sending with the save request is outdated), you will receive the error CKErrorServerRecordChanged

There is no magic happening behind the scenes in such case, iCloud doesn’t make assumptions about how you want to resolve conflicts, you need to handle this yourself

However, the SDK provides you the necessary information in the userInfo:

CKRecordChangedErrorClientRecordKey – what you tried to save
CKRecordChangedErrorAncestorRecordKey – the original version
CKRecordChangedErrorServerRecordKey – what is currently on the server

Usually you will want to resolve the conflict by applying the same changes that you did on the original record to the current server version of the record

CloudKit Operations

Batch operations:

If you create and save a lot of records in one go, each of them will create a separate network request, and making a lot of requests in a short period means you’re likely to hit some kind of rate limit and they will be queued up

To avoid making multiple similar requests, you can use the CKOperation API

Almost every convenience API method that works on one record at a time has a CKOperation counterpart that allows you to work on a batch of records

For saving multiple records, use CKModifyRecordsOperation:

class CKModifyRecordsOperation: CKDatabaseOperation {
  convenience init(recordsToSave: [CKRecord]?, recordIDsToDelete: [CKRecordID]?)
}

Note: there are certain limits on how large batch operations you can make (number of items in the request and total request size)

This doesn’t include the size of saved binary assets, just the record field data

If you hit this limit, you will get the CKErrorLimitExceeded error

In that case, the best solution is usually to try to divide the batch in half and make two requests

If one or more records in the batch can’t be saved, you will get CKErrorPartialFailure

From this error’s userInfo you can get a dictionary with specific record errors under CKPartialErrorsByItemIDKey

In a standard zone, in such scenario some records will be saved and those with errors won’t

In a custom zone you can make an atomic update – in this case, in case of a problem with some of the records, no records will actually be saved, but instead you will get an error CKErrorBatchRequestFailed for those that could have been saved but weren’t

Queries:

If you expect a query to return a large number of records, but you only need a small number of them at a time, you can use the CKQueryOperation.resultsLimit property

Also available on CKFetchRecordChangesOperation, CKFetchNotificationChangesOperation

When limiting the number of records, you will usually also want to set the query's sortDescriptors to e.g. sort records by oldest or newest first

You can use the creationDate key which is automatically added to all saved records regardless of type

To implement pagination and get further pages beyond the first one, use the CKQueryCursor object that you get in response to the queryCompletionBlock callback

Then, initialize the next CKOperation passing it the cursor object in the argument to the initializer

If you don’t need all information contained in a record immediately, you can also use the desiredKeys property to only download the keys you want

E.g. download the record’s thumbnail image but not a full-size photo

Also available on CKFetchRecordsOperation, CKFetchRecordChangesOperation

Maintaining a local cache:

Let's say you want to keep some subset of all data completely cached on all local devices for quicker access (e.g. user’s personal notes)

You have two options:

use CKQueryOperation to fetch all records and synchronize them manually
make a custom zone and use delta downloads using CKFetchRecordChangesOperation

When saving fetched records to a local database, you should save CKRecord’s system fields like the change tag together with your own data

To do that, you can use encodeSystemFieldsWithCoder:

let data = NSMutableData()
let archiver = NSKeyedArchiver(forWritingWithMutableData: data)
archiver.requiresSecureCoding = true
record.encodeSystemFieldsWithCoder(archiver)
archiver.finishEncoding()

When restoring a record from the local storage, you don’t have to set all its data fields – it’s fine to only set those you want to change

To synchronize any changes from the server, create a subscription subscribing to a given record type using silent notifications, and use CKFetchRecordChangesOperation to fetch all recent changes when notified

Subscriptions (CKSubscription) are persistent queries on the server that send remote notifications about a relevant change – either in a specific record set (query subscription) or in the whole zone (zone subscription)

To get CloudKit subscription notifications, you need to follow the usual setup for push notifications:

have push notification capability enabled for your app
call registerForRemoteNotifications()
call registerUserNotificationSettings(…) if you want to show notifications to the user

To ask for silent subscription notifications, configure the CKNotificationInfo object appropriately:

set the shouldSendContentAvailable key
do not set any of the UI-related keys: alertBody, shouldBadge, soundName

Notification priorities: a notification is high priority if it has any UI keys set, otherwise it’s medium priority

For silent notifications, the UIApplicationDelegate will receive the following callback:

func application(application: UIApplication,
	didReceiveRemoteNotification: [NSObject: AnyObject],
	fetchCompletionHandler: (UIBackgroundFetchResult) -> Void) { … }

Remember that push notification delivery in general is “best effort” – pushes can be dropped if many are received in a short period of time or because of network issues

Silent notifications may also be additionally delayed if the system is waiting for better conditions

When you receive a notification, use CKFetchNotificationChangesOperation to check the server’s notification collection for any notifications you might have missed

You may want to use a UIApplication background task (beginBackgroundTaskWithName(…)) for syncing tasks

Interactive notifications: you can now make CloudKit notifications interactive (e.g. show action buttons) by setting the category key on CKNotificationInfo

Other performance tips

CloudKit is a highly asynchronous API, most operations require a network call and take some time to execute

You will often want to make a series of operations that have some dependencies between them

Things to keep in mind:

however you implement task handling, remember to always handle all errors
never block the main thread with an operation in progress

Don’t nest calls to the convenience API methods, creating a “callback hell”

Don’t use locks/semaphores to wait for an API call to finish

Instead, use the addDependency() API in CKOperation to add dependencies between operations:

let firstFetch = CKFetchRecordsOperation(…)
let secondFetch = CKFetchRecordsOperation(…)
secondFetch.addDependency(firstFetch)

let queue = NSOperationQueue()
queue.addOperations([firstFetch, secondFetch], waitUntilFinished: false)

Use the qualityOfService property on NSOperation to indicate which operations are something you need in the UI and which are low priority background operations

there used to be a usesBackgroundSession property on CKOperation too, but it’s deprecated now ⭢ use quality of service for this

QoS = .utility and .background use discretionary networking, use .userInteractive and .userInitiated for high priority tasks

Note: .background QoS is the default if you don’t change it!

update: now it's .utility

Advanced CloudKit

2020-11-03T14:57:14Z

CloudKit API is designed to be asynchronous, all calls return through a callback, because they all require a network connection

The main API (“operational API”) is based on NSOperation

You use it by creating special NSOperation objects for a given use case, e.g. CKFetchRecordsOperation, and specifying parameters and callbacks in their properties

Apart from the final result callback, you can set callbacks e.g. for reporting download progress or to get records one by one as they’re downloaded

Operation lifecycle (cancelling, suspending etc.) can be managed through standard NSOperation methods and NSOperationQueue

There are separete fetch/modify operation types for records, subscriptions, zones, users and notifications

You can set dependencies between operations (also if they’re in different queues), e.g. make a fetch operation and then a modify operation that needs to wait for the object to load

Operations can also have different priority levels

Starting an operation:

(ℹ️ Note: this wasn’t in the video, but it really should have been, because it's completely not obvious.)

How to start an operation once you prepare the CKOperation object:

1) Use the database’s built-in queue:

let fetchOperation = ...
CKContainer.default().privateCloudDatabase.add(fetchOperation)

2) Use your own operation queue and assign a reference to the database:

let operationQueue = NSOperationQueue()

let fetchOperation = ...
fetchOperation.database = CKContainer.default().privateCloudDatabase
operationQueue.addOperation(fetchOperation)

Custom zones

Custom zones (in the private database) let you compartmentalize data and add some special features

Records can’t be moved between zones or have cross-zone relationships

There are some operations that can only be done in custom zones:

Atomic commits:

Objects in the CloudKit database have relationships between them, and you want to keep all data consistent

Atomic commits are kind of like transactions in a relational database: batch operations succeed or fail together

Only available in the private database (because public database may be accessed by millions of users at the same time)

If an operation fails, you get a CKErrorPartialFailure response, with the user info containing info about errors on specific records (CKPartialErrorsByItemID)

Error CKErrorBatchRequestFailed means that this record wasn’t saved because of a problem with another record in the batch

Delta downloads:

Allow you to download a list of all changes since the last time the app was online, to let you perform a full sync

When a device connects, you can send a “change token” to the server asking for all changes since that version

This lets you implement an offline cache of the whole dataset and sync any changes when possible

To do that:

track all local changes
send changes to the server when connected
resolve conflicts
fetch server changes with CKFetchRecordChangesOperation
remember the received new server change token and send it back next time

Zone subscriptions:

Let you subscribe for notifications about any change in the zone

When you get a notification, you request a delta download

Advanced record operations

Record changes:

When you change some fields in a CKRecord, the changes are automatically tracked locally and only the changed fields are transmitted when you save it

By default CloudKit performs a “locked update”, which makes sure that the update is only saved on the server if the record wasn’t modified in the meantime by another client (this uses record change tokens)

After you execute a save, the server returns your record with a new change token – so you should use that returned version for any subsequent changes

Unlocked update ⭢ just overwrites server data regardless what is there

Locked update ⭢ if the record was changed in the meantime, you get back an error (CKErrorServerRecordChanged)

The userInfo of the CKErrorServerRecordChanged error contains info that lets you perform a 3-way merge:

CKRecordChangedErrorClientRecordKey – what you tried to save
CKRecordChangedErrorAncestorRecordKey – the original version
CKRecordChangedErrorServerRecordKey – what is currently on the server

Based on the values from these 3 copies of the record you can decide what state the record should be in, and then retry the save

You can modify the behavior with “save policies”:

SaveIfServerUnchanged ⭢ default, performs a locked update and sends only changed keys
SaveChangedKeys ⭢ unlocked update, sends only changed keys
SaveAllKeys ⭢ unlocked update, overwrites all keys in the record (note: this doesn’t affect keys that aren’t present in the local copy at all)

You should almost always use the default locked update (SaveIfServerUnchanged), use unlocked updates only to forcefully resolve serious conflicts

Use SaveAllKeys if the user requests to overwrite server data with local data

Partial records:

The desiredKeys field present in most operation types lets you specify that you only want to download selected keys from the server

This is useful if the whole record is very large and you don’t need all of it

Partial records can be normally saved after a change

CloudKit data modeling

References:

Forward reference ⭢ a parent object keeps an array of references to children in its property

Backward reference ⭢ only child objects have a reference to the parent

It’s recommended to use backward references – with a forward reference you need to update the parent object every time a new child is added, and you will run into conflicts if multiple clients are adding records

To get a list of all children using backward references, make a query for all child records with a predicate “owner = X”

References give you cascading deletes – when you delete the parent object, all child objects and their children are deleted

If an object has two parent references, it’s deleted when the first parent is deleted

When batch uploading a tree of objects, CloudKit makes sure that parent objects are uploaded first so that you don’t get inconsistent data during upload (important in the public database)

Your data objects:

CloudKit is only a transport mechanism and requires you to keep and manage your own local copy of all data

It’s recommended that you don’t subclass CK* objects to build your models – make your own completely independent model classes and translate to/from CloudKit objects when fetching and saving

Handling push notifications:

You need to remember that push notifications in general aren’t guaranteed to be delivered

The server only stores one push per client, so if you reconnect e.g. after a flight, you might miss some previous notifications

You can find pushes that you’ve missed in a “Notification Collection” where every notification is saved

The Notification Collection works kind of like delta updates – you ask for notifications since a given change token and you get a list of everything added since then

You can mark a notification as read, which notifies all other clients that they can ignore it

You should check the Notification Collection every time you get a push, since you never know what you might have missed (this doesn’t only happen with airplane mode)

The iCloud Dashboard

The dashboard lets you browse data saved by your app – the whole public database and the private database for your developer account (but not anyone else’s private database)

You can view saved records, run queries with any filters, and add new records

You can define roles in the public database and define for each model who can create/read/modify records (e.g. specify that records are publicly readable but only an admin can create them)

You will also see a list of all user ids and first/last names of those users that marked themselves as discoverable

Schema:

The CloudKit database has two separate “environments”: development and production

The schema for each record type is “just in time” during development, i.e. when you save a new type of record, it automatically creates a new schema for that record type, recording every field type, and when you save a record with a new field, it adds a field to the list

However, once you’re ready to release a new version of your app, you need to save the schema to production and at that point it’s locked – a production version of the app can’t save records or fields that aren’t defined in the schema

CloudKit also automatically creates indexes for each field in each record type – when you’re done with development, you can delete some indexes that you won’t need so they don’t waste space in the production database

Tips & tricks

Please handle all errors :)

Remember that you can get partial errors (when atomic commits aren’t used), so some records might be saved while others aren’t

Retry any “server busy” errors (CKErrorRetryAfterKey tells you the amount of time you should wait)

Don’t waste space in your users’ iCloud in private databases, they may be paying real money for it

Limits in the public database are mostly to prevent abuse, they should be fine for most normal use (the limits scale with the number of users)

iCloud Storage Overview

2020-11-01T20:52:45Z

iCloud Storage APIs allow you to store your app’s data in iCloud

System services sync your data automatically even when your app isn’t running

3 different types of storage:

key-value storage – simple storage for things like preferences, game state (“it’s so simple, we actually don’t have another session talking about it”)
document storage – a filesystem in the cloud scoped for your application, where you can store any kinds of files and folders, synced between devices; ideal for productivity apps like iWork
Core Data storage – an extension for Core Data that lets you store Core Data databases in the cloud [note: deprecated in 2016]

What iCloud handles for you:

account setup – users don’t need to create a new account for your service, they already have an iCloud account
APIs for your apps integrated into the OSX/iOS SDKs
server code you don’t have to write, for things like load balancing, replication, backup and recovery
the personnel handling the servers, support etc.

“If you happen to know some friends, who like writing server code that scales to hundreds of millions of users, send them our way – we’re hiring” :D

How it works:

Key-value storage:

Accessed through NSUbiquitousKeyValueStore

Lets you put simple plist-type values into the cloud

It talks to a key-value service running on the device which talks to iCloud on your behalf

Document storage:

Use UIDocument (iOS) / NSDocument (OSX) / UIManagedDocument (Core Data)

You can also use lower level file storage APIs like NSFileCoordination, NSFilePresenter, NSMetadataQuery

All these APIs (also the Core Data iCloud API) talk to OS’s document service which talks to the iCloud for you

You never actually interact with the iCloud servers yourself, you just use these APIs in the SDK and system services

How to enable iCloud in your project:

your app needs to be distributed through the App Store [note: no longer true for Mac apps]
enable the relevant entitlement: com.apple.developer.ubiquity-kvstore-identifier and/or com.apple.developer.ubiquity-container-identifiers

Working with Key Value Storage:

NSUbiquitousKeyValueStore

Lets you store simple plist values – strings, numbers, booleans, dictionaries and arrays of those

Similar API to UserDefaults

Simple conflict resolution: if two devices independently change or add a value for a given key, the latest change wins

Works even if iCloud isn’t configured – in this case it just acts as local user defaults that don’t sync

Improvements from last year’s initial release:

increased capacity – now up to 1024 keys, up to 1 MB per application (not part of the total user quota); it’s fine to have just one key of 1 MB
improved responsiveness (allows around 15 requests every 90 seconds)

How to set up:

// get a reference to the store
NSUbiquitousKeyValueStore *store = [NSUbiquitousKeyValueStore defaultStore];

// observe changes
NSNotificationCenter *center = [NSNotificationCenter defaultCenter];
[center addObserver:self
           selector:@selector(kvStoreDidChange:)
               name:NSUbiquitousKeyValueStoreDidChangeExternallyNotification
             object:nil];

// ask for any changes since the last launch
[store synchronize];

Making changes:

[store setObject:someObject forKey:@"someKey"];
[store setBool:YES forKey:@"someOtherKey"];

[store synchronize];

It’s recommended to also store another copy of the same data locally, so that you can do conflict resolution manually in case if just keeping the latest change isn’t always the right strategy for your app

Handling notifications about a change:

- (void)kvStoreDidChange:(NSNotification *)notification {
    NSDictionary *userInfo = [notification userInfo];

    // get change reason
    int reason = [userInfo objectForKey:NSUbiquitousKeyValueStoreChangeReasonKey];
    NSArray *changedKeys = [userInfo objectForKey:NSUbiquitousKeyValueStoreChangedKeysKey];

    // … store values locally, do conflict resolution etc.
}

Working with document storage:

Apart from your app’s container, each app has a separate “ubiquity container” (or iCloud container)

The app can put any files inside that container, and whatever is put there is synced with other devices via iCloud, kind of like Dropbox

When you put a file in the ubiquity container, the file is broken into chunks and the chunks that are new or modified are uploaded to iCloud – so if you only change a few bytes of the file, most of it doesn’t need to be uploaded again

Metadata about the file is uploaded first – info about the file name, type, size etc.

So every device knows about each file that’s uploaded, but it doesn’t necessarily have to download each file – the decision is made independently on the device depending on the platform and settings: OSX usually downloads all files if it has space, iOS only downloads files on demand

Once a file is downloaded to the device, all later changes are also automatically synced

Local peer to peer communication is used if possible – e.g. if you have a file on the Mac, your iPhone will copy it from the Mac instead of the iCloud network

Automatic conflict resolution – if a file is edited on two devices in parallel, the system picks a winner automatically, but your application gets access to both versions and can override it if needed

URL publishing: you can make the current version of the document public and available through a URL which you can share with others (if it’s changed later in the iCloud, the URL still downloads that previous version)

URLs are not permanent, they expire after some time

Detecting an iCloud account:

Unlike key-value storage, using this API requires that the user has an iCloud account

You can check for an “Ubiquity Identity Token” to see if they have an account configured

The token is anonymous so it doesn’t tell you anything about the user, but it will change if the user switches to another account (you also get a notification then)

The token is also unique to your app and to this specific device, so the same app will get a different token from that user on another device

id token = [[NSFileManager defaultManager] ubiquityIdentityToken];
if (token) {
    // cache the token
    // the next time the app launches, check if it has changed
}

// register for the identity changed notification
NSNotificationCenter *center = [NSNotificationCenter defaultCenter];
[center addObserver:self
           selector:@selector(handleUbiquityIdentityChanged:)
               name:NSUbiquityIdentityDidChangeNotification
             object:nil];

When the account changes, clear any local caches specific to this account and refresh the UI

To access the ubiquity container, you ask for the container URL – note that the container will be created on demand the first time you ask for it; this should ideally not be called on the main thread

dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
    containerURL = [fileManager URLForUbiquityContainerIdentifier:nil];
});

Types of documents you can store in the document storage:

normal files, also symlinks
directories of files
packages – bundles of files that act as a single document
Core Data stores

File extended attributes are also synced

Watch out for filesystem case sensitivity issues – users running on a Mac might have a case-sensitive filesystem

For packages, iCloud updates only the files from a package that have been changed, but it handles updates to the whole package atomically, so you will not get a package in an inconsistent state

Core Data:

Useful for so-called “shoebox style applications”, like iPhoto or iTunes, which work with a single database in app’s own format

The Core Data store remains local and only change logs are uploaded to iCloud

Not recommended to use binary and XML stores, because in those cases every change modifies the whole file (only use those for small data sets that don’t change often) – use SQLite stores for iCloud sync instead

UIManagedDocument – a subclass of UIDocument for managing Core Data stores that supports syncing them with iCloud

Note: NSPersistentDocument, the AppKit equivalent, does not support iCloud

Designing your document format for iCloud:

Design with network efficiency in mind – don’t write a lot of changes very often

Keep in mind any possible differences between platforms

Plan for future app upgrades – include version number in the format and keep compatibility if possible

Beware of sync loops – when one instance of the app receives a change, merges it and writes back the result, and the other side does the same and triggers a change in the first copy again

“And you have two versions of the app playing ping-pong with the user’s iCloud account”

Avoid making rapid changes to the file, e.g. updating some position tag while the user is scrolling the document

Don’t put the last open date into the document itself, so that opening it doesn’t count as making a change

Use iCloud for user data only: don’t put any caches, temporary files or auto-generated content there

Think about privacy when allowing the user to publish a document: don’t include any sensitive info or things they might not be aware they’re publishing (e.g. undo history) in the publicly accessible view

APIs:

NSFileManager

NSFileCoordinator & NSFilePresenter

NSMetadataQuery – use a live metadata query to be notified of new files and changes before the file contents are downloaded

NSDocument & UIDocument

UIManagedDocument

NSDocument/UIDocument handle most of the integration with iCloud for you:

using the ubiquity container
coordinating with the OS
tracking files and their versions
resolving conflicts

Tips:

subclass native document types
use the default auto-save behavior
if the app provides a prepopulated Core Data store on first launch, use a migration instead of copying a packaged store file
track documents using NSMetadataQuery
if it makes sense for your app, manually control conflict resolution (UIDocumentStateInConflict); avoid user involvement if possible

Tips for debugging:

test with multiple devices
monitor network traffic
use Airplane Mode to create conflicts artificially
there will be a configuration profile available that makes the document service log additional messages
developer.icloud.com – a web tool that shows you all iCloud storage on your account from various apps [note: not available anymore, replaced with CloudKit dashboard, but it only shows CloudKit data]

Introducing CloudKit

2020-11-01T20:52:04Z

CloudKit lets you write client applications without having to build and host a server part to handle things like database, accounts or push notifications

Usage is free for the developer up to pretty big limits

CloudKit gives you more direct access to iCloud servers

It’s the framework that’s used behind the scenes by iCloud Photo Library and iCloud Drive

Uses the same iCloud account as iCloud documents or key-value storage

Two types of databases:

public – accessible to everyone
private – private data of a specific user

It’s only a transport technology, it does not deal with local data persistence – you need to decide how you store the data that you load from the cloud

To enable iCloud in your app, set it up in the Capabilities tab in Xcode just like with other iCloud APIs

Containers:

Each app’s data is kept in a separate container (CKContainer)

Containers give you the safety that your app’s data will not be mixed with someone else’s app’s data

A container’s ID needs to be unique in the whole iCloud, so use reverse-domain style identifiers

By default each app has one container of its own, but apps can additionally use shared containers

Containers are managed by the developer through the WWDR portal

Databases:

A container contains one shared public database for everyone, and separate private databases for each user

An app running on the device has access to one public and one private database (CKDatabase)

The database is the initial entry point to CloudKit (from a container)

CKDatabase *publicDatabase = [[CKContainer defaultContainer] publicCloudDatabase];
CKDatabase *privateDatabase = [[CKContainer defaultContainer] privateCloudDatabase];

Private database:

requires a logged in iCloud account
data stored counts against the user’s iCloud account quota
default permission for data is user readable
the data your users store in your app’s CloudKit container is *not* accessible to you

Public database:

can be accessed anonymously even if the user isn’t logged in
data stored counts against the developer’s app quota
default permission for data is world readable
permissions can be customized using iCloud Dashboard Roles

Records:

A record (CKRecord) is a single “object” in the CloudKit database, essentially a list of key-value pairs

Records have a Record Type (~ table name)

There is no defined up front schema, you can just save a record of any type with any keys and the schema will be updated based on that

→ note: this only works in development, the schema is fixed in production, see "Advanced CloudKit"

Records can have metadata: who created it & modified it and when, also includes a “change tag” (version id) – for determining if two sides have the same version of a record

Record values can be: strings, numbers, dates, NSData, CLLocation, CKReference, CKAsset, arrays of any of these

- (instancetype)initWithRecordType:(NSString *)recordType;
- (id)objectForKey:(NSString*)key;
- (void)setObject:(id)object forKey:(NSString *)key;
- (NSArray *)allKeys;

Subscripts also work:

CKRecord *party = [[CKRecord alloc] initWithRecordType:@"Party"];
party[@"start"] = [NSDate date];

Record zones:

Records are grouped within a database inside “zones” (CKRecordZoneID)

The public database has one zone, the private database has one default zone, but it can have additional custom zones

Record identifiers:

Record identifier (CKRecordID) is a tuple grouping: a “record name” + zone ID

You can provide a recordID when creating a record instance

If you don’t provide a recordID, a random UUID will be assigned

References:

A reference (CKReference) is a pointer from one record to another, as an id of the “parent” record contained in a child record’s field

References allow you to do cascade deletes, deleting child records when parent is deleted

You can create a reference from a CKRecord object or from a CKRecordID if you know the ID but don’t have the object in memory

Assets:

An asset (CKAsset) is an unstructured piece of data, basically a binary file

Assets are downloaded and uploaded from/to files on disk, not from memory

An asset is always owned by a record, and is deleted when the record is deleted

Transport of assets is optimized so that only the minimal amount of data is transferred

APIs:

There are two different APIs for managing CloudKit data: “operational API” and “convenience API”

The operational API has every possible operation you might need, the convenience API is more convenient

Start with the convenience API, use operational API for tweaking and overriding options if needed

CloudKit APIs for saving/fetching data are asynchronous – there is no SDK-managed local data, everything needs to go over the network unless you manually cache it

In CloudKit it’s absolutely necessary to properly handle error cases – every network call can fail and your app needs to be prepared for this

Convenience API:

[publicDatabase saveRecord:obj completionHandler: { … }];
[publicDatabase retchRecordWithID:recordID completionHandler: { … }];

Queries:

For any large database, or the shared public database, you shouldn’t try to keep a copy of the whole database on disk and sync all of it, but instead fetch what you need on demand – for this you can use queries

A query (CKQuery) allows you to fetch a list of records matching some conditions

Query can specify a RecordType, NSPredicate and optionally NSSortDescriptors

A subset of NSPredicate language is supported, if something is not supported you’ll get an exception

Predicates such as “equal”, “greater than”, “distance to location”, string tokenizing, and OR / AND are supported

[publicDatabase performQuery:query inZoneWithID:nil completionHandler: { … }];

Subscriptions:

If you repeatedly run the same query, polling for the same data, you can ask the server to run the query for you and notify you immediately when a new record is added

You do that by creating a subscription (CKSubscription)

A subscription includes: RecordType, NSPredicate and push configuration (CKNotificationInfo)

Your app is notified of changes through a push notification with some additional data

CKSubscription *subscription =
  [[CKSubscription alloc] initWithRecordType:@"Party"
                                   predicate:predicate
                                     options:CKSubscriptionOptionsFiresOnRecordCreation];

[publicDatabase saveSubscription:subscription completionHandler: { … }];

Pushes are handled through the usual push API:

- (void)application:(UIApplication *)application
    didReceiveRemoteNotification:(NSDictionary *)userInfo;

Build a CKNotification object from the user info:

CKNotification *cloudKitNotification =
    [CKNotification notificationFromRemoteNotificationDictionary:userInfo];

NSString *alertBody = cloudKitNotification.alertBody;

if (cloudKitNotification.notificationType == CKNotificationTypeQuery) {
    CKQueryNotification *queryNotification = cloudKitNotification;
    CKRecordID *recordID = [queryNotification recordID];
}

Handling user accounts:

Your application does not get direct access to any user identifiers like iCloud email address

Instead, in each container each user is represented as a unique ID within that container that doesn’t change unless the user switches to another account

The same user will have a different ID in a different CloudKit container

The ID is an instance of CKRecordID

[[CKContainer defaultContainer] fetchUserRecordIDWithCompletionHandler: { … }];

Each user has a user record representing them (which is almost like any other record) with their user id and record type = CKRecordTypeUserRecord, one in the private database, and another with the same ID in the public database

You can set and read any key-value data on this record like on other records

However, these records aren’t created by you and can’t be queried to get a list of users

[publicDatabase fetchRecordWithID:userRecordID completionHandler: { … }];

User discovery:

You can ask the user to allow you to make them discoverable by other users (they get a request popup)

If they agree, they can be looked up by user ID, specific email, or by fetching a list of all users matching your user’s contacts from the address book (this doesn’t give your app access to the address book itself, just a list of matching users)

You get back record IDs, first & last names of users, but no emails

[defaultContainer discoverAllContactUserInfosWithCompletionHandler: { … }];

Returns an array of CKDiscoveredUserInfo objects with properties userRecordID, firstName, lastName

[note: this API has been replaced since then with a new one that returns CKUserIdentity objects]

When to use CloudKit vs. other APIs?

CloudKit doesn’t replace or deprecate any existing iCloud APIs [yet ;P], it’s just an additional tool

Key-value store:

asynchronous, small amounts of data
mostly for application preferences

iCloud Drive:

works on files and folders
on OSX it makes a full offline cache of the drive
good for document-centric apps

iCloud Core Data:

built on top of iCloud Drive
good for keeping private, structured data (custom databases) in sync
note: the whole data set is downloaded to each device

CloudKit:

good for sharing public data between users, both structured data and large files
good for large data sets where not every device needs to have a copy of the whole database
for attaching some data to the user’s identity and sharing info between users that know each other
more low-level, your app is in control of when any information is downloaded or uploaded to the iCloud servers, and has responsibility for handling sync

TypeScript on Corona Charts

2020-10-15T15:43:06Z

Back in spring I built a website that lets you browse charts of coronavirus cases for each country separately, or to compare any chosen countries or regions together on one chart. I spent about a month of time working on it, but I mostly stopped around early May, since I ran out of feature ideas and the pandemic situation was getting better (at least in Europe). The traffic that was huge at the beginning (over 10k visits daily at first) gradually fell to something around 1-1.5k over a few months, and I was only checking the page myself now and then. So it seemed like it wouldn’t be needed for much longer…

“Oh, my sweet summer child”, I kinda want to tell the June me 😬

So now that autumn is here and winter is coming, I suddenly found new motivation to work on the charts site again. But instead of adding a bunch of new features right away, I figured that maybe some refactoring would make sense first. I initially built this page as a sort of hackathon-style prototype (“let’s see if I can build this in a day”), but it grew much more complex since then, to reach around 2k lines of plain JavaScript - all in one file and on one level.

I started thinking about how I can make this easier to manage, and somehow I got the idea to try TypeScript.

Why add static typing?

I used to believe that static typing was just unnecessary complication.

My first programming experiments back in school were in Pascal and very bad C++. At the university, pretty much everything was either plain C or Java (or C#, depending on which group you picked). It was only near the end of the studies that I suddenly discovered Python and later Ruby, and it was like a breath of fresh air. I also read Paul Graham’s book “Hackers and Painters”, which made a big impression on me, steered me away from big corpos towards the startup world (for which I’m forever grateful), and also showed me how much better dynamically typed languages (specifically Lisp) were than statically typed ones.

I spent the next few years writing mostly Ruby and some JavaScript for the frontend, and I loved it. I still love Ruby and to this day I use it for all my scripting and server-side code.

However, at some point I also started building Mac and iOS apps, first in ObjC, and then in Swift. ObjC just felt like so much unnecessary boilerplate, and it really was. Then Swift came and simplified everything, but it exchanged the boilerplate for a very strict type system, much stricter than anything I’ve seen before. It was annoying to have to explain the compiler which property can be nil and when and what to do with it, or what to do if this JSON array does not contain what I think it does.

But after using Swift for a few years, I really appreciate the feeling of safety it gives you. You have to put in more work up front, but once you do, and once it compiles, you can be sure that whole categories of possible errors have already been eliminated before you even run the app. You have to do fewer build - run - find an error - fix the error cycles while building new features, and it’s also great while refactoring. And most importantly, it makes it harder to break one part of the code while changing another - we don’t always test every part and every single path in the app after every change, so such accidentally introduced errors can make their way to production and to users before they’re discovered.

Long story short, I miss this feeling a bit when working with Ruby and JavaScript now. It would be nice to have someone or something look over my code as I’m working on it, and not only the parts that are currently executed.

Learning TypeScript

TypeScript is not a difficult language, it’s not even a completely new language - it’s like JavaScript with some extra features and a compiler. So if you know JavaScript, you just need to learn the syntax for adding types, and the type declarations is the only thing you need to change in your code.

The official TypeScript site has a great docs section, and you can learn everything you need there. Start with the TypeScript for JS programmers intro and then go through the whole handbook and possibly the reference part if you want more, and that’s it.

Setting up the editor

A normal person would just download VS Code… however, that’s not me. I just refuse to run any IDE or editor without a fully native UI look & feel, so that leaves me with very little choice for those moments when I’m not using Xcode. For Ruby and JavaScript, I use TextMate, which I’ve been using non stop since 2008 (now the version 2 for the last few years).

There is a TextMate bundle for TypeScript, however, it only provides code highlighting and formatting. To simplify running the compiler while I’m in the editor, I manually added two actions using the bundle editor so that I don’t need to switch to the terminal to run npx tsc every time:

1) Compile to first error (shows output in a tooltip):

#!/usr/bin/env ruby18
require ENV['TM_SUPPORT_PATH'] + '/lib/textmate'

tsc = File.expand_path(File.join(ENV['TM_PROJECT_DIRECTORY'], 'node_modules', '.bin', 'tsc'))

ENV['PATH'] += ':/usr/local/bin'

result = `#{tsc} --noEmit`
first_line = result.each_line.first

if first_line.to_s.strip.length > 0
  puts first_line

  if first_line =~ /\((\d+)\,(\d+)\)\:/
    TextMate.go_to :line => $1.to_i
  end
else
  puts "Build OK"
end

2) Compile file (shows output in a special new window):

#!/usr/bin/env ruby18
require ENV['TM_SUPPORT_PATH'] + '/lib/textmate'
require ENV["TM_SUPPORT_PATH"] + "/lib/tm/executor"

tsc = File.expand_path(File.join(ENV['TM_PROJECT_DIRECTORY'], 'node_modules', '.bin', 'tsc'))

ENV['PATH'] += ':/usr/local/bin'

TextMate::Executor.run(tsc, '--noEmit')

This is a bit hacky, but it works for me. It only supports a scenario with one TypeScript file for now - there’s apparently no way to make tsc both use the tsconfig.json config to configure compiler options, but also specify a specific filename on the command line. And I’d love to have real autocompletion too, but you can’t have everything…

(If you know a good programmer’s editor with a native Mac UI, please let me know!)

Setting up the compiler & build

Once you download the TypeScript compiler node module, you can run npx tsc --init to create a tsconfig.json file at the root of your project. In this file you can specify where to look for .ts files, how modern JavaScript it should output (I chose es2018 since I don’t need to support any old browsers), and turn specific checks on and off. I’ve experimented a lot with the compiler options, and eventually I’ve left everything on except strict, strictNullChecks and strictPropertyInitialization (which requires strictNullChecks). The null checks add a ton of additional errors everywhere, and would require me to unwrap everything with ! like in Swift on every step (e.g. every call to querySelector, querySelectorAll, parentNode and such things), and I decided it’s just not worth the effort. It could possibly make sense if I was using some framework that was abstracting all interaction with DOM like React.

If you use any external libraries, like Chart.js in my case, you will also want to download their .d.ts definition files before you start working on your code - otherwise the compiler will keep telling you “I have no idea what this chart thing is and whether it has such property”.

As for running the build, npx tsc does everything once you configure it in tsconfig.json - the problem is how to make it run when it needs to, and what to do with what it outputs…

Again, a normal person would just set it up in some Gulp, Grunt, Webpack or whatever it is that people use in JavaScript land this month 😛 In my case however, I have a Ruby project built on top of Sinatra that has no JavaScript build system (since I have very little JavaScript in general outside of the Corona Charts page) and even no proper asset pipeline configured. So I could either set up some new build system just for this, or write some kind of hack. You can guess what I picked.

Since I only have one TypeScript file for now, and it’s only used on one page, I realized I can just manually check the timestamp in the controller action and rebuild if the .ts file is newer:

get '/corona/' do
  # ...
  unless production?
    original = 'public/javascripts/corona.ts'
    compiled = 'public/javascripts/corona.js'

    if File.mtime(original) > File.mtime(compiled)
      puts "Compiling #{original}..."
      `npx tsc`
      puts "Done"
    end
  end

  erb :corona, layout: false
end

That’s for development - for production, I simply run it as one of the deployment phases in my Capistrano script:

after 'deploy:update_code', 'deploy:install_node_packages', 'deploy:compile_typescript'

task :install_node_packages do
  run "cd #{release_path}; npm install"
end

task :compile_typescript do
  run "cd #{release_path}; ./node_modules/.bin/tsc"
end

Updating the code

It took me a good day or two to update all the code to silence all TypeScript errors. The good thing is that even though you get errors, the TypeScript compiler still outputs proper JavaScript that looks like what you had before (as long as it makes any sense at all), it just can’t promise it will work, so you could possibly deploy it as is, treat the errors like warnings in Xcode and get rid of them gradually. But ideally you want to not have any errors at all, since just like with warnings in Xcode, once you have too many of them, you stop noticing the important ones.

The changes I had to make can be grouped in a few categories:

deleting some unused code, variables and parameters (that I haven’t realized were unused)
creating type definitions for all informal data structures used in the code - e.g. declaring that a DataSeries is a hash mapping a string to an array of exactly 4 numbers, what the structure of the downloaded JSON is, or that the valueMode parameter can only be “confirmed”, “deaths” or “active” (it’s so cool that you can have such specific types!):
```
type ChartMode = "total" | "daily";
type ValueMode = "confirmed" | "deaths" | "active";
type ValueIndex = 0 | 1 | 2 | 3;

type RankingItem = [number, Place];
type DataPoint = [number, number, number, number]
type DataSeries = Record<string, DataPoint>;

interface DataItem {
  place: Place;
  data: DataSeries;
  ...
}
```

defining all global variables as explicitly typed properties on Window:

interface Window {
  colorSet: string[];
  coronaData: DataItem[];
  chart: Chart;
  ...
}

declaring all custom properties I set on built-in objects like DOM elements, or method extensions added to built-in types like Object or Array:
```
interface HTMLAnchorElement {
  country: string;
  region: string;
  place: Place;
}
```

declaring class variables and their types:

class Place {
  country?: string;
  region?: string;
  title?: string;
  ...
}

declaring the type of all function parameters (you don’t usually need to specify return types, those are inferred):
```
function datasetsForSingleCountry(place: Place, dates: string[], json: DataSeries) {
  ...
}
```

casting some DOM objects to a more specific type like HTMLInputElement when I want to use the value:

let checkbox = document.getElementById('show_trend') as HTMLInputElement;
window.showTrend = checkbox.checked;

providing a type for local vars initialized with [] or {}:

let ranking: RankingItem[] = [];
let autocompleteList: string[] = [];

It’s a lot of changes in total when you look at the diff, but most of it is things I needed to write once somewhere at the top. When I write a new function now, I usually just need to add types to the parameters in the function header.

Was it worth it?

So far - I’d say, absolutely. Like I wrote above, when adding new functions I usually just need to declare parameter types, unless I start adding completely new types, but most of the time I operate on the ones I already have. Like in most modern languages, you usually don’t need to define a type for a local variable like let thing = getThing(), because the compiler knows that it’s of type Thing. And if you return it, it knows this function always returns a Thing when it’s called elsewhere.

So it doesn’t add much overhead for new code, but it does give me this nice feeling that someone is checking what I write. I’ve done one refactoring since then, modifying the structure of the JSON file to make it smaller, since it naturally got way larger over the last few months (1.2 MB uncompressed, although I managed to compress it to 190 KB now using Brotli compression set at max level).

I changed the declaration of window.coronaData at the top to be an object instead of an array, and the DataSeries to be an array instead of a hash. And the compiler immediately showed me every single place in the code that was using these objects and had to be updated to the new format. I didn’t have to use the editor search to hunt down every single place of use, and worry that I might have missed one that I’ll only discover after some thorough testing (or a complaint from a user). Once it compiled, it was basically done and it worked from the first run.

So am I going to use TypeScript now for every 1-page-long piece of JS that adds some animations to a blog? Of course not. But does it make sense to use it in a webapp with dozens of features that builds and manages the whole UI in JavaScript? I think it does.

WatchKit Adventure #4: Tables and Navigation

2020-09-10T11:36:23Z

< Previously on WatchKit Adventure…

Two weeks ago I posted the first part of a tutorial about how to build an Apple Watch app UI using WatchKit, using a WKInterfaceController and a storyboard. We’ve built the main screen for the SmogWatch app, showing a big colored circle with the PM10 value inside and a chart showing data from the last few hours.

Here’s the second part: today we’re going to add a second screen that lets the user choose which station they want to load the data from. So far I’ve used a hardcoded ID of the station that’s closest to me, but there are 8 stations within Krakow and the system includes a total of 20 in the region, so it would be nice to be able to choose a different one.

(I initially wanted to also include a selection of the measured pollutant - from things like sulphur oxides, nitrogen oxides, benzene etc. - and I’ve actually mostly implemented it, but that turned out to be way more complex than I thought, so I dropped this idea.)

The starting point of the code (where the previous part ends) is available here.

Preparing the data

Since the list of stations doesn’t change often, we can hardcode a list of stations with their names, locations and IDs in a plist file that we’ll bundle inside the app. The list is generated using a Ruby script, in case it needs to be updated later - you can just download the plist and add it to the Xcode project.

At runtime, the list will be available in the DataStore:

struct Station: Codable {
    let channelId: Int
    let name: String
    let lat: Double
    let lng: Double
}

class DataStore {
    // ...

    lazy private(set) var stations: [Station] = loadStations()

    private func loadStations() -> [Station] {
        let stationsFile = Bundle.main.url(forResource: "Stations", withExtension: "plist")!
        let data = try! Data(contentsOf: stationsFile)

        return try! PropertyListDecoder().decode([Station].self, from: data)
    }
}

Handling secondary screens

When we want to add an additional screen to the app that shows some secondary information or less commonly used feature like this, there are generally three ways we can handle it:

we can add an explicit button somewhere on the screen that opens it (usually in the bottom part)
we can put it on another page in the page-based layout (e.g. like sharing and awards in the Activity app)
or we can put it as an action in the menu accessed through Force Touch

The third option (Force Touch menus) is going away now. In the watchOS 7 betas, Apple has removed all Force Touch interactions from the OS and their own apps, the APIs for using it in third party apps (addMenuItem in WKInterfaceController) are deprecated, and it’s highly likely that the upcoming Series 6 watch will not include it as a hardware feature.

Hiding some actions in a menu had the advantage that it didn’t clutter the main view, but it also made those actions less discoverable and harder to use for those who need them. I personally always had a problem with the Force Touch menus in that I rarely remembered that they exist, and I often not realized that an app had some extra features if it put them there… So I guess it’s for the better, although it will probably take some time to adjust.

Using a page view controller and putting settings on the second page could work, but it doesn’t feel to me like this feature is important enough to get its whole new section in the main app navigation. I don’t think this is something people will do often - normally they should just select a station close to their home and never change it again. There will probably be some users who might want to often switch between stations to compare the values, but that would rather be a minority. (If you do want to use a page view controller, adding it is kind of unintuitive: there is no “Page View Controller” in the library, but instead you drag a segue from the first screen to the second and choose “Next page”.)

So I’ve decided that in this case it’s not a problem to have an additional button at the bottom of the main screen, which opens the list in a separate view.

The second choice is how to show the screen: do we show it as a modal, or push it onto the navigation stack (the latter only possible if we aren’t using a page-based layout)? Again, both could work and it’s mostly a matter of preference. But I think in this case a pushed view integrates better with the rest of the app.

Opening the list view

So let’s look at the storyboard again. We’d like to have a button below the chart that looks like a single table cell, which says “Choose Station” and shows the current station below, and opens a selection dialog when pressed.

In WatchKit, buttons work in an interesting way: a button can show a text or an image as its title, but it can also include… a group, which itself can contain any structure you want, however complex. You could even wrap the whole screen in one group which acts as a button if you want (though I’m not sure what happens if you put a button into a group in a button - it might be like when you type Google into Google…). By the way, SwiftUI, which started its life on watchOS, kind of took over this idea - the Button takes a closure that returns its label and you can also put almost anything there.

So let’s add a button at the bottom of the view here, and change its Content type to Group. The group is horizontal by default, so change its Layout to Vertical. Next, drag two labels inside: a top label “Choose Station”, with a Body font, and a bottom label that shows the name of a station, with a Footnote font and Light Gray color. Like on iOS, you can also configure labels so that they’re able to shrink if the text is too long - lower the Min Scale slightly to 0.9, since the station names are kind of long.

Notice that while a button normally has a dark gray background by default when showing a text, it lost the background when we switched it into the group mode. I’d like it to look like the standard buttons used e.g. in the Settings app’s various dialogs, but I don’t think there is any way to restore this default shape and color other than trying to manually recreate it. (Technically, we could probably implement it as a one-row table instead… but that would be too much extra work.)

So here’s how we’ll do it: add another plain button to the view. Select the new group, open the select field for the Color property (not Background - that’s used when you have an image background) and choose “Custom”. Now, in the system color picker use the “eyedropper” tool at the bottom to read the color value from the standard button:

Now you can delete the plain button. Let’s also give the group button custom insets to have some padding around the text: 8 on the top, bottom and right, and 10 on the left. And connect the lower label to an outlet in the InterfaceController, since we’re going to need it later:

@IBOutlet var stationNameLabel: WKInterfaceLabel!

Pushing the list view

Now, we’re going to add a second screen to our app. Drag a new interface controller from the library and put it on the storyboard on the right side of the main screen. Then, drag a segue from the button to the new screen - you’ll probably have to use the element sidebar on the left, since if you drag from the button’s rendering in the scene, it selects the group and not its parent button. Select the “Push” segue type (for a modal, you’d do it the same way, but with a “Modal” type segue instead).

Alternatively, we could leave the new screen disconnected, assign it an identifier, and then open the screen manually in code using the pushController(withName:context:) method, or presentController(withName:context:) in case of a modal, from the IBAction triggered from the button. But this sounds like more work, and I generally like to use segues whenever possible, since the storyboard then shows a clearer picture of the whole flow of the app.

The pushed view shows a “<” back button at the top, and you can put a title there. However, these titles work differently than on iOS: on iOS, you usually have a “< Back” button on the left, and a title in the center. On the Watch, there isn’t enough space for that - so the title shown after the “<” sign is supposed to be the name of the current view, not the view that you can get back to.

In this case, let’s make it say simply “Station”, since “Choose Station” is a bit too long (you need to leave space for the clock on the right). You can type it into the Title field, or just double-click the top area where the title should be (though it won’t be displayed on the storyboard).

Designing table cells

To display a list of things from which you can select one, the obvious choice is a table view. On watchOS it works somewhat similarly to iOS, with some exceptions - there are no sections, there’s only one single section of cells (although you could simulate sections by making different cells that act as section headers). But like on iOS, you use custom classes to handle the cells - like UITableViewCell, here they’re called “Row Controllers”; cells also have identifiers, and you can use different kinds of cells in one table.

To start, drag a table from the library into the second view. You automatically get one standard row type created for you, but you can add more.

Add a label to the table row’s group, use a standard Body font, but set Min Scale to 0.8 to accomodate longer names. By default the label will put itself in the top-left corner, so set its Vertical Alignment to center.

We need one more thing though: it would be nice to show a checkmark symbol on the right when you select a cell - just like in the Settings app:

Unfortunately, there doesn’t seem to be any equivalent of UITableViewCell.AccessoryType here - if you want to have a checkmark, you need to add it manually as a normal label.

So, add another label to the same row group, set its title to the unicode symbol “✓” (which looks very similar to the checkmark in the settings), and “borrow” the green color from the system checkmark in the Settings using the same eyedropper method as before. Set its Vertical Alignment to center too.

We have one problem though: the first label takes all available space, and the checkmark is pushed to the edge:

Sadly, there are no “compression priorities” here like in AutoLayout, so there’s no way to tell WatchKit to give the checkmark all the space it needs and then leave the rest to the title. What we can do instead is assign the checkmark a Fixed width which is then enforced - 15 seems about right; it’s not an elegant solution, but it works. Set also its internal Alignment (the text alignment, in the Label section, not the position alignment) to right so that the symbol stays at the right edge, even if we gave it too much space.

Handling the channel ID

Let’s look at our data & networking code for a moment. When the user picks a station, we’re going to store the channel ID in the DataStore. We also need to make sure that when the channel ID is changed, the old data is reset, because it was loaded from another station so it’s no longer relevant:

private let selectedChannelIdKey = "SelectedChannelId"

class DataStore {
    // ...

    var selectedChannelId: Int? {
        get {
            return defaults.object(forKey: selectedChannelIdKey) as? Int
        }
        set {
            if newValue != selectedChannelId {
                defaults.set(newValue, forKey: selectedChannelIdKey)
                invalidateData()
            }
        }
    }

    func invalidateData() {
        defaults.removeObject(forKey: savedPointsKey)
        defaults.removeObject(forKey: lastUpdateDateKey)
    }
}

We also need to actually use the new channel ID when making the request for the data. This is a bit too long to paste here, so here’s the relevant change in the KrakowPiosDataLoader class. Instead of the hardcoded ID of one specific station that I’ve used before, we’ll now be passing the ID of the selected station to the query.

The table controller

We’ll handle the table view in code in a new interface controller, which we’ll call StationListController. Create such class in Xcode (inherit from WKInterfaceController - there’s no special “table view controller”) and assign it to the new scene on the storyboard. Also add this outlet and connect it:

@IBOutlet weak var table: WKInterfaceTable!

We’ll also need a row controller class (it’s required, and there is no default base class with standard outlets like UITableViewCell that we could use anyway). Use NSObject as the base class and call it StationListRow - like table view cells, it will be very simple:

class StationListRow: NSObject {
    @IBOutlet weak var titleLabel: WKInterfaceLabel!
    @IBOutlet weak var checkmark: WKInterfaceLabel!

    func showStation(_ station: Station) {
        titleLabel.setText(station.name)
        checkmark.setHidden(true)
    }

    func setCheckmarkVisible(_ visible: Bool) {
        checkmark.setHidden(!visible)
    }
}

Assign the class name to the row on the storyboard and connect the outlets. A row also needs an identifier - call it BasicListRow (yes, there will be another :).

Now, in the StationListController, the interesting stuff happens in the awake(withContext:) method. What is a context, you might ask? It’s a cool idea that Apple kind of expanded on in the iOS 13 SDK, in the form of UIStoryboard.instantiateViewController, which lets you have a custom initializer in a UIViewController in which you can receive any required data, while still using storyboards and segues to navigate to the view controller.

In WatchKit, instead of custom initializers you have a single context object - but this context could be anything you want, including any complex structure and non-ObjC types. You can use this object to pass all required data from the parent/presenting controller to the presented controller, and extract it in awake(withContext:).

We’ll use the context to pass the new view controller the list of all stations and the id of the currently selected one (if any). And it turns out that it’s also possible to pass blocks this way, so we can include a simple block that will act as a selection callback - this way we can avoid building a “delegate protocol” to pass the response back.

Let’s prepare a simple struct for the context data:

struct StationListContext {
    let items: [Station]
    let selectedId: Int?
    let onSelect: ((Station) -> ())
}

The selection controller will get this data from the main interface controller, which assigns it in the contextForSegue(withIdentifier:) method:

override func contextForSegue(withIdentifier segueIdentifier: String) -> Any? {
    if segueIdentifier == "ChooseStation" {
        return StationListContext(
            items: dataStore.stations,
            selectedId: dataStore.selectedChannelId,
            onSelect: { _ in }
        )
    }

    return nil
}

For this to work, you need to select the segue on the storyboard and assign it the identifier ChooseStation.

In the StationListController, we’ll receive the data from the context in awake(withContext:):

class StationListController: WKInterfaceController {
    var selectedRowIndex: Int? = nil
    var items: [Station] = []
    var selectionHandler: ((Station) -> ())?

    override func awake(withContext context: Any?) {
        let context = context as! StationListContext

        items = context.items
        selectionHandler = context.onSelect
        ...

Notice that we don’t need to call super() in awake(withContext:) - the same is true for willActivate, didAppear etc. If you look at the documentation of those, it always says “The super implementation of this method does nothing” there.

To initialize the table, we call the setNumberOfRows method to set the row count, and then we iterate over the rows to initialize their contents (there is no “cell reuse” and initializing cells during scrolling, it’s all done up front). If you wanted to have multiple types of rows in one table, then you need to call setRowTypes instead and pass it an array with as many repeated identifiers as you want to have rows.

table.setNumberOfRows(items.count, withRowType: "BasicListRow")

for i in 0..<items.count {
    let row = table.rowController(at: i) as! StationListRow
    row.showStation(items[i])
}

We’ll also preselect the row of the currently selected station, if we can find it (we pass the channel ID from the parent controller, but here we’ll store the index of the row, so that we can later deselect it easily).

if context.selectedId != nil {
    if let index = items.firstIndex(where: { $0.channelId == context.selectedId }) {
        let row = table.rowController(at: index) as! StationListRow
        row.setCheckmarkVisible(true)
        selectedRowIndex = index
    }
}

To handle row selection, we’ll implement table(_:didSelectRowAt:) (you don’t need to assign any delegate/data source properties to the table or add any protocols, it works automatically):

override func table(_ table: WKInterfaceTable, didSelectRowAt rowIndex: Int) {
    if let previous = selectedRowIndex, previous != rowIndex {
        listRowController(at: previous).setCheckmarkVisible(false)
    }

    listRowController(at: rowIndex).setCheckmarkVisible(true)
    selectedRowIndex = rowIndex

    selectionHandler?(items[rowIndex])
}

func listRowController(at index: Int) -> StationListRow {
    return table.rowController(at: index) as! StationListRow
}

As you can see: we manually hide the checkmark in the previously selected row whose index we’ve saved, we show the checkmark on the current row, we store the row index, and we pass the Station back through the callback block.

Finally, we need to handle the selection in the parent controller when the callback is called. Specifically, we’ll need to:

save the channel ID of the new station in the DataStore, so that further requests to the web service will load data from that station
update the displayed station ID in the button at the bottom
show in the UI that we have no data from that station yet
request to reload the data immediately

Here’s how we do it:

func setSelectedStation(_ station: Station) {
    dataStore.selectedChannelId = station.channelId
    stationNameLabel.setText(station.name)

    updateDisplayedData()
    gradeLabel.setText("Loading")

    KrakowPiosDataLoader().fetchData { success in
        self.updateDisplayedData()
    }
}

And remember to call this in the callback block from StationListContext:

return StationListContext(
    items: dataStore.stations,
    selectedId: dataStore.selectedChannelId,
    onSelect: { station in
        self.setSelectedStation(station)
    }
)

The end result should look like this 🙂

And one more thing: we need to also remember to initialize the selected station label when the view is first loaded. We’ll make it say “not selected” if nothing was selected yet:

// call in awake(withContext:)

func updateStationInfo() {
    guard let channelId = dataStore.selectedChannelId else { return }

    if let station = dataStore.stations.first(where: { $0.channelId == channelId }) {
        stationNameLabel.setText(station.name)
    } else {
        stationNameLabel.setText("not selected")
    }
}

User location

There’s still something we could do to improve the user experience: why ask the user which station they want to load the data from, when in the majority of cases they’ll only be interested in the one that’s closest to them? And why make them scroll through the whole list, if some of the stations are 100 km away from them?

We can solve this if we ask the user for location access - after all, almost every Apple Watch has built-in GPS, and those that don’t are connected to an iPhone that has one.

On watchOS we ask for location exactly like on iOS - so we can follow the instructions I wrote here a few years ago:

add an NSLocationWhenInUseUsageDescription key to the Info.plist (e.g. “SmogWatch uses location data to pick a station that’s closest to you.”)
add a reference to a CLLocationManager in the InterfaceController and make it its delegate
ask for location access when the main screen opens:

var userLocation: CLLocation?

override func willActivate() {
    askForLocationIfNeeded()
}

func askForLocationIfNeeded() {
    guard userLocation == nil, CLLocationManager.locationServicesEnabled() else { return }

    switch CLLocationManager.authorizationStatus() {
    case .notDetermined:
        locationManager.requestWhenInUseAuthorization()
    case .authorizedAlways, .authorizedWhenInUse:
        locationManager.requestLocation()
    default:
        break
    }
}

We’re going to store the location in userLocation when we find it, so that we can use it in the station selection screen.

⚠️ One warning here - I initially added askForLocationIfNeeded() to didAppear so that we only ask for location once the UI appears, and I was expecting didAppear to always be called following willActivate - but it doesn’t seem to work this way. From my testing right now, it seems that didAppear is only called when the app is launched and when you return to the interface controller from the pushed view, but not when the app is closed and reopened. If you add something to one of these two methods, make sure you test exactly in which cases they get called.

Next, if the user grants us location access after the launch, we ask for location data then:

func locationManager(
  _ manager: CLLocationManager,
  didChangeAuthorization status: CLAuthorizationStatus)
{
    switch CLLocationManager.authorizationStatus() {
    case .authorizedAlways, .authorizedWhenInUse:
        locationManager.requestLocation()
    default:
        break
    }
}

We only ask for a single location, we don’t need to track it continuously. And we can also set desiredAccuracy to kCLLocationAccuracyHundredMeters when setting up the CLLocationManager - we won’t need more precision than that, and we should get the location much faster this way.

When we get the location, we save it in the property userLocation mentioned earlier (we also need to handle an error case - you actually get an exception immediately if you don’t). Also, most importantly, if there is no station selected yet, but we have the user location, we can preselect the closest one automatically - that way, the user will see some data almost immediately, without having to configure the app first, and it’s very likely it will be exactly the data that they want 👍

func locationManager(_ manager: CLLocationManager, didUpdateLocations locations: [CLLocation]) {
    guard let currentLocation = locations.last else { return }

    userLocation = currentLocation

    if dataStore.selectedChannelId == nil {
        let closestStation = stationsSortedByDistance(from: currentLocation).first!
        setSelectedStation(closestStation)
    }
}

func locationManager(_ manager: CLLocationManager, didFailWithError error: Error) {
    NSLog("CLLocationManager error: %@", "\(error)")
}

func stationsSortedByDistance(from userLocation: CLLocation) -> [Station] {
    return dataStore.stations.sorted { (s1, s2) -> Bool in
        let d1 = CLLocation(latitude: s1.lat, longitude: s1.lng).distance(from: userLocation)
        let d2 = CLLocation(latitude: s2.lat, longitude: s2.lng).distance(from: userLocation)

        return d1 < d2
    }
}

We can then pass the saved location to the stations list, where we’ll use it to show the distance to each station, and we can also pass it a list of locations sorted by location:

override func contextForSegue(withIdentifier segueIdentifier: String) -> Any? {
    if segueIdentifier == "ChooseStation" {
        let stations: [Station]

        if let currentLocation = userLocation {
            stations = stationsSortedByDistance(from: currentLocation)
        } else {
            stations = dataStore.stations
        }

        return StationListContext(
            items: stations,
            selectedId: dataStore.selectedChannelId,
            userLocation: userLocation,
            onSelect: { station in
                self.setSelectedStation(station)
            }
        )
    }

    return nil
}

Add a userLocation: CLLocation? property to the SelectionListContext, from which we’ll read it in the controller’s initializer.

Showing distances in the list

Let’s look back on our storyboard again. We want to have a second label below the station title that shows the distance to the station - that is, if we know user’s location, otherwise we show the old version.

It’s possible that we could somehow make this work with a single cell type, but I figured that a much easier way would be to have two different cells managed by the same class. So make a duplicate of our BasicListRow, give it an identifier ListRowWithDistance and keep the same class name.

In order to have 3 elements in the cell positioned correctly, we’re going to need two groups: one horizontal, dividing the checkmark on the right from the two labels on the left, and then an inner vertical group that arranges the two labels. So change the cell this way:

Wrap the left label in a Vertical group.
Add a second label inside that inner group. Make sure it’s below the first one (you can set their vertical alignment, or you can just make sure they’re in the right order in the view tree).
The table row’s main group has a fixed “Default” height configured when created - but with two labels, this default height is too little. So change the height setting to Size to fit.
Give the lower label a Light Gray color and a Footnote font. Make it say e.g. “3.2 km”, and assign it to an outlet in the StationListRow:

@IBOutlet weak var distanceLabel: WKInterfaceLabel!

Give the vertical group Insets of 3 at the top and bottom, and change its Width setting to “Size to fit content” - otherwise it will take whole cell width by default and push the checkmark out.
Customize the outer (horizontal) group’s Spacing to 0; we can allow less space between the checkmark and the edge of the title label now, because the checkmark will be positioned in the vertical center, so slightly lower than the label.
The checkmark’s properties should stay as before.

We can then add another helper method to StationListRow to set the distance. We’re going to use MeasurementFormatter here in order to automatically display kilometers or miles, and we’ll also make sure to only print 1 decimal digit, since we asked for a slightly less precise location (the default is something like “2.456 km”):

let measurementFormatter: MeasurementFormatter = {
    let numberFormatter = NumberFormatter()
    numberFormatter.maximumFractionDigits = 1

    let measurementFormatter = MeasurementFormatter()
    measurementFormatter.numberFormatter = numberFormatter
    return measurementFormatter
}()

func setDistance(_ distance: Double) {
    let text = measurementFormatter.string(
        from: Measurement(value: distance, unit: UnitLength.meters)
    )
    distanceLabel.setText(text)
}

And now in the awake(withContext:) method in StationListController we can choose between the two types of cells depending on whether we have the location or not, and if we do, calculate the distance to each station and show it in the lower label:

let rowType = (context.userLocation == nil) ? "BasicListRow" : "ListRowWithDistance"
table.setNumberOfRows(items.count, withRowType: rowType)

for i in 0..<items.count {
    let row = listRowController(at: i)
    row.showStation(items[i])

    if let location = context.userLocation {
        let itemLocation = CLLocation(latitude: items[i].lat, longitude: items[i].lng)
        row.setDistance(location.distance(from: itemLocation))
    }
}

You should now see something like this:

We’ve reached the end of this tutorial. For the next episode, I will try to rewrite the whole UI again from scratch in SwiftUI and compare how much effort it requires to build the same kind of UI in the new framework :)

The final version of the code after a completed tutorial is available on a branch here, and the slightly different real version on the master would be more or less at this commit.