<![CDATA[Misleading Metaphors]]>https://ayewo.com/https://ayewo.com/favicon.pngMisleading Metaphorshttps://ayewo.com/Ghost 5.53Wed, 21 Feb 2024 19:20:21 GMT60<![CDATA[Programmatic Creation of the Ghost Admin User]]>Problem

You want to completely automate the installation of Ghost including creation of the admin user?

You search on Google and land on this exactly worded forum post from 2019 with no ready-to-use answer.

Solution

You can do it using curl:

curl 'http://localhost:2368/ghost/api/admin/authentication/
]]>https://ayewo.com/programmatic-creation-of-the-ghost-admin-user/64a33e7c3b652d31711a981aMon, 03 Jul 2023 21:35:05 GMTProblem

You want to completely automate the installation of Ghost including creation of the admin user?

You search on Google and land on this exactly worded forum post from 2019 with no ready-to-use answer.

Solution

You can do it using curl:

curl 'http://localhost:2368/ghost/api/admin/authentication/setup/' \
-X 'POST' \
-H 'Content-Type: application/json; charset=UTF-8' \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'Host: localhost:2368' \
-H 'Origin: http://localhost:2368' \
-H 'Referer: http://localhost:2368/ghost/' \
-H 'Connection: keep-alive' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'App-Pragma: no-cache' \
--data-binary '{"setup":[{"name":"Admin","email":"email@ghost.example.com","password":"<ghost-admin-password>","blogTitle":"Blog Title"}]}'

You only need to change the following values in the payload:

  • email;
  • password,
  • blogTitle.

This solution was used in a [:larger article] so you may want to read it as well.

PostScript

The Ghost CTO did offer the following suggestion, 2 years later:

You can automate booting Ghost to initialise the DB and then calling the setup API endpoint to finalise setup so that the window of opportunity is a few ms.

I did all of my research on a server with a public IP, and since it would take me more than a few ms before I could figure out how to correctly create the admin user programmatically, I had to take some precautions.

Rather than expose the unfinished Ghost Admin setup page over the public Internet, I decided to leave it accessible only from localhost and used SSH tunneling to access it from my machine.

To figure out the right incantations to correctly create an admin user from the CLI using curl, I installed a development instance of Ghost:

su - ghost-mgr
sudo mkdir -p /var/www/local && cd /var/www/local
sudo chown -R ghost-mgr:ghost-mgr /var/www/local
ghost install local

This will create an entry in ~/.ghost/config:

{
  "instances": {
    "ghost1-ayewo-com": {
      "cwd": "/var/www/ghost"
    },
    "ghost-local": {
      "cwd": "/var/www/local"
    }
  }
}

Each time you use the ghost-cli to interact with one or more instances of Ghost on your machine, it will update its entry inside /home/ghost-mgr/.ghost/config:

ghost ls
┌──────────────────┬────────────────┬─────────┬─────────┬─────┬──────┬─────────────────┐
│ Name             │ Location       │ Version │ Status  │ URL │ Port │ Process Manager │
├──────────────────┼────────────────┼─────────┼─────────┼─────┼──────┼─────────────────┤
│ ghost1-ayewo-com │ /var/www/ghost │ 5.46.1  │ stopped │ n/a │ n/a  │ n/a             │
├──────────────────┼────────────────┼─────────┼─────────┼─────┼──────┼─────────────────┤
│ ghost-local      │ /var/www/local │ 5.51.2  │ stopped │ n/a │ n/a  │ n/a             │
└──────────────────┴────────────────┴─────────┴─────────┴─────┴──────┴─────────────────┘

After figuring out the right curl incantations, I tested it against the development instance of Ghost and quickly encountered a bug where Ghost couldn't write to the SQLite database due to some file permissions issue.

So I tried this quick fix since Ghost in production requires this folder to be owned by a non-privileged user ghost (not ghost-mgr):

sudo chown -R ghost:ghost /var/www/local/content/

Ultimately I reverted it because the development instance of Ghost complained:

✖ Starting Ghost: ghost-local
A SystemError occurred.

Message: The content folder is not owned by the current user.
Ensure the content folder has correct permissions and try again.

The error went away after a ghost stop/ghost start cycle (I think?), so a workaround might be to simply restart Ghost immediately after installation of the development instance i.e.:

# after you install a development instance with
ghost install local

# restart it before continuing
ghost restart ghost-local

Your admin interface is located at:

    http://localhost:2368/ghost/

Note: The presence of this development instance will always cause Ghost to assume any attempts to run ghost start inside /var/www/ghost (the production instance path) or even invoking the name of the production instance explicitly (e.g. cd /var/www/ghost && ghost start ghost1-ayewo-com), is an attempt at running the development instance installed at /var/www/local.

Each time I tried to start the production instance while the development instance was still installed, this terrible assumption caused Ghost to output an inscrutable error even when the JSON at /var/www/ghost/config.production.json is perfectly valid:

1) Validating config

Error detected in the development configuration.

Message: Config file is not valid JSON

It always prefaced its outputs with "Running in development mode" even when the config file inside /var/www/ghost is explicitly named config.production.json and not config.development.json.

The fix was to relocate the development instance folder /var/www/local to another location where the ghost-cli cannot find it before continuing.

]]><![CDATA[How to Host a New Ghost Blog on AWS]]>https://ayewo.com/how-to-host-a-new-ghost-blog-on-aws/64a2981b3b652d31711a9611Mon, 03 Jul 2023 17:53:37 GMTThe instructions in this article is a chimera between the documentation for installing Ghost on Ubuntu and the Ghost 1-Click app on the DigitalOcean Marketplace.

These instructions have been made into an open source project called ghost.sh.

The project will convert all the steps into an easy-to-use 1-Click script that can be used to set up a brand new Ghost blog on any cloud provider in just a few minutes. I also wrote an announcement post that delves into my [:motivation behind] the project.

1. EC2

Create a t2.micro instance either using the AWS CLI or the AWS Console.

  • Pick an Ubuntu AMI. I used an Ubuntu 22.04 server for my EC2 instance based on this AMI: ami-09744628bed84e434.
  • Enable Elastic IP when setting up your instance. This allows you to hold on to the same public IP (between reboots) for as long as you need.
  • In the settings for your Security Group (SG), be sure to open ports 80 (http), 443 (https) and 22 (ssh).

Ghost can automate the request for a Let's Encrypt certificate to secure your HTTP traffic but the domain verification process can only be done over HTTP which is why you must have port 80 open in your SG.

Note
I originally started with a t2.micro EC2 instance but noticed that ghost install was quite slow. It took up to 40 mins to download ~450MB on the instance so I restarted the process on a t2.small instance. Also, for some reason, once ghost setup is complete, the installation ballooned in size to ~880MB but the issue appears to have went away with the switch to a t2.small.

2. DNS

2.1. Login to your domain registrar and associate your Elastic IP with your domain. I use namecheap.com so from the "Advanced DNS" page for my domain, I added an A Record that mapped the Elastic IP of the t2.small instance to ghost.example.com and used a TTL of 1 minute so the changes can be picked up quickly.

While I wait for the DNS changes to propagate, I'll append an entry to my local /etc/hosts so that ghost.example.com is resolved immediately by my machine. (Don't worry, we will check that nslookup ghost.example.com works correctly later.)

2.2. Open Terminal tab, then run: sudo vim /etc/hosts to append a temporary mapping for the Elastic IP to your domain.

18.1.1.1  ghost.example.com

2.3. Next, login to the EC2 instance using the domain name (you could also use the Elastic IP directly).

# Unlike DigitalOcean which uses 'root@' for remote SSH, AWS AMIs use 'ubuntu@' 
ssh -v -i ~/.ssh/ec2-keypair.pem ubuntu@ghost.example.com

Welcome to Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-1031-aws x86_64)

  ...
  System information as of Mon May  1 17:26:05 UTC 2023

  System load:  0.0               Processes:             95
  Usage of /:   5.3% of 28.89GB   Users logged in:       0
  Memory usage: 21%               IPv4 address for eth0: 172.31.17.58
  Swap usage:   0%

...
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

2.4. While logged into the EC2 instance, confirm that the DNS changes have now propagated:

nslookup ghost.example.com
Server:   127.0.0.53
Address:  127.0.0.53#53

Non-authoritative answer:
Name: ghost.example.com
Address: 18.1.1.1

3. Ghost

These steps are based directly on the instructions at How to install Ghost on Ubuntu.

3.1. Create a sudoer and Update the Server

# Launch the 'screen' command with logging in case you are working from a flaky connection like me.
cd /tmp && screen -L -S a01 -Logfile a01.log  # -Logfile <log-file> only works in screen v4.06.02+
cd /tmp && echo "logfile a01.log" >> a01.rc && screen -c /tmp/a01.rc -L -S a01  # hack for screen v4.03.01 & older 

# Ghost UNIX account setup
# Create a new user and follow prompts
sudo adduser ghost-mgr

# Add user to superuser group to unlock admin privileges
sudo usermod -aG sudo ghost-mgr

# Remove the password you picked in 'adduser'
sudo passwd -d ghost-mgr

# Then log in as the new user
su - ghost-mgr

# You can also use
# sudo -i -u ghost-mgr

# Update the server
# Update package lists
sudo apt-get update

# Update installed packages
sudo apt-get upgrade
Note regarding sudo apt-get upgrade Towards the end, the command produced the following output: 
Restarting services...
 systemctl restart multipathd.service packagekit.service polkit.service rsyslog.service ssh.service
Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart networkd-dispatcher.service
 systemctl restart systemd-logind.service
 systemctl restart unattended-upgrades.service
 systemctl restart user@1000.service

No containers need to be restarted.


No user sessions are running outdated binaries.


No VM guests are running outdated hypervisor (qemu) binaries on this host.

I tried again and the failures persisted.

Restarting services...
 /etc/needrestart/restart.d/dbus.service
 systemctl restart networkd-dispatcher.service systemd-logind.service unattended-upgrades.service user@1000.service
Job for user@1000.service failed because the control process exited with error code.
See "systemctl status user@1000.service" and "journalctl -xeu user@1000.service" for details.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

The fix for the issues above was to simply reboot the EC2 instance: sudo reboot

3.2. Auxilliary Tools Setup

# Install net-tools (for later use of netcat)
sudo apt install net-tools -y

# Install zip (for later use of zip)
sudo apt install zip -y

# Install tree (for later use)
sudo apt install tree -y

# Install dependencies
# Install NGINX
sudo apt install nginx -y
-> nginx is already the newest version (1.18.0-6ubuntu14.3).

# Check installed version
nginx -v
-> nginx version: nginx/1.18.0 (Ubuntu)

# Check listening ports
sudo netstat -pant | grep nginx
-> tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      498/nginx: master p
-> tcp6       0      0 :::80                   :::*                    LISTEN      498/nginx: master p

# Install certbot
sudo apt install certbot python3-certbot -y

# Check installed version
certbot --version
certbot 1.21.0

# View the firewall list
sudo ufw app list
Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH

# Allow HTTP and HTTPS connections for nginx
sudo ufw allow 'Nginx Full'
-> Rules updated
-> Rules updated (v6)

# Check the firewall status
sudo ufw status
Status: inactive

# Enable it if inactive
sudo ufw enable

# Check that it has been enabled
sudo ufw status verbose

# Install MySQL
sudo apt-get install mysql-server -y

# Check installed version
mysql --version
-> mysql  Ver 8.0.32-0ubuntu0.22.04.2 for Linux on x86_64 ((Ubuntu))

# Check listening ports
sudo netstat -pant | grep mysqld
-> tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1583/mysqld
-> tcp        0      0 127.0.0.1:33060         0.0.0.0:*               LISTEN      1583/mysqld

# Add the NodeSource APT repository for Node 16
curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash

# Install Node.js
sudo apt-get install nodejs -y 

# Check installed version
node --version && npm --version
-> v16.20.0
-> 8.19.4

# List currently installed global 'npm' packages
npm list -g
/usr/lib
├── corepack@0.17.0
└── npm@8.19.4

# Install Ghost-CLI globally
sudo npm install ghost-cli@latest -g

# Check installed version
ghost --version
-> Ghost-CLI version: 1.24.0

# Confirm the package was globally installed
npm list -g
/usr/lib
├── corepack@0.17.0
├── ghost-cli@1.24.0
└── npm@8.19.4
Note regardingsudo npm install ghost-cli@latest -g 
# Install Ghost-CLI globally
sudo npm install ghost-cli@latest -g

The sudo prefix is necessary because npm will need write access to the system-wide node_modules/ path which is only writable by users with sudo privileges as can be seen below.

# the system-wide `node_modules/` path can be found using `npm root -g`
npm root -g 
-> /usr/lib/node_modules

# this path is only writable by root (and users with `sudo` privileges)
ls -l /usr/lib/node_modules/
-> drwxr-xr-x 4 root root 4096 May  2 19:28 corepack
-> drwxr-xr-x 7 root root 4096 May  2 19:28 npm

3.3. Install Ghost

# Create Ghost install path: 
sudo mkdir -p /var/www/ghost

# The folder is still owned by 'root'
ll /var/www/ghost/
-> drwxr-xr-x 2 root root 4096 May  2 20:24 ./
-> drwxr-xr-x 4 root root 4096 May  2 20:24 ../

# Change the owner to our sudoer 'ghost-mgr'
sudo chown ghost-mgr:ghost-mgr /var/www/ghost

# Set the correct permissions
sudo chmod 775 /var/www/ghost

# Confirm the folder permissions were updated
ll /var/www/ghost/
-> drwxrwxr-x 2 ghost-mgr ghost-mgr 4096 May  2 20:24 ./
-> drwxr-xr-x 4 root      root      4096 May  2 20:24 ../

# Change to the folder
cd /var/www/ghost

# Install Ghost with one final, non-interactive command
ghost install --no-setup
->
✔ Checking system Node.js version - found v16.20.0
✔ Checking current folder permissions
✔ Checking memory availability
✔ Checking free space
✔ Checking for latest Ghost version
✔ Setting up install directory
✔ Downloading and installing Ghost v5.46.1
✔ Finishing install process

3.4 MySQL Setup

3.4.1. We will use the following configuration with the MySQL database server

Name Value
DB host localhost
DB user ghost_dba
DB name ghost_production
DB pswd <ghost-mysql-password>

3.4.2. Create a MySQL DBA account specifically for Ghost
The MySQL DBA account root has no password set, so create a DBA user exclusively for Ghost

mkdir -p /tmp/ghost/mysql/ && cd /tmp/ghost/mysql/
cat <<EOF > dba.sql
CREATE USER 'ghost_dba'@'localhost' IDENTIFIED WITH mysql_native_password BY '<ghost-mysql-password>';  
GRANT ALL PRIVILEGES ON *.* TO 'ghost_dba'@'localhost';
FLUSH PRIVILEGES;
quit
EOF

sudo mysql < /tmp/ghost/mysql/dba.sql > /tmp/ghost/mysql/dba.out

3.5. Ghost Config

# Change back to the Ghost install path
cd /var/www/ghost

# The docs is wrong on usage of the '--log' parameter. It says --log '["stdout", "file"]' but what works in v5.x is below
ghost config --url https://ghost.example.com --ip 127.0.0.1 --port 2368 --log "stdout" --log "file" --db mysql --dbhost localhost --dbname ghost_production --dbuser ghost_dba --dbpass <ghost-mysql-password>

# We will temporarily use 'localhost' instead of 'ghost.example.com' to avoid exposing '/ghost/' admin page on the Internet
ghost config --url http://localhost:2368 --ip 127.0.0.1 --port 2368 --log "stdout" --log "file" --db mysql --dbhost localhost --dbname ghost_production --dbuser ghost --dbpass <ghost-mysql-password>


# This will create a `config.production.json` in /var/www/ghost which you can review
cat /var/www/ghost/config.production.json
{
  "url": "http://localhost",
  "server": {
    "port": 2368,
    "host": "127.0.0.1"
  },
  "database": {
    "client": "mysql",
    "connection": {
      "host": "localhost",
      "user": "ghost_dba",
      "password": "<ghost-mysql-password>",
      "database": "ghost_production"
    }
  },
  "mail": {
    "transport": "Direct"
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/www/ghost/content"
  }
}

Note
specifying ghost config --url http://127.0.0.1:2368 ... will create a config.development.json instead of config.production.json which is why I had to use the production URL https://ghost.example.com in ghost config.

3.6. Ghost Setup

3.6.1. Finish setting up Ghost non-interactively:

# setup (will print the ghost admin URL to the console)
ghost setup --no-prompt --no-setup-mysql -V


# check the service is up
sudo netstat -pant | grep node
-> tcp        0      0 127.0.0.1:2368          0.0.0.0:*               LISTEN      2762/node
-> tcp        0      0 127.0.0.1:32904         127.0.0.1:3306          ESTABLISHED 2762/node
-> tcp        0      0 127.0.0.1:32890         127.0.0.1:3306          ESTABLISHED 2762/node

There are two ways by which you can complete the Ghost admin user setup process:

  • remotely from your browser using SSH tunneling or;
  • locally from the CLI using curl[1].

3.6.2. Open another Terminal tab and connect to Ghost on localhost using SSH tunneling:

ssh -i ~/.ssh/ec2-keypair.pem -v -L 2368:localhost:2368 ubuntu@ghost.example.com

Next visit localhost:2368 on your browser to complete the setup process:

open http://localhost:2368/ghost/#/setup

3.6.3. Or, use the following curl command to achieve the same thing from the CLI:

curl 'http://localhost:2368/ghost/api/admin/authentication/setup/' \
-X 'POST' \
-H 'Content-Type: application/json; charset=UTF-8' \
-H 'Accept: application/json, text/javascript, */*; q=0.01' \
-H 'Host: localhost:2368' \
-H 'Origin: http://localhost:2368' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15' \
-H 'Referer: http://localhost:2368/ghost/' \
-H 'Connection: keep-alive' \
-H 'X-Requested-With: XMLHttpRequest' \
-H 'App-Pragma: no-cache' \
--data-binary '{"setup":[{"name":"Admin","email":"email@ghost.example.com","password":"<ghost-admin-password>","blogTitle":"Blog Title"}]}'

{"users":[{"id":"1","name":"Admin","slug":"admin","email":"email@ghost.example.com","profile_image":null,"cover_image":null,"bio":null,"website":null,"location":null,"facebook":null,"twitter":null,"accessibility":null,"status":"active","meta_title":null,"meta_description":null,"tour":null,"last_seen":null,"comment_notifications":true,"free_member_signup_notification":true,"paid_subscription_started_notification":true,"paid_subscription_canceled_notification":false,"mention_notifications":true,"milestone_notifications":true,"created_at":"2023-06-14T15:49:42.000Z","updated_at":"2023-06-17T07:58:16.000Z","url":"https://ghost.example.com/author/admin/"}]}

Remember to change the email, password, blogTitle in the curl command accordingly to match your desired values.

3.6.4. Now that an admin user has been set, switch to a production URL

# First stop Ghost
ghost stop

# Update the production URL
ghost config --url https://ghost.example.com

# Your config should now look identical to the following output
cat /var/www/ghost/config.production.json
{
  "url": "https://ghost.example.com",
  "server": {
    "port": 2368,
    "host": "127.0.0.1"
  },
  "database": {
    "client": "mysql",
    "connection": {
      "host": "localhost",
      "user": "ghost_dba",
      "password": "<ghost-mysql-password>",
      "database": "ghost_production"
    }
  },
  "mail": {
    "transport": "Direct"
  },
  "logging": {
    "transports": [
      "file",
      "stdout"
    ]
  },
  "process": "systemd",
  "paths": {
    "contentPath": "/var/www/ghost/content"
  }
}

3.6.5. Start Ghost

ghost start

+ sudo systemctl is-active ghost_localhost
✔ Checking system Node.js version - found v16.20.0
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
+ sudo systemctl is-active ghost_localhost
✔ Validating config
✔ Checking folder permissions
✔ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking binary dependencies
✔ Checking systemd unit file
✔ Checking systemd node version - found v16.20.0
+ sudo systemctl start ghost_localhost
+ sudo systemctl is-enabled ghost_localhost
✔ Starting Ghost: localhost

------------------------------------------------------------------------------

Your admin interface is located at:

    https://ghost.example.com/ghost/

# check the service is up
sudo netstat -pant | grep node

3.6.6. View your brand new Ghost blog

open https://ghost.example.com/ghost/

4. Tidy Up

4.1. Clean up on the server

# check disk usage 
cd /home/ghost-mgr/ && du -shL .cache/
-> 819M  .cache/

# free up some space
ghost buster

# remove scratch folder
rm -rf /tmp/ghost

4.2. Clean up locally

# Remove the temporary mapping of the Elastic IP
sudo vim /etc/hosts

# check that the domain is now being resolved correctly
nslookup ghost.example.com

  1. For almost 4 years, no definitive answer has been given to the question on how to create the admin user programmatically so I knew I had to figure it out so I could completely automate the process of launching a Ghost blog. I show how to [:create the admin programmatically using curl] in a separate post. Compared to WordPress, Ghost's DevOps story isn't particularly great and I'm hoping ghost.sh will slowly help fix that. ↩︎

]]><![CDATA[Ghost.sh]]>

This article is the umbrella article for a 3-part series on how move (and upgrade) a live Ghost blog hosted on a DigitalOcean droplet to an EC2 VM on AWS, with minimal downtime.

Other articles planned/published are:

]]>https://ayewo.com/ghost-sh/64a486403b652d31711a987aSat, 01 Jul 2023 10:50:00 GMT

This article is the umbrella article for a 3-part series on how move (and upgrade) a live Ghost blog hosted on a DigitalOcean droplet to an EC2 VM on AWS, with minimal downtime.

Other articles planned/published are:

Before I dive in, I'd like to go on a small detour on what motivated this project.

Ghost's Adoption is Growing but There's Still Friction

Ghost is now at version 5.x and their documentation story has improved, but unfortunately, their docs are still lacking on real world information on self-hosting like: how to migrate between hosts, or the 1-Click launch of a new blog, from start to finish, on any cloud provider.

Information on how to do any of those things is scattered across forum posts, GitHub and random blogs. I'm hoping my publication of this information in one place with save you all the hair-pulling I had to endure to get this working.

(The blog you are currently reading was simultaneously upgraded from Ghost v3.x to v5.x and moved from DigitalOcean to AWS.)

They Select for Folks with Ability to Pay

Ghost is a COSS (Commercial Open Source Software) project—it is completely open source and free to use, and if you choose to self-host it, the amount of friction you will face is no accident.

The project's way of making money is by offering a paid service called Ghost(Pro) designed to remove friction for both blogging first-timers and blogging veterans looking to switch to Ghost.

Ghost(Pro) is essentially a DFY (Done For You) service that handles blog migration and blog hosting for 4 different personas, which is why there are 4 different price points. (Pre-pandemic there were only 3 price points.)

From the outside, it seems Ghost is squarely focused on attracting users with a high likelihood of converting into paying customers in the future, albeit to the detriment of others. This is a reasonable stance for a small team.

But, this stance has a glaring casualty: it limits broader adoption of Ghost as an OSS project.

Highly technical users i.e. technical end users with more time on their hands than money (hi! 👋, that's me 😁) who adopt Ghost and self-host it end up paying with a huge chunk of their time[1][2] because the project is not (yet) interested in eliminating friction for self-hosters since they don't contribute directly to their bottomline. Perhaps they hope the community will rise up to fill the gap?

They Reduce Friction for First-Timers

To their credit, the Ghost team maintains a 1-Click app on the DigitalOcean marketplace for first-timers looking for a quick and easy way to self-host Ghost.

In fact, this is how I got started with Ghost in 2020, and I've been on this setup for 2+ years.

They Ignore Friction for Veterans

The 1-Click app caters quite well to the needs of first-time users seeking to launch a brand new blog. But, it is woefully inadequate if you are not a first-timer:

  • the 1-Click app or ghost-cli are often not enough[3] to upgrade an existing blog;
  • the 1-Click app itself is not available outside of the DigitalOcean marketplace and thus cannot be used on other cloud providers like AWS, Azure or GCP;
  • the 1-Click app does not have support for importing or migrating an existing blog to DigitalOcean, or indeed to any of the other cloud providers.

Regarding the first point, I'll share two instances from my experience.

There's a lot of Housekeeping to keep Ghost up-to-date

First Example: Things get out of Sync Fairly Quickly

Back in December 2020, I used the 1-Click app to stand up the latest version of Ghost at the time (v3.38.3) to host https://ayewo.com/ on a $5/mo droplet on DigitalOcean.

By November 2021, Ghost was now on v4.x.x. When I tried to perform an upgrade from v3.38.3 to v4.x.x, I was met with this inscrutable error (at the time):

ghost update

You are running an outdated version of Ghost-CLI.
It is recommended that you upgrade before continuing.
Run `npm install -g ghost-cli@latest` to upgrade.

+ sudo systemctl is-active ghost_ayewo-com
✔ Checking system Node.js version
✔ Ensuring user is not logged in as ghost user
✔ Checking if logged in user is directory owner
✔ Checking current folder permissions
✔ Checking folder permissions
✖ Checking file permissions
✔ Checking content folder ownership
✔ Checking memory availability
✔ Checking free space
One or more errors occurred.

1) Checking file permissions

Message: Your installation folder contains some directories or files with incorrect permissions:
- ./content/themes/Alto-master/assets/js/lib/owl.carousel.min.js
- ./content/themes/Alto-master/assets/js/lib/jquery.slicknav.min.js
- ./content/themes/Alto-master/assets/js/lib/jquery.fitvids.js
- ./content/themes/Alto-master/assets/fonts/Alto.woff
- ./content/themes/Alto-master/assets/fonts/selection.json
- ./content/themes/Alto-master/assets/fonts/Alto.svg
- ./content/themes/Alto-master/assets/fonts/Alto.ttf
Run sudo find ./ ! -path "./versions/*" -type f -exec chmod 664 {} \; and try again.

Debug Information:
    OS: Ubuntu, v18.04.4 LTS
    Node Version: v12.18.0
    Ghost Version: 3.38.3
    Ghost-CLI Version: 1.15.2
    Environment: production
    Command: 'ghost update'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/api/ghost-cli/ for troubleshooting.

Of course the error was due to a file permission issue as hinted in the middle of the output, but realizing this required familiarity with Ghost's deployment that I didn't yet have.

(The file permissions issue must have arisen silently when I tried out the Alto theme a while back. I uploaded it from the Ghost Admin page, applied it site-wide but didn't like how it looked, so I switched back to the default theme (Casper) and promptly forgot to delete its files from my server.)

Second Example: Breaking Changes

Ghost v5.x introduced a breaking change where support for MySQL versions below v8.x was dropped. The official docs on how to do the migration had suggestions that boiled down to taking the server offline to do an OS upgrade before applying some essential DDL changes.

I didn't bother with the official docs due to the uncertainty involved in an upgrade to the latest Ubuntu Linux version. How much downtime am I looking at? Plus the follow-up DDL changes looked like open heart surgery, if I got it wrong.

They did helpfully link to an alternative approach that minimized server downtime but I couldn't use it because my goal was to upgrade Ghost and move hosts to a different cloud provider.

This second example is a mixed bag. The Ghost project is slowly maturing which is good, but they could do a better job of making breaking changes easy to deal with.

Pets vs Cattle

The instructions certainly got me thinking: if we flipped our thinking by treating the server used for the deployment like cattle instead of pets, the whole migration experience could be:

  • simplified and;
  • automated.

Reducing Friction for First-Timers & Veterans

The ghost.sh[4] project is intended to plug those gaps: collect the most common step-by-step instructions necessary to launch a Ghost blog (whether brand new or existing) and turn those instructions into a 1-Click script that can be used on any cloud provider.

The project is open source: ghost.sh.

  1. For instance, see this recent HN comment thread where the following frustrations were shared: "As someone who's manually setup Ghost blog, I'd heavily advise towards just paying for Ghost hosting. I wasted hours setting it up, and I gave up when it came time to configure emailing." ↩︎

  2. Another comment from the same thread: "I had a heck of a time setting it up to self host myself and I am technically inclined. There was one thing misconfigured by default that took me a few hours to catch. Now I’m trying to figure out why the VM stops responding randomly…free sounded better to me at the time" ↩︎

  3. There are some neat aspects of the ghost-cli that I learned of while working on this project, but overall, I think there's still room for more community participation. WordPress has had automatic background updates since v3.7 (released in 2013) and there are a handful of community efforts to automate the installation of WordPress (e.g. AbhishekGhosh/Ubuntu-18.04-WordPress-Autoinstall-Bash-Script, in addition multiple hosting providers offering 1-Click installation of WordPress. The few automation-focused community projects for Ghost that I came across were either archived (e.g. systemd-ghost), abandoned (e.g. ghost-on-heroku), or too brittle for production use (e.g. ghost-on-github-pages ) or requires quite a few steps to get started (e.g. ethibox/awesome-stacks). Although ghost-on-aws doesn't provide code and has not been updated in 3 years, I still found the high-level instructions to be useful as a checklist when planning to deploy Ghost to AWS. ↩︎

  4. Originally named the project Ghost(Ultra) since Ultra is the next step (hypothetically) after Ghost, Ghost(Pro), Ghost(Max). After writing out a dozen names, landed on the much neater Ghost.sh. The lesson is to write out a dozen names before settling on a name. ↩︎

]]><![CDATA[Make sudo Spiffy on macOS with TouchID]]>About five months ago, I followed the steps in this article: Use TouchID to Authenticate sudo on macOS via HN so that I could use my fingerprint to complete actions that require sudo privileges.

Other than using TouchID to login at the start of my day, I rarely use the

]]>https://ayewo.com/sudo-on-macos-with-touchid/6499621cf941aa2920349d22Sat, 21 Jan 2023 15:55:48 GMT

About five months ago, I followed the steps in this article: Use TouchID to Authenticate sudo on macOS via HN so that I could use my fingerprint to complete actions that require sudo privileges.

Other than using TouchID to login at the start of my day, I rarely use the TouchID button on my MacBook Pro for much else, so I liked the added convenience this feature offered.

I have been putting off updating to macOS Ventura for several weeks now, but I made a mental note: whenever I do succumb to doing an major OS update, I would re-apply my TouchID changes because such changes do not survive a major OS update.

Fast-forward to today and I only just realized that my changes had been overwritten. At the time I enabled TouchID with sudo on August 27, 2022, I was on macOS Monterey 12.3.1. About a month later, on September 26, I updated to Monterey 12.6.

It turns out that, even for a minor OS update, those changes will be overwritten by macOS. I had incorrectly assumed such changes will only be overwritten by a major update i.e. moving from Big Sur to Monterey or Monterey to Ventura.

I'm still trying to come up with a safe and repeatable way to automatically apply these changes after an OS update, but for the time being, I'm simply going to spell out all the steps in this article so I can quickly copy-paste just the bits that I need for my future self.

Step 1: view the contents of /etc/pam.d/sudo:

cat /etc/pam.d/sudo
# sudo: auth account password session
auth       sufficient     pam_smartcard.so
auth       required       pam_opendirectory.so
account    required       pam_permit.so
password   required       pam_deny.so
session    required       pam_permit.so

Step 2: make a backup of /etc/pam.d/sudo so any typos can easily be reverted (ask me how I know 🤦‍♂️). Note that macOS wont allow you write to /etc/ so choose a location that is writable like /Users/<account> :

cp /etc/pam.d/sudo ~/etc-pam.d-sudo

Step 3: add the following line: auth sufficient pam_tid.so directly below the first line auth sufficient pam_smartcard.so inside the file /etc/pam.d/sudo:

# even though it's a small edit, doing it manually is error-prone
sudo vim /etc/pam.d/sudo

or, if you are lazy like me, use this one-liner instead:

# this one-liner is much safer
sudo perl -pi -e 's/(pam_smartcard.so)/$1\nauth       sufficient     pam_tid.so/' /etc/pam.d/sudo

Step 4: review the updated file /etc/pam.d/sudo to confirm your changes:

cat /etc/pam.d/sudo
# sudo: auth account password session
auth       sufficient     pam_smartcard.so
auth       sufficient     pam_tid.so
auth       required       pam_opendirectory.so
account    required       pam_permit.so
password   required       pam_deny.so
session    required       pam_permit.so

Done!

]]><![CDATA[An Obscure Error While Downloading a YouTube Video for Offline Viewing]]>https://ayewo.com/how-to-fix-a-failing-offline-download-using-youtube-dl/6499621cf941aa2920349d21Thu, 22 Dec 2022 21:56:30 GMT

Lenny Rachitsky recently did a lengthy podcast interview with Ethan Smith who runs Graphite, an SEO agency.

The video is more than 1:30 hrs long so I decided I was going to watch it offline. Several attempts to download the video from YouTube failed but I managed to find a workaround.

The Download Failed on macOS and Ubuntu

My first attempt at downloading the video on macOS using youtube-dl led to the following error:

[download]   0.4% of 645.08MiB at 51.43KiB/s ETA 03:33:17[download] Got server HTTP error: Downloaded 2451832 bytes, expected 676416237 bytes. Retrying (attempt 1 of 10)...

I was too impatient to try to figure out why the download failed, so I SSHed into a spare Ubuntu Linux VPS I had running on AWS and installed the youtube-dl binary. This second attempt also failed. The error message was identical to the error I was shown on macOS.

A quick check revealed that I had version v2021.12.17 of youtube-dl installed on both macOS and Ubuntu.

At the time of my attempt, the current date was 2022.12.17 which coincided with the month and day (but not the year) of release for the version I had installed (v2021.12.17). In other words, the only difference was the year of release (2021 vs 2022).

Since the version of youtube-dl I had installed was released exactly a year ago from the current date, the quickest way to fix the issue was to download the latest version of youtube-dl, but it turns out that v2021.12.17 is indeed the latest version.

Then, I remembered that the open source project for youtube-dl had been having management issues. Collectively, these issues led the community to shift their focus to an actively maintained fork called yt-dlp.

So, I downloaded the latest binary for yt-dlp which was released on GitHub only a month ago. The download also failed. The error output from yt-dlp was identical to the output above from youtube-dl.

Slowing Down to Diagnose the Error

Based on the hint contained in the error, my next suspicion was that there was some kind of mismatch between the HTTP range request that youtube-dl was sending and what YouTube's servers were expecting.

I condensed the error down to the following string and did a quick search on Google:

Got error: Downloaded 2451832 bytes, expected 676416237 bytes.

In return, Google gave the following (non-) result:

An Obscure Error While Downloading a YouTube Video for Offline Viewing
About 0 results (0.44 seconds)

Although there were no results for my query, Google did helpfully include 4 screenshots, two of which linked to GitHub. I clicked through to the first image which led me to this closed issue on the youtube-dl project.

This comment, left in 2017, provided a valuable hint: that the issue is from YouTube and not due to a bug in the youtube-dl code. The workaround is to use a different format.

Since I didn't specify any format in all of my attempts, I decided to skim the CLI arguments supported by youtube-dl so I could specify a format explicitly.

youtube-dl --help


Video Format Options:
    -f, --format FORMAT                  Video format code, see the "FORMAT SELECTION" for all the info
    --all-formats                        Download all available video formats
    --prefer-free-formats                Prefer free video formats unless a specific one is requested
    -F, --list-formats                   List all available formats of requested videos

Armed with this information, I turn on verbose mode (with the "-v" option) and passed the "-F" option to youtube-dl so it can list the video formats supported by YouTube for the video in question:

youtube-dl -v "https://www.youtube.com/watch?v=mOY159TlYXM" -F
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=mOY159TlYXM', '-F']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: fba051f98
[debug] Python version 3.10.5 (CPython) - macOS-12.6-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] mOY159TlYXM: Downloading webpage
[info] Available formats for mOY159TlYXM:
... (truncated due to poor formatting) ...
... (see the next screen shot) ...
An Obscure Error While Downloading a YouTube Video for Offline Viewing

It turns out that when I don't explicitly specify a format, youtube-dl defaults to downloading the last format in the list (format code: 22; file format: mp4 at a 1280x720 resolution).

It seems there's problem with how that video format is encoded/served by YouTube's servers because any attempts to download it explicitly leads to the error I mentioned above. (Highly likely that YT has checks in place to surpress encoding errors when videos are consumed normally by a human.)

Once I picked the mp4 format at a lower resolution (format code: 18, file format: mp4 at a 640x360 resolution), the download worked fine. The command I used was:

youtube-dl -v "https://www.youtube.com/watch?v=mOY159TlYXM" -f 18

HTH!

]]><![CDATA[How to get delv working on macOS]]>https://ayewo.com/how-to-get-delv-working-on-macos/6499621cf941aa2920349d1fMon, 05 Sep 2022 16:29:26 GMT

While reading this HN comment, I learnt of a CLI tool named delv.

delv's history paints it as the spiritual successor to dig, plus its name pays homage to a popular nursey rhyme. Interesting.

Similar to dig ,delv is a DNS lookup and validation utility. Apple shipped the delv binary with my machine running macOS Monterey (v12.3.1) but I had trouble getting it to work.

When I ran the first command in that HN comment: delv MX ycombinator.com @9.9.9.9 I got an inscrutable output:

delv MX ycombinator.com @9.9.9.9
;; none:29: no crypto support
delv: No trusted keys were loaded

Some Attempts to fix the Problem

I did a search on the error message and came across a mailing list discussion from 3 years ago which mentioned the -a anchor-file option from the man page.

Based on the explanation in that man entry, the default behavior of delv is to look for key information in a default location /etc/bind.keys, if no -a anchor-file option is specified, so I decided to create that file since it was non-existent on my machine.

Just to be sure, I first check to see what version of delv I have installed:

delv -v                                                        
delv 9.10.6

So, I use the term "/etc/bind.keys" to search for a valid copy of that file online and landed on the homepage of the ISC (Internet Systems Consortium). It is hosted on their FTP site and I was able to download a valid copy (v9.11) with the following commands: 

cd /tmp
curl -LO https://downloads.isc.org/isc/bind9/keys/9.11/bind.keys.v9_11
sudo mv /tmp/bind.keys.v9_11 /etc/bind.keys

With a valid /etc/bind.keys in place, I ran the delv command again and the error message was slightly different:

delv MX ycombinator.com @9.9.9.9 
;; /private/etc/bind.keys:29: no crypto support
delv: No trusted keys were loaded

Based on additional searches, the real issue is the "no crypto support" part of the error message. This ServerFault answer suggests that the Apple-provided version of the delv binary was not set to support SSL when it was compiled.

Apple's inclusion of OSS network utilities can sometimes be half-hearted, so rather than play whack-a-mole trying to fix it, I decided to use Homebrew.

Fixing the Problem

bind is an OSS collection of network utilities (sometimes called bind-utils or bind-tools) from the ISC that includes CLI tools like dig, host, nslookup and of course delv.

So, I search for "homebrew bind" and the first result gave me a simple way to install it on macOS in only 2 steps.

Step 1: Install bind by executing brew install bind:

brew install bind
...
==> Caveats
==> bind
To restart bind after an upgrade:
  sudo brew services restart bind
Or, if you don't want/need a background service you can just run:
  /Users/mac/homebrew/opt/bind/sbin/named -f -L /Users/mac/homebrew/var/log/named/named.log

Step 2: Next, open a new terminal window, and check the version of delv that was installed is newer than the Apple-provided version of 9.10.6:

delv -v
delv 9.18.4

The newer delv binary can be found at this path: /Users/mac/homebrew/Cellar/bind/9.18.4/bin/delv on my Mac and because my Homebrew installation was setup to put Homebrew binaries ahead of Apple-provided binaries in my PATH, the shell found it before it found the older version provided by Apple.

You can read more about how I set this up towards the end of my article on using a newer version of rsync on macOS.

Using the new version of delv

Re-running the original command in the HN comment earlier–delv MX ycombinator.com @9.9.9.9  now works correctly:

delv MX ycombinator.com @9.9.9.9 
; unsigned answer
ycombinator.com.	300	IN	MX	10 aspmx.l.google.com.
ycombinator.com.	300	IN	MX	20 alt1.aspmx.l.google.com.
ycombinator.com.	300	IN	MX	20 alt2.aspmx.l.google.com.
ycombinator.com.	300	IN	MX	30 aspmx4.googlemail.com.

The second command in the HN comment–delv NS ycombinator.com @9.9.9.9 also worked:

delv NS ycombinator.com @9.9.9.9
; unsigned answer
ycombinator.com.	10267	IN	NS	ns-225.awsdns-28.com.
ycombinator.com.	10267	IN	NS	ns-556.awsdns-05.net.
ycombinator.com.	10267	IN	NS	ns-1411.awsdns-48.org.
ycombinator.com.	10267	IN	NS	ns-1914.awsdns-47.co.uk.
]]><![CDATA[Touchpoint Reputation in B2B SaaS]]>https://ayewo.com/the-concept-of-touchpoint-reputation/6499621cf941aa2920349d1cTue, 16 Aug 2022 21:54:39 GMTRand Fishkin recently posted the following tweet:

Source: Rand Fishkin

I think the conversation sparked by his thread speaks to the existence of a invisible but important quality of contact forms on B2B websites.

A contact form, a contact telephone, a contact email etc are merely different ways by which a website visitor can establish initial contact with a company they intend to do business with. Collectively, NNgr0up refers to them as touchpoints that are part of a customer's journey.  

I'm writing this short piece because the Twitter thread generated an interesting thought in my head–it seems that touchpoints have a reputation which shapes peoples behavior.

Depending on how a touchpoint's reputation is perceived, visitors might decide to either use it or seek out an alternative to accomplish the task at hand.

For instance, if a contact form:

  • is asking for too much info (i.e. it will take too much time to complete), or;
  • is a bag of uncertainty–it offers no hint as to when you’ll hear back from the business (i.e. no reassuring note like "we typically respond within 24-48 hrs");

a visitor is likely to abort the task at hand or seek out alternative touchpoints known to deliver fast answers.

Another angle worth considering is past experience—if visitors have taken the pains to fill long contact forms and not hear back for weeks, this negative experience will add up and ultimately turn visitors away from bothering with such forms altogether.

They'd rather seek out alternative touchpoints such as the telephone, email or LinkedIn messages (mentioned in Rand's tweet) which are known to have a reputation of being manned by an actual human, ensuring that they can get a response faster.

John Xie mentioned Taskade’s counterexample of having good contact form usage by site visitors:

Source: John Xie

I think there are two possible explanations at play here:

  • they kept the Typeform-based contact form on their pricing page simple;
  • Typeform-based contact forms in general tend to enjoy a good reputation amongst netizens;

For #1, the contact form has 4 mandatory questions and 1 optional question so the form should only take about a minute complete.

For #2, businesses that care about providing a smooth form experience often use Typeform over bland competitors, so I'm guessing that when visitors see that a Typeform-like experience is available, they are much more likely to put up with it because it feels different:

Source: Dan Cave

Perhaps there is a direct correlation between businesses that use Typeform-based contact forms and businesses that close the feedback loop quickly when they receive enquiries from website visitors?

(PS: I hope to someday follow-up this post with a substantive look at the evolution of email as a touchpoint. There was a period when inboxes got overran by unsolicited email, before spam filters got good enough to combat the problem.)

Additional Reading

This article: After self-hosting my email for twenty-three years I have thrown in the towel via HN on September 5, 2022 had an interesting discussion that touched on a number of concepts, all related to reputation:

  • IP reputation can make or break email deliverability: 5 Ways to Check Your Sending Reputation;
  • IP range reputation is now being used as a substitute for IP reputation amongst signals for combating spam since (temporary) blockage of an entire IP block (or range) can be effective against spam. This makes sense since if you are on shared infrastructure where there are no anti-spam policies in place, or where neighboring IPs repeatedly get pwned to send out spam, the only way to get your provider's attention is to drop all traffic originating from their IP range–since your provider is unaware or does not care enough to put checks in place.
]]><![CDATA[3 Steps to Fix a Failed iCloud Backup on Your iPhone]]>https://ayewo.com/3-steps-to-fix-your-failed-iphone-cloud-backup/6499621cf941aa2920349d1aThu, 21 Jul 2022 21:10:39 GMTSynopsis3 Steps to Fix a Failed iCloud Backup on Your iPhone

Several weeks ago, I noticed that backups from my iPhone had stopped syncing to iCloud. While trying to fix the issue, I encountered the following error messages on the iCloud Backup screen:

  • The last backup could not be completed because of poor network conditions.
  • Some files were unavailable during the last backup.

My situation was a bit unique because I use a second iPhone as a WiFi hotspot, so it took me some time to resolve the issue.

Background

This is a story about two iPhones: the first is running iOS 15.5 and the second is running iOS 12.5.5. The second iPhone is stuck on an old iOS version due to the device being a fairly old but still functional iPhone 6, which I bought new back in 2015.

In spite of several years of constant use, the second iPhone has refused to die, so I decided to use it as a Personal Hotspot—to provide (nearly) unmetered Internet access to the first iPhone over WiFi.

And for over 7 months, it was quite reliable as a WiFi hotspot before I noticed in late June that iCloud Backups had started failing on the first iPhone. It had stopped syncing my backups to iCloud for about two weeks.

My Theory as to the Culprit

The second iPhone no longer receives updates—it has been stuck on iOS 12.5.5 since September 2021 (and started duty as a WiFi hotspot in early December 2021), which rules it out as the culprit.

My suspicions turned to the first iPhone because I remember updating it to iOS 15.5 in early June, around the time of WWDC 2022 (6 Jun – 10 Jun 2022). The iCloud Backup screen showed June 10 as the last backup date, which lines up nicely with my recollection of when I updated iOS.

Fixing the iCloud Backup Error

I scanned the security content as well as the release notes for iOS 15.5 to see if there were any changes to the Personal Hotspot feature or something less visible like an update to the networking stack, but I didn’t find anything to latch on, so I knew I had to copy-paste the iCloud Backup error message into Google Search to make any progress.

Step 1

The first error I was shown was: “The last backup could not be completed because of poor network conditions”, so I switched to a WiFi connection and restarted the backup process. This worked because I was shown a different error after the restarted backup failed.

So, step 1 is to change from Personal Hotspot to a 'proper' WiFi connection.

Steps 2 & 3

The new error read: “Some files were unavailable during the last backup” and after skimming several articles, I tried out the following suggestions:

So, I did everything from rebooting the iPhone (multiple times), close several apps (almost 2 dozen of them), to attempting to sign-out and sign-back-in with my Apple ID. iOS showed a scary warning about deleting files stored on the device if I were to proceed with signing out so I aborted this suggestion.

I tried again, and after about an hour or so, the backup completed successfully 💯!

No Official Fix Yet

Today, I put the second iPhone in Personal Hotspot mode so I could use it to access the Internet on the first iPhone. I attempted an iCloud Backup and I received the same error as I got the first time: "The last backup could not be completed because of poor network conditions", which makes the error reproducible.

I have updated to iOS 15.6 which was released yesterday (July 20) and the error remains.

Epilogue: the Root Cause‌

iCloud Backups via another iPhone in Personal Hotspot mode used to work fine prior to iOS 15.5. It seems that Apple may have introduced a silent, or more correctly, undocumented, change in May 2022 when iOS 15.5 was released.

An iPhone running iOS 15.5 now treats a WiFi connection from another iPhone in Personal Hotspot mode differently from a regular WiFi connection, which explains why my iCloud Backups started failing after I updated from iOS 15.4 to 15.5.

I'm fairly certain because the first step in the instructions above, where I switched from using the second iPhone in Personal Hotspot mode to a "proper" WiFi connection, was really just me popping out the LTE SIM from the iPhone and plopping it into another phone also in hotspot mode, but running a different OS (Android instead of iOS).

]]><![CDATA[Exploding Career Opportunities in DevRel]]>https://ayewo.com/exploding-career-opportunities-in-devrel/6499621cf941aa2920349d19Sat, 22 Jan 2022 22:09:13 GMT

I've been tracking job openings in DevRel (Developer Relations) and adjacent roles in a private document in Notion for a while now. I've been using these job ads as a proxy–to help me discover companies that I wasn't aware of as a maker of tools for developers.

DevScape

Why? Back in July 2021, I bought a domain–devscape.co–where I plan to write in-depth about the developer landscape:

Tweet where I announce https://devscape.co

DevScape would also make for a great home for several of the things that I have been working on on the side, including a Playbook for DevRel practioners, growth strategy tear downs and potentially–a job board. If the experiment works out, DevScape could make money from subscriptions because the developer landscape is a fast growing niche.

Dev Rel Employers Often Lead to Dev Tool Makers

Using devrel job ads as a proxy for locating dev tool companies strewn across the dev tool landscape, I have been recording new finds in a free-form document in Notion but have come to realize that I'm slowly setting myself for a world of hurt later.

Going over my notes helped me realize that there are so many such companies that it makes more sense to use a tabular structure versus a free-form structure. A table is easier to scan, sort and reorganize than free-form copy-pasta. So, that's exactly what I did yesterday and today: move the data from Notion to Airtable.

Dev Rel Job Titles

The most common job title for the role of a developer that markets other developers is developer advocate, but quite a few companies opt for a different title.

Some examples of such titles include developer relations engineer, open source evangelist, community manager, developer relations leader etc, perhaps to emphasize how each role will require specific skills like professional coding experience, high visibility in the OSS (Open Source Software) community, past experience managing a community, or past experience managing a team, respectively.

A sampling of the titles I currently track include:

  • Developer Advocate
  • Developer Educator
  • Developer Relations Engineer
  • Developer Evangelist
  • Open Source Evangelist
  • Technical Evangelist
  • Data Advocate
  • Data Evangelist
  • Developer Relations Manager
  • Developer Advocacy Leader

Airtable

Today, I'd like to share my Airtable publicly, in the hope that it will be useful to veterans looking for a new job and newbies looking to make the jump to devrel.

Without further ado, the Airtable is embedded below:

I plan to regularly update the table, so be sure to bookmark this page.

(PS: The curator of DevRel Weekly: Mary Thengvall also maintains a public bookmark of devrel roles that is worth checking out.)

]]><![CDATA[Engineering vs. Technology]]>In this article, I attempt to chronicle how I arrived at what I think is a layman explanation of the difference between the terms engineering and technology .

An Incomplete Devil’s Dictionary

My goal with this series is to offer layman definitions of words used in the technology industry.

]]>https://ayewo.com/engineering-vs-technology-company/6499621cf941aa2920349d0bThu, 02 Dec 2021 01:16:38 GMTIn this article, I attempt to chronicle how I arrived at what I think is a layman explanation of the difference between the terms engineering and technology .

An Incomplete Devil’s Dictionary

My goal with this series is to offer layman definitions of words used in the technology industry. I try to be clear and concise, in the style of the concise (and witty) definitions that appear in The Devil’s Dictionary. The Devil's Dictionary was first published in 1911 by Ambrose Bierce.

A Long Primer

Two years ago, I read and thoroughly enjoyed Herbert Simon's The Sciences of the Artificial (SotA). In terms of information density, SotA is a phenomenal book–it is almost as dense as another personal favorite of mine: The Structure of Scientific Revolutions (SoSR) by Thomas Kuhn.

Both books were first published in the 1960s–SoSR was first published in 1962, while SotA was first published in 1969. I read the 1970 edition (aka 2nd edition) of SoSR, while I read the 1996 edition (aka 3rd edition) of SotA.

Why did I choose to read books from 50-60 years ago?

A Getaway for Looming Questions

I forget where I first saw the recommendations to read SoSR, but I suspect it must have been on HackerNews.

Every once in a while, someone would leave a comment pointing out the human tendency to make assumptions about the history of science, rather than consult the actual history of science. Recommending SoSR was often the antidote to that malaise. The nudge to read it often increased when the concept of paradigm shifts which was made famous by the book came up for discussion. I finally read it in 2018.

Reading SoSR helped answer several open questions that had been floating in my head for years, and more importantly, it created a new set of open questions deserving of their own answers.

Journeying With a Clear Thinker

SoSR absolutely showed how clear thinking can elevate your writing–reading it set the bar pretty high for what I now consider a good book. Today, it's hard to find books that can match the information density of SoSR, but it didn't stop me from being on the look out for other books of similar calibre.

Journeying With Another Clear Thinker

I came across the recommendation to read SotA in this strong nod to Herbert Simon as a thinker by Alan Kay on Quora:

Try “The Sciences of the Artificial” by Herb Simon. A much stronger way to think about computing — and what “Computer Science” might mean — by a much stronger thinker than most today.

I read SotA in 2019 and agree with Alan Kay–Herbert Simon was indeed a strong thinker though I prefer the term clear thinker.

Granted, many readers will consider them old books because of the time they were written, nonetheless, I think reading them and taking copious notes will sharpen one's thinking. I strongly recommend both books to anyone that wants to learn how to think from first principles, rather than by analogy.

A Small but Asute Observation

For several years now, my daily routine often involves doing a ton of reading online about the technology industry. Topics are wide and varied, but reading SotA forced me to reflect on a subtle but important observation.

Conflating Engineering with Technology

A lot of writers, especially online, use the words engineering and technology interchangeably even though they are not synonyms. This contrasts sharply with how Herbert Simon used them in his book.

In SotA, science, engineering and technology roughly approximate to the following distinct concepts: analysis, synthesis and tooling, respectively.

An Initial But Incorrect Diagnosis

It turns out that my initial diagnosis was wrong. Or more accurately, the premise was correct but the conclusion was wrong.

Yes, writers are increasingly using the words engineering and technology interchangeably, but not as synonyms.

Resolving the Misdiagnosis

To illustrate the issue, lets attempt to answer the following question which should have a fairly simple answer: what kind of company is Google? Is Google an engineering company or a technology company?

Google describes (1, 2) itself as an engineering company on the Engineering & Design section of its careers site:

Google is and always will be an engineering company. We hire people with a broad set of technical skills who are ready to tackle some of technology’s greatest challenges and make an impact on millions, if not billions, of users.

Wikipedia describes Google as a technology company:

Google LLC is an American multinational technology company that specializes in Internet-related services and products, which include online advertising technologies, a search engine, cloud computing, software, and hardware.

While Google says it is an engineering company, Wikipedia disagrees, describing them as a technology company instead. Which characterization is more accurate, the one by Google or Wikipedia?

To answer this follow-on question, we need to agree on a layman definition for each word in order to understand how the underlying activities differ. In other words, what is the defining trait that allows us to tell apart engineering companies from their technology counterparts?

Verbose Definitions

The individual definitions of engineering and technology on Wikipedia are too verbose to be useful here for two reasons:

  • their definitions partially depend on a definition for science;
  • the definition of science is too academic to expressed in a concise way.

I remember repeatedly pausing my reading to search for a modern layperson explanation of how the two concepts differ, but none of the definitions I came across were intuitive or concise.

Concise Definitions

So I took a stab at it and decided to maintain a layperson list of word definitions on my device. By the time I had finished reading SotA, I had come up with a fairly stable definition for the terms engineering and technology. For completeness, I added concise definitions for science and mathematics, which altogether make up the STEM acronym.

STEM: Science, Technology, Engineering, and Mathematics
A clear and concise definition for each discipline that makes up the STEM acronym.
Misleading MetaphorsSaïd

Resolving the Question

Armed with the following characterizations:

  • engineering is preoccupied with working within constraints;
  • technology is preoccupied with transcending constraints (with tools);

let's attempt to answer our last question: "which characterization is more accurate, the one by Google or Wikipedia?"

I'd say that Google is engaged in both kinds of activities as is evidenced by the copy on the Consumer Hardware section of their careers site:

Our Consumer Hardware team researches, designs, and develops new technologies and hardware to make users’ interaction with computing faster, more powerful, and seamless. Whether finding ways to capture and sense the world around us, advancing form factors, or improving interaction methods, our Consumer Hardware team is making people’s lives better through technology.

Google is a multi-faceted entity which is why it can be referred to as an engineering company, a technology company or a monopoly. Whether a label is appropriate is context-dependent.

Essentially, the real-world is complex. Calling Google an engineering company or a technology company is a matter of perspective. Both are correct.

 

]]><![CDATA[STEM: Science, Technology, Engineering, and Mathematics]]>https://ayewo.com/stem/6499621cf941aa2920349d18Wed, 01 Dec 2021 15:56:38 GMTMotivation

I have yet to come across a satisfactory definition for any of the individual words that make up the acronym STEM: science, technology, engineering, and mathematics, so I'd like to offer my own.

Science

Science is preoccupied with the discovery of constraints.

Engineering

Engineering is preoccupied with working within constraints.

Technology

Technology is preoccupied with transcending constraints (with tools).

Mathematics

Mathematics is preoccupied with expressing constraints (with symbols).

Summary

All four (4) definitions highlight an important theme that exists across STEM fields: the concept of constraints.

Each definition re-frames, in a handful of words, the four STEM fields as variations on one underlying human endeavor—the human need to deal with constraints.

A side-effect of this re-framing using the concept of constraints is that the definitions are fairly easy to recall–at least for me.

]]><![CDATA[rsync on macOS: Matters Arising]]>https://ayewo.com/switching-to-a-newer-version-of-rsync-on-macos/6499621cf941aa2920349d17Sun, 21 Nov 2021 11:27:26 GMT

I recently had to transfer large files on Windows to macOS Catalina (10.15.7) using rsync, but I met a few snags on the way. I already wrote about my experience using rsync for the first time on Windows.

This article is the concluding part that delves into the issues I encountered on the macOS side of the transfer.

Too many bytes?

The transfer went smoothly for the most part, until rsync got to an ~18GB .vdi file used by one of my VirtualBox VMs:

bash -c "rsync -avzh -P --remove-source-files --stats --timeout=60  VirtualBox 'mac@10.0.0.104:/Users/mac/Windows8-Lenovo/'"
(mac@10.0.0.104) Password:
building file list ...
65 files to consider
VirtualBox/Ubuntu 18.04 LTS + Upwork/Ubuntu 18.04 LTS.vdi
          9.63G  54%   68.12MB/s    0:01:55
rsync: [sender] write error: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]
rsync: connection unexpectedly closed (1747 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(453) [generator=2.6.9]
rsync error: error in rsync protocol data stream (code 12) at io.c(823) [sender=3.2.3]

I scanned the error message and figured it might have been due to a minor network issue, so I re-attempted the transfer two more times and the error was fairly reproducible.

Here's the output from the 2nd try:

65 files to consider
VirtualBox/Ubuntu 18.04 LTS + Upwork/Ubuntu 18.04 LTS.vdi
          9.66G  54%   10.34MB/s    0:12:37
rsync: [sender] write error: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]
rsync: connection unexpectedly closed (1747 bytes received so far) [generator]

Here's the output from the 3rd try:

65 files to consider
VirtualBox/Ubuntu 18.04 LTS + Upwork/Ubuntu 18.04 LTS.vdi
          9.62G  54%   68.44MB/s    0:01:54
rsync: [sender] write error: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]
rsync: connection unexpectedly closed (1747 bytes received so far) [generator]

So, after 3 attempts, the transfer fails at exactly 54% of the way during the transfer of an 18GB file.

Filesystem limitation?

I've experienced a similar kind of error a long time ago while transferring a large file to FAT-based filesystem used by Windows, so my first thought was that this was also a filesystem limitation issue.

The difference here was that Windows was the source and not the destination of the large file transfer, so I decided to search for the first error message in the output "rsync: [sender] write error: Broken pipe (32)".

Unable to rsync due to broken pipe

The first result on Google: Unable to rsync due to broken pipe turned out to be especially useful in pointing me towards the underlying issue–it turned out I was using two different versions of rsync.

Just to be sure, I went ahead to re-read the error output much more closely:

rsync: [sender] write error: Broken pipe (32)

rsync error: timeout in data send/receive (code 30) at /AppleInternal\

/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/\

rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]

rsync: connection unexpectedly closed (1747 bytes received so far) [generator]

rsync error: error in rsync protocol data stream (code 12) at \

/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/\

Sources/rsync/rsync-54.120.1/rsync/io.c(453) [generator=2.6.9]

rsync error: error in rsync protocol data\

stream (code 12) at io.c(823) [sender=3.2.3]

On the sending machine (Windows) I had rsync v3.2.3, while on the receiving machine (macOS) I had rsync v2.6.9 installed:

rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Fixing the problem

The fix was fairly easy–I had to update the dated v2.6.9 copy of rsync on macOS to match v3.2.3 on Windows.

brew install rsync

There was a slight problem though, because of how my Homebrew installation was setup to put all binaries under ~/homebrew/bin.

Version 2.6.9 of rsync at /usr/bin/rsync was getting found first in my PATH instead of v3.2.3 that I had just installed to ~/homebrew/bin/rsync.

Fixing a latent problem

I had been putting off the Homebrew PATH issue for a while due to my unfamiliarity with the default zsh on macOS, but this time, I decided I was going to properly research how zsh differs from the more familiar bash, so I could fix the PATH issue the correct way, rather than cargo-cult magical incantations from the Internet.

I've distilled my research into two comment lines just before the actual fix for zsh:

# In https://apple.stackexchange.com/a/425894 it mentions /etc/zprofile (different from my pre-existing ~/.zshenv and ~/.zshrc)

# In /etc/zprofile it says to add customizations to ~/.zprofile instead, so this is where our Homebrew's rsync v3.2.3 can overshadow Apple's v2.6.9 in the PATH

echo "export PATH=/Users/mac/homebrew/bin:$PATH" > ~/.zprofile

Running rsync --version from anywhere now gives the newer version:

rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, no atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, no prealloc, stop-at, crtimes, file-flags
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.
]]><![CDATA[7 Step Guide to Installing rsync on Windows]]>I recently needed to move several gigabytes of data from an old laptop running Windows 8.1 to another laptop running macOS and my options were to use an external disk or a network transfer. I wanted the whole process to be unattended so I naturally gravitated towards doing the

]]>https://ayewo.com/how-to-install-rsync-on-windows/6499621cf941aa2920349d16Wed, 13 Oct 2021 13:04:17 GMT

I recently needed to move several gigabytes of data from an old laptop running Windows 8.1 to another laptop running macOS and my options were to use an external disk or a network transfer. I wanted the whole process to be unattended so I naturally gravitated towards doing the transfer over my WiFi network.

For the network transfer, the easiest method I could think of was to use scp but it turns out that there's no built-in support to "move" files with scp. My options were to write a script to delete each file after getting copied or use a different tool with file move semantics.

A couple of Google searches later, the most up-voted tool on Serverfault for the task is rsync. I'ved used rsync before but I have never used it on a Windows box so this was an interesting option to try out.

Some additional searches led me to this answer how to use rsync from Windows PC to remote Linux server? that I was hoping would be a drop-in solution to my situation.

Unfortunately, I couldn't get rsync working on my first try because the instructions in the most-upvoted answer along with other answers on the question no longer work in 2021. Even this somewhat useful article on installing rsync on Windows from more than a year ago, which I found on Google, was almost completely out of date.

I had to figure out a few missing steps to get rsync working on Windows, so I decided to publish step-by-step instructions to help my future self and others in a similar situation. I tested the commands below on Windows 8.1, nonetheless they should work on Windows 10 or 11.

The macOS side of things are fairly straight forward so I wont go into that here.

Instructions

  1. Install Git for Windows to the path "C:\Program Files\Git".
    In my case I already had it installed so installation merely updated my version to v2.33.0.2.

2. Visit https://repo.msys2.org/msys/x86_64/ and download the following three (3) file archives:

3. The .zst extension indicates that the files were compressed using the fast Zstandard compression algorithm open sourced by Facebook so I had to download another (open source) tool to help me extract those files locally: PeaZip for Windows v8.2.0.

4. Create a temporary folder called "C:\tmp" then use PeaZip to extract the 3 files we downloaded earlier into this folder, in two (2) steps.

The 1st step is to use PeaZip to convert .tar.zst files to .tar files:

  • rsync-3.2.3-1-x86_64.pkg.tar.zst -> rsync-3.2.3-1-x86_64.pkg.tar
  • libxxhash-0.8.0-1-x86_64.pkg.tar.zst -> libxxhash-0.8.0-1-x86_64.pkg.tar
  • libzstd-1.5.0-1-x86_64.pkg.tar.zst -> libzstd-1.5.0-1-x86_64.pkg.tar

The 2nd step is to use PeaZip to extract the contents of each .tar file. Note that all 3 archives contain a folder named \usr with different contents, so be sure to click "yes" when prompted to overwrite the contents of the C:\tmp\usr folder during the extraction process.

5. Now, you need to move the C:\tmp\usr folder to its final destination i.e. C:\Program Files\Git, or the location where you have Git for Windows installed.

Essentially, we need to merge the contents of C:\tmp\usr with C:\Program Files\Git\usr so that the rsync binary (and its dependencies) will end up at the following path C:\Program Files\Git\usr\bin\rsync.exe.

6. The final step is to confirm that you have a working installation of rsync from a command prompt by executing these 2 commands:

cd "C:\Program Files\Git\usr\bin" and rsync --version

rsync.exe --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, no hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, prealloc, stop-at, no crtimes
Optimizations:
    no SIMD, asm, openssl-crypto
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

7. You are done. Enjoy!

]]><![CDATA[Can Apple Watch Measure Your Body Weight? [Updated for 2022]]]>https://ayewo.com/can-the-apple-watch-measure-your-body-weight/6499621cf941aa2920349d15Sun, 19 Sep 2021 13:12:32 GMT

As you can see in the screen shot below, the Apple Health app shows a fairly accurate measure of my weight that was added by the Apple Watch. Can the Apple Watch estimate a person's weight now? What's going on?

Can Apple Watch Measure Your Body Weight? [Updated for 2022]

TL;DR

The short answer is no. The Apple Watch is not equipped with sensors for measuring body weight, at least not yet. The rest of this post explains why a fairly accurate reading of your body weight in the Apple Health app was added by your new Apple Watch.

Apple Watch Series 6

I recently got an new Apple Watch Series 6 which I finished setting up in early August, but I was surprised to see a fairly accurate measure of my weight in the Health app that was added by the Apple Watch, more than a week after I set up the Apple Watch for the first time.

I have a habit of measuring my weight at the end of my workouts using a regular body weight scale i.e. it is not a smart scale, but the day the entry was added by the AW was actually the first time I wore the AW prior to a full workout. In fact, the time the entry was added was during my warm ups—stretches that I do prior to that day's workout to stave off injuries. At the time, I noticed that the LED lights at the back of the AW turned on a few times. The LED lights are used to monitor heart rate.

Searching for Clues

So, I went searching online to see if the ability to measure, or at least estimate a wearer's weight is a hidden feature of the AW. If this were true, I'd be curious to learn how what sensors are packed into the AW to enable such a neat feature.

After much searching, I came back empty handed.

A Plausible Explanation

Yesterday, I repeated the search using a slightly better set of keywords (in hindsight): "apple watch series 6 measure weight" and the 8th link on Google was to this MacRumors forum discussion from 2015 where nicho posted this helpful answer:

If I remember right, when you set up your watch it checks your height and weight and such. It then puts that info into the health app.

Essentially, what nicho is saying is that, as part of the first time set up dance, the AW does three things, it:

  • reads your existing height and weight data stored in Apple Health;
  • uses the existing height and weight data for some calculations;
  • adds back your existing height and weight data to Apple Health.

Looking more closely at the screenshot, the weight added by the AW is really just the previous day's reading, which I added manually to the Health app. I have gone on several more workouts with the AW on my wrist and no additional estimates of my weight have been added by the AW, since that one entry.

I checked Body Measurements -> Weight, to look more closely at the data inside the Health app and the only difference between the existing data and the data added by the AW is that the name of the AW is recorded as the "Source" of the data in the Health app, while the "Was User Entered" flag is set to "No".

I also checked Body Measurements -> Height, and the AW did in fact add a new entry on that date and time for my height, similar to what it did for my weight.

So, my guess is that the AW didn't finish setting itself up the first time I wore it, until I triggered something (the hear rate sensor?)—which was several days later.

Mystery solved!

The Newer Apple Watch Models Have More Sensors

At the Apple Event on September 7, 2022, Apple announced updates to the Apple Watch product line where a new sensor for wrist temperature sensing was added to two brand new Apple Watch models:

This is a cool new feature but there's still no support for body weight estimation using only an Apple Watch.

]]><![CDATA[RE: Apple, Its Control Over the iPhone, The Internet, And The Metaverse]]>Note to readers: This is a rejoinder to Matthew Ball's epic 15000-word essay of the same title, published on February 2, 2021. He was gracious enough to reply to my initial concerns on Twitter, and he did stop by to read this.

It is not often that someone

]]>https://ayewo.com/re-apple-its-control-over-the-iphone-the-internet-and-the-metaverse/6499621cf941aa2920349d12Tue, 09 Feb 2021 17:12:49 GMT

Note to readers: This is a rejoinder to Matthew Ball's epic 15000-word essay of the same title, published on February 2, 2021. He was gracious enough to reply to my initial concerns on Twitter, and he did stop by to read this.

It is not often that someone takes effort to pen a long-form piece of about 30-pages on Apple (Matthew says the essay took several months to write), but because it concerns Apple, the details matter.

I have never worked with or for Apple, neither do I own Apple stock, but as someone who has been in the industry for long enough, I can say that Apple would easily be at the top of a list of the most detailed-oriented companies, if such a list were to exist. Perhaps this is also why their actions and in-actions are grossly misunderstood.

Chapter One: The Creation of Today’s Internet and the Needs of Tomorrow’s

In chapter one, Matthew writes that:

Services like Zoom also work because they leverage actively maintained standards that are free to use and designed to support any device.

I totally understand that this essay may have been written to be accessible to a lay person, but the problem with this claim is that, it doesn't tell the full story.

A lot of video conference call invites are accepted via email, and if you are doing it for the first time, you'll likely not have the Zoom client installed on your machine, so yes, Zoom has to "leverage actively maintained standards" to make a good first impression, when all a first-time user has installed is a web browser.

But as your needs expand, so will your desire to explore the best they have to offer–at which point they can nudge you towards their more powerful services, which are of course not built on (open) standards.

He also writes that:

But just imagine, for example, how the internet might differ if it had been created by multinational media conglomerates in order to sell things, serve ads or harvest user data for profits. Downloading a .JPG could cost money, with a .PNG costing 50% more. Teleconferencing software might have required the use of a broadband operator’s app or portal (e.g. Welcome to your Xfinity Browser™, click here for Xfinitybook™ or XfinityCalls™ powered by Zoom™). Websites might only work in Internet Explorer or Chrome - and need to pay a given browser an annual fee for the privilege.

The fascinating thing about this passage is that, every single hypothetical idea he doesn't wish to entertain has been used in the past, successfully.

Yes, downloading a .JPG or images in general, did in fact cost money when Internet usage wasn't yet wide spread. Although, you didn't pay for each image directly, the cost of your web browsing was reflected in your phone bill, because back then, the main way to access the Internet was via phone modems.

To corroborate what I'm saying, here's a June 28, 1994 entry on the topic from the NY Times titled PERSONAL COMPUTERS; You Can't Roller-Skate On Electronic Highway. Below is an excerpt:

Although the general rule in modems is "the faster the better," there are several reasons a cautious computer user might wisely choose a slower 14.4 kilobit-per-second modem over the V.34 speedsters.
The first is cost. As with any new technology, the pioneers pay the price. The new modems typically cost $500 or more, as against $100 to $150 for the 14.4-kilobit modems.

Due to the cost implications, it was not uncommon to browse the Web image-free via the "Do not show any images" browser setting, especially because a lot of dial-up modems were very slow anyway.

If you substitute Xfinity with AOL, the hypothetical scenario he paints about portals was in fact the lived experience for millions of first-time Internet users, as succinctly captured by this NY Times article from 1996:

But the follow-up study also found that 20 percent of the people who said they had access to the Internet in the first survey said they no longer had access six months later. Of those who did visit the Internet, most did so through commercial on-line services like America Online, the study found. America Online promises to connect users with a minimum of technical difficulty, which attracts inexperienced computer users.
''It makes sense, because the usage of people who go on line through a commercial service tends to be more personal and more periodic,'' said Thomas E. Miller, vice president of the emerging technologies research group of Find/SVP, a market research firm in New York. ''AOL has succeeded in penetrating a downscale audience.''

With respect to web browsers:

Websites might only work in Internet Explorer or Chrome -

Best viewed with Internet Explorer was an actual thing, it even had a logo to go with it that should be instantly recognizable to early netizens.

RE: Apple, Its Control Over the iPhone, The Internet, And The Metaverse
Best viewed with Internet Explorer badge

... and need to pay a given browser an annual fee for the privilege.

Today, Matthew may balk at the idea of paying for a web browser, but people back then certainly didn't. Many may know Marc Andreessen as promoter of VC firm Andreessen Horowitz (a16z), but in the Feb. 19, 1996 issue of Time Magazine, he appeared on the cover.

RE: Apple, Its Control Over the iPhone, The Internet, And The Metaverse
Barefoot Marc Andreessen on the cover of Time

What was his claim to fame? You guessed it–a successful IPO from an Internet business that initially made money from selling web browsers for $40 a pop!

Chapter Two - Three

Similar to Chapter 1, there are several historical inaccuracies in Chapters 2 - 3.

It's okay to make mistakes since no one can know everything about our computing past. I'll skip them, particularly because it is merely part of the build up to the meat of the essay. Besides, I'm trying to avoid making this piece too long.

Chapter Four: Apple’s Deficient Monopoly Defenses

Lastly, it’s notable that while MacOS does not have iOS’s restrictions on software/app installations, it remains secure and safe to use. This is because the majority of security is held at the kernel/OS-level. To this end, malicious apps (and updates) have lengthy history of making it through the App Store review. This is inevitable given the volume of submissions, time spent reviewing code, and human error rates, but crucially, these bad actor apps don’t destroy the device or devour private files (as is often the case with Windows malware). This is because of iOS’s system and API-level security, which is a stronger and more scalable solution - and one that offers consumers more choice and developers more capabilities.

The reason why there's a disparity in security incidents between macOS and iOS isn't far fetched. As far as devices go, there are way more iOS users than there are macOS users, which causes attackers to focus on iOS because of the incredible ROI.

It is true that iOS doesn't allow third-party browser engines, but there are tricky engineering details behind why Apple made this trade off. I'd love to delve into it here, but based on the outline I wrote, it deserves to be written as a standalone piece. Including it here would make this rejoinder nearly as long as the original ~30-page essay.

Chapter Five: The Importance of Prioritizing Overall Prosperity

Why does this matter? How much should we really care about how apps are made and who gets paid for them? Or the platforms that are used or allowed? Or who can advantage which identity solution, service, or standard? The answer is that we have to care. In 2021, almost every company is a digital company, and the scope, significance, and complexity of the digital/virtual world continues to expand.
To prosper, new winners must be able to emerge at every layer of the “new economy” and without being constrained by what a single company envisions or wants. The current situation isn’t terrible — great things are happening in terms of users, creators, and innovation. But the very decisions that have made Apple so successful today also limit the number of creators that participate in the virtual economy, handicap the creativity of their products and business models, and make every transaction more expensive. Most important, it is blocking the organic evolution of the overall Internet.

I don't understand the logic here. Let's zoom out by focusing on another technology company that makes physical goods and is using an integrated approach: Tesla.

Tesla, like Apple, tries to own every part of the experience: the car, the (expensive) battery, the car charging network, the in-car operating system called Tesla Software (a corollary to iOS), a soon to be launched app store, and of course they own what would have been the car distribution via traditional car dealerships.

Similar to Apple, there is one exception to this integrated approach that they do not try to own or control: the ecosystem around accessories. Tesla has a growing ecosystem for accessories:

Today, Elon Musk is worth $186.5B, a large part is due to his Tesla holding, and if they continue on their current trajectory (read: their competitors take too long to figure out electric in a way that is viable), they might go on to dominate not just the electric car, but the car industry as a whole.

The Tesla in-car OS currently supports apps for Spotify, Netflix, YouTube, and Hulu, and I imagine that list will continue to grow. Assuming they go on to dominate the car industry, somehow by some logic that is not clear to me, they should be compelled to allow on all Teslas, third-party OSes or third-party app stores that compete with Tesla OS or its app store?

In other words, Apple CarPlay and Android Auto, which are competing car OSes designed for general purpose cars (each with its own marketplace), should somehow be allowed to be installed on Teslas because Tesla is hypothetically the dominant car maker, but not on competitor cars because those are not monopolies?

Tesla's hypothetical blocking of third-party app stores would somehow be "blocking the organic evolution of the overall car industry"? Even though they are not the only car seller, they just happen to be the most dominant? Which law or moral code would they be violating?

No car-maker has ever been subjected to such logic, at least with respect to their in-car OS, so I don't understand this logic at all or how it should somehow apply to Apple's iOS.

If Apple's iOS wielded a monopoly on all mobile devices, irrespective of device maker similar to Microsoft Windows at its peak, then this would be a clear case of monopoly power being wielded that needs to be curbed.

I suspect people are leaning too much on the desktop PC and its attendant metaphors, to reason about how the ecosystem should be around mobile products.

Chapter Six: Solving the Apple Problem

Apple has the right to run its own store, offer its own standards, and develop services that are exclusive to its hardware. The problems arise from Apple’s forced bundling of hardware, an operating system, distribution system, payment solution and services. As a result, there is a straightforward remedy — forcing Apple into competition in app distribution and payments.

I'm not sure I understand what is being advocated here. Apple's operating philosophy has always been to offer an integrated experience as opposed to the less integrated experience of Wintel; where Microsoft makes the operating system (OS), Intel makes the computer chip, while Original Equipment Manufacturers or OEMs make the rest of the computer.

This is not some new approach, it has always been how they approach the design their products. In fact, at the 2007 launch of the iPhone (0:40s YouTube clip), Steve Jobs attributed how they think about technology products to a quote from 40 years ago (Apple is 45 years old) by Alan Kay, a highly respected computing pioneer:

People who are really serious about software should make their own hardware. – Alan Kay

When the dominant metaphor was desktop computing, Apple competed against Windows on the desktop by offering an integrated experience and nearly went bankrupt in 1997. The market voted with their wallets because they favored the less integrated approach of Wintel. Of course, there were a lot of factors at play.

Today, the dominant metaphor is mobile computing. In 2007, Apple was the underdog with 0% market share in mobile, but it didn't deter them from using the same integrated approach against two massive incumbents–smartphone vendors Nokia and BlackBerry.

Nokia, with the largest market share, made the device, open-source OS, app store and allowed the loading of apps from third-party app stores–if you know how.

BlackBerry, with the next largest market share, also made the device, proprietary OS and app store. They didn't officially allow loading of apps from third-party app stores, but like Nokia, the process was convoluted anyway.

Each had a lot of similarities to Apple's integrated approach of software, hardware and services, but Nokia and BlackBerry lost to Apple, not because they were a monopoly (they were the underdog), but because the market valued Apple's interpretation of what an integrated experience should look like more than theirs. Simple market economics is at play here, nothing else.

Users who value the existence of a viable third-party app store ecosystem over an integrated experience can always use Google Android. Anyone else who feels left out, can always seek VC funding to start an Android or iPhone killer, and let the market decide.

]]>