Skip to content

First deployment on Hop3

This runbook takes you from an empty server to production, with Umami's and Matomo's history imported. Each step says how to check it worked. Steps marked (Hop3) use the hop3 CLI from your machine; where this document doesn't give the exact command, hop3 help <command> does: Hop3's commands take the app with --app abilian-analytics, never as a positional argument.

0. Before you start

  • DNS for stats.hop3.abilian.com, the app's hostname ([domains] in hop3.toml), pointing at the Hop3 server. The tracker is served from it, so it must stay stable: the snippets name it.
  • Access to Umami's PostgreSQL database and Matomo's MySQL or MariaDB database, to take dumps (step 5).
  • The list of projects you want, with the hostnames each one claims. Umami has one "website" per hostname; here a project groups several (e.g. abilian.com claiming *.abilian.com takes lab.abilian.com and the econ lab). Decide before the imports: they file the history into the projects that exist.
  • Locally, everything green: make lint, make test, make test-postgres, and hop3check (says clean).

1. Deploy (Hop3)

From the project directory, deploy the app: make deploy (hop3 deploy; check hop3 help deploy). hop3.toml declares the rest:

  • the hostname, stats.hop3.abilian.com, served over HTTPS;
  • a PostgreSQL addon, which sets the DATABASE_URL the app requires;
  • a generated SECRET_KEY;
  • alembic upgrade head before each start;
  • one process, myxo serve layout.toml, which serves the web app and runs the background jobs;
  • the geoip volume, where the GeoIP databases persist across deploys;
  • a health check on /health.

Check:

  • hop3 status --app abilian-analytics: the web process is up.
  • https://stats.hop3.abilian.com/health answers {"status":"ok"}.
  • hop3 logs --app abilian-analytics: the app logged two lines, geoip country: https://download.db-ip.com/… and geoip asn: https://download.db-ip.com/…. If it logged jobs.geoip failed instead, the server can't reach download.db-ip.com: fix the outbound network, then run abilian-analytics geoip update; the job also tries again daily.

2. The first account

Run it in the app's environment on the server, quoting the whole command:

hop3 run --app abilian-analytics 'abilian-analytics user create you@abilian.com --superuser --generate'

Every abilian-analytics … command below runs the same way.

It prints a random password, once. Sign in at https://stats.hop3.abilian.com/auth/login, then change the password on your account page (top right menu → Password and API token).

Other people's accounts: Administration → Users (a generated password is shown once, to pass on).

Check: Administration → Instance health. Expect green for GeoIP countries, GeoIP networks, daily salts and the database schema; "No hit received yet" for collection is normal at this point.

3. Projects

Projects → New project, one per project decided in step 0, with the hostnames it claims (one per line; example.com also covers www.example.com; *.example.com covers all subdomains, not example.com itself, which needs its own line).

4. The tracking snippet

The Tracking page of each project's settings shows it; it is the same for all sites:

<script defer src="https://stats.hop3.abilian.com/t.js"></script>

Add it to each site, next to Umami's (keep Umami running for now, step 6).

  • Sites with a Content-Security-Policy must allow the app's origin in script-src (the script) and connect-src (the hits).
  • Static sites: in the base template. Single-page apps: route changes are counted automatically.

Check, for each site, after a visit in a normal browser window:

  • the project's report shows the visit within seconds ("Online now");
  • the visit has a country, and a network, in Visits. If every visit shows the same network, the server's own, the app doesn't see visitors' addresses: Hop3's nginx must append the client address to X-Forwarded-For, and the app reads the right-most entry.
  • Nothing shows? Administration → Unclaimed hostnames: a hostname listed there sends hits but no project claims it; attach it. Still nothing: the browser's developer tools show whether t.js loaded and whether the POST to /t was blocked (CSP, ad blocker).

5. Umami's and Matomo's history

Umami first: on each site, Matomo's history stops where Umami's starts, as Umami's stops where our tracker's starts. Each import takes a dry run, then the real one.

  1. Dump Umami's database, as plain SQL (about 250 MB today):
    pg_dump --format=plain --no-owner <umami database> > umami.sql
    
  2. Dump Matomo's tables, on the Matomo server. Its config/config.ini.php gives, under [database], the database's name (dbname), the account to use and the tables' prefix (tables_prefix, matomo_ here):
    mysqldump -u <user> -p --single-transaction --quick --hex-blob --no-tablespaces \
      <dbname> matomo_site matomo_log_visit matomo_log_link_visit_action matomo_log_action \
      | gzip > matomo.sql.gz
    
  3. Copy both to the server, somewhere the app's user can read (e.g. scp umami.sql matomo.sql.gz <server>:/tmp/).
  4. A dry run, which changes nothing and reports what it would do:
    hop3 run --app abilian-analytics 'abilian-analytics import umami /tmp/umami.sql --owner you@abilian.com --dry-run'
    
    Read the report: the projects it would create (one per website that no project claims), the visits and events it would import, per hostname, and what it would skip (hits after the first of a source that comes before, hostnames no project claims). If it would create a project you don't want, add that hostname to the right project (step 3) and run the dry run again. If a hostname you want is listed as claimed by no project (Matomo's websites without a main URL), add it to a project and run the dry run again.
  5. Then for real (without --dry-run). It takes a few minutes; if hop3 run gives up before, run it on the server in the background, with nohup … > import.log 2>&1 &.
  6. Steps 4 and 5 again for Matomo: abilian-analytics import matomo /tmp/matomo.sql.gz --owner you@abilian.com.
  7. Then fetch the projects' icons: hop3 run --app abilian-analytics 'abilian-analytics icons fetch'.
  8. Delete the dumps, from the server and wherever else they were copied: they hold the tools' raw data, and Matomo's the visitors' IP addresses.

Each import:

  • applies today's filtering rules on what the tools kept: the browser's name (headless Chrome, which Matomo counts, is left out), the country, the referrer; not the network, as there is no IP address;
  • stops, for each hostname, at the first hit of a source that comes before it, so nothing is counted twice;
  • can be run again: it replaces what the previous import from that tool brought in. Importing Umami again later, with more recent data, leaves Matomo's history as it is.

Check: the overview shows history before today; compare a few projects' monthly visits with Umami's and Matomo's over the same range. Expect ours to be somewhat lower: the tools count some of what we leave out (spam referrers, blocked countries, headless browsers).

6. Two weeks side by side

Both trackers run. Compare the figures from time to time: the gap should be explained by what Excluded traffic shows (bots, hosting networks, spam). A network that looks like a hosting provider among the counted ones: leave it out from the Excluded traffic page (for the project, or for every project, as an administrator).

Then remove Umami's snippet from the sites, and stop Umami. Keep its database (or the dump) until you no longer need to compare.

7. Then

Day to day covers upgrades, backups and the instance's health; troubleshooting, what to do when something looks off.