First deployment on Hop3¶
This runbook takes you from an empty server to production, with Umami's and Matomo's history imported. Each step says how to check it worked. Steps marked (Hop3) use the hop3 CLI from your machine; where this document doesn't give the exact command, hop3 help <command> does: Hop3's commands take the app with --app abilian-analytics, never as a positional argument.
0. Before you start¶
- DNS for
stats.hop3.abilian.com, the app's hostname ([domains]inhop3.toml), pointing at the Hop3 server. The tracker is served from it, so it must stay stable: the snippets name it. - Access to Umami's PostgreSQL database and Matomo's MySQL or MariaDB database, to take dumps (step 5).
- The list of projects you want, with the hostnames each one claims. Umami has one "website" per hostname; here a project groups several (e.g. abilian.com claiming
*.abilian.comtakes lab.abilian.com and the econ lab). Decide before the imports: they file the history into the projects that exist. - Locally, everything green:
make lint,make test,make test-postgres, andhop3check(saysclean).
1. Deploy (Hop3)¶
From the project directory, deploy the app: make deploy (hop3 deploy; check hop3 help deploy). hop3.toml declares the rest:
- the hostname,
stats.hop3.abilian.com, served over HTTPS; - a PostgreSQL addon, which sets the
DATABASE_URLthe app requires; - a generated
SECRET_KEY; alembic upgrade headbefore each start;- one process,
myxo serve layout.toml, which serves the web app and runs the background jobs; - the
geoipvolume, where the GeoIP databases persist across deploys; - a health check on
/health.
Check:
hop3 status --app abilian-analytics: thewebprocess is up.https://stats.hop3.abilian.com/healthanswers{"status":"ok"}.hop3 logs --app abilian-analytics: the app logged two lines,geoip country: https://download.db-ip.com/…andgeoip asn: https://download.db-ip.com/…. If it loggedjobs.geoip failedinstead, the server can't reachdownload.db-ip.com: fix the outbound network, then runabilian-analytics geoip update; the job also tries again daily.
2. The first account¶
Run it in the app's environment on the server, quoting the whole command:
hop3 run --app abilian-analytics 'abilian-analytics user create you@abilian.com --superuser --generate'
Every abilian-analytics … command below runs the same way.
It prints a random password, once. Sign in at https://stats.hop3.abilian.com/auth/login, then change the password on your account page (top right menu → Password and API token).
Other people's accounts: Administration → Users (a generated password is shown once, to pass on).
Check: Administration → Instance health. Expect green for GeoIP countries, GeoIP networks, daily salts and the database schema; "No hit received yet" for collection is normal at this point.
3. Projects¶
Projects → New project, one per project decided in step 0, with the hostnames it claims (one per line; example.com also covers www.example.com; *.example.com covers all subdomains, not example.com itself, which needs its own line).
4. The tracking snippet¶
The Tracking page of each project's settings shows it; it is the same for all sites:
Add it to each site, next to Umami's (keep Umami running for now, step 6).
- Sites with a Content-Security-Policy must allow the app's origin in
script-src(the script) andconnect-src(the hits). - Static sites: in the base template. Single-page apps: route changes are counted automatically.
Check, for each site, after a visit in a normal browser window:
- the project's report shows the visit within seconds ("Online now");
- the visit has a country, and a network, in Visits. If every visit shows the same network, the server's own, the app doesn't see visitors' addresses: Hop3's nginx must append the client address to
X-Forwarded-For, and the app reads the right-most entry. - Nothing shows? Administration → Unclaimed hostnames: a hostname listed there sends hits but no project claims it; attach it. Still nothing: the browser's developer tools show whether
t.jsloaded and whether the POST to/twas blocked (CSP, ad blocker).
5. Umami's and Matomo's history¶
Umami first: on each site, Matomo's history stops where Umami's starts, as Umami's stops where our tracker's starts. Each import takes a dry run, then the real one.
- Dump Umami's database, as plain SQL (about 250 MB today):
- Dump Matomo's tables, on the Matomo server. Its
config/config.ini.phpgives, under[database], the database's name (dbname), the account to use and the tables' prefix (tables_prefix,matomo_here): - Copy both to the server, somewhere the app's user can read (e.g.
scp umami.sql matomo.sql.gz <server>:/tmp/). - A dry run, which changes nothing and reports what it would do: Read the report: the projects it would create (one per website that no project claims), the visits and events it would import, per hostname, and what it would skip (hits after the first of a source that comes before, hostnames no project claims). If it would create a project you don't want, add that hostname to the right project (step 3) and run the dry run again. If a hostname you want is listed as claimed by no project (Matomo's websites without a main URL), add it to a project and run the dry run again.
- Then for real (without
--dry-run). It takes a few minutes; ifhop3 rungives up before, run it on the server in the background, withnohup … > import.log 2>&1 &. - Steps 4 and 5 again for Matomo:
abilian-analytics import matomo /tmp/matomo.sql.gz --owner you@abilian.com. - Then fetch the projects' icons:
hop3 run --app abilian-analytics 'abilian-analytics icons fetch'. - Delete the dumps, from the server and wherever else they were copied: they hold the tools' raw data, and Matomo's the visitors' IP addresses.
Each import:
- applies today's filtering rules on what the tools kept: the browser's name (headless Chrome, which Matomo counts, is left out), the country, the referrer; not the network, as there is no IP address;
- stops, for each hostname, at the first hit of a source that comes before it, so nothing is counted twice;
- can be run again: it replaces what the previous import from that tool brought in. Importing Umami again later, with more recent data, leaves Matomo's history as it is.
Check: the overview shows history before today; compare a few projects' monthly visits with Umami's and Matomo's over the same range. Expect ours to be somewhat lower: the tools count some of what we leave out (spam referrers, blocked countries, headless browsers).
6. Two weeks side by side¶
Both trackers run. Compare the figures from time to time: the gap should be explained by what Excluded traffic shows (bots, hosting networks, spam). A network that looks like a hosting provider among the counted ones: leave it out from the Excluded traffic page (for the project, or for every project, as an administrator).
Then remove Umami's snippet from the sites, and stop Umami. Keep its database (or the dump) until you no longer need to compare.
7. Then¶
Day to day covers upgrades, backups and the instance's health; troubleshooting, what to do when something looks off.