1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
|
sacredheart-selfhosted
======================
An Ansible framework for selfhosted infrastructure, based on
[Rocky Linux](https://rockylinux.org/) and [FreeIPA](https://www.freeipa.org/).
## What is this?
This is a collection of Ansible roles for provisioning self-hosted applications,
configuring Linux infrastructure, and managing Linux desktops.
I use these Ansible playbooks to manage my entire digital footprint, including
baremetal servers, virtual machines, and desktop workstations.
Although this project is not intended to be a turn-key appliance, it should be
easy to adapt it to your own environment with some basic sysadmin skills. I've
provided an example inventory to get you started.
## Features
Modular Ansible roles are used to create virtual machines, apply common base
configuration, and configure each service. A summary of the included roles is
provided in the table below. For a complete listing, check out the [roles](roles)
directory.
Each role has its own README file that describes how it should be invoked and what
variables it accepts.
Role | Description
--------------------------------------------|-------------
[proxmox\_instance](roles/proxmox_instance) | Automatically provisions a [Proxmox](https://www.proxmox.com/) VM with the given hardware and cloud-init configuration
[common](roles/common/meta/main.yml) | Meta-role that pulls in common configuration roles (local repos, freeipa client, DNS/NTP, SSH keys, etc)
[freeipa\_server](roles/freeipa_server) | [FreeIPA](https://www.freeipa.org/) provides provides identity management, access control, certificate management, and single sign-on
[yum\_mirror](roles/yum_mirror) | Mirrors all package repositories locally
[dnf\_automatic](roles/dnf_automatic) | Automatically updates packages, performs reboots and service restarts when necessary
[rsyslog\_server](roles/rsyslog_server) | Centralized syslog storage using [Rsyslog](https://www.rsyslog.com/)
[postfix\_server](roles/postfix_server) | Public-facing mail server using [Postfix](https://www.postfix.org/)
[dovecot](roles/dovecot) | [Dovecot](https://www.dovecot.org/) IMAP server, with full text and attachment search
[rspamd](roles/rspamd) | [Rspamd](https://rspamd.com/) spam filtering system
[sabredav](roles/sabredav) | [sabre/dav](https://sabre.io/) CalDAV and CardDAV server with custom FreeIPA integration
[prosody](roles/prosody) | [Prosody](https://prosody.im/) XMPP server
[synapse](roles/synapse) | [Matrix](https://matrix.org/) homeserver
[mastodon](roles/mastodon) | [Mastodon](https://joinmastodon.org/) server
[gitolite](roles/gitolite) | Git repository with [Gitolite](https://gitolite.com/gitolite/index.html) access control
[cgit](roles/cgit) | [cgit](https://git.zx2c4.com/cgit/) web frontend for Git
[vaultwarden](roles/vaultwarden) | [Bitwarden-compatible password manager](https://github.com/dani-garcia/vaultwarden)
[ttrss](roles/ttrss) | [Tiny Tiny RSS](https://tt-rss.org/) feed aggregator
[mediawiki](roles/mediawiki) | [MediaWiki](https://www.mediawiki.org/) wiki platform
[jellyfin](roles/jellyfin) | [Jellyfin](https://jellyfin.org/) media system
[invidious](roles/invidious) | [Invidious](https://invidious.io/) open source YouTube frontend
[nitter](roles/nitter) | [Nitter](https://github.com/zedeus/nitter) open source Twitter frontend
[teddit](roles/teddit) | [Teddit](https://codeberg.org/teddit/teddit) open source Reddit frontend
[hastebin](roles/hastebin) | [Hastebin](https://github.com/toptal/haste-server) open source pastebin
[psitransfer](roles/psitransfer) | [PsiTransfer](https://github.com/psi-4ward/psitransfer) public file sharing
[nfs\_server](roles/nfs_server) | Configures ZFS datasets, NFS exports, SMB shares, ACLs, and autofs maps.
[syncthing](roles/syncthing) | Per-user [Syncthing](https://syncthing.net/) instances that sync files to your NFS home directory
[asterisk](roles/asterisk) | [Asterisk](https://www.asterisk.org/) PBX for VOIP phones
[nsd](roles/nsd) | Authoritative DNS server
[nagios\_server](roles/nagios_server) | Monitors all hosts and services, automatically generated configuration
[znc](roles/znc) | [ZNC](https://znc.in/) IRC bouncer
[cups\_server](roles/cups_server) | Centralized network printing
[unifi](roles/unifi) | [UniFi](https://www.ui.com/) controller for managing Ubiquiti access points
[freeradius](roles/freeradius) | WPA Enterprise authentication for WiFi using FreeIPA credentials or SSL certificates
## Design Choices
### Ansible
All configuration is performed using Ansible playbooks. The Ansible [playbooks](playbooks)
are just the orchestration layer for the modular Ansible [roles](roles),
which do all the heavy lifting. All host metadata is stored in the [inventory](inventory-example).
Why Ansible? Ansible is awful. And full of YAML. And SLOW.
Unfortunately, it seems to be the least awful option out there:
1. No agent. If you can SSH to a host you can configure it with Ansible.
2. An insane number of community modules (admittedly, of varying quality...)
3. Easy escape hatches. If you're knee deep in a YAML tarpit, it's easy to
write a custom Python filter or module to do what you want.
4. There's an extremely convenient [FreeIPA project](https://github.com/freeipa/ansible-freeipa)
that lets you manage your entire FreeIPA domain with Ansible.
When you self-host, 80% of your effort is figuring out exactly what arcane Unix
incantations are needed to make the stupid $THING work in a reproducible way.
And since Ansible is ultimately just a YAML listing of tasks to execute, it
provides a nice self-documenting way of doing that.
Ansible has very little magic, so the YAML is often obtuse. But a positive
side of its stupid simplicity is that anyone can look at the task files and
easily deduce exactly what steps are needed to configure a given thing.
### Rocky Linux
All the roles are designed for [Rocky Linux 9](https://rockylinux.org/) (a
few roles still require version 8 due to package availability).
Why Rocky Linux? Mostly due to the 10-year support cycle. I've been self-hosting
for a long time, and I'm now at a stage of life where fiddling with config files
to keep up with frequent version updates is no longer enjoyable. I take comfort
in knowing that I can coast on this setup for a decade before I need to get in
the weeds again.
I don't have any strong opinions about Rocky vs Alma; at the time, I thought
Rocky Linux had a cooler logo 😎
All my hosts run with SELinux **enabled**, because I dislike [making Dan Walsh weep](https://stopdisablingselinux.com/).
Often this required writing my own SELinux modules, but I made it work.
I chose a RedHat-based distro for the first-class FreeIPA support.
### FreeIPA
I use FreeIPA for 100% of authentication and authorization logic, and Kerberos/GSSAPI
for single sign-on (where possible).
All my desktop computers also run Rocky Linux, and are joined to my FreeIPA
domain. When you log in with GDM, you'll get a Kerberos ticket that is used by
[Firefox](roles/firefox) and other applications to automatically authenticate
you without having to type your password again.
For services that don't support Kerberos (or devices that don't support it, like
smartphones), everything falls back to username/password authentication over TLS.
Authorization is performed using FreeIPA group memberships. This is especially
handy since FreeIPA supports nested groups. For example, everyone in my family
is a member of the FreeIPA group `mylastname`. If I want to grant them access
to `myapp`, I'll make a FreeIPA group called `role-myapp-access`, and then add
the `mylastname` group as a member.
FreeIPA is also used to provision TLS certificates for all internal hosts. For
non-managed devices like smartphones, you'll have to install the local FreeIPA
Root CA. (There is also a [certbot role](roles/certbot) for public-facing
services.)
### KVM Virtual Machines
Each of my applications runs on a dedicated Proxmox KVM virtual machine. The
[common](roles/common) role spins up a dedicated [Proxmox instance](roles/proxmox_instance)
on the fly when configuring a new VM for the first time.
You can certainly use any of the included roles on non-Proxmox hosts, and they
will work fine. Proxmox is just what I decided to use for my own homelab.
Why Proxmox? It's free, and I was already familiar with it.
I still run everything on KVM virtual machines. I briefly looked into Linux
containers, but decided against them. My environment relies heavily on NFS and
automount, both of which only barely work within containers at this point.
There's no Docker whatsoever in this project. Everything is installed from
official repos or [EPEL](https://docs.fedoraproject.org/en-US/epel/), and
and managed using plain old systemd units. For services that lack official RPMs,
the software is built locally from the upstream source repository during the
playbook.
### Networking
Each role that exposes a network service uses `0.0.0.0` for all available
interfaces.
It is assumed that you already have a working network. Other than setting VLAN
tags and `cloud-init` IP configuration for virtual machines, none of the playbooks
in this project touch your network infrastructure.
For my homelab, I don't expose anything to the internet unless absolutely
necessary. I run an [OPNsense](https://opnsense.org/) firewall and configure all
mobile devices with a persistent Wireguard VPN back to my intranet.
You can configure your network and VLANs however you see fit. I actually run everything
from a small rack of used eBay gear in my basement! I have a residential cable
internet connection, with a block of static IPv4 addresses from my ISP, and it
works fine for everything so far.
I use RFC1918 local IP addresses for all my VMs. For services that need to be publicly
accessible, like [SMTP](roles/postfix_server), [Asterisk](roles/asterisk), and [XMPP](roles/prosody),
I add a static IP alias to the WAN interface of my firewall and use a [1:1 NAT](https://docs.opnsense.org/manual/nat.html#one-to-one)
mapping.
### Monitoring
I use [Nagios](roles/nagios_server). I know. I KNOW! I'm sorry.
It's honestly perfect for my use case. I have a bunch of static VMs that once
built, basically never change. The [configs](roles/nagios_server/templates/etc/nagios/objects)
are all generated automatically from my Ansible inventory, and I get an email
whenever something goes wrong.
I don't use Nagios for any metrics gathering--only health checks. In addition
to the usual ping/disk usage/load/network interface/certificate validity checks,
I also have a few custom plugins that check for [failed systemd units](roles/nagios_client/files/usr/lib64/nagios/plugins/check_systemd),
[dead asterisk endpoints](roles/nagios_server/files/usr/lib64/nagios/plugins/check_asterisk_endpoints),
and other random stuff.
### Backup and Restore
In my environment, periodic backups are performed by the [archiver](roles/archive_server).
Basically, applications run periodic [archive jobs](roles/archive_job) that
write data to `/var/spool/archive`, and a special process `rsync`'s this data
to a central location each night.
In addition, [backup](playbooks/util/backup.yml) and [restore](playbooks/util/restore.yml)
playbooks are provided.
The [backup playbook](playbooks/util/backup.yml) will export all transient state
for each service (email inboxes, FreeIPA domain, PostgreSQL databases, etc) to
tarballs on the Ansible controller.
The [restore playbook](playbooks/util/restore.yml) will safely perform the
service-specific tasks necessary to restore each backup.
In a disaster recovery scenario, I can rebuild all my VMs from scratch using the
[site.yml playbook](playbooks/site.yml) playbook. Then, I can use the [restore playbook](playbooks/util/restore.yml)
(along with my most recent backup) to restore all the data for each service.
|
