Nix

Stay frosty like Tony

A home for my system configurations and home lab using Nix Flakes.

Be warned, I'm still learning and experimenting.

Features and Todo

Bedrock (Networking)

Features:

• ACME certificates for TLS on services
• Reverse proxy with mTLS protection to foundational services
• Private CA TLS certificates for direct service access (k8s, OPNsense, Proxmox, access point)
• Wireguard VPN remote access
• Unencrypted client DNS trapped, filtered, and upgraded to DoT
• Malicious traffic dropped with dynamic list updates
• DDNS
• QEMU guest agent
• Prometheus metrics export
• Wake-on-LAN GUI
• Configuration backed up to two locations

Todo:

• Test local DNS from VPNs
• Host an authoritative DNS
• Find a DDNS provider that supports the generic update mechanism, not a proprietary API. Switch to Inadyne DDNS client for that?
• Host authoritative DNS server, maybe Hickory. See ns-global for some resiliency.
• Review net.inet.tcp.tso for VM safety/perf
• Add dNAT port forwarding for Proxmox managment GUI from 443 to 8006
• Tune Wireguard
• Add IPsec
• Fix/add OpenVPN ref ref
• Figure out why DNAT of DNS traffic to loopback doesn't work and has to be LAN IP address
• Figure out how to make the configuration work when the v6 prefix changes
• Add compatibility option/translation layer for IPv6->IPv4
• Remove IPv4
• See about getting my own AS and IPv6 prefix
• Maybe Tailscale OPNsense Seems to be more peer-to-peer and rather point-to-site suits me here.

Substratum (Virtualization and Systems)

Features:

• iPXE/TFTP with Netboot for multi-option
• Automatic disk resize on NixOS VMs
• Mix of VMs and physical nodes
• Standardized configuration and configuration management of all nodes

Todo:

• Convert nodes to use ssh certificates for client authentication and server certificates instead of TOFU
• Swap my user to a lower privilege one on Proxmox and OPNsense
• See about more modern watchdog options - apparently this one is ancient 32 bit PCI
• Debug watchdog not stopping on control node reboot.
• Either stabilize or hardware watchdog Topton N100
• Work out watchdog on OPNsense/BSD
• Set up OpenAMT for out-of-band management.
• See about TFTP on IPv6. Issue 1 Issue 2

Subsoil (Foundational Services)

Features:

• Caddy reverse proxy
• Prometheus+Alertmanager+Grafana monitoring stack
• Garage S3 cluster
• Valheim server
• Nix binary cache
• Kanidm identity management
• Step-CA certificate authority
• OIDC SSO for Grafana and Step-CA

Todo:

• Advanced monitoring (Mimir, Tempo, Loki, Trickster, Victoria Metrics, etc)
• Add rules for k8s apiserver, maybe mixins
• Configure what can be for Otel
• Spire for node identity
• Stop Spire agent dying if stale join token
• Secrets (Vault/OpenBao?)
• More identity integration. Done:
  • Grafana To-do:
  • Proxmox (may be limited to authentication)
  • OPNsense (LDAP only) Not possible:
  • Garage
• Switch routing to dynamic subdomains.
• Add Uptime Kuma publicly
• Deploy external dead man's switch and route Alertmanager to it.
• Find a nice way to make foundational services upstream in Nginx config either nicer or subsume it.
• Look into different Nix store cache, maybe Attic or Harmonia

Topsoil (Kubernetes)

Features:

• Custom, bare-metal deployment configuration
• Private CA certificates
• Single-stack IPv6 with native routing/no overlay
• Dynamic BGP peering of nodes with router/OPNsense
• CoreDNS inside cluster
• OIDC SSO

Todo:

• Setup External-DNS operator
• Setup Step-CA as a certificate provider for Cert-Manager
• Cluster-wide backup system, Velero?
• Do dynamically-delegated prefixes for node pod CIDRs. Honestly I'm not sure this is a value-add but it would be cool. See diagram below.
• Set up IPv6 public ingress and firewalling
• Enable k8s native tracing
• Enable k8s native secret encryption
• Enable KubeletPSI feature
• Use the kubernetes mkCert and mkKubeConfig functions example
• Look into kubernetes managing it's own etcd+cluster CAs.
• Check Kubelet cert rotation is working docs link
• See about CSR auto-approval project
• Add WASM runtime
• Find some kind of dynamic PV/storage option. I'm thinking Longhorn. post 1 post 2 Maybe OpenEBS.
• Play around with Timoni, Kluctl, etc
• Add tracing endpoint for Containerd, maybe monitor better article prom docs
• "Package" an app using generic Helm charts
• Amend systemd-resolved to prefer (or only) use v6 DNS upstream. Issue
• Write a custom cloud provider using SSH and WoL.
• Adjust the custom cloud provider to use OpenAMT.
• Pull k8s module out into it's own flake/repo/overlay?
• Add SLOs to service monitoring sloth
• Use sig-addonmanager to bootstrap a CD tool and a CNI? Seems like a pretty unused subproject and I don't mind for home leaving a couple manual steps.

• Cilium with OpnSense blog
• k8s setup with BGP and /64s

Organics (Applications and nice-to-haves)

• Look into buildEnv over devShell
• Get a container image build with nix going Jamey blog Amos's example

Implementation Notes

See also:

• DNS
• CoreDNS
• Cilium
• MacOS

Bedrock

Substratum

Pre-requisites:

• NixOS flashed to USB

HP EliteDesk 800 G3 Micro/Mini.

1. Mash F10/Esc to hit the bios (this was a thowback and a pain to do). Or just use systemctl reboot --firmware-setup ~ Future Ariel.
2. Update the BIOS
3. Load the settings from HpSetup.txt OR follow along the rest.
4. Configure the following
   • Ensure legacy boot is enabled.
   • I disabled secure boot and MS certificate in case
   • Turn off fast boot (might be optional)
   • Add boot delay 5 seconds (purely QoL)
   • Ensure USB takes priority over local disk
   • I disabled prompt on memory change so if I add RAM later I don't have to displace the system.
   • I disabled Intel's sgx or whatnot. Don't trust it after the RST debacle.
5. Save and reboot
6. Hit escape to select boot option of USB (esc maybe not required)
7. Follow the instructions to install NixOS
   • 23.05 (but higher is fine)
   • User nixos
   • Same password for root
   • Auto login (QoL but consult your threat model)
8. Use nix-shell to obtain Git and Helix
9. Clone this flake repo from github
10. Copy the machine-specific disk config from /etc/nixos/hardware-configuration.nix. Place it in the machine's hardware-configuration.nix in the flake repo. (This step may no longer be necessary)
11. Nix rebuild switch to the flake's config.
12. Confirm SSH remote access is working.
13. Reboot and enter bios.
    • Turn fast boot back on
    • Set boot delay to 0
    • Disable UEFI boot priority. If we need to boot from USB we'll reenter the BIOS.
14. Save BIOS changes and one last confirmation that the system boots and is remotable.
15. Move the machine to it's final home.
16. Remotely retrieve the hardware configuration and commit it to the flake repo.

Topton N100 (CW-AL-4L-V1.0 N100)

1. Download BIOS update and place on Ventoy USB.

2. Mash F10 to enter BIOS, boot update. Enter 1 and it should update.

3. Reset and wait, it will beep and hang and reboot but eventually it should come good.

4. Set USB boot precendence above internal drive/s

5. Boot Proxmox installer and walk through Set static IP with netmask as same as router's DHCP netmask. Best I can tell this is required to send traffic back to origin.

6. Highly recommended but optionally, trust your SSH keys. curl https://github.com/arichtman.keys >> ~/.ssh/authorized_keys

7. Optionally, add static DHCP lease to the router. If you do this, you can also optionally remove the fixed interface configuration. Edit /etc/network/interfaces and switch the virtual bridge network configuration from manual to dhcp.

8. Optionally, install trusted certificates. Instructions are on my blog.

9. Run some of the proxmox helper scripts At least the post install one to fix sources. I also ran the microcode update, CPU scaling governor, and kernel cleanup (since I had been operating for a while).

10. Enable IOMMU. First, check GRUB/systemd efibootmgr -v. If GRUB, sed -i -r -e 's/(GRUB_CMDLINE_LINUX_DEFAULT=")(.*)"/\1\2 intel_iommu=on"/' /etc/default/grub

11. echo 'vfio vfio_iommu_type1 vfio_pci vfio_virqfd' >> /etc/modules

12. Reboot to check config

13. Set BIOS settings:

    • Boot:
      • Disable beep
      • Enable fast boot
      • Enable network stack
    • Chipset:
      • PCH-IO:
        • Enable Wake on lan and BT
        • Enable TCO timer

14. Install Prometheus node exporter, apt install prometheus-node-exporter.

15. Install Avahi daemon to enable mDNS, apt install avahi-daemon.

16. Install grub package so actual grub binaries get updates, apt install grub-efi-amd64.

17. Optionally comment out the Cron job on reboot that sets it to power save.

18. Disable IPMI service since we don't have support, systemctl disable openipmi.

19. Optionally install the PVE Prometheus exporter.

If I check /etc/grub.d/000_ proxmox whatever it says update-grub isn't the way and to use proxmox-boot-tool refresh. It also looks like there's a specific proxmox grub config file under /etc/default/grub.d/proxmox-ve.cfg. I don't expect it hurts much to have iommu on as a machine default, and we're not booting anything else... Might tidy up the sed config command though. Looking at the systemd we could probably do both without harm. That one's also using the official proxmox command.

References:

• Proxmox package repo docs
• Servethehome net passthru tutorial
• Reddit BIOS post
• Actual BIOS download
• Grub forum post
• Arch wiki on CPU scaling
• Proxmox performance tuning
• Proxmox CPU selection tutorial
• PVE Prometheus Exporter install
• PVE Prom Exporter TLS verify issue

Proxmox Disk Setup

We did run mkfs -t ext4 but it didn't allow us to use the disk in the GUI. So using GUI we wiped disk and initialized with GPT.

For the USB rust bucket we found the device name with fdisk -l. Then we mkfs -t ext4 /dev/sdb, followed by a mount /dev/sdb /media/backup. Never mind, same dance with the GUI, followed by heading to Node > Disks > Directory and creating one.

Use blkid to pull details and populate a line in /etc/fstab for auto remount of backup disk. Ref

/etc/fstab:

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
UUID=C61A-7940 /boot/efi vfat defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
UUID=b35130d3-6351-4010-87dd-6f2dac34cfba /mnt/pve/Backup ext4 defaults,nofail,x-systemd.device-timeout=5 0 2

Well-known IPv6 Addressing

Setting this MAC will result in a tasteful link-local IP, fe80::b00b:1eff:fee1:900d.

/etc/network/interfaces:

iface vmbr0 inet static
        hwaddress ether b2:0b:1e:e1:90:0d

Re-IDing a Proxmox VM

I used this to shift OPNsense to 999 and any templates to >=1000.

1. Stop VM
2. Get storage group name lvs -a
3. Rename disk lvrename prod vm-100-disk-0 vm-999-disk-0
4. Enter /etc/pve/nodes/proxmox/qemu-server
5. Edit conf file to use renamed disk.
6. Move conf file to new id

• Proxmox vmid change knowledge base article

Virtual node disk resize

nix-shell -p cloud-utils
growpart /dev/sda 1
resize2fs /dev/sda1

Opnsense

VM Setup

1. Download iso and unpack
2. Move iso to /var/lib/vz/template/iso
3. Create VM with adjustments: I'm trying 2 cores now, utilization was low but we had spikes which I suspect were system stuff
   • Start at boot
   • SSD emulation, 48GiB
   • 1 socket+ 2 cores, NUMA enabled
   • 2048 MiB RAM
4. Use proxmox under datacenter to configure a backup schedule. The following should keep a rolling 4-weekly history.
   • Sunday 0100
   • Notify only on fail
   • Keep weekly 4
5. Boot machine and follow installer
6. Add PCIe ethernet controllers

Base OS Setup

1. Boot system and root login
2. Assign WAN and LAN interfaces to ethernet controllers
3. Check for updates either opkg update or maybe system > firmware
4. Add static DHCP leases for any machines using static IPs so Upbound will serve records for them
5. Install an intermediate cert and it's corrosponding bundle under system > trust
6. Switch to using the TLS certificate under System > Settings > Administration
7. Set both interfaces to delete protected and IPv6 SLAAC
8. Under System > General:
   • set hostname
   • set domanin
   • configure DNS servers
   • Disallow DNS override on WAN
9. Reporting > netflow set capture on
10. System > settings > cron
    • once daily to update the block lists
    • once weekly after the backup is taken (this ensures we can restore)

Tuning

• Follow Ben Tasker's stuff

Set tunable kernel.ipc.maxsockbuf to 33554432 (2 * 16777216 - the failing requested amount). Ref

DNS Configuration

Local Resolver

1. Configure Upbound DNS service
   • enable DNSSEC
   • enable DHCP lease registration No longer works with Kea DHCP and ISC DHCP is deprecated.
   • Disallow system nameservers in DoT and add records with blank domains+port 853
   • Enable blocklists
     • OIDS Ads
     • Steven Black
     • Hagezi Multi Pro++
   • Enable data capture
2. Firewall trap DNS. Create a NAT port-forward:
   • LAN interface
   • IPv4+6
   • TCP+UDP
   • Invert
   • Destination LAN net
   • from dns to dns
   • Redirect target Localhost:53
3. Test the trap.
   • DNS redirection:
     1. Unbound host override bing.com to something
     2. Check this returns the override dig +trace @4.4.4.4 bing.com
     3. Revert the override
   • Ad blocking
     1. Grab some addresses from one of the blocklists manually
     2. Attempt to resolve them

Authoritative Nameserver

** Under construction **

• If putting the nameserver on it's own domain, we need glue records. OnlyDomains insists on 2 NS entries and only supports IPv4. We may need to do this with a subdomain.
• If using another domain, we can have 2 x addresses that are both DDNSed on the secondary domain. So we might DDNS ns-{0,1}.richtman.dev, then have richtman.au root NS entries be those.
• Current ISP does not provision static IPv6 prefix delegations, but the allocation is fairly static and our v4 is already out of CGNAT so pretty stable. Without guarantees about IP stability, the glue records could fail hard if anything changed.
• Bind must be on port 53 to be a public nameserver. DHCP DNS configuration can still point to the router's IP and port 53, since we have the NAT rule redirecting it. The NAT rule will need it's destination port changed.
• It makes the most sense to have LAN devices still using Unbound, rather than Bind with recursive queries allowed.
• Small optimization is to set Unbound query forwarding for the domain to point at Bind on localhost, saves traversing from root.
• I had some issues allowlisting the LAN for recursive queries. I set it to All for now but this will need to be ironed out. Leaving a recursing resolver on the open internet is a no-go.
• Will need to add Firewall quick rule inbound on WAN to allow traffic.
• It might be possible to hook up a custom DDNS account type to hit the OPNsense API. Blog post

1. Install Bind plugin
2. Add ACLs.
   • All: 0.0.0.0/0, ::/0
   • Link-local_v6: fe80::/10
   • Dual-stack_loopback: 127.0.0.0/24, ::1/128
3. Add primary zone. Set the TTL and negative TTL low for testing. DNS server should be the FQDN of the nameserver. Allow query - All.
4. Add the minimum records for the zone to be valid.
   • NS type record at the apex (empty name), value of the zone domain.
   • AAAA type record (or A but), also at the apex, value of the public IP of the nameserver.
5. Generate new RNDC secret in CLI, save in GUI. tsig-keygen -a hmac-sha512 rndc-key
6. Add script to export the Bind RNDC secret for DynDns to use, /usr/local/etc/rc.syshook.d/config/99-bind-ddclient-key
7. Add Dynamic DNS account type nsupdatev6, with the password as /usr/local/etc/namedb/rndc.conf.key. Confirm at /usr/local/etc/ddclient.json.

• GitHub issue about default secret
• Blog about OPNsense Bind config
• Blog on BIND RNDC
• Another blog BIND RFC2136
• ddclient protocols
• GitHub comment external-dns config
• GitHub issue Bind bugs with dynamic DNS
• Tool to externally query my DNS
• Hellthread on the DDNS transition

Firewall Configuration

1. Firewall
   • Add aliases for static boxes, localhost
2. Block bad IPs
   1. Add aliases for IP lists (see references)
   2. Add firewall rules: Either floating destination FireHOL level 3 + Spamhaus OR WAN outbound FireHOL Level 1 + WAN inbound all bad IPs

• Unbound DoT tutorial
• DNS tutorial
• OPNsense forum thread
• Blocking blog post
• Fedi posts about it alt server Spamhaus

OpenVPN

Follow one of the 6000 tutorials AKA yes, I forgot to document it.

• OpenVPN setup guide

WireGuard

Follow tutorial AKA forgot to document it, see configuration file in repo.

BGP

/usr/local/etc/rc.syshook.d/start/99-frr-conf:

#!/bin/sh
# Overwrites FRR configuration and restarts it

cp ~/frr.conf /usr/local/etc/frr/frr.conf
service frr restart

~/frr.conf

Well-known IPv6 Addressing

Ideally, we'd change the MAC itself - but that doesn't seem to work. Strangely, this results in no EUI-64 link-local address.

/usr/local/etc/rc.syshook.d/early/99-ll-alias:

#!/bin/sh
# Sets an additional link-local IPv6 address on the LAN interface

ifconfig igc1 inet6 fe80::1ced:c0ff:fed0:0dad alias

Note: I did try this in the prior file but it didn't seem to take.

/usr/local/etc/rc.syshook.d/start/98-ll-route:

#!/bin/sh

# Required else the default route points to WAN interface, so LL traffic dies
route -6 change default fe80::1ced:c0ff:fed0:dad%igc1

Alacritty terminal

/usr/local/etc/rc.syshook.d/update/99-alacritty-terminal:

#!/bin/sh
# Configures terminal for Alacritty

curl -sSL https://raw.githubusercontent.com/alacritty/alacritty/master/extra/alacritty.info -o alacritty.info
infotocap alacritty.info >> /usr/share/misc/termcap
cap_mkdb /usr/share/misc/termcap
rm alacritty.info

• Blog
• Hooks docs

Nginx for Kanidm

Nginx configuration on OPNsense requires modification for Kanidm Oauth to work!

1. Inspect the generated configuration file, either via SSH or Web console
2. Locate the server block for the Kanidm HTTP server
3. There should be a line like include $UUID_post/*.conf, note the directory name.
4. Create such a directory in /usr/local/etc/nginx, and add a .conf file with the following: proxy_pass_header X-KANIDM-OPID;
5. Restart the nginx service

Plugins

• NextCloud backup, configure with an app key.
• Git backup. Create an uninitialized repository and provide API key and HTTPS URL.
• Prometheus exporter for monitoring.
• DynamicDNS client, configure with AWS Access Key.
• tftp plugin (unmaintained but workable) src. Make directory /usr/local/tftp and download netboot.xyz.kpxe. I also downloaded netboot.xyz.efi for good measure. Enable TFTP and set listening IP to 0.0.0.0. This defaulted to 127.0.0.1 which may have worked but I didn't test.
• ACME client tutorial
• Install os-wol to wake on lan. Add all physical machines to the list of known, you can use ISC DHCP leases to find all the MACs in one place.
• Optionally: themes (rebellion)

Notes:

I will revisit the resources supplied after running the box for a bit.

CPU seems fine, spikey with what I think are Python runtime startups from the control layer. RAM looks consistently under about 1Gb so I'll trim that back from the recommended minimum 2Gb. We're doing pretty well on space too but I'm less short on that.

References:

• Reddit performance comment

Disaster Recovery / Restoring from Backup

1. Access Proxmox directly
2. Visit Backup storage and confirm thre is an intact and appropriate snapshot image to restore.
3. Disable VM protection and delete it.
4. Select backup image and restore to VM ID 999. Enable start on boot and disable unique feature.

Virtualized NixOS Node Bootstrap

1. Use GUI installer to deploy, the CLI one's a pain, 12GiB storage minimum, 4 cores/8GiB ram (min 4+ GiB). We'll scale it down later, it bombs completely without plenty of RAM.
2. Thing's a pain to bootstrap and the web console is limited Sudo edit /etc/nixos/configuration.nix
   • Enable the openssh service
   • security.sudo.wheelNeedsPassword = false
   • Disable OSprober by removing the line
   • Set disk source to /dev/sda
   • I'm not sure those last 2 make much of a difference once the machine is under flake control
3. Rebuild
4. Bounce the machine so it releases using the new hostname
5. Then pull down some keys to get in, it should have already DHCP'd over the bridge network. mkdir ~~/.ssh && curl https://github.com/arichtman.keys -o ~/.ssh/authorized_keys && chmod 600 ~~/.ssh/authorized_keys
6. Upgrade the system sudo nixos-rebuild switch --upgrade --upgrade-all
7. Reboot to test
8. Clear history sudo nix-rebuild list-generations sudo rm /nix/var/nix/profiles/system-#-profile sudo nix-collect-garbage --delete-old
9. Adjust hardware down to 1/2GiB.
10. Make template

• Nixos VM tutorial

Subsoil

Trust chain setup

Arguably this mingles with substratum, as PKI/trust/TLS is required or very desirable for VPN/HTTPS etc. SPIFFE/SPIRE will address this somewhat.

1. Create root CA xkcdpass --delimiter - --numwords 4 > root-ca.pass step certificate create "ariel@richtman.au" ./root-ca.pem ./root-ca-key.pem --profile root-ca --password-file ./root-ca.pass
2. Distribute the intermediate certificates and keys
3. Secure the root CA, it's a bit hidden but Bitwarden does take attachments.
4. Publish the root CA, with my current setup this meant uploading it to s3.
5. Update the sha256 for the root certificate fetchUrl call

• Smallstep documentation
• Certificate creation/authorization tutorial

Garage setup

garage layout assign --zone garage.services.richtman.au --capacity 128GB $(garage node id 2>/dev/null)
garage layout apply --version 1

Kanidm setup

export KANIDM_URL=https://id.richtman.au
export GRAFANA_FQDN=grafana.services.richtman.au
kanidm system oauth2 create grafana $GRAFANA_FQDN https://$GRAFANA_FQDN
kanidm system oauth2 set-landing-url grafana "https://${GRAFANA_FQDN}/login/generic_oauth"

kanidm group create 'grafana_superadmins'
kanidm group create 'grafana_admins'
kanidm group create 'grafana_editors'
kanidm group create 'grafana_users'

kanidm system oauth2 update-scope-map grafana grafana_users email openid profile groups
kanidm system oauth2 enable-pkce grafana

kanidm system oauth2 update-claim-map-join 'grafana' 'grafana_role' array
kanidm system oauth2 update-claim-map 'grafana' 'grafana_role' 'grafana_superadmins' 'GrafanaAdmin'
kanidm system oauth2 update-claim-map 'grafana' 'grafana_role' 'grafana_admins' 'Admin'
kanidm system oauth2 update-claim-map 'grafana' 'grafana_role' 'grafana_editors' 'Editor'

# Note: I'm not sure I need *all* of these
kanidm group add-members grafana_superadmins arichtman
kanidm group add-members grafana_admins arichtman
kanidm group add-members grafana_editors arichtman
kanidm group add-members grafana_users arichtman

kanidm system oauth2 get grafana
kanidm system oauth2 show-basic-secret grafana

Grafana side: Official docs examples

Topsoil

• Kubernetes the hard way

Cluster access bootstrap

# Create a client certificate with admin
step certificate create cluster-admin cluster-admin.pem cluster-admin-key.pem \
  --ca ca.pem --ca-key ca-key.pem --insecure --no-password --template granular-dn-leaf.tpl --set-file dn-defaults.json --not-after 8760h \
  --set organization=system:masters
# Construct the kubeconfig file
# Here we're embedding certificates to avoid breaking stuff if we move or remove cert files
kubectl config set-cluster home --server https://fat-controller.systems.richtman.au:6443 --certificate-authority ca.pem --embed-certs=true
kubectl config set-credentials home-admin --client-certificate cluster-admin.pem --client-key cluster-admin-key.pem --embed-certs=true
kubectl config set-context --user home-admin --cluster home home-admin

Cluster access (normal)

1. Create private key openssl genpkey -out klient-key.pem -algorithm ed25519
2. Create CSR openssl req -new -config klient.csr.conf -key klient-key.pem -out klient.csr
3. export KLIENT_CSR=$(base64 klient.csr | tr -d "\n")
4. Submit the CSR to the cluster envsubst -i klient-csr.yaml | kubectly apply -f -
5. Approve the request kubectl certificate user approve

Cluster node roles

For security reasons, it's not possible for nodes to self-select roles. We can label our nodes using label.sh.

hmmm, deleting the nodes (reasonably) removes labels. ...and since they can't self-identify, we have to relabel every time. I expect taints would work the same way, so we couldn't use a daemonset or spread topology with labeling privileges since it wouldn't know what to label the node. Unless... we deploy it with a configMap? That's kinda lame. I suppose all the nodes that need this are dynamic, ergo ephemeral and workers, so we could make something like that. Heck, a static pod would work fine for this and be simple as. But then it'd be a pod, which is a continuous workload, which we really don't need. A job would suit better, but then it's like, why even run this on the nodes themselves? Have the node self-delete (it'll self-register again anyway), and have the admin box worry about admin like labelling. I wonder if there's any better way security-wise to have nodes be trusted with certain labels. Already they need apiServer-trusted client certificates, it'd be cool if the metadata on those could determine labels.

There is a way to tell the Kubelet to register with labels but it's limited to a specific group. I doubt the Kubelet has an option to open that up and since we're getting denied even starting the binary it's probably not settable on the APIserver.

Node CSRs piling up

kubectl get csr --no-headers -o jsonpath='{.items[*].metadata.name}' | xargs -r kubectl certificate approve

• GitHub comment

Notes

Checking builds manually: nix build .#nixosConfigurations.fat-controller.config.system.build.toplevel

Deploying from laptop: nixos-rebuild-ng test --build-host fc --target-host fc --flake . --sudo

Minimal install ~3.2 gigs Lab-node with master node about 3.2 gb also, so will want more headroom.

Add to nomicon

• fakesha256
• nix-prefetch-url > hash.txt

Android phone setup

Using tasker

Profile: AutoPrivateDNS
         State: Wifi Connected [ SSID:sugar_monster_house MAC:* IP:* Active:Any ]
     Enter: Anon
         A1: Custom Setting [ Type:Global Name:private_dns_mode Value:off Use Root:Off Read Setting To: ]
     Exit: Anon
         A1: Custom Setting [ Type:Global Name:private_dns_mode Value:hostname Use Root:Off Read Setting To: ]

Secure setting accessibility_display_daltonizer_enabled to 0 or 1 for color toggle.

nix shell nixpkgs#android-tools -c adb shell pm grant net.dinglisch.android.taskerm android.permission.WRITE_SECURE_SETTINGS

References:

• StackExchange
• AdGuard instructions for secret settings
• HN post

Desktops

Mac

Trust chain system install: sudo security add-trusted-cert -r trustRoot -k /Library/Keychains/System.keychain -d ~/Downloads/root-ca.pem

Old MBP setup

OPNsense/openssl's ciphers are too new, to install client certificate you may need to pkcs12 bundle legacy. openssl pkcs12 -export -legacy -out Certificate.p12 -in certificate.pem -inkey key.pem

• StackOverflow post

MBP M2 setup

1. Update everything softwareupdate -ia
2. Optionally install rosetta softwareupdate --install-rosetta --agree-to-license I didn't explicitly install it but it's on there somehow now. There was some mention that it auto-installs if you try running x86_64 binaries.
3. Determinant systems install nix
4. Until this is resolved nix-darwin/nix-darwin#149 sudo mv /etc/nix/nix.conf /etc/nix/.nix-darwin.bkp.nix.conf
5. Nix-Darwin build and run installer

nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer
./result/bin/darwin-installer

edit default configuration.nix? n
# Accept the option to manage nix-darwin using nix-channel or else it bombs
manage using channels? y
add to bashrc y
add to zshrc? y
create /run? y
# a nix-channel call will now fail

Bootstrapping:

1. do the xcode-install method
2. Build manually once nix build github:arichtman/nix#darwinConfigurations.macbook-pro-work.system
3. Switch manually once ./result/sw/bin/darwin-rebuild switch --flake .#macbook-pro-work
4. If bootstrapped, build according to flake ./result/sw/bin/darwin-rebuild switch --flake github:arichtman/nix

To do: look into Nix VMs on Mac

Universal Blue

some very wip notes about the desktop.

• Installer with nVidia drivers worked ok in simplified mode

• Despite the claims of signing automation for secure boot it still needs to be disabled, 'less you like 800x600.

• Bluetooth pair the speaker though you may have to change the codec in settings > sound

• I ran bluetoothctl trust $MAC to try and start off autoconnect

• I fiddled about in display settings to get orientation of monitors correct

• sudo visudo and swap the commented lines for wheel to enable NOPASSWD.

• Suppress/fix warnings about running Nix commands as myself: Added trusted-users = @wheel to /etc/nix/nix.custom.conf (DetSys thing not to use /etc/nix/nix.conf). Note: might be able to specify this at install time...

• Enable composefs transient root, then install DetSys Nix (for SELinux support). /etc/ostree/prepare-root.conf:

  [composefs]
  enabled = yes
  [root]
  transient = true

  Then sudo rpm-ostree initramfs-etc --reboot --track=/etc/ostree/prepare-root.conf. Reference

• Used nix develop to bootstrap

• home-manager switch --flake . -b backup

• Installed my root certificate sudo curl https://www.richtman.au/root-ca.pem -o /etc/pki/ca-trust/source/anchors/root-ca.pem sudo update-ca-trust

• Set my shell to Zsh sudo usermod --shell $(which zsh) arichtman. Note: not sure how this is going, obvs that path isn't in /etc/shells, but I can't see any bash-default-shell in rpm-ostree. Reboot and see if it applies on login.

• Install system level layers with zsh and alacritty. sudo rpm-ostree install -y --idempotent zsh alacritty

• Fix failure to wake from sleep. /usr/lib/systemd/system/service.d/50-keep-warm.conf:

  # Disable freezing of user sessions to work around kernel bugs.
  # See https://bugzilla.redhat.com/show_bug.cgi?id=2321268
  [Service]
  Environment=SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=0

  Reference

• Enabled WoL tutorial

Desktop Todo

• Set resolved's upstream DNS from DHCPv4, figure out what to do about v6 dynamic DNS server.
• Get CLI clipboard access post
• Learn about universal blue/ostree and decide if I want to keep this
• look into errors running tracker-miner-fs-3.service
• Work out how to uninstall nano-default-editor rpm-ostree override remove
• Fix failing Alacritty launchpad launch
• Fix failing systemd-remount-fs.service

Batocera

Fix terminfo for Alacritty: curl -sSL https://raw.githubusercontent.com/alacritty/alacritty/master/extra/alacritty.info | tic -x -

• Blog
• Alacritty docs

Nix References

• Opinionated flake structure
• Home-manager configuration options
• Misterio77's starter configs
• Just generally sucking at it, spelunking nixpkgs and NixOS-WSL source Nix files
• Jake Hamilton videos
• Nebucatnetzer's config
• Post about inline nix use helm
• Wrapper library