Simplifying OCI Buildout: Automated with Ansible

This repo serves to build out, restore, or recover from disaster on OCI always free instances.

Pre Requisites

• You must have an OCI instance spun up and accessible

• You must have the target machines setup and have ssh key based authentication setup.

• You must have ansible installed on your local machine. You do not need Ansible installed on the target hosts.

To Use These Playbooks

Simply clone this repository and follow the readme below.

File structure:

├── ansible.cfg
├── inventory.yml
└── playbooks
    ├── vault
        ├── caddy
            └── Caddyfile
        ├── compose
            └── docker-compose.yml
        ├── ddns
            └── ddns.config
        └── restic
            └── restic.yml
    ├── docker_status.yml
    ├── docker_update_containers.yml
    ├── deploy.yml
    ├── redeploy.yml
    ├── disaster.yml
    ├── clean.yml
    ├── fresh_install.yml
    ├── install_docker.yml
    ├── os_family_discovery.yml
    ├── ping_test.yml
    └── update_upgrade.yml

ansible.cfg

This is the ansible configuration file. It tells ansible basic information about where to find certain files and how to run

inventory.yml

This file contains the inventory for all of our hosts and information about those hosts.

playbooks

This directory contains all of our plalybooks, which we will touch on later in the blog post when we get to them.

vault

This contains encrypted files and variables to be used with the playbooks.

Introduction

This repo has been built to jump start or rebuild my remote homelab. It contains the following containers

Vaultwarden Caddy UptimeKuma Syncthing Watchtower Cloudflare-ddns Webhook

To get started, clone the github repo that goes along with this post:

git clone https://madereddy.com/git/jeet/OCI_Build.git

cd OCI_Build

Section 1: Setting Up SSH Key-Based Authentication

Before jumping into Ansible playbooks, it's crucial to establish a secure and efficient way to connect to your remote machines. SSH key-based authentication offers a more secure alternative to traditional password-based methods.

Add the Public Key to the /.ssh/authorized_keys which is found on the ansible machine.

ssh-copy-id -i ~/.ssh/id_ed25519 user@remote1

If the ansible machine is not available, you can directly modify the /.ssh/authorized_keys file and add the public key found on Vaultwarden Secure Notes.

Restart the SSHD Service

sudo systemctl restart sshd

Verify you can log in using the Private Key.

Section 2: Crafting Your Ansible Inventory File

An inventory file in Ansible is where you define the hosts and groups for your automation tasks. Let's create an inventory.yml file that categorizes your machines into groups. Fill in your host information, ip address, and user name & key file in the inventory.yml file. If you're unsure how to group your machines, just put them all in the same group for now and we will regroup them in the next section.

Section 3: Discovering OS Families with Ansible

Now that you have your inventory set, let's start with the `os_family_discovery.yml`` playbook. This playbook will help you identify the operating system family of your hosts, which is crucial for tailoring further automation tasks to specific OS types.

Run this playbook against all of your hosts to get information on which family they belong to.

ansible-playbook playbooks/os_family_discovery.yml

Now that you know what family each host is, I recommend going back to the inventory.yml and grouping the hosts based on family (Debian, RedHat, etc.). If you want to furthur subdivide your hosts you can have a host in multiple groups as well.

Section 4: The Fresh Install - Setting Up New Machines

Next up is the fresh_install.yml playbook. This playbook is designed to install a suite of essential utilities on new machines, whether they're running Debian/Ubuntu or RedHat/CentOS. Notably, this playbook also imports the install_docker.yml playbook, automating Docker installation as part of the setup process.

ansible-playbook playbooks/fresh_install.yml --ask-become

This will prompt you to input the BECOME password, which is the sudo password for your target machine

Section 5a: Fresh Install Docker Containers on Prod

After setting up your machines, let's focus on Docker with the docker_status.yml playbook. This playbook checks the status of your Docker containers, ensuring they are running as expected.

If you have hosts already running docker, you will want to add them to the inventory.yml file in a group named Docker.

If you don't have any hosts running docker, pick a host that you just ran the fresh install script on and add it to the Docker group in your inventory.yml. Then execute the deploy playbook which will stand up your first Docker container.

ansible-playbook playbooks/deploy.yml --ask-vault-pass

You can now run docker_status.yml against your docker hosts to check the status of your containers. This playbook will return all green if your containers are all good, and it will fail if any container is in status "exited"

ansible-playbook playbooks/docker_status.yml

To clean up the build you can run clean.yml against the docker hosts.

ansible-playbook playbooks/clean.yml --ask-become

Section 5b: Restore Docker Containers to Prod from Backup

If restoring your new machine. You can run playbook redeploy to restore from your OCI Backup which uses Restic to backup and restore.

ansible-playbook playbooks/redeploy.yml --ask-vault-pass

To clean up the build you can run clean.yml against the docker hosts.

ansible-playbook playbooks/clean.yml --ask-become

Section 6: Updating hosts

Lastly, we have the update_upgrade.yml playbook. This playbook is crucial for updating underlying host machines.

ansible-playbook playbooks/update_upgrade.yml --ask-become

---

Recovering from Disaster

This is assuming you have an already configured DR Instance in OCI with Docker installed and the ansible inventory updated properly.

Step 1: Initiate DR

The playbook will restore the backup stored in OCI onto the DR instance.

ansible-playbook playbooks/disaster.yml --ask-vault-pass

Step 2: Validate Recovery

Once the containers are online and stable you will need to check a few things.

1. Verify that Vaultwarden is running and accessible at bitwarden.madereddy.com

2. Verify that UptimeKuma is running and accessible at outsideuptime.madereddy.com

3. Verify that Syncthing is running and syncing at oracle.madereddy.com

If any of those sites have an issue with the SSL Cert recycle Caddy to have ACME issue a new cert.

Step 3: Fix Ansible Inventory

Update the Ansible inventory with the new Prod server information.

Step 4: Build a new DR Server

Build an always free OCI Server in a different Availability Domain then the current Prod server.

Ampere Servers are recommened with 1CPU and 6GB of RAM, but if they are not available then select the E1 instance family.

Copy your SSH to the instance during build or follow Section 1: Setting Up SSH Key-Based Authentication from above.

Step 5: Verify Access and update Ansible Inventory

Verify access to the new instance. Once confirmed, update the Ansible Invetory file with the new DR server.

Step 6: Prepare the new DR Server

Follow Section 4: The Fresh Install - Setting Up New Machines to make sure the required packages are available for the next DR.