madereddy
191 lines
7.5 KiB
Markdown
191 lines
7.5 KiB
Markdown
# Simplifying OCI Buildout: Automated with Ansible
|
|
|
|
This repo serves to build out, restore, or recover from disaster on OCI always free instances.
|
|
|
|
### Pre Requisites
|
|
|
|
- You must have an OCI instance spun up and accessible
|
|
|
|
- You must have the target machines setup and have ssh key based authentication setup.
|
|
|
|
- You must have ansible installed on your local machine. You do not need Ansible installed on the target hosts.
|
|
|
|
### To Use These Playbooks
|
|
Simply clone this repository and follow the readme below.
|
|
|
|
### File structure:
|
|
|
|
``````
|
|
├── ansible.cfg
|
|
├── inventory.yml
|
|
└── playbooks
|
|
├── vault
|
|
├── caddy
|
|
└── Caddyfile
|
|
├── compose
|
|
└── docker-compose.yml
|
|
├── ddns
|
|
└── ddns.config
|
|
└── restic
|
|
└── restic.yml
|
|
├── docker_status.yml
|
|
├── docker_update_containers.yml
|
|
├── deploy.yml
|
|
├── redeploy.yml
|
|
├── disaster.yml
|
|
├── clean.yml
|
|
├── fresh_install.yml
|
|
├── install_docker.yml
|
|
├── os_family_discovery.yml
|
|
├── ping_test.yml
|
|
└── update_upgrade.yml
|
|
|
|
``````
|
|
|
|
#### ansible.cfg
|
|
This is the ansible configuration file. It tells ansible basic information about where to find certain files and how to run
|
|
#### inventory.yml
|
|
This file contains the inventory for all of our hosts and information about those hosts.
|
|
#### playbooks
|
|
This directory contains all of our plalybooks, which we will touch on later in the blog post when we get to them.
|
|
#### vault
|
|
This contains encrypted files and variables to be used with the playbooks.
|
|
|
|
## Introduction
|
|
|
|
This repo has been built to jump start or rebuild my remote homelab. It contains the following containers
|
|
|
|
Vaultwarden
|
|
Caddy
|
|
UptimeKuma
|
|
Syncthing
|
|
Watchtower
|
|
Cloudflare-ddns
|
|
Webhook
|
|
|
|
To get started, clone the github repo that goes along with this post:
|
|
|
|
`git clone https://madereddy.com/git/jeet/OCI_Build.git`
|
|
|
|
`cd OCI_Build`
|
|
|
|
## Section 1: Setting Up SSH Key-Based Authentication
|
|
|
|
Before jumping into Ansible playbooks, it's crucial to establish a secure and efficient way to connect to your remote machines. SSH key-based authentication offers a more secure alternative to traditional password-based methods.
|
|
|
|
Add the Public Key to the /.ssh/authorized_keys which is found on the ansible machine.
|
|
|
|
```
|
|
ssh-copy-id -i ~/.ssh/id_ed25519 user@remote1
|
|
|
|
```
|
|
|
|
If the ansible machine is not available, you can directly modify the /.ssh/authorized_keys file and add the public key found on Vaultwarden Secure Notes.
|
|
|
|
Restart the SSHD Service
|
|
|
|
```
|
|
sudo systemctl restart sshd
|
|
```
|
|
|
|
Verify you can log in using the Private Key.
|
|
|
|
## Section 2: Crafting Your Ansible Inventory File
|
|
|
|
An inventory file in Ansible is where you define the hosts and groups for your automation tasks. Let's create an inventory.yml file that categorizes your machines into groups. Fill in your host information, ip address, and user name & key file in the `inventory.yml` file. If you're unsure how to group your machines, just put them all in the same group for now and we will regroup them in the next section.
|
|
|
|
## Section 3: Discovering OS Families with Ansible
|
|
|
|
Now that you have your inventory set, let's start with the `os_family_discovery.yml`` playbook. This playbook will help you identify the operating system family of your hosts, which is crucial for tailoring further automation tasks to specific OS types.
|
|
|
|
Run this playbook against all of your hosts to get information on which family they belong to.
|
|
|
|
`ansible-playbook playbooks/os_family_discovery.yml`
|
|
|
|
Now that you know what family each host is, I recommend going back to the `inventory.yml` and grouping the hosts based on family (Debian, RedHat, etc.). If you want to furthur subdivide your hosts you can have a host in multiple groups as well.
|
|
|
|
## Section 4: The Fresh Install - Setting Up New Machines
|
|
|
|
Next up is the `fresh_install.yml` playbook. This playbook is designed to install a suite of essential utilities on new machines, whether they're running Debian/Ubuntu or RedHat/CentOS. Notably, this playbook also imports the `install_docker.yml` playbook, automating Docker installation as part of the setup process.
|
|
|
|
`ansible-playbook playbooks/fresh_install.yml --ask-become`
|
|
|
|
This will prompt you to input the BECOME password, which is the sudo password for your target machine
|
|
|
|
## Section 5a: Fresh Install Docker Containers on Prod
|
|
|
|
After setting up your machines, let's focus on Docker with the `docker_status.yml` playbook. This playbook checks the status of your Docker containers, ensuring they are running as expected.
|
|
|
|
If you have hosts already running docker, you will want to add them to the `inventory.yml` file in a group named `Docker`.
|
|
|
|
If you don't have any hosts running docker, pick a host that you just ran the fresh install script on and add it to the `Docker` group in your `inventory.yml`. Then execute the `deploy` playbook which will stand up your first Docker container.
|
|
|
|
`ansible-playbook playbooks/deploy.yml --ask-vault-pass`
|
|
|
|
You can now run `docker_status.yml` against your docker hosts to check the status of your containers. This playbook will return all green if your containers are all good, and it will fail if any container is in status "exited"
|
|
|
|
`ansible-playbook playbooks/docker_status.yml`
|
|
|
|
To clean up the build you can run `clean.yml` against the docker hosts.
|
|
|
|
`ansible-playbook playbooks/clean.yml --ask-become`
|
|
|
|
## Section 5b: Restore Docker Containers to Prod from Backup
|
|
|
|
If restoring your new machine. You can run playbook `redeploy` to restore from your OCI Backup which uses Restic to backup and restore.
|
|
|
|
`ansible-playbook playbooks/redeploy.yml --ask-vault-pass`
|
|
|
|
To clean up the build you can run `clean.yml` against the docker hosts.
|
|
|
|
`ansible-playbook playbooks/clean.yml --ask-become`
|
|
|
|
## Section 6: Updating hosts
|
|
|
|
Lastly, we have the `update_upgrade.yml` playbook. This playbook is crucial for updating underlying host machines.
|
|
|
|
`ansible-playbook playbooks/update_upgrade.yml --ask-become`
|
|
|
|
---
|
|
|
|
# Recovering from Disaster
|
|
|
|
This is assuming you have an already configured DR Instance in OCI with Docker installed and the ansible inventory updated properly.
|
|
|
|
### Step 1: Initiate DR
|
|
|
|
The playbook will restore the backup stored in OCI onto the DR instance.
|
|
|
|
`ansible-playbook playbooks/disaster.yml --ask-vault-pass`
|
|
|
|
### Step 2: Validate Recovery
|
|
|
|
Once the containers are online and stable you will need to check a few things.
|
|
|
|
1. Verify that Vaultwarden is running and accessible at bitwarden.madereddy.com
|
|
|
|
2. Verify that UptimeKuma is running and accessible at outsideuptime.madereddy.com
|
|
|
|
3. Verify that Syncthing is running and syncing at oracle.madereddy.com
|
|
|
|
If any of those sites have an issue with the SSL Cert recycle Caddy to have ACME issue a new cert.
|
|
|
|
### Step 3: Fix Ansible Inventory
|
|
|
|
Update the Ansible inventory with the new Prod server information.
|
|
|
|
### Step 4: Build a new DR Server
|
|
|
|
Build an always free OCI Server in a different Availability Domain then the current Prod server.
|
|
|
|
Ampere Servers are recommened with 1CPU and 6GB of RAM, but if they are not available then select the E1 instance family.
|
|
|
|
Copy your SSH to the instance during build or follow `Section 1: Setting Up SSH Key-Based Authentication` from above.
|
|
|
|
### Step 5: Verify Access and update Ansible Inventory
|
|
|
|
Verify access to the new instance. Once confirmed, update the Ansible Invetory file with the new DR server.
|
|
|
|
### Step 6: Prepare the new DR Server
|
|
|
|
Follow `Section 4: The Fresh Install - Setting Up New Machines` to make sure the required packages are available for the next DR. |