First Commit

README.md

@@ -0,0 +1,110 @@
+# Simplifying Homelab Management: Automate with Ansible
+
+This repo serves to work with my blog post on using Ansible to automate your homelab. Instructions to use are as follows and the blog post is below
+
+### NOTE: This was built against an ubuntu install. RHEL9 doesn't officially support docker so the docker install playbook currently doesn't work on RHEL machines
+
+### Pre Requisites
+
+- You must have the target machines setup and have ssh key based authentication setup. If you don't have that completed yet you can follow along with the blog post below.
+
+- You must have ansible installed on your local machine. You do not need Ansible installed on the target hosts.
+
+### To Use These Playbooks
+Simply clone this repository and follow the blog post below to get started
+
+### File structure:
+
+``````
+├── ansible.cfg
+├── inventory.yml
+└── playbooks
+    ├── docker_status.yml
+    ├── docker_update_containers.yml
+    ├── first_container.yml
+    ├── fresh_install.yml
+    ├── install_docker.yml
+    ├── install_vim.yml
+    ├── os_family_discovery.yml
+    ├── ping_test.yml
+    └── update_upgrade.yml
+``````
+
+#### ansible.cfg
+This is the ansible configuration file. It tells ansible basic information about where to find certain files and how to run
+#### inventory.yml
+This file contains the inventory for all of our hosts and information about those hosts. We will go into detail later.
+#### playbooks
+This directory contains all of our plalybooks, which we will touch on later in the blog post when we get to them.
+
+## Introduction
+
+Welcome back to our Homelab posts! Today, we're venturing into the realm of automation with Ansible. Ansible is an open-source automation tool that makes it easier to configure and manage computers and servers. It uses a simple, human-readable language to automate the setup and management of your systems, eliminating repetitive tasks and ensuring consistency across your environment. Whether you're managing a few machines or a large-scale network, Ansible's versatility allows for efficient scaling and control, making it an indispensable tool in any homelab setup.
+
+In this post, we dive into Ansible to simplify and streamline your homelab operations. We'll begin by discussing the basics of setting up SSH key-based authentication and creating an Ansible inventory file. From there, we'll progress through various playbooks, covering everything from OS family discovery to Docker container management. By automating these essential tasks, your homelab will become more efficient, secure, and easier to maintain.
+
+To get started, clone the github repo that goes along with this post:
+
+`git clone https://github.com/jonezy35/ansible-homelab.git`
+
+`cd ansible-homelab`
+
+## Section 1: Setting Up SSH Key-Based Authentication
+
+Before jumping into Ansible playbooks, it's crucial to establish a secure and efficient way to connect to your remote machines. SSH key-based authentication offers a more secure alternative to traditional password-based methods. I've detailed the setup process in a previous post, which you can find [here](https://medium.com/@jonezy7173_88832/using-ssh-key-based-authentication-remote-machines-and-github-f7fe6d142be8). This setup is essential for seamless Ansible operations.
+
+## Section 2: Crafting Your Ansible Inventory File
+
+An inventory file in Ansible is where you define the hosts and groups for your automation tasks. Let's create an inventory.yml file that categorizes your machines into groups. Fill in your host information, ip address, and user name & key file in the `inventory.yml` file. If you're unsure how to group your machines, just put them all in the same group for now and we will regroup them in the next section.
+
+## Section 3: Discovering OS Families with Ansible
+
+Now that you have your inventory set, let's start with the `os_family_discovery.yml`` playbook. This playbook will help you identify the operating system family of your hosts, which is crucial for tailoring further automation tasks to specific OS types.
+
+Run this playbook against all of your hosts to get information on which family they belong to.
+
+`ansible-playbook playbooks/os_family_discovery.yml`
+
+Now that you know what family each host is, I recommend going back to the `inventory.yml` and grouping the hosts based on family (Debian, RedHat, etc.). If you want to furthur subdivide your hosts you can have a host in multiple groups as well.
+
+## Section 4: The Fresh Install - Setting Up New Machines
+
+Next up is the `fresh_install.yml` playbook. This playbook is designed to install a suite of essential utilities on new machines, whether they're running Debian/Ubuntu or RedHat/CentOS. Notably, this playbook also imports the `install_docker.yml` playbook, automating Docker installation as part of the setup process.
+
+`ansible-playbook playbooks/fresh_install.yml --ask-become`
+
+This will prompt you to input the BECOME password, which is the sudo password for your target machine
+
+## Section 5: Monitoring Docker Containers
+
+After setting up your machines, let's focus on Docker with the `docker_status.yml` playbook. This playbook checks the status of your Docker containers, ensuring they are running as expected.
+
+If you have hosts already running docker, you will want to add them to the `inventory.yml` file in a group named `Docker`.
+
+If you don't have any hosts running docker, pick a host that you just ran the fresh install script on and add it to the `Docker` group in your `inventory.yml`. Then execute the `first_container` playbook which will stand up your first Docker container.
+
+`ansible-playbook playbooks/first_container.yml`
+
+Now add the following to your `compose_file_paths` for the docekr host in your `inventory.yml` file:
+
+`~/firstContainer/docker-compose.yml`
+
+You can now run `docker_status.yml` against your docker hosts to check the status of your containers. This playbook will return all green if your containers are all good, and it will fail if any container is in status "exited"
+
+`ansible-playbook playbooks/docker_status.yml`
+
+## Section 6: Keeping Docker Containers Up-to-Date
+
+Lastly, we have the `docker_update_containers.yml` playbook. This playbook is crucial for updating your Docker containers with the latest images. It also re-imports the docker_status.yml playbook to check the status of containers after the update.
+
+At this point you should have all of your hosts running docker containers in the `Docker` group of your `inventory.yml` file.
+
+To update your containers this playbook brings your containers down, deletes their images, and brings them back up to pull the updated images. 
+
+**YOU MUST HAVE PERSISTANT STORAGE SETUP! IF YOU DON'T THEN THIS PLAYBOOK WILL <u>DELETE ALL OF YOUR DATA FROM YOUR CONTAINERS**</u> - consider yourself warned.
+
+`ansible-playbook playbooks/docker_update_containers.yml`
+
+## Conclusion:
+
+Throughout this post, we've explored a series of Ansible playbooks designed to automate various aspects of your homelab. From initial setup with SSH key-based authentication and inventory creation to managing and updating Docker containers, these playbooks are the building blocks for a highly efficient homelab environment. Remember, Ansible's flexibility allows you to expand and customize these automations to suit your specific needs. Dive in, experiment, and watch your homelab thrive with the power of automation!

ansible.cfg

@@ -0,0 +1,3 @@
+[defaults]
+inventory = ./inventory.yml
+host_key_checking = False

docker/docker-compose.yaml

@@ -0,0 +1,92 @@
+version: "3"
+volumes:
+  vw-logs:
+  f2b-data:
+services:
+  uptime-kuma:
+    image: louislam/uptime-kuma:latest
+    container_name: uptime
+    restart: unless-stopped
+    ports:
+      - "3031:3031"
+    volumes:
+      - ./uptime:/app/data
+  caddy:
+    image: caddy:latest
+    container_name: caddy
+    cap_add:
+      - NET_ADMIN
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro
+      - ./caddy:/config
+      - ./caddy:/data
+    restart: unless-stopped
+  watchtower:
+      image: containrrr/watchtower:latest
+      container_name: watchtower
+      volumes:
+          - /var/run/docker.sock:/var/run/docker.sock
+      environment:
+          - WATCHTOWER_CLEANUP=true
+          - WATCHTOWER_SCHEDULE= 0 0 4 * * * #At 04:00 AM, only on Friday Change to suit
+      restart: unless-stopped
+  vaultwarden:
+    image: vaultwarden/server:latest
+    container_name: bitwarden
+    environment:
+      - SIGNUPS_ALLOWED=false #use true if needing to create account
+      - DOMAIN=https://bitwarden.madereddy.com
+      - "YUBICO_CLIENT_ID=47188"
+      - "YUBICO_SECRET_KEY=VBO1BoGv47JwFudR4SaiwZ9gwG4="
+      - LOG_FILE=/data/vaultwarden.log
+      - ROCKET_PORT=8080
+      - PUSH_ENABLED=true
+      - "PUSH_INSTALLATION_ID=caf6a736-022d-442e-a37b-b06f0125aa9c"
+      - "PUSH_INSTALLATION_KEY=A20fZMOjId4OENWSvyEw"
+      - "ADMIN_TOKEN=8qwvHlTb1nC2KmTwYFfgfTvRdJF0CWL3x5eCpIksMCmQ9mVHEeYdbpeQo53a2lkC"
+    volumes:
+      - ./bitwarden:/data
+      - vw-logs:/var/log/vw-logs
+    restart: unless-stopped
+  cloudflare-ddns:
+    image: timothyjmiller/cloudflare-ddns:latest
+    container_name: cloudflare-ddns
+    security_opt:
+      - no-new-privileges:true
+    environment:
+      - PUID=1001
+      - PGID=1001
+    volumes:
+      - ./ddns/config.json:/config.json
+    restart: unless-stopped
+  syncthing:
+    image: lscr.io/linuxserver/syncthing:latest
+    container_name: syncthing
+    environment:
+      - PUID=1001
+      - PGID=1001
+      - TZ=Etc/UTC
+    volumes:
+      - ./syncthing/config:/config
+      - ./:/data1
+    ports:
+      - 8384:8384
+      - 22000:22000/tcp
+      - 22000:22000/udp
+      - 21027:21027/udp
+    restart: unless-stopped
+  webhook:
+      image: ncarlier/webhookd:edge-distrib
+      container_name: webhook
+      user: 1001:999
+      volumes:
+        - /var/run/docker.sock:/var/run/docker.sock
+        - ./webhook:/scripts/hook
+      ports:
+        - 8080:8080
+      environment:
+        - WHD_HOOK_TIMEOUT=600
+      restart: unless-stopped

inventory.yml

@@ -0,0 +1,28 @@
+all:
+  machines:
+    BareMetal:
+      hosts:
+        host1:
+          ansible_host: 192.168.1.79
+          ansible_user: jeet
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519
+#        host2:
+#          ansible_host: <ip>
+#          ansible_user: <user>
+#          ansible_ssh_private_key_file: ~/.ssh/<private key>
+    Docker:
+      hosts:
+        host1:
+          ansible_host: 192.168.1.79
+          ansible_user: jeet
+          ansible_ssh_private_key_file: ~/.ssh/id_ed25519
+          compose_file_paths: 
+            - /path/to/docker-compose.yml
+#        host2:
+#          ansible_host: <ip>
+#          ansible_user: <user>
+#          ansible_ssh_private_key_file: ~/.ssh/<private key>
+#          compose_file_paths: 
+#            - /path/to/docker-compose.yml
+#            - /path/to/docker-compose.yml
+#            - /path/to/docker-compose.yml

playbooks/docker_status.yml

@@ -0,0 +1,28 @@
+---
+- name: Check and Report Status of Docker Containers
+  hosts: Docker
+  gather_facts: yes
+  vars:
+    exited_containers: []
+
+  tasks:
+    - name: Check container status
+      ansible.builtin.shell: |
+        docker ps -a --format "{{ '{{' }}.Names{{ '}}' }}\t{{ '{{' }}.Status{{ '}}' }}\t{{ '{{' }}.Ports{{ '}}' }}"
+      register: container_status
+
+    - name: Collect exited containers
+      set_fact:
+        exited_containers: "{{ exited_containers + [item.split('\t')[0]] }}"
+      loop: "{{ container_status.stdout_lines }}"
+      when: "'Exited' in item.split('\t')[1]"
+
+    - name: Display container status for each host
+      ansible.builtin.debug:
+        msg: "Container: {{ item.split('\t')[0] }}, Status: {{ item.split('\t')[1] }}, Ports: {{ item.split('\t')[2] or 'None' }}"
+      loop: "{{ container_status.stdout_lines }}"
+
+    - name: Fail with summary of exited containers
+      ansible.builtin.fail:
+        msg: "Exited containers found: {{ exited_containers | join(', ') }}"
+      when: exited_containers | length > 0

playbooks/docker_update_containers.yml

@@ -0,0 +1,52 @@
+---
+- name: Update Docker Containers With New Images
+  hosts: Docker
+  gather_facts: yes
+
+  tasks:
+    - name: Stop and remove Docker containers
+      ansible.builtin.command: docker compose down
+      args:
+        chdir: "{{ compose_file_path | dirname }}"
+      loop: "{{ compose_file_paths }}"
+      ignore_errors: yes
+      loop_control:
+        loop_var: compose_file_path
+
+    - name: Run docker compose up
+      ansible.builtin.command: docker compose up -d
+      args:
+        chdir: "{{ compose_file_path | dirname }}"
+      loop: "{{ compose_file_paths }}"
+      loop_control:
+        loop_var: compose_file_path
+
+    - name: Pause for 60 seconds to allow containers to stabilize
+      ansible.builtin.pause:
+        seconds: 60
+
+    - name: Check container status
+      ansible.builtin.shell: docker compose ps -q | xargs -n1 docker container inspect --format '{{ "{{" }} .State.Running {{ "}}" }}'
+      args:
+        chdir: "{{ compose_file_path | dirname }}"
+      loop: "{{ compose_file_paths }}"
+      register: container_status
+      ignore_errors: yes
+      loop_control:
+        loop_var: compose_file_path
+
+    - name: Conditional restart of containers
+      ansible.builtin.command: docker compose up -d
+      args:
+        chdir: "{{ result_item.item.path | dirname }}"
+      loop: "{{ container_status.results }}"
+      when: "'false' in result_item.stdout"
+      register: restart_status
+      loop_control:
+        loop_var: result_item
+
+    - name: Pause for 60 seconds to allow containers to stabilize
+      ansible.builtin.pause:
+        seconds: 60
+
+- import_playbook: docker_status.yml

playbooks/first_container.yml

@@ -0,0 +1,22 @@
+- name: Build OCI Stack
+  hosts: Docker
+  tasks:
+    - name: Clone repo
+      command: wget http://192.168.1.157:3000/jeet/OCI_Build.git 
+
+    - name: Start container using Docker Compose
+      ansible.builtin.command:
+        cmd: docker compose up -d
+        chdir: ~/docker
+      ignore_errors: yes
+
+    - name: Pause for 30 seconds to allow containers to stabilize
+      ansible.builtin.pause:
+        seconds: 60
+
+    - name: Check container status
+      ansible.builtin.shell: docker compose ps -q | xargs -n1 docker container inspect --format '{{ "{{" }} .State.Running {{ "}}" }}'
+      args:
+        chdir: ~/firstContainer
+      register: container_status
+      ignore_errors: yes

playbooks/fresh_install.yml

@@ -0,0 +1,27 @@
+---
+- name: Install various utilities on Debian/Ubuntu
+  hosts: all
+  become: yes
+  gather_facts: yes
+
+  tasks:
+    - name: Update apt cache (Debian/Ubuntu)
+      apt:
+        update_cache: yes
+        cache_valid_time: 3600 # Cache valid for 1 hour
+
+    - name: Install packages for Debian/Ubuntu
+      apt:
+        name:
+          - build-essential
+          - git
+          - curl
+          - wget
+          - htop
+          - tar
+          - net-tools
+          - unzip
+        state: present
+
+# This will run after the previous play is completed
+- import_playbook: install_docker.yml

playbooks/install_docker.yml

@@ -0,0 +1,59 @@
+- name: Install Docker on Debian, Ubuntu, or Raspbian
+  hosts: all
+  become: yes
+  gather_facts: yes
+
+  tasks:
+    - name: Install Docker dependencies for Debian/Ubuntu/Raspbian
+      apt:
+        name:
+          - apt-transport-https
+          - ca-certificates
+          - curl
+          - gnupg
+          - lsb-release
+        state: present
+        update_cache: yes
+      when: ansible_os_family == "Debian"
+
+    - name: Add Docker GPG key for Debian/Ubuntu/Raspbian
+      apt_key:
+        url: https://download.docker.com/linux/ubuntu/gpg
+        state: present
+      when: ansible_os_family == "Debian"
+
+    - name: Add Docker repository for Debian/Ubuntu
+      apt_repository:
+        repo: "deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable"
+        state: present
+        update_cache: yes
+      when: ansible_os_family == "Debian" and ansible_distribution == "Ubuntu"
+
+    - name: Add Docker repository for Raspbian/Debian
+      apt_repository:
+        repo: "deb [arch=arm64] https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable"
+        state: present
+        update_cache: yes
+      when: ansible_os_family == "Debian" and ansible_architecture == "aarch64"
+
+    - name: Install Docker for Debian/Ubuntu/Raspbian
+      apt:
+        name: docker-ce
+        state: present
+        update_cache: yes
+
+    - name: Add authenticated user to Docker group
+      user:
+        name: "{{ ansible_user }}"
+        groups: docker
+        append: yes
+
+    - name: Ensure Docker service is enabled and started
+      systemd:
+        name: docker
+        enabled: yes
+        state: started
+
+    - name: Reset connection to refresh user group membership
+      meta: reset_connection
+

playbooks/ping_test.yml

@@ -0,0 +1,5 @@
+---
+- hosts: all
+  tasks:
+    - name: Test Ping
+      ping:

playbooks/update_upgrade.yml

@@ -0,0 +1,13 @@
+---
+- hosts: all
+  become: yes
+  gather_facts: true
+
+  tasks:
+    - name: "Updating and Upgrading Apt Packages"
+      apt:
+        update_cache: yes
+        upgrade: safe
+
+    # Specific adjustments for Raspbian can be made here, if necessary
+    # Raspbian will typically be covered by the Debian task, but if you have specific needs, you can specify them here.