Automated Server Patching with Ansible: Zero-Downtime Update Strategy

The Patching Problem

Unpatched servers are the number one attack vector. Yet manual patching is tedious, error-prone, and often postponed. The solution is automation with Ansible — idempotent, agentless, and simple enough that your playbooks serve as documentation.

Ansible Inventory Setup

Start by organizing your servers into groups:

# inventory/hosts.ini
[webservers]
web1.example.com
web2.example.com

[databases]
db1.example.com
db2.example.com

[monitoring]
monitor.example.com

[all:vars]
ansible_user=deploy
ansible_python_interpreter=/usr/bin/python3

[webservers:vars]
patch_group=group_a

[databases:vars]
patch_group=group_b

The Patching Playbook

Here is a comprehensive playbook that handles the entire patching lifecycle:

---
# playbooks/patch-servers.yml
- name: Automated Server Patching
  hosts: all
  become: true
  serial: "50%"  # Rolling update — patch 50% at a time
  max_fail_percentage: 0
  vars:
    reboot_timeout: 600
    snapshot_before_patch: true
    notify_channel: "#ops-alerts"

  pre_tasks:
    - name: Check if server is in maintenance window
      assert:
        that:
          - maintenance_window | default(true)
        fail_msg: "Server is not in maintenance window. Skipping."

    - name: Create pre-patch snapshot (if LXC/VM)
      delegate_to: proxmox_host
      command: >
        pct snapshot {{ inventory_hostname_short }}
        pre-patch-{{ ansible_date_time.date }}
      when: snapshot_before_patch
      ignore_errors: true

    - name: Record pre-patch package versions
      shell: dpkg -l > /tmp/pre-patch-packages.txt
      changed_when: false

  tasks:
    - name: Update apt cache
      apt:
        update_cache: true
        cache_valid_time: 3600

    - name: Upgrade all packages
      apt:
        upgrade: safe
        autoremove: true
        autoclean: true
      register: apt_result

    - name: Display upgraded packages
      debug:
        msg: "{{ apt_result.stdout_lines | default([]) }}"
      when: apt_result.changed

    - name: Check if reboot is required
      stat:
        path: /var/run/reboot-required
      register: reboot_required

    - name: Reboot server if required
      reboot:
        reboot_timeout: "{{ reboot_timeout }}"
        msg: "Ansible patching reboot"
      when: reboot_required.stat.exists

    - name: Wait for server to be ready
      wait_for_connection:
        delay: 10
        timeout: 300

  post_tasks:
    - name: Verify critical services are running
      systemd:
        name: "{{ item }}"
        state: started
      loop: "{{ critical_services | default(['docker', 'ssh']) }}"
      register: service_check

    - name: Record post-patch package versions
      shell: dpkg -l > /tmp/post-patch-packages.txt
      changed_when: false

    - name: Generate patch diff report
      shell: >
        diff /tmp/pre-patch-packages.txt
        /tmp/post-patch-packages.txt || true
      register: patch_diff
      changed_when: false

    - name: Save patch report
      copy:
        content: |
          Patch Report for {{ inventory_hostname }}
          Date: {{ ansible_date_time.iso8601 }}
          Reboot Required: {{ reboot_required.stat.exists }}
          Changes:
          {{ patch_diff.stdout }}
        dest: "/var/log/patch-reports/{{ ansible_date_time.date }}.txt"

  handlers:
    - name: Send notification
      uri:
        url: "https://notify.example.com/ops"
        method: POST
        body_format: json
        body:
          topic: ops-patches
          title: "Patch complete: {{ inventory_hostname }}"
          message: "{{ apt_result.changed | ternary('Packages updated', 'No updates') }}"

Rolling Updates Strategy

The key to zero-downtime patching is the serial directive:

# Patch one server at a time
serial: 1

# Patch 50% at a time (good for load-balanced groups)
serial: "50%"

# Progressive — start slow, then speed up
serial:
  - 1
  - "30%"
  - "100%"

Security-Only Updates

Sometimes you want only security patches, not feature updates:

- name: Install security updates only
  apt:
    upgrade: dist
    default_release: "{{ ansible_distribution_release }}-security"
    update_cache: true

Or use unattended-upgrades for automatic security patches:

- name: Configure unattended upgrades
  apt:
    name:
      - unattended-upgrades
      - apt-listchanges
    state: present

- name: Enable automatic security updates
  copy:
    content: |
      APT::Periodic::Update-Package-Lists "1";
      APT::Periodic::Unattended-Upgrade "1";
      APT::Periodic::AutocleanInterval "7";
    dest: /etc/apt/apt.conf.d/20auto-upgrades
    mode: "0644"

Rollback Playbook

If patching breaks something, roll back from the snapshot:

---
- name: Rollback Server Patch
  hosts: "{{ target_host }}"
  become: true
  tasks:
    - name: Restore from pre-patch snapshot
      delegate_to: proxmox_host
      command: >
        pct rollback {{ inventory_hostname_short }}
        pre-patch-{{ rollback_date }}
      when: use_snapshot | default(false)

    - name: Or downgrade specific packages
      apt:
        name: "{{ packages_to_downgrade }}"
        state: present
        force: true
      when: packages_to_downgrade is defined

Compliance Reporting

Generate a compliance report showing patch status across your fleet:

- name: Patch Compliance Report
  hosts: all
  become: true
  gather_facts: true
  tasks:
    - name: Check available updates
      shell: apt list --upgradable 2>/dev/null | tail -n +2 | wc -l
      register: available_updates
      changed_when: false

    - name: Check last patch date
      stat:
        path: /var/log/apt/history.log
      register: apt_history

    - name: Compile report
      set_fact:
        patch_status:
          hostname: "{{ inventory_hostname }}"
          os: "{{ ansible_distribution }} {{ ansible_distribution_version }}"
          kernel: "{{ ansible_kernel }}"
          pending_updates: "{{ available_updates.stdout }}"
          last_patched: "{{ apt_history.stat.mtime | default('never') }}"
          uptime_days: "{{ ansible_uptime_seconds | int // 86400 }}"

    - name: Write consolidated report
      delegate_to: localhost
      lineinfile:
        path: ./reports/patch-compliance.csv
        line: "{{ patch_status.hostname }},{{ patch_status.os }},{{ patch_status.pending_updates }},{{ patch_status.uptime_days }}"
        create: true
      run_once: false

Scheduling with Cron or Systemd Timers

# Run patching every Sunday at 3 AM
0 3 * * 0 cd /opt/ansible && ansible-playbook playbooks/patch-servers.yml -i inventory/hosts.ini >> /var/log/ansible-patching.log 2>&1

Best Practices

1. Always snapshot before patching — LXC and ZFS make this instant
2. Use serial for rolling updates — never patch all servers at once
3. Verify services after patching — automated health checks in post_tasks
4. Keep a patch log — compliance requires proof of patching
5. Test in staging first — use separate inventory groups
6. Set a reboot timeout — servers that fail to come back should alert

At TechSaaS, we manage patching across all our infrastructure with Ansible. Combined with ZFS snapshots for instant rollback, we patch confidently knowing we can revert in seconds.

Need automated patching for your infrastructure? Contact [email protected].