bike/lwvwv
1
0
Fork 0
Code Issues 2 Pull Requests Projects Releases Wiki Activity
lwvwv/automation/README.md

11 KiB
Raw Blame History

LWVWV Member Subscription Automation

Part of the LWVWV project. This directory contains Docker-based automation for subscribing LWVWV members to Mailman 3 mailing lists via the mailman-connector API service.

For standalone script usage (without Docker), see the main project README.

Overview

This automation coordinates:

  1. Downloading the latest member roster from the LWVWV portal
  2. Converting the roster to CSV format (email,first_name,last_name)
  3. Subscribing members to the Mailman list via the connector API

Architecture

┌─────────┐     ┌─────────────────────────────┐     ┌──────────────────┐
│ Ofelia  │────>│  lwvwv-subscriber-cron      │────>│  LWVWV Portal    │
│Scheduler│cron │  (Docker Container)         │     │  (download CSV)  │
└─────────┘     └─────────────────────────────┘     └──────────────────┘
                         │
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
   ┌──────────┐   ┌──────────────┐   ┌──────────────┐
   │download- │   │google-civic- │   │subscribe-    │
   │roster.js │──>│api.pl        │──>│members.sh    │
   │(portal)  │   │(email-csv)   │   │(connector)   │
   └──────────┘   └──────────────┘   └──────────────┘
                                             │
                                     ▼
                                     ┌──────────────────────────────────┐
                                     │ [mailman-connector](https://git. │
                                     │ bikeshopi.dev/bike/mailman-       │
                                     │ connector) - REST API for batch   │
                                     │ Mailman 3 subscriptions           │
                                     └──────────────────────────────────┘

Files

File Description
Dockerfile Container image definition
docker-compose.yml Service orchestration with Ofelia labels
entrypoint.sh Parses ../env and exports variables
run-subscription.sh Master orchestration script
start-automation.sh Quick-start helper script
README.md This file

Environmental Variables

Single Source of Truth: ../env

All configuration is stored in one file: ../env (in the parent directory)

This file uses Perl-style syntax (which existing scripts expect):

$VARIABLE_NAME = "value";

How It Works

  1. ../env is the single source of truth for all configuration
  2. The entrypoint.sh script parses ../env and exports variables as environment variables
  3. All scripts can then access them via standard environment variable syntax ($VARIABLE_NAME)

Configuration Variables

Required for Subscription

Variable Description Example
$CONNECTOR_URL URL of mailman-connector https://mailman-connector.example.org
$CONNECTOR_PASSWORD Secret password for connector API your_secret_password
$MAILMAN_LIST_ID Target mailing list ID(s). Format: list_id:league_id or just list_id. For multiple lists, use comma-separated values. members.lists.lwvwv.org:WV000 or list1:WV000, list2:WV103

Required for Portal Download

Variable Description Example
$PORTAL_URL LWVWV portal URL https://portal.lwv.org
$MEMBERSHIP_URL Membership page URL https://portal.lwv.org/groups/...

Email Notifications (Optional)

Variable Description Example
$SMTP_HOST SMTP server mail.example.org
$SMTP_PORT SMTP port 587
$SMTP_USER SMTP username example@example.org
$SMTP_PASSWORD SMTP password your_smtp_password
$EMAIL_FROM Sender address example@example.org
$EMAIL_TO Recipient for alerts hello@example.org

Google API (for district lookup)

Variable Description Example
$key Google Civic API key AIzaSy...
$STATE State code WV
$stateLeagueID State league ID WV000
$localLeagueIDs Local league IDs `WV102

Example ../env File

# Portal Configuration
$PORTAL_URL = "https://portal.lwv.org";
$MEMBERSHIP_URL = "https://portal.lwv.org/groups/53a93df1/league_membership_state_view";

# Connector Settings
$CONNECTOR_URL = "https://mailman-connector.example.org";
$CONNECTOR_PASSWORD = "your_secret_password";
$MAILMAN_LIST_ID = "members.lists.example.org";

# Email Notification Settings
$SMTP_HOST = "mail.example.org";
$SMTP_PORT = "587";
$SMTP_USER = "example@example.org";
$SMTP_PASSWORD = "your_smtp_password";
$EMAIL_FROM = "example@example.org";
$EMAIL_TO = "hello@example.org";

# Google API Configuration
$key = "LALKDdkdk_12LSL_RKFDL";
$STATE = "WV";
$stateLeagueID = "WV000";
$localLeagueIDs = "WV102|WV103|WV112";

Multi-List Configuration with League Filtering

When managing multiple local leagues, you can subscribe different member groups to different lists using the colon-separated format:

# Format: list_id:league_id
# State list gets WV000 members, Morgantown list gets WV103 members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103";

How it works:

  • list_id: The Mailman list identifier (e.g., members.lists.lwvwv.org)
  • league_id: The LWV League ID to filter members (e.g., WV000, WV103, or ALL)
  • If :league_id is omitted, defaults to ALL (all members)

Examples:

# Single list, all members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org";

# Single list, only state (WV000) members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000";

# Multiple lists with different league filters
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103, huntington.lists.lwvwv.org:WV102";

The script filters the roster by League ID before subscribing, so each list only receives its members.

Quick Start

1. Configure Environment

Ensure ../env exists with your settings:

cd automation
# The env file should already exist in the parent directory
ls ../env

2. Build and Start

docker compose up -d

3. Verify Container is Running

docker compose ps | grep lwvwv-subscriber

Usage

Manual Trigger (for testing)

# Run the full workflow (downloads roster, converts, subscribes)
docker compose exec lwvwv-subscriber /app/run-subscription.sh

# With a specific CSV file (skips download)
docker compose exec lwvwv-subscriber /app/run-subscription.sh /path/to/members.csv

# Run subscribe-members.sh directly with env vars
docker compose exec lwvwv-subscriber /app/subscribe-members.sh /tmp/rosters/members.csv

Scheduled Execution (via Ofelia)

The container includes Ofelia labels for automatic scheduling in Robfig CRON expression format:

  • Default: 4x daily at 00:00, 06:00, 12:00, 18:00
  • Format: Cron expression 0 0 0,6,12,18 * * *

To change schedule, edit docker-compose.yml:

labels:
  ofelia.enabled: "true"
  ofelia.job-exec.lwvwv-subscribe.schedule: "0 0 0,6,12,18 * * *"

docker-compose.yml for Ofelia job scheduler

# This service provides ofelia, which is a job scheduler (cron).
# 
# There should be at least on job in one of the services to make this 
# meaningfull to run.
# 
# It is accessible everywhere since it is bound to the docker socket.
# Cron jobs can be conveniently created with labels.
#
# https://github.com/mcuadros/ofelia
#
# When you add/change a cron job simply -
# docker compose restart

services:
  ofelia:
    container_name: ofelia
    image: mcuadros/ofelia:latest
    command: daemon --docker
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Standalone Usage (without Docker)

The scripts also work outside Docker using positional arguments:

# subscribe-members.sh with all positional args
./subscribe-members.sh members.csv mypassword members.lists.example.org https://connector.example.com

# With some env vars, some positional
CONNECTOR_PASSWORD=mypassword ./subscribe-members.sh members.csv

Logs

# View container logs
docker compose logs lwvwv-subscriber

# Follow logs in real-time
docker compose logs -f lwvwv-subscriber

# View Ofelia scheduler logs in Ofelia directory
docker compose logs ofelia

Troubleshooting

Session Expired

If portal download fails with "Session expired":

# Run save-session.js on host to refresh session
cd ..
node save-session.js
# Then restart container
docker compose restart

Missing Environment Variables

If you see errors like CONNECTOR_URL not set:

  1. Check that ../env exists and contains the variable
  2. Verify the entrypoint is parsing it correctly: 
    docker compose exec lwvwv-subscriber env | grep CONNECTOR

Manual CSV Subscription

To subscribe members without downloading:

# Place CSV file in shared volume
cp members.csv automation/rosters/

# Run with specific file
docker compose exec lwvwv-subscriber /app/run-subscription.sh /tmp/rosters/members.csv

Network Requirements

The container needs access to:

  • LWV Portal (https://portal.lwv.org)
  • SMTP server (if using email alerts)

Security Notes

  • ../env is mounted read-only (:ro) in the container
  • Sensitive data is never committed to git
  • Session file (.session.json) is also mounted read-only
  • All scripts use the single ../env file - no duplicated configuration

See Also

  • Main Project README - Standalone script usage, Google Civic API documentation, and general project information
  • mailman-connector - The REST API service used for batch Mailman 3 subscriptions