lwvwv/automation/README.md

LWVWV Member Subscription Automation

Automated Docker container for subscribing LWVWV members to Mailman 3 mailing lists.

Overview

This automation coordinates:

1. Downloading the latest member roster from the LWVWV portal
2. Converting the roster to CSV format (email,first_name,last_name)
3. Subscribing members to the Mailman list via the connector API

Architecture

┌─────────┐     ┌─────────────────────────────┐     ┌──────────────────┐
│ Ofelia  │────>│  lwvwv-subscriber-cron      │────>│  LWVWV Portal    │
│Scheduler│cron │  (Docker Container)         │     │  (download CSV)  │
└─────────┘     └─────────────────────────────┘     └──────────────────┘
                         │
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
   ┌──────────┐   ┌──────────────┐   ┌──────────────┐
   │download- │   │google-civic- │   │subscribe-    │
   │roster.js │──>│api.pl        │──>│members.sh    │
   │(portal)  │   │(email-csv)   │   │(connector)   │
   └──────────┘   └──────────────┘   └──────────────┘
                                             │
                                             ▼
                                    ┌──────────────┐
                                    │mailman-      │
                                    │connector     │
                                    │(batch sub)   │
                                    └──────────────┘

Files

File	Description
Dockerfile	Container image definition
docker-compose.yml	Service orchestration with Ofelia labels
entrypoint.sh	Parses ../env and exports variables
run-subscription.sh	Master orchestration script
start-automation.sh	Quick-start helper script
README.md	This file

Environmental Variables

Single Source of Truth: ../env

All configuration is stored in one file: ../env (in the parent directory)

This file uses Perl-style syntax (which existing scripts expect):

$VARIABLE_NAME = "value";

How It Works

1. ../env is the single source of truth for all configuration
2. The entrypoint.sh script parses ../env and exports variables as environment variables
3. All scripts can then access them via standard environment variable syntax ($VARIABLE_NAME)

Configuration Variables

Required for Subscription

Variable	Description	Example
$CONNECTOR_URL	URL of mailman-connector	https://mailman-connector.example.org
$CONNECTOR_PASSWORD	Secret password for connector API	your_secret_password
$MAILMAN_LIST_ID	Target mailing list ID	members.lists.example.org

Required for Portal Download

Variable	Description	Example
$PORTAL_URL	LWVWV portal URL	https://portal.lwv.org
$MEMBERSHIP_URL	Membership page URL	https://portal.lwv.org/groups/...

Email Notifications (Optional)

Variable	Description	Example
$SMTP_HOST	SMTP server	mail.example.org
$SMTP_PORT	SMTP port	587
$SMTP_USER	SMTP username	example@example.org
$SMTP_PASSWORD	SMTP password	your_smtp_password
$EMAIL_FROM	Sender address	example@example.org
$EMAIL_TO	Recipient for alerts	hello@example.org

Google API (for district lookup)

Variable	Description	Example
$key	Google Civic API key	AIzaSy...
$STATE	State code	WV
$stateLeagueID	State league ID	WV000
$localLeagueIDs	Local league IDs	`WV102

Example ../env File

# Portal Configuration
$PORTAL_URL = "https://portal.lwv.org";
$MEMBERSHIP_URL = "https://portal.lwv.org/groups/53a93df1/league_membership_state_view";

# Connector Settings
$CONNECTOR_URL = "https://mailman-connector.example.org";
$CONNECTOR_PASSWORD = "your_secret_password";
$MAILMAN_LIST_ID = "members.lists.example.org";

# Email Notification Settings
$SMTP_HOST = "mail.example.org";
$SMTP_PORT = "587";
$SMTP_USER = "example@example.org";
$SMTP_PASSWORD = "your_smtp_password";
$EMAIL_FROM = "example@example.org";
$EMAIL_TO = "hello@example.org";

# Google API Configuration
$key = "LALKDdkdk_12LSL_RKFDL";
$STATE = "WV";
$stateLeagueID = "WV000";
$localLeagueIDs = "WV102|WV103|WV112";

Quick Start

1. Configure Environment

Ensure ../env exists with your settings:

cd automation
# The env file should already exist in the parent directory
ls ../env

2. Build and Start

docker-compose up -d

3. Verify Container is Running

docker ps | grep lwvwv-subscriber

Usage

Manual Trigger (for testing)

# Run the full workflow (downloads roster, converts, subscribes)
docker exec lwvwv-subscriber /app/run-subscription.sh

# With a specific CSV file (skips download)
docker exec lwvwv-subscriber /app/run-subscription.sh /path/to/members.csv

# Run subscribe-members.sh directly with env vars
docker exec lwvwv-subscriber /app/subscribe-members.sh /tmp/rosters/members.csv

Scheduled Execution (via Ofelia)

The container includes Ofelia labels for automatic scheduling:

• Default: 4x daily at 00:00, 06:00, 12:00, 18:00
• Format: Cron expression 0 0,6,12,18 * * *

To change schedule, edit docker-compose.yml:

labels:
  ofelia.enabled: "true"
  ofelia.job-exec.lwvwv-subscribe.schedule: "0 0,6,12,18 * * *"

Standalone Usage (without Docker)

The scripts also work outside Docker using positional arguments:

# subscribe-members.sh with all positional args
./subscribe-members.sh members.csv mypassword members.lists.example.org https://connector.example.com

# With some env vars, some positional
CONNECTOR_PASSWORD=mypassword ./subscribe-members.sh members.csv

Logs

# View container logs
docker logs lwvwv-subscriber

# Follow logs in real-time
docker logs -f lwvwv-subscriber

# View Ofelia scheduler logs
docker logs ofelia

Troubleshooting

Session Expired

If portal download fails with "Session expired":

# Run save-session.js on host to refresh session
cd ..
node save-session.js
# Then restart container
docker-compose restart

Missing Environment Variables

If you see errors like CONNECTOR_URL not set:

1. Check that ../env exists and contains the variable
2. Verify the entrypoint is parsing it correctly:

   docker exec lwvwv-subscriber env | grep CONNECTOR
   

Manual CSV Subscription

To subscribe members without downloading:

# Place CSV file in shared volume
cp members.csv automation/rosters/

# Run with specific file
docker exec lwvwv-subscriber /app/run-subscription.sh /tmp/rosters/members.csv

Network Requirements

The container needs access to:

• LWV Portal (https://portal.lwv.org)
• SMTP server (if using email alerts)

Security Notes

• ../env is mounted read-only (:ro) in the container
• Sensitive data is never committed to git
• Session file (.session.json) is also mounted read-only
• All scripts use the single ../env file - no duplicated configuration