bike/lwvwv
1
0
Fork 0
Code Issues 2 Pull Requests Projects Releases Wiki Activity

Adds automation in a containerized environment

Browse Source
This commit is contained in:
bike
2026-05-04 23:35:54 -04:00
parent 1dca98898e
commit 203bffbbf8
10 changed files with 733 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
automation/README.md
+245
View File
@@ -0,0 +1,245 @@
# LWVWV Member Subscription Automation
Automated Docker container for subscribing LWVWV members to Mailman 3 mailing lists.
## Overview
This automation coordinates:
1. **Downloading** the latest member roster from the LWVWV portal
2. **Converting** the roster to CSV format (email,first_name,last_name)
3. **Subscribing** members to the Mailman list via the connector API
## Architecture
```
┌─────────┐ ┌─────────────────────────────┐ ┌──────────────────┐
│ Ofelia │────>│ lwvwv-subscriber-cron │────>│ LWVWV Portal │
│Scheduler│cron │ (Docker Container) │ │ (download CSV) │
└─────────┘ └─────────────────────────────┘ └──────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────────┐ ┌──────────────┐
│download- │ │google-civic- │ │subscribe- │
│roster.js │──>│api.pl │──>│members.sh │
│(portal) │ │(email-csv) │ │(connector) │
└──────────┘ └──────────────┘ └──────────────┘
│
▼
┌──────────────┐
│mailman- │
│connector │
│(batch sub) │
└──────────────┘
```
## Files
| File | Description |
|------|-------------|
| `Dockerfile` | Container image definition |
| `docker-compose.yml` | Service orchestration with Ofelia labels |
| `entrypoint.sh` | Parses ../env and exports variables |
| `run-subscription.sh` | Master orchestration script |
| `start-automation.sh` | Quick-start helper script |
| `README.md` | This file |
## Environmental Variables
### Single Source of Truth: `../env`
All configuration is stored in **one file**: `../env` (in the parent directory)
This file uses **Perl-style syntax** (which existing scripts expect):
```perl
$VARIABLE_NAME = "value";
```
### How It Works
1. **`../env`** is the **single source of truth** for all configuration
2. The **entrypoint.sh** script parses `../env` and exports variables as environment variables
3. All scripts can then access them via standard environment variable syntax (`$VARIABLE_NAME`)
### Configuration Variables
#### Required for Subscription
| Variable | Description | Example |
|----------|-------------|---------|
| `$CONNECTOR_URL` | URL of mailman-connector | `https://mailman-connector.example.org` |
| `$CONNECTOR_PASSWORD` | Secret password for connector API | `your_secret_password` |
| `$MAILMAN_LIST_ID` | Target mailing list ID | `members.lists.example.org` |
#### Required for Portal Download
| Variable | Description | Example |
|----------|-------------|---------|
| `$PORTAL_URL` | LWVWV portal URL | `https://portal.lwv.org` |
| `$MEMBERSHIP_URL` | Membership page URL | `https://portal.lwv.org/groups/...` |
#### Email Notifications (Optional)
| Variable | Description | Example |
|----------|-------------|---------|
| `$SMTP_HOST` | SMTP server | `mail.example.org` |
| `$SMTP_PORT` | SMTP port | `587` |
| `$SMTP_USER` | SMTP username | `example@example.org` |
| `$SMTP_PASSWORD` | SMTP password | `your_smtp_password` |
| `$EMAIL_FROM` | Sender address | `example@example.org` |
| `$EMAIL_TO` | Recipient for alerts | `hello@example.org` |
#### Google API (for district lookup)
| Variable | Description | Example |
|----------|-------------|---------|
| `$key` | Google Civic API key | `AIzaSy...` |
| `$STATE` | State code | `WV` |
| `$stateLeagueID` | State league ID | `WV000` |
| `$localLeagueIDs` | Local league IDs | `WV102|WV103|WV112` |
### Example `../env` File
```perl
# Portal Configuration
$PORTAL_URL = "https://portal.lwv.org";
$MEMBERSHIP_URL = "https://portal.lwv.org/groups/53a93df1/league_membership_state_view";
# Connector Settings
$CONNECTOR_URL = "https://mailman-connector.example.org";
$CONNECTOR_PASSWORD = "your_secret_password";
$MAILMAN_LIST_ID = "members.lists.example.org";
# Email Notification Settings
$SMTP_HOST = "mail.example.org";
$SMTP_PORT = "587";
$SMTP_USER = "example@example.org";
$SMTP_PASSWORD = "your_smtp_password";
$EMAIL_FROM = "example@example.org";
$EMAIL_TO = "hello@example.org";
# Google API Configuration
$key = "LALKDdkdk_12LSL_RKFDL";
$STATE = "WV";
$stateLeagueID = "WV000";
$localLeagueIDs = "WV102|WV103|WV112";
```
## Quick Start
### 1. Configure Environment
Ensure `../env` exists with your settings:
```bash
cd automation
# The env file should already exist in the parent directory
ls ../env
```
### 2. Build and Start
```bash
docker-compose up -d
```
### 3. Verify Container is Running
```bash
docker ps | grep lwvwv-subscriber
```
## Usage
### Manual Trigger (for testing)
```bash
# Run the full workflow (downloads roster, converts, subscribes)
docker exec lwvwv-subscriber /app/run-subscription.sh
# With a specific CSV file (skips download)
docker exec lwvwv-subscriber /app/run-subscription.sh /path/to/members.csv
# Run subscribe-members.sh directly with env vars
docker exec lwvwv-subscriber /app/subscribe-members.sh /tmp/rosters/members.csv
```
### Scheduled Execution (via Ofelia)
The container includes Ofelia labels for automatic scheduling:
- **Default**: 4x daily at 00:00, 06:00, 12:00, 18:00
- **Format**: Cron expression `0 0,6,12,18 * * *`
To change schedule, edit `docker-compose.yml`:
```yaml
labels:
ofelia.enabled: "true"
ofelia.job-exec.lwvwv-subscribe.schedule: "0 0,6,12,18 * * *"
```
### Standalone Usage (without Docker)
The scripts also work outside Docker using positional arguments:
```bash
# subscribe-members.sh with all positional args
./subscribe-members.sh members.csv mypassword members.lists.example.org https://connector.example.com
# With some env vars, some positional
CONNECTOR_PASSWORD=mypassword ./subscribe-members.sh members.csv
```
## Logs
```bash
# View container logs
docker logs lwvwv-subscriber
# Follow logs in real-time
docker logs -f lwvwv-subscriber
# View Ofelia scheduler logs
docker logs ofelia
```
## Troubleshooting
### Session Expired
If portal download fails with "Session expired":
```bash
# Run save-session.js on host to refresh session
cd ..
node save-session.js
# Then restart container
docker-compose restart
```
### Missing Environment Variables
If you see errors like `CONNECTOR_URL not set`:
1. Check that `../env` exists and contains the variable
2. Verify the entrypoint is parsing it correctly:
```bash
docker exec lwvwv-subscriber env | grep CONNECTOR
```
### Manual CSV Subscription
To subscribe members without downloading:
```bash
# Place CSV file in shared volume
cp members.csv automation/rosters/
# Run with specific file
docker exec lwvwv-subscriber /app/run-subscription.sh /tmp/rosters/members.csv
```
## Network Requirements
The container needs access to:
- LWV Portal (`https://portal.lwv.org`)
- SMTP server (if using email alerts)
## Security Notes
- `../env` is mounted **read-only** (`:ro`) in the container
- Sensitive data is never committed to git
- Session file (`.session.json`) is also mounted read-only
- All scripts use the single `../env` file - no duplicated configuration
automation/docker-compose.yml
+28
View File
@@ -0,0 +1,28 @@
services:
lwvwv-subscriber:
build:
context: ..
dockerfile: Dockerfile
image: lwvwv-subscriber
container_name: lwvwv-subscriber
volumes:
# Mount parent directory env file (source of truth)
- ../env:/app/env:ro
# Mount session file for portal authentication
- ../.session.json:/app/.session.json:ro
# Optional: Persist roster files for debugging
- ./rosters:/tmp/rosters
networks:
- default
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
# Ofelia labels for scheduling (4x daily)
labels:
ofelia.enabled: "true"
ofelia.job-exec.lwvwv-subscribe.schedule: "0 0,6,12,18 * * *"
ofelia.job-exec.lwvwv-subscribe.container: "lwvwv-subscriber"
ofelia.job-exec.lwvwv-subscribe.command: "/app/run-subscription.sh"
restart: always
automation/entrypoint.sh
Executable
+27
View File
@@ -0,0 +1,27 @@
#!/bin/bash
# Entrypoint script for LWVWV subscriber container
# Parses ../env (Perl format) and exports variables as environment variables
# Then executes the main command
ENV_FILE="/app/env"
if [ -f "$ENV_FILE" ]; then
echo "Loading configuration from $ENV_FILE..."
while IFS= read -r line || [ -n "$line" ]; do
# Skip empty lines and comments
[[ -z "$line" || "$line" =~ ^# ]] && continue
# Parse Perl format: $VAR = "value" or $VAR = "value";
if [[ $line =~ ^\$([A-Za-z_][A-Za-z0-9_]*)[[:space:]]*=[[:space:]]*\"([^\"]*)\"[[:space:]]*\;?$ ]]; then
varname="${BASH_REMATCH[1]}"
varvalue="${BASH_REMATCH[2]}"
export "$varname=$varvalue"
fi
done < "$ENV_FILE"
echo "Configuration loaded successfully"
else
echo "Warning: $ENV_FILE not found"
fi
# Execute the main command
exec "$@"
automation/run-subscription.sh
Executable
+198
View File
@@ -0,0 +1,198 @@
#!/bin/bash
source "$(dirname "$0")/entrypoint.sh" 2>/dev/null || true
# Master orchestration script for LWVWV member subscription automation
# This script coordinates the entire workflow:
# 1. Download roster from LWVWV portal (if download-roster.js is available)
# 2. Convert roster to CSV format, filtered by League ID per list
# 3. Subscribe members to Mailman list(s) via connector API
# 4. Cleanup temporary files
#
# Environment Variables (loaded from /app/env via entrypoint.sh):
# CONNECTOR_URL - URL of mailman-connector
# CONNECTOR_PASSWORD - Secret password for connector API
# MAILMAN_LIST_ID - Target mailing list ID(s). Format: "list_id:league_id, list_id:league_id"
# If :league_id is omitted, defaults to "ALL" (all members)
# EMAIL_TO - Email for failure notifications
# SMTP_HOST - SMTP server for notifications
#
# Examples:
# Single list, all members: "members.lists.lwvwv.org"
# Single list, WV000 only: "members.lists.lwvwv.org:WV000"
# Multiple lists: "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103"
set -e # Exit on error
# Configuration from environment
WORK_DIR="${WORK_DIR:-/tmp/rosters}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
ROSTER_FILE="$WORK_DIR/roster_$TIMESTAMP.csv"
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}
error_exit() {
echo "[ERROR] $1" >&2
# Send alert email if configured
if [ -n "$EMAIL_TO" ] && [ -n "$SMTP_HOST" ]; then
echo "$1" | s-nail -s "LWVWV Subscription Failed" "$EMAIL_TO" 2>/dev/null || true
fi
exit 1
}
# Create work directory
mkdir -p "$WORK_DIR"
# Determine CSV input source or download
if [ -n "$1" ]; then
# CSV file provided as argument
ROSTER_FILE="$1"
log "Using provided roster file: $ROSTER_FILE"
if [ ! -f "$ROSTER_FILE" ]; then
error_exit "Roster file not found: $ROSTER_FILE"
fi
else
# No CSV provided - try to download from portal
if [ ! -f "./download-roster.js" ]; then
error_exit "No CSV file provided and download-roster.js not found"
fi
# Step 1: Download roster
log "Downloading roster from LWVWV portal..."
if ! node download-roster.js; then
error_exit "Failed to download roster"
fi
# Find the downloaded file
ROSTER_FILE=$(ls -t league_membership_state_view_*.csv 2>/dev/null | head -1)
if [ -z "$ROSTER_FILE" ]; then
error_exit "No roster file found after download"
fi
log "Downloaded roster: $ROSTER_FILE"
fi
# Validate required environment variables
if [ -z "$CONNECTOR_URL" ]; then
error_exit "CONNECTOR_URL not set. Ensure /app/env is mounted and contains \$CONNECTOR_URL"
fi
if [ -z "$CONNECTOR_PASSWORD" ]; then
error_exit "CONNECTOR_PASSWORD not set. Ensure /app/env is mounted and contains \$CONNECTOR_PASSWORD"
fi
if [ -z "$MAILMAN_LIST_ID" ]; then
error_exit "MAILMAN_LIST_ID not set. Ensure /app/env is mounted and contains \$MAILMAN_LIST_ID"
fi
if [ ! -f "./google-civic-api.pl" ]; then
error_exit "google-civic-api.pl not found for roster conversion"
fi
if [ ! -f "./subscribe-members.sh" ]; then
error_exit "subscribe-members.sh not found"
fi
# Parse multiple list configurations (comma-separated)
# Format: list_id:league_id or just list_id (defaults to ALL)
IFS=',' read -ra LIST_CONFIGS <<< "$MAILMAN_LIST_ID"
log "=========================================="
log "Starting batch subscription"
log "=========================================="
log "Roster File: $ROSTER_FILE"
log "Connector URL: $CONNECTOR_URL"
log "Total Lists: ${#LIST_CONFIGS[@]}"
log "=========================================="
# Track success/failure per list
SUCCESS_COUNT=0
FAILED_LISTS=()
# Helper function to increment counter (avoids set -e issues with arithmetic)
increment_count() {
SUCCESS_COUNT=$((SUCCESS_COUNT + 1))
}
# Loop through each list configuration
for CONFIG in "${LIST_CONFIGS[@]}"; do
# Trim whitespace
CONFIG=$(echo "$CONFIG" | xargs)
# Parse list_id and league_id (format: list_id:league_id or just list_id)
if [[ "$CONFIG" =~ ^([^:]+):(.*)$ ]]; then
LIST_ID="${BASH_REMATCH[1]}"
LEAGUE_ID="${BASH_REMATCH[2]}"
else
LIST_ID="$CONFIG"
LEAGUE_ID="ALL"
fi
log ""
log "----------------------------------------"
log "Processing List: $LIST_ID"
log "League Filter: $LEAGUE_ID"
log "----------------------------------------"
# Generate filtered CSV for this list
FILTERED_CSV="$WORK_DIR/members_${LEAGUE_ID}_${TIMESTAMP}.csv"
log "Filtering roster for League ID: $LEAGUE_ID..."
if ! ./google-civic-api.pl "$LEAGUE_ID" "$ROSTER_FILE" email-csv > "$FILTERED_CSV" 2>/dev/null; then
log "✗ Failed to filter roster for $LEAGUE_ID"
FAILED_LISTS+=("$LIST_ID")
rm -f "$FILTERED_CSV"
continue
fi
# Check if any members were found
MEMBER_COUNT=$(wc -l < "$FILTERED_CSV" | tr -d ' ')
log "Found $MEMBER_COUNT members for $LEAGUE_ID"
if [ "$MEMBER_COUNT" -eq 0 ]; then
log "⚠ No members found for League ID $LEAGUE_ID, skipping $LIST_ID"
rm -f "$FILTERED_CSV"
continue
fi
# Subscribe members to this list
log "Subscribing $MEMBER_COUNT members to $LIST_ID..."
if ./subscribe-members.sh "$FILTERED_CSV" "$CONNECTOR_PASSWORD" "$LIST_ID" "$CONNECTOR_URL"; then
log "✓ Successfully subscribed to $LIST_ID"
increment_count
# Clean up filtered CSV on success
rm -f "$FILTERED_CSV"
else
log "✗ Failed to subscribe to $LIST_ID"
FAILED_LISTS+=("$LIST_ID")
# Keep filtered CSV for debugging on failure
log " Filtered CSV retained: $FILTERED_CSV"
fi
done
# Summary
log ""
log "=========================================="
log "Subscription Summary"
log "=========================================="
log "Total Lists: ${#LIST_CONFIGS[@]}"
log "Successful: $SUCCESS_COUNT"
log "Failed: ${#FAILED_LISTS[@]}"
log "=========================================="
# Cleanup original roster if we downloaded it
if [ -z "$1" ] && [ -f "$ROSTER_FILE" ]; then
log "Cleaning up roster file..."
rm -f "$ROSTER_FILE"
fi
# If any lists failed, report error
if [ ${#FAILED_LISTS[@]} -gt 0 ]; then
error_exit "Failed to subscribe to the following list(s): ${FAILED_LISTS[*]}"
fi
log "Subscription process completed successfully"
exit 0
automation/start-automation.sh
Executable
+93
View File
@@ -0,0 +1,93 @@
#!/bin/bash
# Quick start script for LWVWV automation
# Usage: ./start-automation.sh [command]
#
# Commands:
# start - Start the automation container
# stop - Stop the automation container
# restart - Restart the automation container
# logs - View container logs
# run-now - Trigger subscription immediately
# status - Check container status
set -e
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_DIR"
show_help() {
cat << EOF
LWVWV Automation Quick Start
Usage: $0 [command]
Commands:
start Start the automation container
stop Stop the automation container
restart Restart the automation container
logs View container logs
run-now Trigger subscription immediately
status Check container status
build Rebuild the container image
help Show this help message
Examples:
$0 start # Start automation
$0 run-now # Run subscription now
$0 logs # View logs
$0 logs -f # Follow logs
EOF
}
case "${1:-help}" in
start)
echo "Starting LWVWV automation container..."
docker-compose up -d
echo "Container started. Use '$0 logs' to view logs."
;;
stop)
echo "Stopping LWVWV automation container..."
docker-compose down
echo "Container stopped."
;;
restart)
echo "Restarting LWVWV automation container..."
docker-compose restart
echo "Container restarted."
;;
logs)
echo "Viewing container logs..."
shift # Remove 'logs' from args
docker-compose logs "$@"
;;
run-now)
echo "Triggering subscription immediately..."
docker exec lwvwv-subscriber /app/run-subscription.sh
;;
status)
echo "Container status:"
docker-compose ps
;;
build)
echo "Rebuilding container image..."
docker-compose build --no-cache
echo "Build complete. Use '$0 start' to start."
;;
help|--help|-h)
show_help
;;
*)
echo "Unknown command: $1"
show_help
exit 1
;;
esac