lwvwv/automation/README.md

# LWVWV Member Subscription Automation

Part of the [LWVWV project](../). This directory contains Docker-based automation for subscribing LWVWV members to Mailman 3 mailing lists via the [mailman-connector](https://git.bikeshopi.dev/bike/mailman-connector) API service.

For standalone script usage (without Docker), see the [main project README](../README.md).

## Overview

This automation coordinates:
1. **Downloading** the latest member roster from the LWVWV portal
2. **Converting** the roster to CSV format (email,first_name,last_name)
3. **Subscribing** members to the Mailman list via the connector API

## Architecture

```
┌─────────┐     ┌─────────────────────────────┐     ┌──────────────────┐
│ Ofelia  │────>│  lwvwv-subscriber-cron      │────>│  LWVWV Portal    │
│Scheduler│cron │  (Docker Container)         │     │  (download CSV)  │
└─────────┘     └─────────────────────────────┘     └──────────────────┘
                         │
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
   ┌──────────┐   ┌──────────────┐   ┌──────────────┐
   │download- │   │google-civic- │   │subscribe-    │
   │roster.js │──>│api.pl        │──>│members.sh    │
   │(portal)  │   │(email-csv)   │   │(connector)   │
   └──────────┘   └──────────────┘   └──────────────┘
                                             │
                                     ▼
                                     ┌──────────────────────────────────┐
                                     │ [mailman-connector](https://git. │
                                     │ bikeshopi.dev/bike/mailman-       │
                                     │ connector) - REST API for batch   │
                                     │ Mailman 3 subscriptions           │
                                     └──────────────────────────────────┘
```

## Files

| File | Description |
|------|-------------|
| `Dockerfile` | Container image definition |
| `docker-compose.yml` | Service orchestration with Ofelia labels |
| `entrypoint.sh` | Parses ../env and exports variables |
| `run-subscription.sh` | Master orchestration script |
| `start-automation.sh` | Quick-start helper script |
| `README.md` | This file |

## Environmental Variables

### Single Source of Truth: `../env`

All configuration is stored in **one file**: `../env` (in the parent directory)

This file uses **Perl-style syntax** (which existing scripts expect):
```perl
$VARIABLE_NAME = "value";
```

### How It Works

1. **`../env`** is the **single source of truth** for all configuration
2. The **entrypoint.sh** script parses `../env` and exports variables as environment variables
3. All scripts can then access them via standard environment variable syntax (`$VARIABLE_NAME`)

### Configuration Variables

#### Required for Subscription
| Variable | Description | Example |
|----------|-------------|---------|
| `$CONNECTOR_URL` | URL of mailman-connector | `https://mailman-connector.example.org` |
| `$CONNECTOR_PASSWORD` | Secret password for connector API | `your_secret_password` |
| `$MAILMAN_LIST_ID` | Target mailing list ID(s). Format: `list_id:league_id` or just `list_id`. For multiple lists, use comma-separated values. | `members.lists.lwvwv.org:WV000` or `list1:WV000, list2:WV103` |

#### Required for Portal Download
| Variable | Description | Example |
|----------|-------------|---------|
| `$PORTAL_URL` | LWVWV portal URL | `https://portal.lwv.org` |
| `$MEMBERSHIP_URL` | Membership page URL | `https://portal.lwv.org/groups/...` |

#### Email Notifications (Optional)
| Variable | Description | Example |
|----------|-------------|---------|
| `$SMTP_HOST` | SMTP server | `mail.example.org` |
| `$SMTP_PORT` | SMTP port | `587` |
| `$SMTP_USER` | SMTP username | `example@example.org` |
| `$SMTP_PASSWORD` | SMTP password | `your_smtp_password` |
| `$EMAIL_FROM` | Sender address | `example@example.org` |
| `$EMAIL_TO` | Recipient for alerts | `hello@example.org` |

#### Google API (for district lookup)
| Variable | Description | Example |
|----------|-------------|---------|
| `$key` | Google Civic API key | `AIzaSy...` |
| `$STATE` | State code | `WV` |
| `$stateLeagueID` | State league ID | `WV000` |
| `$localLeagueIDs` | Local league IDs | `WV102|WV103|WV112` |

### Example `../env` File

```perl
# Portal Configuration
$PORTAL_URL = "https://portal.lwv.org";
$MEMBERSHIP_URL = "https://portal.lwv.org/groups/53a93df1/league_membership_state_view";

# Connector Settings
$CONNECTOR_URL = "https://mailman-connector.example.org";
$CONNECTOR_PASSWORD = "your_secret_password";
$MAILMAN_LIST_ID = "members.lists.example.org";

# Email Notification Settings
$SMTP_HOST = "mail.example.org";
$SMTP_PORT = "587";
$SMTP_USER = "example@example.org";
$SMTP_PASSWORD = "your_smtp_password";
$EMAIL_FROM = "example@example.org";
$EMAIL_TO = "hello@example.org";

# Google API Configuration
$key = "LALKDdkdk_12LSL_RKFDL";
$STATE = "WV";
$stateLeagueID = "WV000";
$localLeagueIDs = "WV102|WV103|WV112";
```

### Multi-List Configuration with League Filtering

When managing multiple local leagues, you can subscribe different member groups to different lists using the colon-separated format:

```perl
# Format: list_id:league_id
# State list gets WV000 members, Morgantown list gets WV103 members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103";
```

**How it works:**
- **list_id**: The Mailman list identifier (e.g., `members.lists.lwvwv.org`)
- **league_id**: The LWV League ID to filter members (e.g., `WV000`, `WV103`, or `ALL`)
- If `:league_id` is omitted, defaults to `ALL` (all members)

**Examples:**
```perl
# Single list, all members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org";

# Single list, only state (WV000) members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000";

# Multiple lists with different league filters
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103, huntington.lists.lwvwv.org:WV102";
```

The script filters the roster by League ID before subscribing, so each list only receives its members.

## Quick Start

### 1. Configure Environment

Ensure `../env` exists with your settings:

```bash
cd automation
# The env file should already exist in the parent directory
ls ../env
```

### 2. Build and Start

```bash
docker compose up -d
```

### 3. Verify Container is Running

```bash
docker compose ps | grep lwvwv-subscriber
```

## Usage

### Manual Trigger (for testing)

```bash
# Run the full workflow (downloads roster, converts, subscribes)
docker compose exec lwvwv-subscriber /app/run-subscription.sh

# With a specific CSV file (skips download)
docker compose exec lwvwv-subscriber /app/run-subscription.sh /path/to/members.csv

# Run subscribe-members.sh directly with env vars
docker compose exec lwvwv-subscriber /app/subscribe-members.sh /tmp/rosters/members.csv
```

### Scheduled Execution (via Ofelia)

The container includes Ofelia labels for automatic scheduling in [Robfig CRON expression](https://pkg.go.dev/github.com/robfig/cron@v1.2.0#hdr-CRON_Expression_Format) format:
- **Default**: 4x daily at 00:00, 06:00, 12:00, 18:00
- **Format**: Cron expression `0 0 0,6,12,18 * * *`

To change schedule, edit `docker-compose.yml`:
```yaml
labels:
  ofelia.enabled: "true"
  ofelia.job-exec.lwvwv-subscribe.schedule: "0 0 0,6,12,18 * * *"
```

<details>

<summary>

```
docker-compose.yml for Ofelia job scheduler
```

</summary>

```
# This service provides ofelia, which is a job scheduler (cron).
# 
# There should be at least on job in one of the services to make this 
# meaningfull to run.
# 
# It is accessible everywhere since it is bound to the docker socket.
# Cron jobs can be conveniently created with labels.
#
# https://github.com/mcuadros/ofelia
#
# When you add/change a cron job simply -
# docker compose restart

services:
  ofelia:
    container_name: ofelia
    image: mcuadros/ofelia:latest
    command: daemon --docker
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
```
</details>
 
### Standalone Usage (without Docker)

The scripts also work outside Docker using positional arguments:

```bash
# subscribe-members.sh with all positional args
./subscribe-members.sh members.csv mypassword members.lists.example.org https://connector.example.com

# With some env vars, some positional
CONNECTOR_PASSWORD=mypassword ./subscribe-members.sh members.csv
```

## Logs

```bash
# Follow Ofelia scheduler logs in real-time in Ofelia directory
docker compose logs -f ofelia 

# View Ofelia scheduler logs in Ofelia directory
docker compose logs ofelia
```

## Troubleshooting

### Session Expired

If portal download fails with "Session expired":
```bash
# Run save-session.js on host to refresh session
cd ..
node save-session.js
# Then restart container
docker compose restart
```

### Missing Environment Variables

If you see errors like `CONNECTOR_URL not set`:
1. Check that `../env` exists and contains the variable
2. Verify the entrypoint is parsing it correctly:
   ```bash
   docker compose exec lwvwv-subscriber env | grep CONNECTOR
   ```

### Manual CSV Subscription

To subscribe members without downloading:
```bash
# Place CSV file in shared volume
cp members.csv automation/rosters/

# Run with specific file
docker compose exec lwvwv-subscriber /app/run-subscription.sh /tmp/rosters/members.csv
```

## Network Requirements

The container needs access to:
- LWV Portal (`https://portal.lwv.org`)
- SMTP server (if using email alerts)

## Security Notes

- `../env` is mounted **read-only** (`:ro`) in the container
- Sensitive data is never committed to git
- Session file (`.session.json`) is also mounted read-only
- All scripts use the single `../env` file - no duplicated configuration


## See Also

- **[Main Project README](../README.md)** - Standalone script usage, Google Civic API documentation, and general project information
- **[mailman-connector](https://git.bikeshopi.dev/bike/mailman-connector)** - The REST API service used for batch Mailman 3 subscriptions