lwvwv/README.md

# League of Women Voters Member Management Tools

Custom-developed software for the League of Women Voters of West Virginia (LWVWV) and other LWV state/local leagues. This toolkit automates the workflow from portal membership exports to Mailman 3 list subscriptions, with optional legislative district lookups.

## What This Project Does

Managing membership across multiple mailing lists and tracking legislative districts is time-consuming. This software automates:

1. **Data Retrieval** - Download member rosters from the LWV portal
2. **Data Processing** - Convert CSV formats and query Google Civic API for district info
3. **List Management** - Subscribe members to appropriate Mailman 3 lists

## Three Ways to Use This Software

### Standalone Scripts (Manual)
- Download rosters: `download-roster.js`
- Process/district lookup: `google-civic-api.pl`
- Subscribe one list: `subscribe-members.sh`

### Full Automation (Docker + Ofelia)
- Complete hands-off operation
- Scheduled roster downloads
- Multi-list subscription with league filtering
- Error notifications via email

### Hybrid Approach
- Use standalone scripts for one-off tasks
- Use Docker automation for recurring operations

## For Other LWV Leagues

While originally developed for LWVWV, this software can be adapted by any LWV state or local league. The main adaptations needed are:
- Update `$STATE`, `$stateLeagueID`, and `$localLeagueIDs` in `env`
- Configure your LWV portal `$MEMBERSHIP_URL`
- Set up your Mailman lists and `$MAILMAN_LIST_ID`

See [Configuration](#required-file) for details.

## REQUIRED FILE

A file called `env` is required in the same directory as the programs. This file contains configuration for both the Perl and JavaScript scripts.

### Complete env File Example

```
# Portal Configuration (for download-roster.js and save-session.js)
$PORTAL_URL = "https://portal.lwv.org";
$MEMBERSHIP_URL = "https://portal.lwv.org/groups/43a93df1-901a-4676-88c3-f4ea430d4884/league_membership_state_view";

# Email Notification Settings (for download-roster.js - error notifications via s-nail)
$SMTP_HOST = "mail.bikelover.org";
$SMTP_PORT = "587";
$SMTP_USE_STARTTLS = "true";
$SMTP_AUTH = "login";
$SMTP_USER = "your-email@lwv.org";
$SMTP_PASSWORD = "your-password";
$SSL_VERIFY_IGNORE = "ignore";
$EMAIL_FROM = "your-email@lwv.org";
$EMAIL_TO = "recipient@lwv.org";

# Google API Configuration (for google-civic-api.pl)
$key = "your-google-api-key";
$STATE = "WV";
$stateLeagueID = "WV000";
$localLeagueIDs = "WV102|WV103|WV112";
```

### Configuration Details

| Variable | Required | Description |
|----------|----------|-------------|
| `PORTAL_URL` | Yes | Base URL of the LWV portal (default: https://portal.lwv.org) |
| `MEMBERSHIP_URL` | Yes | Your league's membership page URL. Find this by navigating to your league's membership page in the portal and copying the URL |
| `SMTP_HOST` | No | SMTP server hostname for error email notifications (requires s-nail) |
| `SMTP_PORT` | No | SMTP server port (default: 587) |
| `SMTP_USE_STARTTLS` | No | Enable STARTTLS for SMTP (default: true) |
| `SMTP_AUTH` | No | SMTP auth type (default: login) |
| `SMTP_USER` | No | SMTP authentication username |
| `SMTP_PASSWORD` | No | SMTP authentication password |
| `SSL_VERIFY_IGNORE` | No | SSL verify setting for SMTP (default: ignore) |
| `EMAIL_FROM` | No | Sender email address for error notifications |
| `EMAIL_TO` | No | Recipient email address for error notifications |
| `MAILMAN_LIST_ID` | Yes | Target Mailman list ID(s) with optional League ID filter. Format: `list_id:league_id`. Examples: `members.lists.lwvwv.org:WV000` or `list1:WV000, list2:WV103`. If league omitted, defaults to ALL. Used by run-subscription.sh |
| `CONNECTOR_URL` | Yes | URL of mailman-connector API. Used by run-subscription.sh |
| `CONNECTOR_PASSWORD` | Yes | Secret password for connector API. Used by run-subscription.sh |
| `key` | Optional | Google Civic Information API key. See [Google API credentials](https://console.cloud.google.com/apis/credentials) - restrict key to the Google Civic Information API. Required for google-civic-api.pl. |
| `STATE` | Required* | Two-letter state code (e.g., WV). Required for google-civic-api.pl. |
| `stateLeagueID` | Required* | State league ID (e.g., WV000). Required for google-civic-api.pl. |
| `localLeagueIDs` | Required* | Pipe-separated list of local league IDs. Required for google-civic-api.pl. |

*Required for google-civic-api.pl


## JavaScript Scripts (download-roster.js and save-session.js)

These scripts automate downloading the membership roster CSV from the LWV portal.

### Prerequisites

```bash
# Install Node.js dependencies
npm install playwright
npx playwright install chromium

# Install s-nail (required for error email notifications)
# On Debian/Ubuntu:
sudo apt-get install s-nail
```

### First-Time Setup (Run Once)

Before running the download script for the first time, you need to save your browser session:

```bash
node save-session.js
```

This will:
1. Open a browser window
2. Prompt you to log in using the **magic link** method (email)
3. Wait for you to complete login and press Enter
4. Save your session to `.session.json`

The session is valid for 10 years (indefinite), but if it ever expires or fails, running `save-session.js` again will refresh it.

### Downloading the Roster

To download the latest membership CSV:

```bash
node download-roster.js
```

This will:
1. Load your saved session
2. Navigate to the membership page
3. Click Export → Download
4. Save the CSV file with timestamp (e.g., `league_membership_state_view_2026-04-15T05-00-24.csv`)

### Testing Error Email

To test that error emails work correctly:

```bash
# Temporarily rename the session file
mv .session.json .session.json.bak

# Run download - should fail and send error email
node download-roster.js

# Restore the session file
mv .session.json.bak .session.json
```

### Error Notifications

When `download-roster.js` fails (due to missing/expired session, download error, etc.), it will send an error email via **s-nail** to the address specified in `EMAIL_TO`. The email will include:
- The error message
- Instructions to run `save-session.js` to refresh the session

**Note:** s-nail must be installed for error emails to work. See Prerequisites above.


## Perl Script (google-civic-api.pl)

The original script for processing LWVWV membership data. It queries the Google Civic Information API for legislative district information and converts roster CSV files to various formats.

### Description

Copyright (C) 2025 - by Jonathan Rosenbaum

This may be freely redistributed under the terms of the GNU General Public License

### Usage

```bash
./google-civic-api.pl google ./roster-file  (queries Google Civic Api Delegate and Senate District for all LWVWV members)
./google-civic-api.pl WV000 '*.csv' (show all information for members at large, but do not query Google)
./google-civic-api.pl WV000 '*.csv' email (only show email addresses for members at large, but do not query Google)
```

### Arguments

1. **Query Type:**
   - `google` - All members with senate/delegate district query
   - `ALL` - All members without district query
   - `League ID` - Specific league (e.g., WV000, WV102, WV103, WV112)

2. **Roster file** - Path to CSV file

3. **Output format** (optional):
   - `email` - Output only email addresses
   - `email-csv` - Output CSV format: email,first_name,last_name

### Output Formats

**Full CSV (google type):**
```
Name,Email,Phone,Address,Delegate District,Senate District,League ID,Join Date
```

**Email only:**
```
First Last <email@example.com>
```

**Email CSV (for subscription):**
```
email@example.com,FirstName,LastName
```

### Examples

```bash
# Query Google API for all members with district info
./google-civic-api.pl google '*.csv' > 2023-districts.csv

# Get emails for state members only (WV000)
./google-civic-api.pl WV000 '*.csv' email > state-emails.txt

# Generate subscription CSV for all members
./google-civic-api.pl ALL '*.csv' email-csv > members.csv
```

### Column Name Based Parsing

The script now uses column names from the CSV header rather than fixed positions, making it robust against column reordering. If required columns are missing, it will display an error showing which columns were expected and which are available.


## Docker Automation

Copyright (C) 2025 - by Jonathan Rosenbaum

## Automation

### run-subscription.sh

Master orchestration script that coordinates the entire subscription workflow:
1. Downloads roster from LWVWV portal (via download-roster.js)
2. Filters roster by League ID for each list (via google-civic-api.pl)
3. Subscribes members to one or more Mailman lists

**Multi-List Support with League Filtering:**

`MAILMAN_LIST_ID` uses colon-separated format to associate each list with a League ID:
```bash
# Format: list_id:league_id
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103"
```

- **list_id**: The Mailman list identifier (e.g., `members.lists.lwvwv.org`)
- **league_id**: The LWV League ID to filter members (e.g., `WV000`, `WV103`, `ALL`)

If `:league_id` is omitted, defaults to `ALL` (all members).

**Examples:**
```bash
# Single list, all members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org"

# Single list, only WV000 (state) members  
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000"

# Multiple lists with different league filters
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103, huntington.lists.lwvwv.org:WV102"
```

**Usage:**
```bash
./run-subscription.sh [csv_file]
```

If no CSV file is provided, the script will download the latest roster from the portal.

### subscribe-members.sh

Low-level script that subscribes members to a **single** Mailman 3 list via the connector API. Called by run-subscription.sh for each list when multiple lists are configured.

**Note:** This script handles ONE list per invocation. For multiple lists, use run-subscription.sh.

**Prerequisites:**
- `google-civic-api.pl` must be in the same directory or PATH
- Membership CSV file from LWVWV portal

**Usage:**
```bash
./subscribe-members.sh <csv_file> <password> [list_id] [connector_url]
```

**Arguments:**
- `csv_file` - Path to CSV file with format: email,first_name,last_name
- `password` - Secret password for the mailman-connector (required)
- `list_id` - Target mailing list (default: members.lists.example.org)
- `connector_url` - URL of the mailman-connector (default: https://mailman-connector.example.com)

**Examples:**
```bash
# Using default list and connector URL
./subscribe-members.sh members.csv mysecretpassword

# With specific list
./subscribe-members.sh members.csv mysecretpassword test.lists.example.org

# With specific list and connector URL
./subscribe-members.sh members.csv mysecretpassword test.lists.example.org https://connector.example.com
```

**How it works:**
1. Runs `./google-civic-api.pl ALL <membership_file> email-csv` to extract emails
2. Converts output to CSV format: `email,first_name,last_name`
3. Sends batch subscribe request to connector


## Related Projects

- **[mailman-connector](https://git.bikeshopi.dev/bike/mailman-connector)** - REST API service for batch-subscribing members to Mailman 3 lists (used by automation)
- **[automation/](automation/)** - Docker-based automation with Ofelia scheduler for hands-off operation. See [automation/README.md](automation/README.md) for Docker setup and configuration.