lwvwv/README.md

League of Women Voters Member Management Tools

Custom-developed software for the League of Women Voters of West Virginia (LWVWV) and other LWV state/local leagues. This toolkit automates the workflow from portal membership exports to Mailman 3 list subscriptions, with optional legislative district lookups.

What This Project Does

Managing membership across multiple mailing lists and tracking legislative districts is time-consuming. This software automates:

1. Data Retrieval - Download member rosters from the LWV portal
2. Data Processing - Convert CSV formats and query Google Civic API for district info
3. List Management - Subscribe members to appropriate Mailman 3 lists

Three Ways to Use This Software

Standalone Scripts (Manual)

• Download rosters: download-roster.js
• Process/district lookup: google-civic-api.pl
• Subscribe one list: subscribe-members.sh

Full Automation (Docker + Ofelia)

• Complete hands-off operation
• Scheduled roster downloads
• Multi-list subscription with league filtering
• Error notifications via email

Hybrid Approach

• Use standalone scripts for one-off tasks
• Use Docker automation for recurring operations

For Other LWV Leagues

While originally developed for LWVWV, this software can be adapted by any LWV state or local league. The main adaptations needed are:

• Update $STATE, $stateLeagueID, and $localLeagueIDs in env
• Configure your LWV portal $MEMBERSHIP_URL
• Set up your Mailman lists and $MAILMAN_LIST_ID

See Configuration for details.

REQUIRED FILE

A file called env is required in the same directory as the programs. This file contains configuration for both the Perl and JavaScript scripts.

Complete env File Example

# Portal Configuration (for download-roster.js and save-session.js)
$PORTAL_URL = "https://portal.lwv.org";
$MEMBERSHIP_URL = "https://portal.lwv.org/groups/53a93df1/league_membership_state_view";

# Email Notification Settings (for download-roster.js - error notifications via s-nail)
$SMTP_HOST = "mail.example.org";
$SMTP_PORT = "587";
$SMTP_USE_STARTTLS = "true";
$SMTP_AUTH = "login";
$SMTP_USER = "your-email@lwv.org";
$SMTP_PASSWORD = "your-password";
$SSL_VERIFY_IGNORE = "ignore";
$EMAIL_FROM = "your-email@lwv.org";
$EMAIL_TO = "recipient@lwv.org";

# Google API Configuration (for google-civic-api.pl)
$key = "your-google-api-key";
$STATE = "WV";
$stateLeagueID = "WV000";
$localLeagueIDs = "WV102|WV103|WV112";

Configuration Details

Variable	Required	Description
PORTAL_URL	Yes	Base URL of the LWV portal (default: https://portal.lwv.org)
MEMBERSHIP_URL	Yes	Your league's membership page URL. Find this by navigating to your league's membership page in the portal and copying the URL
SMTP_HOST	No	SMTP server hostname for error email notifications (requires s-nail)
SMTP_PORT	No	SMTP server port (default: 587)
SMTP_USE_STARTTLS	No	Enable STARTTLS for SMTP (default: true)
SMTP_AUTH	No	SMTP auth type (default: login)
SMTP_USER	No	SMTP authentication username
SMTP_PASSWORD	No	SMTP authentication password
SSL_VERIFY_IGNORE	No	SSL verify setting for SMTP (default: ignore)
EMAIL_FROM	No	Sender email address for error notifications
EMAIL_TO	No	Recipient email address for error notifications
MAILMAN_LIST_ID	Yes	Target Mailman list ID(s) with optional League ID filter. Format: list_id:league_id. Examples: members.lists.lwvwv.org:WV000 or list1:WV000, list2:WV103. If league omitted, defaults to ALL. Used by run-subscription.sh
CONNECTOR_URL	Yes	URL of mailman-connector API. Used by run-subscription.sh
CONNECTOR_PASSWORD	Yes	Secret password for connector API. Used by run-subscription.sh
key	Optional	Google Civic Information API key. See Google API credentials - restrict key to the Google Civic Information API. Required for google-civic-api.pl.
STATE	Required*	Two-letter state code (e.g., WV). Required for google-civic-api.pl.
stateLeagueID	Required*	State league ID (e.g., WV000). Required for google-civic-api.pl.
localLeagueIDs	Required*	Pipe-separated list of local league IDs. Required for google-civic-api.pl.

*Required for google-civic-api.pl

JavaScript Scripts (download-roster.js and save-session.js)

These scripts automate downloading the membership roster CSV from the LWV portal.

Prerequisites

# Install Node.js dependencies
npm install playwright
npx playwright install chromium

# Install s-nail (required for error email notifications)
# On Debian/Ubuntu:
sudo apt-get install s-nail

First-Time Setup (Run Once)

Before running the download script for the first time, you need to save your browser session:

node save-session.js

This will:

1. Open a browser window
2. Prompt you to log in using the magic link method (email)
3. Wait for you to complete login and press Enter
4. Save your session to .session.json

The session is valid for 10 years (indefinite), but if it ever expires or fails, running save-session.js again will refresh it.

Downloading the Roster

To download the latest membership CSV:

node download-roster.js

This will:

1. Load your saved session
2. Navigate to the membership page
3. Click Export → Download
4. Save the CSV file with timestamp (e.g., league_membership_state_view_2026-04-15T05-00-24.csv)

Testing Error Email

To test that error emails work correctly:

# Temporarily rename the session file
mv .session.json .session.json.bak

# Run download - should fail and send error email
node download-roster.js

# Restore the session file
mv .session.json.bak .session.json

Error Notifications

When download-roster.js fails (due to missing/expired session, download error, etc.), it will send an error email via s-nail to the address specified in EMAIL_TO. The email will include:

• The error message
• Instructions to run save-session.js to refresh the session

Note: s-nail must be installed for error emails to work. See Prerequisites above.

Perl Script (google-civic-api.pl)

The original script for processing LWVWV membership data. It queries the Google Civic Information API for legislative district information and converts roster CSV files to various formats.

Description

Copyright (C) 2025 - by Jonathan Rosenbaum

This may be freely redistributed under the terms of the GNU General Public License

Usage

./google-civic-api.pl google ./roster-file  (queries Google Civic Api Delegate and Senate District for all LWVWV members)
./google-civic-api.pl WV000 '*.csv' (show all information for members at large, but do not query Google)
./google-civic-api.pl WV000 '*.csv' email (only show email addresses for members at large, but do not query Google)

Arguments

1. Query Type:

   • google - All members with senate/delegate district query
   • ALL - All members without district query
   • League ID - Specific league (e.g., WV000, WV102, WV103, WV112)

2. Roster file - Path to CSV file

3. Output format (optional):

   • email - Output only email addresses
   • email-csv - Output CSV format: email,first_name,last_name

Output Formats

Full CSV (google type):

Name,Email,Phone,Address,Delegate District,Senate District,League ID,Join Date

Email only:

First Last <email@example.com>

Email CSV (for subscription):

email@example.com,FirstName,LastName

Examples

# Query Google API for all members with district info
./google-civic-api.pl google '*.csv' > 2023-districts.csv

# Get emails for state members only (WV000)
./google-civic-api.pl WV000 '*.csv' email > state-emails.txt

# Generate subscription CSV for all members
./google-civic-api.pl ALL '*.csv' email-csv > members.csv

Column Name Based Parsing

The script now uses column names from the CSV header rather than fixed positions, making it robust against column reordering. If required columns are missing, it will display an error showing which columns were expected and which are available.

Docker Automation

Copyright (C) 2025 - by Jonathan Rosenbaum

Automation

run-subscription.sh

Master orchestration script that coordinates the entire subscription workflow:

1. Downloads roster from LWVWV portal (via download-roster.js)
2. Filters roster by League ID for each list (via google-civic-api.pl)
3. Subscribes members to one or more Mailman lists

Multi-List Support with League Filtering:

MAILMAN_LIST_ID uses colon-separated format to associate each list with a League ID:

# Format: list_id:league_id
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103"

• list_id: The Mailman list identifier (e.g., members.lists.lwvwv.org)
• league_id: The LWV League ID to filter members (e.g., WV000, WV103, ALL)

If :league_id is omitted, defaults to ALL (all members).

Examples:

# Single list, all members
$MAILMAN_LIST_ID = "members.lists.lwvwv.org"

# Single list, only WV000 (state) members  
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000"

# Multiple lists with different league filters
$MAILMAN_LIST_ID = "members.lists.lwvwv.org:WV000, morgantown.lists.lwvwv.org:WV103, huntington.lists.lwvwv.org:WV102"

Usage:

./run-subscription.sh [csv_file]

If no CSV file is provided, the script will download the latest roster from the portal.

subscribe-members.sh

Low-level script that subscribes members to a single Mailman 3 list via the connector API. Called by run-subscription.sh for each list when multiple lists are configured.

Note: This script handles ONE list per invocation. For multiple lists, use run-subscription.sh.

Prerequisites:

• google-civic-api.pl must be in the same directory or PATH
• Membership CSV file from LWVWV portal

Usage:

./subscribe-members.sh <csv_file> <password> [list_id] [connector_url]

Arguments:

• csv_file - Path to CSV file with format: email,first_name,last_name
• password - Secret password for the mailman-connector (required)
• list_id - Target mailing list (default: members.lists.example.org)
• connector_url - URL of the mailman-connector (default: https://mailman-connector.example.com)

Examples:

# Using default list and connector URL
./subscribe-members.sh members.csv mysecretpassword

# With specific list
./subscribe-members.sh members.csv mysecretpassword test.lists.example.org

# With specific list and connector URL
./subscribe-members.sh members.csv mysecretpassword test.lists.example.org https://connector.example.com

How it works:

1. Runs ./google-civic-api.pl ALL <membership_file> email-csv to extract emails
2. Converts output to CSV format: email,first_name,last_name
3. Sends batch subscribe request to connector

Related Projects

• mailman-connector - REST API service for batch-subscribing members to Mailman 3 lists (used by automation)
• automation/ - Docker-based automation with Ofelia scheduler for hands-off operation. See automation/README.md for Docker setup and configuration.