feat: add batch subscribe and list whitelist support
- Add subscribe_batch endpoint for subscribing multiple members from CSV - Add list whitelist: MAILMAN_LIST_ID supports multiple comma-separated lists - Add IP whitelist: REQUESTOR_IP supports multiple comma-separated IPs - Add optional list_id override for targeting specific lists - Add test-subscribe-members2.sh utility script - Update README with comprehensive documentation and examples
This commit is contained in:
parent
4feeb3faf5
commit
0d01f3e0b8
2
.gitignore
vendored
2
.gitignore
vendored
@ -1 +1,3 @@
|
|||||||
.env
|
.env
|
||||||
|
test*csv
|
||||||
|
tests
|
||||||
|
|||||||
399
README.md
399
README.md
@ -1,55 +1,392 @@
|
|||||||
# Mailman Connector
|
# Mailman Connector
|
||||||
|
|
||||||
This program provides an API server to receive requests to subscribe and unsubscribe users from a MailMan 3 email list. The requester must provide a secret password to utilize the service.
|
A secure API bridge for Mailman 3 that enables subscription management without exposing the Mailman REST API publicly. Designed to run alongside Dockerized Mailman Core on the same Docker network.
|
||||||
|
|
||||||
Mailman Connector is designed to run on the same host as a Dockerized version of Mailman and shares the same Docker network (`mailman`). This setup ensures that you don't need to expose Mailman's API publicly. The `HOSTNAME` environment variable should be set to the Docker service name specified for `mailman-core` provided by `maxking/mailman-core`.
|
Mailman Connector is designed to run on the same host as a Dockerized version of Mailman and shares the same Docker network (`mailman`). This setup ensures that you don't need to expose Mailman's API publicly. The `HOSTNAME` environment variable should be set to the Docker service name specified for `mailman-core` provided by `maxking/mailman-core`.
|
||||||
|
|
||||||
## Example Request Code (PHP)
|
## Quick Start
|
||||||
Here's an example of how to send a request using PHP:
|
|
||||||
|
1. **Configure environment variables** in `.env` or `docker-compose.yml`:
|
||||||
|
```bash
|
||||||
|
SECRET_PASSWORD=your_secret_password
|
||||||
|
MAILMAN_USERNAME=rest-admin
|
||||||
|
MAILMAN_PASSWORD=rest_password
|
||||||
|
MAILMAN_LIST_ID=list.domain.org,test.lists.domain.org
|
||||||
|
HOSTNAME=mailman-core
|
||||||
|
PORT=8001
|
||||||
|
HOST_PORT=10000
|
||||||
|
REQUESTOR_IP=1.2.3.4,5.6.7.8
|
||||||
```
|
```
|
||||||
$json = array(
|
|
||||||
|
2. **Start the container**:
|
||||||
|
```bash
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Test with curl** (single subscribe):
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"subscribe":"subscribe","password":"your_secret_password","email":"test@example.com","first_name":"Test","last_name":"User"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Or test batch subscribe (multiple members at once):
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"subscribe_batch":"subscribe_batch","password":"your_secret_password","csv_data":"user1@example.com,John,Doe\nuser2@example.com,Jane,Smith"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Single Subscribe**: Add individual members to mailing lists
|
||||||
|
- **Batch Subscribe**: Subscribe multiple members from CSV data with automatic duplicate detection
|
||||||
|
- **Unsubscribe**: Remove members from mailing lists
|
||||||
|
- **List Whitelist**: Control which mailing lists clients can access via `MAILMAN_LIST_ID`
|
||||||
|
- **IP Restrictions**: Limit access by IP address with support for multiple IPs
|
||||||
|
- **Security**: Secret password authentication, optional IP whitelisting, HTTPS support
|
||||||
|
|
||||||
|
## Environmental Variables
|
||||||
|
|
||||||
|
Set these in your `docker-compose.yml` or `.env` file:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Required
|
||||||
|
SECRET_PASSWORD=your_secret_password
|
||||||
|
MAILMAN_USERNAME=rest-administrator-username
|
||||||
|
MAILMAN_PASSWORD=password-for-restadmin
|
||||||
|
MAILMAN_LIST_ID=list.domain.org
|
||||||
|
HOSTNAME=mailman-core
|
||||||
|
PORT=8001
|
||||||
|
|
||||||
|
# Optional
|
||||||
|
HOST_PORT=10000
|
||||||
|
CONTAINER_PORT=10000
|
||||||
|
REQUESTOR_IP=1.2.3.4,5.6.7.8
|
||||||
|
LETSENCRYPT_HOST=connector.example.com
|
||||||
|
LOGLEVEL=warn
|
||||||
|
ENV=production
|
||||||
|
```
|
||||||
|
|
||||||
|
### MAILMAN_LIST_ID Configuration
|
||||||
|
|
||||||
|
Control which mailing lists clients can access:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Single list (backward compatible)
|
||||||
|
MAILMAN_LIST_ID=list.domain.org
|
||||||
|
|
||||||
|
# Multiple allowed lists (first is default)
|
||||||
|
MAILMAN_LIST_ID=list.domain.org,test.lists.domain.org,dev.lists.domain.org
|
||||||
|
```
|
||||||
|
|
||||||
|
When multiple lists are configured, clients can specify `list_id` in requests. Only whitelisted lists are permitted.
|
||||||
|
|
||||||
|
### REQUESTOR_IP Configuration
|
||||||
|
|
||||||
|
Restrict access by IP address:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Single IP
|
||||||
|
REQUESTOR_IP=1.2.3.4
|
||||||
|
|
||||||
|
# Multiple IPs (comma-separated)
|
||||||
|
REQUESTOR_IP=1.2.3.4,5.6.7.8,9.10.11.12
|
||||||
|
|
||||||
|
# No restriction (allow all)
|
||||||
|
REQUESTOR_IP=
|
||||||
|
```
|
||||||
|
|
||||||
|
## Request Formats
|
||||||
|
|
||||||
|
### Single Subscribe (Default List)
|
||||||
|
|
||||||
|
Subscribe an individual member using the default list (first in `MAILMAN_LIST_ID`):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"subscribe": "subscribe",
|
||||||
|
"password": "your_secret_password",
|
||||||
|
"email": "user@example.com",
|
||||||
|
"first_name": "John",
|
||||||
|
"last_name": "Doe"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Single Subscribe (Override List)
|
||||||
|
|
||||||
|
Subscribe to a specific list (must be in whitelist):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"subscribe": "subscribe",
|
||||||
|
"password": "your_secret_password",
|
||||||
|
"list_id": "test.lists.domain.org",
|
||||||
|
"email": "user@example.com",
|
||||||
|
"first_name": "John",
|
||||||
|
"last_name": "Doe"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch Subscribe (Default List)
|
||||||
|
|
||||||
|
Subscribe multiple members from CSV data:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"subscribe_batch": "subscribe_batch",
|
||||||
|
"password": "your_secret_password",
|
||||||
|
"csv_data": "user1@example.com,John,Doe\nuser2@example.com,Jane,Smith\nuser3@example.com,Bob,Johnson"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Batch Subscribe (Override List)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"subscribe_batch": "subscribe_batch",
|
||||||
|
"password": "your_secret_password",
|
||||||
|
"list_id": "test.lists.domain.org",
|
||||||
|
"csv_data": "user1@example.com,John,Doe\nuser2@example.com,Jane,Smith"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Unsubscribe (Default List)
|
||||||
|
|
||||||
|
Remove a member from the default list:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"password": "your_secret_password",
|
||||||
|
"email": "user@example.com"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Unsubscribe (Override List)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"password": "your_secret_password",
|
||||||
|
"list_id": "test.lists.domain.org",
|
||||||
|
"email": "user@example.com"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing with curl
|
||||||
|
|
||||||
|
### 1. Single Subscribe (Default List)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"subscribe": "subscribe",
|
||||||
|
"password": "secret",
|
||||||
|
"email": "test1@example.com",
|
||||||
|
"first_name": "Alice",
|
||||||
|
"last_name": "Anderson"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Single Subscribe (Override List)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"subscribe": "subscribe",
|
||||||
|
"password": "secret",
|
||||||
|
"list_id": "test.lists.example.org",
|
||||||
|
"email": "test2@example.com",
|
||||||
|
"first_name": "Bob",
|
||||||
|
"last_name": "Brown"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Batch Subscribe (Default List)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"subscribe_batch": "subscribe_batch",
|
||||||
|
"password": "secret",
|
||||||
|
"csv_data": "user1@example.com,John,Doe\nuser2@example.com,Jane,Smith"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Batch Subscribe (Override List)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"subscribe_batch": "subscribe_batch",
|
||||||
|
"password": "secret",
|
||||||
|
"list_id": "test.lists.example.org",
|
||||||
|
"csv_data": "user3@example.com,Mike,Johnson\nuser4@example.com,Sarah,Williams"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Unsubscribe (Default List)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"password": "secret",
|
||||||
|
"email": "test1@example.com"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Unsubscribe (Override List)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST https://mailman-connector.example.com \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"password": "secret",
|
||||||
|
"list_id": "test.lists.example.org",
|
||||||
|
"email": "test2@example.com"
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Request Code (PHP)
|
||||||
|
|
||||||
|
Here's an example of how to send a request using PHP:
|
||||||
|
|
||||||
|
```php
|
||||||
|
<?php
|
||||||
|
$json = array(
|
||||||
'subscribe' => $_POST['email_list_connector'],
|
'subscribe' => $_POST['email_list_connector'],
|
||||||
'password' => $email_list_connector_password,
|
'password' => $email_list_connector_password,
|
||||||
'email' => $_POST['email'],
|
'email' => $_POST['email'],
|
||||||
'first_name' => $_POST['first_name'],
|
'first_name' => $_POST['first_name'],
|
||||||
'last_name' => $_POST['last_name'],
|
'last_name' => $_POST['last_name'],
|
||||||
);
|
);
|
||||||
|
|
||||||
$ch = curl_init();
|
$ch = curl_init();
|
||||||
$curlConfig = array(
|
$curlConfig = array(
|
||||||
CURLOPT_URL => $email_list_connector,
|
CURLOPT_URL => $email_list_connector,
|
||||||
CURLOPT_POST => true,
|
CURLOPT_POST => true,
|
||||||
CURLOPT_SSL_VERIFYPEER => true,
|
CURLOPT_SSL_VERIFYPEER => true,
|
||||||
CURLOPT_RETURNTRANSFER => true,
|
CURLOPT_RETURNTRANSFER => true,
|
||||||
CURLOPT_POSTFIELDS => json_encode($json),
|
CURLOPT_POSTFIELDS => json_encode($json),
|
||||||
);
|
);
|
||||||
|
|
||||||
// $ssl_certificate = SSL_CERTIFICATE;
|
if ($ssl_certificate) {
|
||||||
// if( !defined( 'SSL_CERTIFICATE' ) ) define ("SSL_CERTIFICATE", "/var/www/html/examples/cert.pem");
|
|
||||||
if ($ssl_certificate) {
|
|
||||||
$curlConfig[CURLOPT_CAINFO] = $ssl_certificate;
|
$curlConfig[CURLOPT_CAINFO] = $ssl_certificate;
|
||||||
|
}
|
||||||
|
|
||||||
|
curl_setopt_array($ch, $curlConfig);
|
||||||
|
$result = curl_exec($ch);
|
||||||
|
curl_close($ch);
|
||||||
|
|
||||||
|
echo $result;
|
||||||
|
?>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test Scripts
|
||||||
|
|
||||||
|
### test-subscribe-members.sh
|
||||||
|
|
||||||
|
Test batch subscription from an existing CSV file (standalone, no external dependencies).
|
||||||
|
|
||||||
|
**Usage:**
|
||||||
|
```bash
|
||||||
|
./test-subscribe-members2.sh <csv_file> [list_id]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Examples:**
|
||||||
|
```bash
|
||||||
|
# Using default list
|
||||||
|
./test-subscribe-members2.sh members.csv
|
||||||
|
|
||||||
|
# Using specific list
|
||||||
|
./test-subscribe-members2.sh members.csv test.lists.example.org
|
||||||
|
```
|
||||||
|
|
||||||
|
**CSV Format:**
|
||||||
|
```csv
|
||||||
|
user1@example.com,John,Doe
|
||||||
|
user2@example.com,Jane,Smith
|
||||||
|
user3@example.com,Bob,Johnson
|
||||||
|
```
|
||||||
|
|
||||||
|
**Configuration:**
|
||||||
|
Edit the script or set environment variables:
|
||||||
|
```bash
|
||||||
|
export CONNECTOR_URL="https://mailman-connector.example.com"
|
||||||
|
export CONNECTOR_PASSWORD="your_secret_password"
|
||||||
|
./test-subscribe-members2.sh members.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
## Response Format
|
||||||
|
|
||||||
|
### Success Response (Single Subscribe)
|
||||||
|
|
||||||
|
Returns Mailman 3 API response (member details).
|
||||||
|
|
||||||
|
### Success Response (Batch Subscribe)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"summary": {
|
||||||
|
"total_received": 3,
|
||||||
|
"subscribed": 1,
|
||||||
|
"already_subscribed": 2,
|
||||||
|
"errors": 0
|
||||||
|
},
|
||||||
|
"details": {
|
||||||
|
"subscribed": ["user3@example.com"],
|
||||||
|
"already_subscribed": ["user1@example.com", "user2@example.com"],
|
||||||
|
"errors": []
|
||||||
}
|
}
|
||||||
|
}
|
||||||
curl_setopt_array($ch, $curlConfig);
|
|
||||||
$result = curl_exec($ch);
|
|
||||||
curl_close($ch);
|
|
||||||
|
|
||||||
echo $result;
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Environmental Variables for `docker-compose.yml`
|
### Success Response (Unsubscribe)
|
||||||
Set the following environment variables in your `docker-compose.yml` file:
|
|
||||||
|
Returns empty body on success (HTTP 200).
|
||||||
|
|
||||||
|
### Error Responses
|
||||||
|
|
||||||
|
| HTTP Status | Error | Description |
|
||||||
|
|-------------|-------|-------------|
|
||||||
|
| 400 | Requestor does not match IP | Client IP not in `REQUESTOR_IP` whitelist |
|
||||||
|
| 400 | Invalid JSON format | Malformed JSON in request body |
|
||||||
|
| 403 | Unauthorized | Wrong `password` provided |
|
||||||
|
| 403 | List not allowed | Requested `list_id` not in `MAILMAN_LIST_ID` whitelist |
|
||||||
|
| 404 | Member not found | Email not found during unsubscribe |
|
||||||
|
| 413 | Request entity too large | Request body exceeds 1MB |
|
||||||
|
| 500 | Internal server error | Server or Mailman API error |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
```
|
```
|
||||||
SECRET_PASSWORD=mypassword
|
┌─────────────┐ HTTPS ┌──────────────────┐ HTTP ┌─────────────┐
|
||||||
MAILMAN_USERNAME=rest-administrator-username
|
│ Client │ ──────────────> │ Mailman Connector │ ────────────> │ Mailman Core│
|
||||||
MAILMAN_PASSWORD=password-for-restadmin
|
│ (PHP/curl) │ │ (this service) │ │ (port 8001)│
|
||||||
MAILMAN_LIST_ID=listname.atlists.coollists.org
|
└─────────────┘ └──────────────────┘ └─────────────┘
|
||||||
LETSENCRYPT_HOST=myhost.org # Uses nginxproxy/acme-companion
|
│
|
||||||
LOGLEVEL=warn
|
│ Docker network: mailman
|
||||||
ENV=production
|
│
|
||||||
HOST_PORT=10000
|
┌─────────────┐
|
||||||
CONTAINER_PORT=10000
|
│ Secret │
|
||||||
REQUESTOR_IP=1.1.1.1 # Optional, but limits request to a specific IP
|
│ Password │
|
||||||
HOSTNAME=mailman-core
|
└─────────────┘
|
||||||
PORT=8001
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
1. **Always use HTTPS** in production
|
||||||
|
2. **Use strong SECRET_PASSWORD** (random, 32+ characters)
|
||||||
|
3. **Restrict REQUESTOR_IP** to known client IPs
|
||||||
|
4. **Limit MAILMAN_LIST_ID** to only needed lists
|
||||||
|
5. **Monitor logs** for unauthorized access attempts
|
||||||
|
6. **Keep Mailman Core** on internal Docker network only
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
GNU Lesser General Public License v3.0
|
||||||
|
|||||||
@ -9,7 +9,37 @@ const port = process.env.PORT
|
|||||||
const buf = Buffer.from(username + ":" + password);
|
const buf = Buffer.from(username + ":" + password);
|
||||||
const auth = "Basic " + buf.toString("base64");
|
const auth = "Basic " + buf.toString("base64");
|
||||||
const mailman = require('http');
|
const mailman = require('http');
|
||||||
const requestor = process.env.REQUESTOR_IP
|
// Parse REQUESTOR_IP - support single IP or comma-separated list of IPs
|
||||||
|
const requestor = process.env.REQUESTOR_IP;
|
||||||
|
const requestor_ips = requestor ? requestor.split(',').map(ip => ip.trim()) : [];
|
||||||
|
|
||||||
|
// Parse MAILMAN_LIST_ID - support single or multiple comma-separated lists (whitelist)
|
||||||
|
const mailman_list_ids = process.env.MAILMAN_LIST_ID
|
||||||
|
? process.env.MAILMAN_LIST_ID.split(',').map(id => id.trim().toLowerCase())
|
||||||
|
: [];
|
||||||
|
const default_list_id = mailman_list_ids[0] || null;
|
||||||
|
|
||||||
|
// Helper function to validate and get allowed list_id
|
||||||
|
function get_allowed_list_id(requested_list_id) {
|
||||||
|
// If no lists configured, reject
|
||||||
|
if (!mailman_list_ids.length) {
|
||||||
|
throw new Error("No MAILMAN_LIST_ID configured");
|
||||||
|
}
|
||||||
|
|
||||||
|
// If client didn't specify, use default (first in list)
|
||||||
|
if (!requested_list_id) {
|
||||||
|
return default_list_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if requested list is in allowed list (case-insensitive)
|
||||||
|
const normalized_request = requested_list_id.toLowerCase();
|
||||||
|
if (mailman_list_ids.includes(normalized_request)) {
|
||||||
|
return normalized_request;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Not allowed
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
// Server loop
|
// Server loop
|
||||||
const server = http.createServer(function (request, response) {
|
const server = http.createServer(function (request, response) {
|
||||||
@ -41,8 +71,14 @@ const server = http.createServer(function (request, response) {
|
|||||||
let data = JSON.parse(body);
|
let data = JSON.parse(body);
|
||||||
let realIp = request.headers['x-forwarded-for'] || request.socket.remoteAddress;
|
let realIp = request.headers['x-forwarded-for'] || request.socket.remoteAddress;
|
||||||
|
|
||||||
if ((requestor && realIp.includes(requestor)) || !requestor) {
|
// Check if IP is allowed (no restriction, or matches any IP in the list)
|
||||||
if (data.subscribe === 'subscribe') {
|
const allowed = requestor_ips.length === 0 || requestor_ips.some(ip => realIp.includes(ip));
|
||||||
|
if (allowed) {
|
||||||
|
if (data.subscribe_batch === 'subscribe_batch') {
|
||||||
|
console.log("From IP:", realIp);
|
||||||
|
console.log("Batch subscribe request");
|
||||||
|
handleSubscribeBatch(data, response);
|
||||||
|
} else if (data.subscribe === 'subscribe') {
|
||||||
console.log("From IP:", realIp);
|
console.log("From IP:", realIp);
|
||||||
console.log("Data", data);
|
console.log("Data", data);
|
||||||
handleSubscribe(data, response);
|
handleSubscribe(data, response);
|
||||||
@ -81,8 +117,16 @@ function handleSubscribe(data, response) {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Validate list_id against whitelist
|
||||||
|
let list_id = get_allowed_list_id(data.list_id);
|
||||||
|
if (!list_id) {
|
||||||
|
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||||
|
response.end(JSON.stringify({ error: "List not allowed" }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
let url_object = {
|
let url_object = {
|
||||||
list_id: process.env.MAILMAN_LIST_ID,
|
list_id: list_id,
|
||||||
subscriber: data.email,
|
subscriber: data.email,
|
||||||
display_name: data.first_name + ' ' + data.last_name,
|
display_name: data.first_name + ' ' + data.last_name,
|
||||||
pre_verified: 'True',
|
pre_verified: 'True',
|
||||||
@ -122,7 +166,14 @@ async function handleUnsubscribe(data, response) {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
let member_id = await find_member_id(data.email);
|
// Validate list_id against whitelist
|
||||||
|
let list_id = get_allowed_list_id(data.list_id);
|
||||||
|
if (!list_id) {
|
||||||
|
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||||
|
response.end(JSON.stringify({ error: "List not allowed" }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let member_id = await find_member_id(data.email, list_id);
|
||||||
if (member_id) {
|
if (member_id) {
|
||||||
await unsubscribe_member_id(member_id, response);
|
await unsubscribe_member_id(member_id, response);
|
||||||
} else {
|
} else {
|
||||||
@ -137,9 +188,10 @@ async function handleUnsubscribe(data, response) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Find member in mailing list
|
// Find member in mailing list
|
||||||
function find_member_id(email) {
|
function find_member_id(email, list_id_override) {
|
||||||
return new Promise((resolve, reject) => {
|
return new Promise((resolve, reject) => {
|
||||||
let url_object = { subscriber: email, list_id: process.env.MAILMAN_LIST_ID, role: 'member' };
|
let list_id = list_id_override || process.env.MAILMAN_LIST_ID;
|
||||||
|
let url_object = { subscriber: email, list_id: list_id, role: 'member' };
|
||||||
|
|
||||||
let mailman_options = {
|
let mailman_options = {
|
||||||
hostname: hostname,
|
hostname: hostname,
|
||||||
@ -191,3 +243,170 @@ function unsubscribe_member_id(member_id, response) {
|
|||||||
});
|
});
|
||||||
req.end();
|
req.end();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Get all members from the mailing list (handles pagination)
|
||||||
|
function get_all_members(list_id_override) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
let members = [];
|
||||||
|
let page_token = null;
|
||||||
|
let list_id = list_id_override || process.env.MAILMAN_LIST_ID;
|
||||||
|
|
||||||
|
function fetch_page() {
|
||||||
|
let path = '/3.1/lists/' + list_id + '/members';
|
||||||
|
if (page_token) {
|
||||||
|
path += '?token=' + encodeURIComponent(page_token);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mailman_options = {
|
||||||
|
hostname: hostname,
|
||||||
|
port: port,
|
||||||
|
path: path,
|
||||||
|
method: 'GET',
|
||||||
|
headers: { "Authorization": auth }
|
||||||
|
};
|
||||||
|
|
||||||
|
let req = mailman.request(mailman_options, function (res) {
|
||||||
|
let body = '';
|
||||||
|
res.on('data', (chunk) => { body += chunk; });
|
||||||
|
res.on('end', () => {
|
||||||
|
try {
|
||||||
|
let parsed = JSON.parse(body);
|
||||||
|
if (parsed.entries) {
|
||||||
|
members = members.concat(parsed.entries.map(e => e.address || e.email));
|
||||||
|
}
|
||||||
|
if (parsed.token) {
|
||||||
|
page_token = parsed.token;
|
||||||
|
fetch_page();
|
||||||
|
} else {
|
||||||
|
resolve(members);
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
reject(err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
req.on('error', reject);
|
||||||
|
req.end();
|
||||||
|
}
|
||||||
|
|
||||||
|
fetch_page();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Subscribe a single member
|
||||||
|
function subscribe_single(email, first_name, last_name, list_id_override) {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
let list_id = list_id_override || process.env.MAILMAN_LIST_ID;
|
||||||
|
let url_object = {
|
||||||
|
list_id: list_id,
|
||||||
|
subscriber: email,
|
||||||
|
display_name: (first_name + ' ' + last_name).trim(),
|
||||||
|
pre_verified: 'True',
|
||||||
|
pre_confirmed: 'True',
|
||||||
|
pre_approved: 'True'
|
||||||
|
};
|
||||||
|
|
||||||
|
let mailman_options = {
|
||||||
|
hostname: hostname,
|
||||||
|
port: port,
|
||||||
|
path: '/3.1/members?' + querystring.stringify(url_object),
|
||||||
|
method: 'POST',
|
||||||
|
headers: { "Authorization": auth }
|
||||||
|
};
|
||||||
|
|
||||||
|
let req = mailman.request(mailman_options, function (res) {
|
||||||
|
let body = '';
|
||||||
|
res.on('data', (chunk) => { body += chunk; });
|
||||||
|
res.on('end', () => {
|
||||||
|
if (res.statusCode >= 200 && res.statusCode < 300) {
|
||||||
|
resolve(body);
|
||||||
|
} else {
|
||||||
|
reject(new Error('HTTP ' + res.statusCode + ': ' + body));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
req.on('error', reject);
|
||||||
|
req.end();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle batch subscription requests
|
||||||
|
async function handleSubscribeBatch(data, response) {
|
||||||
|
if (secret_password !== data.password) {
|
||||||
|
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||||
|
response.end(JSON.stringify({ error: "Unauthorized" }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Validate list_id against whitelist
|
||||||
|
let list_id = get_allowed_list_id(data.list_id);
|
||||||
|
if (!list_id) {
|
||||||
|
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||||
|
response.end(JSON.stringify({ error: "List not allowed" }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
console.log("Using list: " + list_id);
|
||||||
|
|
||||||
|
// Parse CSV data: email,first_name,last_name
|
||||||
|
const lines = data.csv_data.split('\n').filter(l => l.trim());
|
||||||
|
const members_to_process = lines.map(line => {
|
||||||
|
const parts = line.split(',').map(p => p.trim());
|
||||||
|
return {
|
||||||
|
email: parts[0],
|
||||||
|
first_name: parts[1] || '',
|
||||||
|
last_name: parts[2] || ''
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
console.log("Processing " + members_to_process.length + " members");
|
||||||
|
|
||||||
|
// Get existing subscribers
|
||||||
|
const existing_emails = await get_all_members(list_id);
|
||||||
|
const existing_set = new Set(existing_emails.map(e => e.toLowerCase()));
|
||||||
|
console.log("Found " + existing_emails.length + " existing subscribers");
|
||||||
|
|
||||||
|
// Filter out already subscribed
|
||||||
|
const to_subscribe = members_to_process.filter(m => !existing_set.has(m.email.toLowerCase()));
|
||||||
|
const already_subscribed = members_to_process.filter(m => existing_set.has(m.email.toLowerCase()));
|
||||||
|
|
||||||
|
console.log("New to subscribe: " + to_subscribe.length);
|
||||||
|
console.log("Already subscribed: " + already_subscribed.length);
|
||||||
|
|
||||||
|
// Subscribe new ones
|
||||||
|
let subscribed = [];
|
||||||
|
let errors = [];
|
||||||
|
|
||||||
|
for (const member of to_subscribe) {
|
||||||
|
try {
|
||||||
|
await subscribe_single(member.email, member.first_name, member.last_name, list_id);
|
||||||
|
subscribed.push(member.email);
|
||||||
|
console.log("Subscribed: " + member.email);
|
||||||
|
} catch (err) {
|
||||||
|
errors.push({ email: member.email, error: err.message });
|
||||||
|
console.error("Failed to subscribe " + member.email + ": " + err.message);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Return results
|
||||||
|
response.writeHead(200, { 'Content-Type': 'application/json' });
|
||||||
|
response.end(JSON.stringify({
|
||||||
|
success: true,
|
||||||
|
summary: {
|
||||||
|
total_received: members_to_process.length,
|
||||||
|
subscribed: subscribed.length,
|
||||||
|
already_subscribed: already_subscribed.length,
|
||||||
|
errors: errors.length
|
||||||
|
},
|
||||||
|
details: {
|
||||||
|
subscribed: subscribed,
|
||||||
|
already_subscribed: already_subscribed.map(m => m.email),
|
||||||
|
errors: errors
|
||||||
|
}
|
||||||
|
}));
|
||||||
|
} catch (error) {
|
||||||
|
console.error("Batch subscribe error:", error);
|
||||||
|
response.writeHead(500, { 'Content-Type': 'application/json' });
|
||||||
|
response.end(JSON.stringify({ error: "Internal server error", message: error.message }));
|
||||||
|
}
|
||||||
|
}
|
||||||
109
test-subscribe-members.sh
Executable file
109
test-subscribe-members.sh
Executable file
@ -0,0 +1,109 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Test script for batch subscribing members from an existing CSV file
|
||||||
|
# Usage: ./test-subscribe-members.sh <csv_file> [list_id]
|
||||||
|
#
|
||||||
|
# Example:
|
||||||
|
# ./test-subscribe-members.sh test.csv
|
||||||
|
# ./test-subscribe-members.sh test.csv listname.atlists.coollists.org
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
CONNECTOR_URL="https://mailman-connector.coollists.org"
|
||||||
|
CONNECTOR_PASSWORD="${CONNECTOR_PASSWORD:-your_secret_password}"
|
||||||
|
LIST_ID="${2:-listname.atlists.coollists.org}"
|
||||||
|
CSV_FILE="$1"
|
||||||
|
|
||||||
|
# Check if CSV file is provided
|
||||||
|
if [ -z "$CSV_FILE" ]; then
|
||||||
|
echo "Usage: $0 <csv_file> [list_id]"
|
||||||
|
echo ""
|
||||||
|
echo "Arguments:"
|
||||||
|
echo " csv_file Path to CSV file with format: email,first_name,last_name"
|
||||||
|
echo " list_id Optional: Target mailing list (default: listname.atlists.coollists.org)"
|
||||||
|
echo ""
|
||||||
|
echo "Examples:"
|
||||||
|
echo " $0 test.csv"
|
||||||
|
echo " $0 members.csv listname.atlists.coollists.org"
|
||||||
|
echo ""
|
||||||
|
echo "Environment Variables:"
|
||||||
|
echo " CONNECTOR_PASSWORD Secret password for connector (can also be set via env)"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if file exists
|
||||||
|
if [ ! -f "$CSV_FILE" ]; then
|
||||||
|
echo "Error: File not found: $CSV_FILE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Validate CSV file format (basic check)
|
||||||
|
if ! grep -q ',' "$CSV_FILE"; then
|
||||||
|
echo "Error: CSV file appears invalid (no commas found)"
|
||||||
|
echo "Expected format: email,first_name,last_name"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Read CSV content
|
||||||
|
CSV_DATA=$(cat "$CSV_FILE")
|
||||||
|
|
||||||
|
# Count members
|
||||||
|
MEMBER_COUNT=$(wc -l < "$CSV_FILE" | tr -d ' ')
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "Mailman Connector - Batch Subscribe Test"
|
||||||
|
echo "=========================================="
|
||||||
|
echo "Connector URL: $CONNECTOR_URL"
|
||||||
|
echo "Target List: $LIST_ID"
|
||||||
|
echo "CSV File: $CSV_FILE"
|
||||||
|
echo "Members: $MEMBER_COUNT"
|
||||||
|
echo "=========================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check if jq is installed
|
||||||
|
if ! command -v jq &> /dev/null; then
|
||||||
|
echo "Error: jq is required but not installed"
|
||||||
|
echo "Install with: apt-get install jq (Debian/Ubuntu) or yum install jq (RHEL/CentOS)"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Send to connector
|
||||||
|
echo "Sending request to connector..."
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
RESPONSE=$(curl -s -X POST "$CONNECTOR_URL" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{
|
||||||
|
\"subscribe_batch\": \"subscribe_batch\",
|
||||||
|
\"password\": \"$CONNECTOR_PASSWORD\",
|
||||||
|
\"list_id\": \"$LIST_ID\",
|
||||||
|
\"csv_data\": $(echo "$CSV_DATA" | jq -R -s .)
|
||||||
|
}")
|
||||||
|
|
||||||
|
# Check if response is valid JSON
|
||||||
|
if echo "$RESPONSE" | jq -e . &> /dev/null; then
|
||||||
|
echo "Response:"
|
||||||
|
echo "$RESPONSE" | jq .
|
||||||
|
|
||||||
|
# Extract summary if available
|
||||||
|
if echo "$RESPONSE" | jq -e '.summary' &> /dev/null; then
|
||||||
|
echo ""
|
||||||
|
echo "Summary:"
|
||||||
|
echo " Total received: $(echo "$RESPONSE" | jq -r '.summary.total_received // "N/A"')"
|
||||||
|
echo " Subscribed: $(echo "$RESPONSE" | jq -r '.summary.subscribed // "N/A"')"
|
||||||
|
echo " Already subscribed: $(echo "$RESPONSE" | jq -r '.summary.already_subscribed // "N/A"')"
|
||||||
|
echo " Errors: $(echo "$RESPONSE" | jq -r '.summary.errors // "N/A"')"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check for errors
|
||||||
|
if echo "$RESPONSE" | jq -e '.error' &> /dev/null; then
|
||||||
|
echo ""
|
||||||
|
echo "ERROR: $(echo "$RESPONSE" | jq -r '.error')"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Error: Invalid response from connector"
|
||||||
|
echo "Response: $RESPONSE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Done!"
|
||||||
Loading…
x
Reference in New Issue
Block a user