feat: add batch subscribe and list whitelist support
- Add subscribe_batch endpoint for subscribing multiple members from CSV - Add list whitelist: MAILMAN_LIST_ID supports multiple comma-separated lists - Add IP whitelist: REQUESTOR_IP supports multiple comma-separated IPs - Add optional list_id override for targeting specific lists - Add test-subscribe-members2.sh utility script - Update README with comprehensive documentation and examples
This commit is contained in:
parent
4feeb3faf5
commit
0d01f3e0b8
2
.gitignore
vendored
2
.gitignore
vendored
@ -1 +1,3 @@
|
||||
.env
|
||||
test*csv
|
||||
tests
|
||||
|
||||
429
README.md
429
README.md
@ -1,55 +1,392 @@
|
||||
# Mailman Connector
|
||||
|
||||
This program provides an API server to receive requests to subscribe and unsubscribe users from a MailMan 3 email list. The requester must provide a secret password to utilize the service.
|
||||
A secure API bridge for Mailman 3 that enables subscription management without exposing the Mailman REST API publicly. Designed to run alongside Dockerized Mailman Core on the same Docker network.
|
||||
|
||||
Mailman Connector is designed to run on the same host as a Dockerized version of Mailman and shares the same Docker network (`mailman`). This setup ensures that you don't need to expose Mailman's API publicly. The `HOSTNAME` environment variable should be set to the Docker service name specified for `mailman-core` provided by `maxking/mailman-core`.
|
||||
|
||||
## Example Request Code (PHP)
|
||||
Here's an example of how to send a request using PHP:
|
||||
```
|
||||
$json = array(
|
||||
'subscribe' => $_POST['email_list_connector'],
|
||||
'password' => $email_list_connector_password,
|
||||
'email' => $_POST['email'],
|
||||
'first_name' => $_POST['first_name'],
|
||||
'last_name' => $_POST['last_name'],
|
||||
);
|
||||
## Quick Start
|
||||
|
||||
$ch = curl_init();
|
||||
$curlConfig = array(
|
||||
CURLOPT_URL => $email_list_connector,
|
||||
CURLOPT_POST => true,
|
||||
CURLOPT_SSL_VERIFYPEER => true,
|
||||
CURLOPT_RETURNTRANSFER => true,
|
||||
CURLOPT_POSTFIELDS => json_encode($json),
|
||||
);
|
||||
|
||||
// $ssl_certificate = SSL_CERTIFICATE;
|
||||
// if( !defined( 'SSL_CERTIFICATE' ) ) define ("SSL_CERTIFICATE", "/var/www/html/examples/cert.pem");
|
||||
if ($ssl_certificate) {
|
||||
$curlConfig[CURLOPT_CAINFO] = $ssl_certificate;
|
||||
}
|
||||
|
||||
curl_setopt_array($ch, $curlConfig);
|
||||
$result = curl_exec($ch);
|
||||
curl_close($ch);
|
||||
|
||||
echo $result;
|
||||
```
|
||||
|
||||
## Environmental Variables for `docker-compose.yml`
|
||||
Set the following environment variables in your `docker-compose.yml` file:
|
||||
```
|
||||
SECRET_PASSWORD=mypassword
|
||||
MAILMAN_USERNAME=rest-administrator-username
|
||||
MAILMAN_PASSWORD=password-for-restadmin
|
||||
MAILMAN_LIST_ID=listname.atlists.coollists.org
|
||||
LETSENCRYPT_HOST=myhost.org # Uses nginxproxy/acme-companion
|
||||
LOGLEVEL=warn
|
||||
ENV=production
|
||||
HOST_PORT=10000
|
||||
CONTAINER_PORT=10000
|
||||
REQUESTOR_IP=1.1.1.1 # Optional, but limits request to a specific IP
|
||||
1. **Configure environment variables** in `.env` or `docker-compose.yml`:
|
||||
```bash
|
||||
SECRET_PASSWORD=your_secret_password
|
||||
MAILMAN_USERNAME=rest-admin
|
||||
MAILMAN_PASSWORD=rest_password
|
||||
MAILMAN_LIST_ID=list.domain.org,test.lists.domain.org
|
||||
HOSTNAME=mailman-core
|
||||
PORT=8001
|
||||
HOST_PORT=10000
|
||||
REQUESTOR_IP=1.2.3.4,5.6.7.8
|
||||
```
|
||||
|
||||
2. **Start the container**:
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
3. **Test with curl** (single subscribe):
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"subscribe":"subscribe","password":"your_secret_password","email":"test@example.com","first_name":"Test","last_name":"User"}'
|
||||
```
|
||||
|
||||
Or test batch subscribe (multiple members at once):
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"subscribe_batch":"subscribe_batch","password":"your_secret_password","csv_data":"user1@example.com,John,Doe\nuser2@example.com,Jane,Smith"}'
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- **Single Subscribe**: Add individual members to mailing lists
|
||||
- **Batch Subscribe**: Subscribe multiple members from CSV data with automatic duplicate detection
|
||||
- **Unsubscribe**: Remove members from mailing lists
|
||||
- **List Whitelist**: Control which mailing lists clients can access via `MAILMAN_LIST_ID`
|
||||
- **IP Restrictions**: Limit access by IP address with support for multiple IPs
|
||||
- **Security**: Secret password authentication, optional IP whitelisting, HTTPS support
|
||||
|
||||
## Environmental Variables
|
||||
|
||||
Set these in your `docker-compose.yml` or `.env` file:
|
||||
|
||||
```yaml
|
||||
# Required
|
||||
SECRET_PASSWORD=your_secret_password
|
||||
MAILMAN_USERNAME=rest-administrator-username
|
||||
MAILMAN_PASSWORD=password-for-restadmin
|
||||
MAILMAN_LIST_ID=list.domain.org
|
||||
HOSTNAME=mailman-core
|
||||
PORT=8001
|
||||
|
||||
# Optional
|
||||
HOST_PORT=10000
|
||||
CONTAINER_PORT=10000
|
||||
REQUESTOR_IP=1.2.3.4,5.6.7.8
|
||||
LETSENCRYPT_HOST=connector.example.com
|
||||
LOGLEVEL=warn
|
||||
ENV=production
|
||||
```
|
||||
|
||||
### MAILMAN_LIST_ID Configuration
|
||||
|
||||
Control which mailing lists clients can access:
|
||||
|
||||
```bash
|
||||
# Single list (backward compatible)
|
||||
MAILMAN_LIST_ID=list.domain.org
|
||||
|
||||
# Multiple allowed lists (first is default)
|
||||
MAILMAN_LIST_ID=list.domain.org,test.lists.domain.org,dev.lists.domain.org
|
||||
```
|
||||
|
||||
When multiple lists are configured, clients can specify `list_id` in requests. Only whitelisted lists are permitted.
|
||||
|
||||
### REQUESTOR_IP Configuration
|
||||
|
||||
Restrict access by IP address:
|
||||
|
||||
```bash
|
||||
# Single IP
|
||||
REQUESTOR_IP=1.2.3.4
|
||||
|
||||
# Multiple IPs (comma-separated)
|
||||
REQUESTOR_IP=1.2.3.4,5.6.7.8,9.10.11.12
|
||||
|
||||
# No restriction (allow all)
|
||||
REQUESTOR_IP=
|
||||
```
|
||||
|
||||
## Request Formats
|
||||
|
||||
### Single Subscribe (Default List)
|
||||
|
||||
Subscribe an individual member using the default list (first in `MAILMAN_LIST_ID`):
|
||||
|
||||
```json
|
||||
{
|
||||
"subscribe": "subscribe",
|
||||
"password": "your_secret_password",
|
||||
"email": "user@example.com",
|
||||
"first_name": "John",
|
||||
"last_name": "Doe"
|
||||
}
|
||||
```
|
||||
|
||||
### Single Subscribe (Override List)
|
||||
|
||||
Subscribe to a specific list (must be in whitelist):
|
||||
|
||||
```json
|
||||
{
|
||||
"subscribe": "subscribe",
|
||||
"password": "your_secret_password",
|
||||
"list_id": "test.lists.domain.org",
|
||||
"email": "user@example.com",
|
||||
"first_name": "John",
|
||||
"last_name": "Doe"
|
||||
}
|
||||
```
|
||||
|
||||
### Batch Subscribe (Default List)
|
||||
|
||||
Subscribe multiple members from CSV data:
|
||||
|
||||
```json
|
||||
{
|
||||
"subscribe_batch": "subscribe_batch",
|
||||
"password": "your_secret_password",
|
||||
"csv_data": "user1@example.com,John,Doe\nuser2@example.com,Jane,Smith\nuser3@example.com,Bob,Johnson"
|
||||
}
|
||||
```
|
||||
|
||||
### Batch Subscribe (Override List)
|
||||
|
||||
```json
|
||||
{
|
||||
"subscribe_batch": "subscribe_batch",
|
||||
"password": "your_secret_password",
|
||||
"list_id": "test.lists.domain.org",
|
||||
"csv_data": "user1@example.com,John,Doe\nuser2@example.com,Jane,Smith"
|
||||
}
|
||||
```
|
||||
|
||||
### Unsubscribe (Default List)
|
||||
|
||||
Remove a member from the default list:
|
||||
|
||||
```json
|
||||
{
|
||||
"password": "your_secret_password",
|
||||
"email": "user@example.com"
|
||||
}
|
||||
```
|
||||
|
||||
### Unsubscribe (Override List)
|
||||
|
||||
```json
|
||||
{
|
||||
"password": "your_secret_password",
|
||||
"list_id": "test.lists.domain.org",
|
||||
"email": "user@example.com"
|
||||
}
|
||||
```
|
||||
|
||||
## Testing with curl
|
||||
|
||||
### 1. Single Subscribe (Default List)
|
||||
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"subscribe": "subscribe",
|
||||
"password": "secret",
|
||||
"email": "test1@example.com",
|
||||
"first_name": "Alice",
|
||||
"last_name": "Anderson"
|
||||
}'
|
||||
```
|
||||
|
||||
### 2. Single Subscribe (Override List)
|
||||
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"subscribe": "subscribe",
|
||||
"password": "secret",
|
||||
"list_id": "test.lists.example.org",
|
||||
"email": "test2@example.com",
|
||||
"first_name": "Bob",
|
||||
"last_name": "Brown"
|
||||
}'
|
||||
```
|
||||
|
||||
### 3. Batch Subscribe (Default List)
|
||||
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"subscribe_batch": "subscribe_batch",
|
||||
"password": "secret",
|
||||
"csv_data": "user1@example.com,John,Doe\nuser2@example.com,Jane,Smith"
|
||||
}'
|
||||
```
|
||||
|
||||
### 4. Batch Subscribe (Override List)
|
||||
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"subscribe_batch": "subscribe_batch",
|
||||
"password": "secret",
|
||||
"list_id": "test.lists.example.org",
|
||||
"csv_data": "user3@example.com,Mike,Johnson\nuser4@example.com,Sarah,Williams"
|
||||
}'
|
||||
```
|
||||
|
||||
### 5. Unsubscribe (Default List)
|
||||
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"password": "secret",
|
||||
"email": "test1@example.com"
|
||||
}'
|
||||
```
|
||||
|
||||
### 6. Unsubscribe (Override List)
|
||||
|
||||
```bash
|
||||
curl -X POST https://mailman-connector.example.com \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"password": "secret",
|
||||
"list_id": "test.lists.example.org",
|
||||
"email": "test2@example.com"
|
||||
}'
|
||||
```
|
||||
|
||||
## Example Request Code (PHP)
|
||||
|
||||
Here's an example of how to send a request using PHP:
|
||||
|
||||
```php
|
||||
<?php
|
||||
$json = array(
|
||||
'subscribe' => $_POST['email_list_connector'],
|
||||
'password' => $email_list_connector_password,
|
||||
'email' => $_POST['email'],
|
||||
'first_name' => $_POST['first_name'],
|
||||
'last_name' => $_POST['last_name'],
|
||||
);
|
||||
|
||||
$ch = curl_init();
|
||||
$curlConfig = array(
|
||||
CURLOPT_URL => $email_list_connector,
|
||||
CURLOPT_POST => true,
|
||||
CURLOPT_SSL_VERIFYPEER => true,
|
||||
CURLOPT_RETURNTRANSFER => true,
|
||||
CURLOPT_POSTFIELDS => json_encode($json),
|
||||
);
|
||||
|
||||
if ($ssl_certificate) {
|
||||
$curlConfig[CURLOPT_CAINFO] = $ssl_certificate;
|
||||
}
|
||||
|
||||
curl_setopt_array($ch, $curlConfig);
|
||||
$result = curl_exec($ch);
|
||||
curl_close($ch);
|
||||
|
||||
echo $result;
|
||||
?>
|
||||
```
|
||||
|
||||
## Test Scripts
|
||||
|
||||
### test-subscribe-members.sh
|
||||
|
||||
Test batch subscription from an existing CSV file (standalone, no external dependencies).
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
./test-subscribe-members2.sh <csv_file> [list_id]
|
||||
```
|
||||
|
||||
**Examples:**
|
||||
```bash
|
||||
# Using default list
|
||||
./test-subscribe-members2.sh members.csv
|
||||
|
||||
# Using specific list
|
||||
./test-subscribe-members2.sh members.csv test.lists.example.org
|
||||
```
|
||||
|
||||
**CSV Format:**
|
||||
```csv
|
||||
user1@example.com,John,Doe
|
||||
user2@example.com,Jane,Smith
|
||||
user3@example.com,Bob,Johnson
|
||||
```
|
||||
|
||||
**Configuration:**
|
||||
Edit the script or set environment variables:
|
||||
```bash
|
||||
export CONNECTOR_URL="https://mailman-connector.example.com"
|
||||
export CONNECTOR_PASSWORD="your_secret_password"
|
||||
./test-subscribe-members2.sh members.csv
|
||||
```
|
||||
|
||||
## Response Format
|
||||
|
||||
### Success Response (Single Subscribe)
|
||||
|
||||
Returns Mailman 3 API response (member details).
|
||||
|
||||
### Success Response (Batch Subscribe)
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"summary": {
|
||||
"total_received": 3,
|
||||
"subscribed": 1,
|
||||
"already_subscribed": 2,
|
||||
"errors": 0
|
||||
},
|
||||
"details": {
|
||||
"subscribed": ["user3@example.com"],
|
||||
"already_subscribed": ["user1@example.com", "user2@example.com"],
|
||||
"errors": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Success Response (Unsubscribe)
|
||||
|
||||
Returns empty body on success (HTTP 200).
|
||||
|
||||
### Error Responses
|
||||
|
||||
| HTTP Status | Error | Description |
|
||||
|-------------|-------|-------------|
|
||||
| 400 | Requestor does not match IP | Client IP not in `REQUESTOR_IP` whitelist |
|
||||
| 400 | Invalid JSON format | Malformed JSON in request body |
|
||||
| 403 | Unauthorized | Wrong `password` provided |
|
||||
| 403 | List not allowed | Requested `list_id` not in `MAILMAN_LIST_ID` whitelist |
|
||||
| 404 | Member not found | Email not found during unsubscribe |
|
||||
| 413 | Request entity too large | Request body exceeds 1MB |
|
||||
| 500 | Internal server error | Server or Mailman API error |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ HTTPS ┌──────────────────┐ HTTP ┌─────────────┐
|
||||
│ Client │ ──────────────> │ Mailman Connector │ ────────────> │ Mailman Core│
|
||||
│ (PHP/curl) │ │ (this service) │ │ (port 8001)│
|
||||
└─────────────┘ └──────────────────┘ └─────────────┘
|
||||
│
|
||||
│ Docker network: mailman
|
||||
│
|
||||
┌─────────────┐
|
||||
│ Secret │
|
||||
│ Password │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Always use HTTPS** in production
|
||||
2. **Use strong SECRET_PASSWORD** (random, 32+ characters)
|
||||
3. **Restrict REQUESTOR_IP** to known client IPs
|
||||
4. **Limit MAILMAN_LIST_ID** to only needed lists
|
||||
5. **Monitor logs** for unauthorized access attempts
|
||||
6. **Keep Mailman Core** on internal Docker network only
|
||||
|
||||
## License
|
||||
|
||||
GNU Lesser General Public License v3.0
|
||||
|
||||
@ -9,7 +9,37 @@ const port = process.env.PORT
|
||||
const buf = Buffer.from(username + ":" + password);
|
||||
const auth = "Basic " + buf.toString("base64");
|
||||
const mailman = require('http');
|
||||
const requestor = process.env.REQUESTOR_IP
|
||||
// Parse REQUESTOR_IP - support single IP or comma-separated list of IPs
|
||||
const requestor = process.env.REQUESTOR_IP;
|
||||
const requestor_ips = requestor ? requestor.split(',').map(ip => ip.trim()) : [];
|
||||
|
||||
// Parse MAILMAN_LIST_ID - support single or multiple comma-separated lists (whitelist)
|
||||
const mailman_list_ids = process.env.MAILMAN_LIST_ID
|
||||
? process.env.MAILMAN_LIST_ID.split(',').map(id => id.trim().toLowerCase())
|
||||
: [];
|
||||
const default_list_id = mailman_list_ids[0] || null;
|
||||
|
||||
// Helper function to validate and get allowed list_id
|
||||
function get_allowed_list_id(requested_list_id) {
|
||||
// If no lists configured, reject
|
||||
if (!mailman_list_ids.length) {
|
||||
throw new Error("No MAILMAN_LIST_ID configured");
|
||||
}
|
||||
|
||||
// If client didn't specify, use default (first in list)
|
||||
if (!requested_list_id) {
|
||||
return default_list_id;
|
||||
}
|
||||
|
||||
// Check if requested list is in allowed list (case-insensitive)
|
||||
const normalized_request = requested_list_id.toLowerCase();
|
||||
if (mailman_list_ids.includes(normalized_request)) {
|
||||
return normalized_request;
|
||||
}
|
||||
|
||||
// Not allowed
|
||||
return null;
|
||||
}
|
||||
|
||||
// Server loop
|
||||
const server = http.createServer(function (request, response) {
|
||||
@ -41,8 +71,14 @@ const server = http.createServer(function (request, response) {
|
||||
let data = JSON.parse(body);
|
||||
let realIp = request.headers['x-forwarded-for'] || request.socket.remoteAddress;
|
||||
|
||||
if ((requestor && realIp.includes(requestor)) || !requestor) {
|
||||
if (data.subscribe === 'subscribe') {
|
||||
// Check if IP is allowed (no restriction, or matches any IP in the list)
|
||||
const allowed = requestor_ips.length === 0 || requestor_ips.some(ip => realIp.includes(ip));
|
||||
if (allowed) {
|
||||
if (data.subscribe_batch === 'subscribe_batch') {
|
||||
console.log("From IP:", realIp);
|
||||
console.log("Batch subscribe request");
|
||||
handleSubscribeBatch(data, response);
|
||||
} else if (data.subscribe === 'subscribe') {
|
||||
console.log("From IP:", realIp);
|
||||
console.log("Data", data);
|
||||
handleSubscribe(data, response);
|
||||
@ -81,8 +117,16 @@ function handleSubscribe(data, response) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Validate list_id against whitelist
|
||||
let list_id = get_allowed_list_id(data.list_id);
|
||||
if (!list_id) {
|
||||
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||
response.end(JSON.stringify({ error: "List not allowed" }));
|
||||
return;
|
||||
}
|
||||
|
||||
let url_object = {
|
||||
list_id: process.env.MAILMAN_LIST_ID,
|
||||
list_id: list_id,
|
||||
subscriber: data.email,
|
||||
display_name: data.first_name + ' ' + data.last_name,
|
||||
pre_verified: 'True',
|
||||
@ -122,7 +166,14 @@ async function handleUnsubscribe(data, response) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
let member_id = await find_member_id(data.email);
|
||||
// Validate list_id against whitelist
|
||||
let list_id = get_allowed_list_id(data.list_id);
|
||||
if (!list_id) {
|
||||
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||
response.end(JSON.stringify({ error: "List not allowed" }));
|
||||
return;
|
||||
}
|
||||
let member_id = await find_member_id(data.email, list_id);
|
||||
if (member_id) {
|
||||
await unsubscribe_member_id(member_id, response);
|
||||
} else {
|
||||
@ -137,9 +188,10 @@ async function handleUnsubscribe(data, response) {
|
||||
}
|
||||
|
||||
// Find member in mailing list
|
||||
function find_member_id(email) {
|
||||
function find_member_id(email, list_id_override) {
|
||||
return new Promise((resolve, reject) => {
|
||||
let url_object = { subscriber: email, list_id: process.env.MAILMAN_LIST_ID, role: 'member' };
|
||||
let list_id = list_id_override || process.env.MAILMAN_LIST_ID;
|
||||
let url_object = { subscriber: email, list_id: list_id, role: 'member' };
|
||||
|
||||
let mailman_options = {
|
||||
hostname: hostname,
|
||||
@ -191,3 +243,170 @@ function unsubscribe_member_id(member_id, response) {
|
||||
});
|
||||
req.end();
|
||||
}
|
||||
|
||||
// Get all members from the mailing list (handles pagination)
|
||||
function get_all_members(list_id_override) {
|
||||
return new Promise((resolve, reject) => {
|
||||
let members = [];
|
||||
let page_token = null;
|
||||
let list_id = list_id_override || process.env.MAILMAN_LIST_ID;
|
||||
|
||||
function fetch_page() {
|
||||
let path = '/3.1/lists/' + list_id + '/members';
|
||||
if (page_token) {
|
||||
path += '?token=' + encodeURIComponent(page_token);
|
||||
}
|
||||
|
||||
let mailman_options = {
|
||||
hostname: hostname,
|
||||
port: port,
|
||||
path: path,
|
||||
method: 'GET',
|
||||
headers: { "Authorization": auth }
|
||||
};
|
||||
|
||||
let req = mailman.request(mailman_options, function (res) {
|
||||
let body = '';
|
||||
res.on('data', (chunk) => { body += chunk; });
|
||||
res.on('end', () => {
|
||||
try {
|
||||
let parsed = JSON.parse(body);
|
||||
if (parsed.entries) {
|
||||
members = members.concat(parsed.entries.map(e => e.address || e.email));
|
||||
}
|
||||
if (parsed.token) {
|
||||
page_token = parsed.token;
|
||||
fetch_page();
|
||||
} else {
|
||||
resolve(members);
|
||||
}
|
||||
} catch (err) {
|
||||
reject(err);
|
||||
}
|
||||
});
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.end();
|
||||
}
|
||||
|
||||
fetch_page();
|
||||
});
|
||||
}
|
||||
|
||||
// Subscribe a single member
|
||||
function subscribe_single(email, first_name, last_name, list_id_override) {
|
||||
return new Promise((resolve, reject) => {
|
||||
let list_id = list_id_override || process.env.MAILMAN_LIST_ID;
|
||||
let url_object = {
|
||||
list_id: list_id,
|
||||
subscriber: email,
|
||||
display_name: (first_name + ' ' + last_name).trim(),
|
||||
pre_verified: 'True',
|
||||
pre_confirmed: 'True',
|
||||
pre_approved: 'True'
|
||||
};
|
||||
|
||||
let mailman_options = {
|
||||
hostname: hostname,
|
||||
port: port,
|
||||
path: '/3.1/members?' + querystring.stringify(url_object),
|
||||
method: 'POST',
|
||||
headers: { "Authorization": auth }
|
||||
};
|
||||
|
||||
let req = mailman.request(mailman_options, function (res) {
|
||||
let body = '';
|
||||
res.on('data', (chunk) => { body += chunk; });
|
||||
res.on('end', () => {
|
||||
if (res.statusCode >= 200 && res.statusCode < 300) {
|
||||
resolve(body);
|
||||
} else {
|
||||
reject(new Error('HTTP ' + res.statusCode + ': ' + body));
|
||||
}
|
||||
});
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
// Handle batch subscription requests
|
||||
async function handleSubscribeBatch(data, response) {
|
||||
if (secret_password !== data.password) {
|
||||
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||
response.end(JSON.stringify({ error: "Unauthorized" }));
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
// Validate list_id against whitelist
|
||||
let list_id = get_allowed_list_id(data.list_id);
|
||||
if (!list_id) {
|
||||
response.writeHead(403, { 'Content-Type': 'application/json' });
|
||||
response.end(JSON.stringify({ error: "List not allowed" }));
|
||||
return;
|
||||
}
|
||||
console.log("Using list: " + list_id);
|
||||
|
||||
// Parse CSV data: email,first_name,last_name
|
||||
const lines = data.csv_data.split('\n').filter(l => l.trim());
|
||||
const members_to_process = lines.map(line => {
|
||||
const parts = line.split(',').map(p => p.trim());
|
||||
return {
|
||||
email: parts[0],
|
||||
first_name: parts[1] || '',
|
||||
last_name: parts[2] || ''
|
||||
};
|
||||
});
|
||||
|
||||
console.log("Processing " + members_to_process.length + " members");
|
||||
|
||||
// Get existing subscribers
|
||||
const existing_emails = await get_all_members(list_id);
|
||||
const existing_set = new Set(existing_emails.map(e => e.toLowerCase()));
|
||||
console.log("Found " + existing_emails.length + " existing subscribers");
|
||||
|
||||
// Filter out already subscribed
|
||||
const to_subscribe = members_to_process.filter(m => !existing_set.has(m.email.toLowerCase()));
|
||||
const already_subscribed = members_to_process.filter(m => existing_set.has(m.email.toLowerCase()));
|
||||
|
||||
console.log("New to subscribe: " + to_subscribe.length);
|
||||
console.log("Already subscribed: " + already_subscribed.length);
|
||||
|
||||
// Subscribe new ones
|
||||
let subscribed = [];
|
||||
let errors = [];
|
||||
|
||||
for (const member of to_subscribe) {
|
||||
try {
|
||||
await subscribe_single(member.email, member.first_name, member.last_name, list_id);
|
||||
subscribed.push(member.email);
|
||||
console.log("Subscribed: " + member.email);
|
||||
} catch (err) {
|
||||
errors.push({ email: member.email, error: err.message });
|
||||
console.error("Failed to subscribe " + member.email + ": " + err.message);
|
||||
}
|
||||
}
|
||||
|
||||
// Return results
|
||||
response.writeHead(200, { 'Content-Type': 'application/json' });
|
||||
response.end(JSON.stringify({
|
||||
success: true,
|
||||
summary: {
|
||||
total_received: members_to_process.length,
|
||||
subscribed: subscribed.length,
|
||||
already_subscribed: already_subscribed.length,
|
||||
errors: errors.length
|
||||
},
|
||||
details: {
|
||||
subscribed: subscribed,
|
||||
already_subscribed: already_subscribed.map(m => m.email),
|
||||
errors: errors
|
||||
}
|
||||
}));
|
||||
} catch (error) {
|
||||
console.error("Batch subscribe error:", error);
|
||||
response.writeHead(500, { 'Content-Type': 'application/json' });
|
||||
response.end(JSON.stringify({ error: "Internal server error", message: error.message }));
|
||||
}
|
||||
}
|
||||
109
test-subscribe-members.sh
Executable file
109
test-subscribe-members.sh
Executable file
@ -0,0 +1,109 @@
|
||||
#!/bin/bash
|
||||
# Test script for batch subscribing members from an existing CSV file
|
||||
# Usage: ./test-subscribe-members.sh <csv_file> [list_id]
|
||||
#
|
||||
# Example:
|
||||
# ./test-subscribe-members.sh test.csv
|
||||
# ./test-subscribe-members.sh test.csv listname.atlists.coollists.org
|
||||
|
||||
# Configuration
|
||||
CONNECTOR_URL="https://mailman-connector.coollists.org"
|
||||
CONNECTOR_PASSWORD="${CONNECTOR_PASSWORD:-your_secret_password}"
|
||||
LIST_ID="${2:-listname.atlists.coollists.org}"
|
||||
CSV_FILE="$1"
|
||||
|
||||
# Check if CSV file is provided
|
||||
if [ -z "$CSV_FILE" ]; then
|
||||
echo "Usage: $0 <csv_file> [list_id]"
|
||||
echo ""
|
||||
echo "Arguments:"
|
||||
echo " csv_file Path to CSV file with format: email,first_name,last_name"
|
||||
echo " list_id Optional: Target mailing list (default: listname.atlists.coollists.org)"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 test.csv"
|
||||
echo " $0 members.csv listname.atlists.coollists.org"
|
||||
echo ""
|
||||
echo "Environment Variables:"
|
||||
echo " CONNECTOR_PASSWORD Secret password for connector (can also be set via env)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if file exists
|
||||
if [ ! -f "$CSV_FILE" ]; then
|
||||
echo "Error: File not found: $CSV_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate CSV file format (basic check)
|
||||
if ! grep -q ',' "$CSV_FILE"; then
|
||||
echo "Error: CSV file appears invalid (no commas found)"
|
||||
echo "Expected format: email,first_name,last_name"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Read CSV content
|
||||
CSV_DATA=$(cat "$CSV_FILE")
|
||||
|
||||
# Count members
|
||||
MEMBER_COUNT=$(wc -l < "$CSV_FILE" | tr -d ' ')
|
||||
|
||||
echo "=========================================="
|
||||
echo "Mailman Connector - Batch Subscribe Test"
|
||||
echo "=========================================="
|
||||
echo "Connector URL: $CONNECTOR_URL"
|
||||
echo "Target List: $LIST_ID"
|
||||
echo "CSV File: $CSV_FILE"
|
||||
echo "Members: $MEMBER_COUNT"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Check if jq is installed
|
||||
if ! command -v jq &> /dev/null; then
|
||||
echo "Error: jq is required but not installed"
|
||||
echo "Install with: apt-get install jq (Debian/Ubuntu) or yum install jq (RHEL/CentOS)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Send to connector
|
||||
echo "Sending request to connector..."
|
||||
echo ""
|
||||
|
||||
RESPONSE=$(curl -s -X POST "$CONNECTOR_URL" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"subscribe_batch\": \"subscribe_batch\",
|
||||
\"password\": \"$CONNECTOR_PASSWORD\",
|
||||
\"list_id\": \"$LIST_ID\",
|
||||
\"csv_data\": $(echo "$CSV_DATA" | jq -R -s .)
|
||||
}")
|
||||
|
||||
# Check if response is valid JSON
|
||||
if echo "$RESPONSE" | jq -e . &> /dev/null; then
|
||||
echo "Response:"
|
||||
echo "$RESPONSE" | jq .
|
||||
|
||||
# Extract summary if available
|
||||
if echo "$RESPONSE" | jq -e '.summary' &> /dev/null; then
|
||||
echo ""
|
||||
echo "Summary:"
|
||||
echo " Total received: $(echo "$RESPONSE" | jq -r '.summary.total_received // "N/A"')"
|
||||
echo " Subscribed: $(echo "$RESPONSE" | jq -r '.summary.subscribed // "N/A"')"
|
||||
echo " Already subscribed: $(echo "$RESPONSE" | jq -r '.summary.already_subscribed // "N/A"')"
|
||||
echo " Errors: $(echo "$RESPONSE" | jq -r '.summary.errors // "N/A"')"
|
||||
fi
|
||||
|
||||
# Check for errors
|
||||
if echo "$RESPONSE" | jq -e '.error' &> /dev/null; then
|
||||
echo ""
|
||||
echo "ERROR: $(echo "$RESPONSE" | jq -r '.error')"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "Error: Invalid response from connector"
|
||||
echo "Response: $RESPONSE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Done!"
|
||||
Loading…
x
Reference in New Issue
Block a user