README

A comprehensive Laravel package for monitoring, logging, and cataloging Shoutbomb notification data with email ingestion, Microsoft Graph integration, and RESTful API.

Overview

This package integrates with Shoutbomb (a third-party notification service for libraries) to provide comprehensive monitoring, logging, and analytics for voice and text notifications. It automatically ingests notification data from multiple sources, validates critical business rules, and provides a RESTful API for dashboard integration.

Key Features

Automated Data Ingestion: Import patron lists, holds exports, and email reports
Email Integration: Microsoft Graph API support for automated email report parsing
Critical Validation: Phone overlap detection between voice and text lists
Polaris PhoneNotices Import: Import native Polaris PhoneNotices.csv for validation and enrichment
Data Enrichment: Cross-validate custom SQL exports against native Polaris notification data
Comprehensive Logging: Track all imports, failures, and system actions
Statistics & Analytics: Daily and monthly statistics with failure rate tracking
RESTful API: Complete API with OpenAPI/Swagger documentation
Dashboard Control: Trigger artisan commands via API endpoints
Polaris Integration: Compatible with Polaris ILS lookup tables
Database Configuration: Override .env settings with database values

Requirements

PHP 8.3 or 8.4
Laravel 11 or 12
MySQL/MariaDB or PostgreSQL
Microsoft Entra (Azure AD) app for email access
Composer 2.x

Installation

See INSTALLATION.md for detailed installation instructions.

Quick Install

composer require dcplibrary/notices-shoutbomb

php artisan shoutbomb:install

The installation wizard will guide you through:

Environment variable setup
Database migration
Polaris lookup table seeding
Microsoft Graph connection testing

Configuration

Configuration can be managed via:

.env file (default values)
Database settings (override .env values)
API endpoints (runtime configuration)

Environment Variables

# Microsoft Entra (Azure AD) Configuration
SHOUTBOMB_MS_TENANT_ID=your-tenant-id
SHOUTBOMB_MS_CLIENT_ID=your-client-id
SHOUTBOMB_MS_CLIENT_SECRET=your-client-secret
SHOUTBOMB_MS_USER_EMAIL=notifications@yourlibrary.org

# Email Configuration
SHOUTBOMB_EMAIL_FOLDER=Inbox/Shoutbomb
SHOUTBOMB_EMAIL_UNREAD_ONLY=true
SHOUTBOMB_EMAIL_MARK_AS_READ=true
SHOUTBOMB_EMAIL_SUBJECT_FILTER=Shoutbomb

# File Import Paths (for local file method)
SHOUTBOMB_IMPORT_PATH=/path/to/imports
SHOUTBOMB_VOICE_PATRONS_PATH=/path/to/voice_patrons.txt
SHOUTBOMB_TEXT_PATRONS_PATH=/path/to/text_patrons.txt
SHOUTBOMB_HOLDS_PATH=/path/to/holds.txt
SHOUTBOMB_OVERDUE_PATH=/path/to/overdue.txt
SHOUTBOMB_RENEW_PATH=/path/to/renew.txt
SHOUTBOMB_PHONE_NOTICES_PATH=/path/to/PhoneNotices.csv

# File Download Method (local, ftp, or smb)
SHOUTBOMB_DOWNLOAD_METHOD=local

# FTP Configuration (for downloading archived files from local FTP server)
SHOUTBOMB_FTP_ENABLED=false
SHOUTBOMB_FTP_HOST=ftp.yourlibrary.org
SHOUTBOMB_FTP_PORT=21
SHOUTBOMB_FTP_USERNAME=shoutbomb_user
SHOUTBOMB_FTP_PASSWORD=your_ftp_password
SHOUTBOMB_FTP_SSL=false
SHOUTBOMB_FTP_PASSIVE=true
SHOUTBOMB_FTP_BASE_PATH=/shoutbomb/logs
SHOUTBOMB_FTP_DOWNLOAD_PATH=/var/www/storage/app/shoutbomb/downloads

# SMB Configuration (for downloading from Windows file shares)
SHOUTBOMB_SMB_ENABLED=false
SHOUTBOMB_SMB_HOST=fileserver.yourlibrary.local
SHOUTBOMB_SMB_SHARE=shoutbomb
SHOUTBOMB_SMB_USERNAME=domain\user
SHOUTBOMB_SMB_PASSWORD=your_smb_password
SHOUTBOMB_SMB_WORKGROUP=WORKGROUP
SHOUTBOMB_SMB_BASE_PATH=ftp/logs
SHOUTBOMB_SMB_DOWNLOAD_PATH=/var/www/storage/app/shoutbomb/downloads

# Statistics & Validation
SHOUTBOMB_STATISTICS_ENABLED=true
SHOUTBOMB_VALIDATION_ENABLED=true
SHOUTBOMB_ALERT_EMAIL=admin@yourlibrary.org

# Data Retention (days)
SHOUTBOMB_RETENTION_DAYS=90

File Download Methods

The package supports three methods for obtaining Shoutbomb export files:

1. Local Filesystem (default)

Files are already present on the local filesystem:

SHOUTBOMB_DOWNLOAD_METHOD=local
SHOUTBOMB_VOICE_PATRONS_PATH=/path/to/voice_patrons_submitted_2025-11-14_04-00-01.txt

2. FTP Download

Download archived files from a local FTP server where WinSCP copies are stored:

SHOUTBOMB_DOWNLOAD_METHOD=ftp
SHOUTBOMB_FTP_ENABLED=true
SHOUTBOMB_FTP_HOST=ftp.yourlibrary.org
SHOUTBOMB_FTP_BASE_PATH=/shoutbomb/logs

Files are expected in format: {type}_submitted_YYYY-MM-DD_HH-MM-SS.txt

Note: PhoneNotices.csv is also stored on the local FTP server in the same location as Shoutbomb archived files, and can be imported using the same path configuration

3. SMB/CIFS Download

Download from Windows file shares (e.g., \\server\shoutbomb\ftp\logs\):

Note: SMB support requires an optional package. Install it first:

composer require icewind/smb:^3.5

Configure SMB settings:

SHOUTBOMB_DOWNLOAD_METHOD=smb
SHOUTBOMB_SMB_ENABLED=true
SHOUTBOMB_SMB_HOST=fileserver.yourlibrary.local
SHOUTBOMB_SMB_SHARE=shoutbomb
SHOUTBOMB_SMB_BASE_PATH=ftp/logs

Test your download configuration:

php artisan shoutbomb:test-download-connection

Usage

Artisan Commands

# Import patron lists and notification exports
php artisan shoutbomb:import-voice-patrons /path/to/file.txt
php artisan shoutbomb:import-text-patrons /path/to/file.txt
php artisan shoutbomb:import-holds /path/to/holds.txt
php artisan shoutbomb:import-overdue /path/to/overdue.txt
php artisan shoutbomb:import-renew /path/to/renew.txt

# Import Polaris PhoneNotices.csv for validation and enrichment
php artisan shoutbomb:import-phone-notices /path/to/PhoneNotices.csv
php artisan shoutbomb:import-phone-notices --path=/path/to/PhoneNotices.csv --replace

# Ingest email reports
php artisan shoutbomb:ingest-email-reports --folder="Inbox/Shoutbomb" --mark-read

# Critical validation
php artisan shoutbomb:validate-phone-overlap --date=2025-11-15

# Generate statistics
php artisan shoutbomb:generate-statistics --date=today

# Cleanup old data
php artisan shoutbomb:cleanup --days=90 --dry-run

API Endpoints

Access the interactive API documentation at:

https://your-app.com/api/shoutbomb/documentation

Statistics & Monitoring

# Get statistics overview
GET /api/shoutbomb/statistics/overview

# Get daily statistics
GET /api/shoutbomb/statistics/daily?from=2025-11-01&to=2025-11-15

# Get alerts
GET /api/shoutbomb/statistics/alerts

# Check phone overlap (CRITICAL)
GET /api/shoutbomb/statistics/phone-overlap?date=2025-11-15

Patron Data

# Search voice patrons
GET /api/shoutbomb/patrons/voice?barcode=12345

# Search text patrons
GET /api/shoutbomb/patrons/text?phone=555-1234

Dashboard Commands

# List available commands
GET /api/shoutbomb/commands

# Import voice patrons
POST /api/shoutbomb/commands/import-voice-patrons
{
  "file_path": "/path/to/file.txt"
}

# Ingest email reports
POST /api/shoutbomb/commands/ingest-emails
{
  "folder": "Inbox/Shoutbomb",
  "mark_as_read": true,
  "limit": 50
}

# Validate phone overlap (CRITICAL)
POST /api/shoutbomb/commands/validate-phone-overlap
{
  "date": "2025-11-15"
}

Critical Validation: Phone Overlap

IMPORTANT: Phone numbers cannot appear on both voice AND text patron lists simultaneously. This breaks Shoutbomb's phone-to-delivery-method mapping.

The package includes critical validation to detect and alert on phone overlap:

# Run validation
php artisan shoutbomb:validate-phone-overlap

# Via API
POST /api/shoutbomb/commands/validate-phone-overlap

Overlap detection will:

Log critical error to Laravel logs
Return detailed overlap information
Set severity to "critical" in API responses
Track in daily statistics

PhoneNotices Validation & Enrichment

The package supports importing native Polaris PhoneNotices.csv exports to validate and enrich Shoutbomb notification data.

Purpose

PhoneNotices.csv is NOT used for sending notifications. It serves three critical purposes:

Validation: Verify that custom SQL queries capture all Polaris notifications
Enrichment: Provide patron names, emails, and metadata missing from Shoutbomb exports
Audit Trail: Cross-reference Shoutbomb deliveries against Polaris expectations

File Format

Export: Native Polaris report (25 fields)
Format: CSV with header row
Delimiter: Comma (,)
Encoding: UTF-8 with BOM
Location: Local FTP server (same location as Shoutbomb archives)

Import Command

# Import PhoneNotices.csv
php artisan shoutbomb:import-phone-notices /path/to/PhoneNotices.csv

# Replace existing records for today
php artisan shoutbomb:import-phone-notices --replace

# Use path from .env configuration
php artisan shoutbomb:import-phone-notices

Cross-Validation Keys

PhoneNotices data links to Shoutbomb exports via:

PatronID - Links to patron barcode
ItemRecordID - Links to item barcodes (holds/overdues)
SysHoldRequestID - Links to hold notifications
Due Date - Links to overdue/renewal dates
Phone Number - Links to delivery records (requires standardization)

Data Enrichment

PhoneNotices provides additional fields not available in Shoutbomb exports:

Patron Names: First name, last name, title
Email Addresses: For multi-channel correlation
Site Information: Branch codes and names
Bibliographic Data: Browse titles, item barcodes
Account Information: Balances, transaction IDs
Language Preferences: ISO 639-2/T codes

Validation Features

The importer automatically validates:

Delivery Method Correlation: Voice (V) → DeliveryOptionID 3/4/5, Text (T) → DeliveryOptionID 8
Date Format Conversion: MM/DD/YYYY → YYYY-MM-DD
Phone Number Standardization: Preserves original format, provides 10-digit accessor
Field Count Validation: Ensures minimum 22 required fields
Import Logging: Tracks records processed, inserted, and failed

Database Schema

PhoneNotices data is stored in the polaris_phone_notices table with:

25 fields from Polaris export
Import metadata (date, timestamp, filename)
Validation tracking (matched_to_shoutbomb, validation_notes)
Foreign key to import_logs
9 strategic indexes for query performance

For detailed field definitions and validation rules, see docs/shoutbomb/POLARIS_PHONE_NOTICES.md.

Testing

The test suite requires PHP PDO SQLite extension (pdo_sqlite). See TESTING.md for installation instructions.

Run the test suite:

composer test

Run specific test suites:

# Unit tests only
vendor/bin/phpunit --testsuite Unit

# Feature tests only
vendor/bin/phpunit --testsuite Feature

For detailed testing information, troubleshooting, and CI/CD setup, see TESTING.md.

Documentation

Installation Guide - Detailed installation and setup
Testing Guide - Running tests, requirements, and troubleshooting
Contributing Guidelines - How to contribute
API Documentation - Interactive Swagger UI
Shoutbomb Documentation - Shoutbomb integration details
Polaris Integration - Polaris ILS integration

Documentation Update Rule

CRITICAL REQUIREMENT: All documentation MUST match the latest procedures and requirements.

Bug fixes and internal changes don't need to be documented themselves, but when changes affect how users interact with the package, documentation must be updated to reflect the current state.

Before committing changes that affect user-facing functionality, run the documentation validation:

composer validate-docs

This checks:

OpenAPI specification is valid and complete
All API routes are documented
Required documentation files exist
Version numbers are consistent
No TODO/FIXME markers remain

See CONTRIBUTING.md for complete documentation requirements.

Available Composer Scripts

composer test              # Run PHPUnit tests
composer validate-docs     # Validate documentation
composer check            # Run all validations (docs + tests)
composer pre-commit       # Pre-commit validation

Installing the Git Pre-commit Hook

Optionally install the pre-commit hook to automatically validate documentation:

cp .github/hooks/pre-commit.example .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

Architecture

Data Flow

Import: Voice/text patron lists and holds exports are imported via artisan commands or API
PhoneNotices Import: Native Polaris PhoneNotices.csv imported for validation and enrichment
Validation: Critical phone overlap validation runs automatically
Email Ingestion: Microsoft Graph API fetches reports from configured mailbox
Parsing: Email parsers extract data from Shoutbomb reports
Cross-Validation: PhoneNotices data cross-referenced with Shoutbomb exports
Storage: All data stored in database with comprehensive logging
Statistics: Daily statistics generated with failure rate tracking
API Access: Dashboard consumes data via RESTful API

Key Components

Services: MicrosoftGraphService, ShoutbombConfigService, StatisticsService
Importers: VoicePatronsImporter, TextPatronsImporter, HoldsImporter, OverdueImporter, RenewImporter, PolarisPhoneNoticeImporter
Parsers: DailyInvalidPhoneParser, VoiceFailureParser, MonthlyStatisticsParser
Commands: 11 artisan commands for all operations
API Controllers: 7 controllers for complete API coverage
Models: 8 Eloquent models with relationships (includes PolarisPhoneNotice)

Contributing

We welcome contributions! Please see CONTRIBUTING.md for:

Code style guidelines
Documentation requirements
Testing requirements
Pull request process
Git commit conventions

Remember: All changes require corresponding documentation updates!

License

This package is open-source software licensed under the MIT license.

Credits

Author: Brian Lashbrook (blashbrook@dcplibrary.org)
Organization: DC Public Library

Support

For issues, questions, or feature requests:

Open an issue on GitHub
Contact: blashbrook@dcplibrary.org

Changelog

See the Git commit history for detailed changes.

Recent Major Updates

2025-11: Polaris PhoneNotices.csv import for validation and enrichment
2025-11: Laravel 11 & 12 support, PHP 8.3-8.4
2025-11: Dashboard-accessible command endpoints
2025-11: Comprehensive OpenAPI documentation
2025-11: Documentation validation system
2025-11: Installation wizard with interactive setup

dcplibrary / notices-shoutbomb

Maintainers

Details