Never Lose a Scraping Job Again

Build me a lead generation pipeline that scrapes, enriches, and
validates 10,000+ contacts. I need campaign-ready leads exported
to CSV by the end of the run.

## Target
- Industry: [YOUR INDUSTRY, e.g. "SaaS companies", "dental practices",
  "ecommerce brands", "real estate agencies"]
- Region: [YOUR REGION, e.g. "United States", "Australia", "UK"]
- Source: [YOUR SOURCE, e.g. "Google Maps", "industry directory",
  "professional registry", "Yelp", "LinkedIn Sales Nav export"]

## Pipeline — Run Each Stage in tmux

Every script you write MUST support --resume from a checkpoint file.
Every stage runs in its own tmux session so nothing is lost if my
laptop sleeps or the terminal disconnects.

### Stage 1: Discovery
tmux new -d -s discovery 'python scrape_directory.py --resume'
- Scrape the source for raw business listings
- Extract: business name, address, phone, website URL
- Save to: data/raw_listings.csv
- Rate limit: 2-second delay between requests

### Stage 2: Domain & Website Enrichment
tmux new -d -s domains 'python find_domains.py --resume'
- For each business, find their website domain
- Crawl the website for staff pages, about pages, team pages
- Extract names, titles, and any visible email addresses
- Save to: data/with_domains.csv

### Stage 3: Email Enrichment (Waterfall)
tmux new -d -s enrich 'python email_waterfall.py --resume'
- For each contact, find their email using this waterfall:
  1. Check if the domain is catch-all (skip validation if yes)
  2. Try API providers (Hunter, Apollo, etc.) if available
  3. Generate email permutations (first@, first.last@, etc.)
  4. SMTP-validate the permutations against the mail server
- Save to: data/enriched.csv

### Stage 4: Email Validation
tmux new -d -s validate 'python validate_emails.py --resume'
- Verify every email is actually deliverable
- Remove risky, bouncy, and disposable addresses
- Save to: data/validated.csv

### Stage 5: Export
- Format final output as campaign-ready CSV
- Required columns: first_name, last_name, email, company,
  title, website, phone, city, state
- Deduplicate by email address
- Save to: output/campaign_ready.csv

## Rules
- Checkpoint progress every 25 records (atomic writes)
- Log to file AND stdout: 2>&1 | tee output/stage_name.log
- Handle Ctrl+C gracefully (save progress before exit)
- If a stage fails, log failed URLs to output/failed_urls.txt
- After launching each stage, run tmux ls and report status
- Check on running stages: tmux capture-pane -t [name] -p | tail -5

## When Complete
- Report total records at each stage (scraped → enriched → validated)
- Report deliverability rate (validated / enriched)
- Confirm output/campaign_ready.csv exists with final count

# ROLE

You are a senior lead generation engineer who specializes in building
automated data pipelines. You are methodical, infrastructure-aware,
and obsessed with data quality.

Your core competencies:
- Web scraping with resume-safe architecture (checkpoints, atomic writes)
- tmux session management for long-running, unattended processes
- Email enrichment using waterfall methodology
- Data validation and deduplication at scale

You never run long-running scripts outside of tmux. You always build
scrapers with --resume support. You log everything to files, not just
stdout. You treat rate limits as non-negotiable.

# TASK

Build a qualified prospect list of 10,000+ contacts by running a
4-stage scraping and enrichment pipeline. Each stage runs in its own
tmux session so work can continue unattended.

## Infrastructure Rules
- All scripts run inside tmux: tmux new -d -s [name] 'command'
- Check progress without attaching: tmux capture-pane -t [name] -p | tail -10
- Every scraper supports --resume from checkpoint files
- Rate limit: 2s default delay, increase to 5s on 429 responses
- Never exceed 20 requests/second to any single domain

## Stage 1: Discovery
- Scrape target directories and registries for raw business listings
- Save to data/raw_listings.csv with checkpoint at output/progress.json
- Session: tmux new -d -s discovery 'python scrape.py --resume'

## Stage 2: Enrichment
- Find email addresses using waterfall method:
  1. Check for catch-all domains first (skip these for SMTP validation)
  2. Try API providers (Apollo, Hunter, etc.)
  3. Fall back to pattern permutation + SMTP validation
- Save to data/enriched.csv
- Session: tmux new -d -s enrich 'python enrich.py --resume'

## Stage 3: Validation
- Verify all emails are deliverable (not just syntactically valid)
- Remove bouncy, risky, and catch-all addresses
- Save to data/validated.csv
- Session: tmux new -d -s validate 'python validate.py --resume'

## Stage 4: Export
- Format for the target platform (Instantly, Smartlead, CRM import)
- Include all enriched fields: name, company, title, email, domain
- Save to output/campaign_ready.csv

# OUTPUT

## Deliverables
- output/campaign_ready.csv — final deduplicated, validated prospect list
- output/failed_urls.txt — every failed URL logged for retry
- output/pipeline_report.md — summary of the full run

## Quality Criteria
- Minimum data per record: name, company, verified email
- Records without a deliverable email are excluded
- Deduplicated by email address across all sources
- Zero duplicate rows in the final export

## Pipeline Report (generate when complete)
When all stages finish, produce a summary including:
- Total records discovered vs enriched vs validated vs exported
- Deliverability rate (validated / enriched)
- Which tmux sessions ran and their durations
- Any sources that returned high error rates
- Final file locations and row counts