AutoResponderProcess — Output Files Data Dictionary#

See repo source for current behavior (ReportGenerator._CSV_COLUMNS / _CSV_COLUMNS_IP4, output_document.py).

This document provides a comprehensive reference for every output file produced by the AutoResponderProcess pipeline. Each file is described with its purpose, generation conditions, format, and a detailed explanation of every column, field, or structural element it contains.

All output files are written to a timestamped run directory:

processing_reports/run_{YYYY-MM-DD_HH-MM-SS}/

Table of Contents#

1. run.log
2. stage_execution.log
3. processing_report.log
4. processing_report.json
5. processing_report_master.csv / processing_report_ip4.csv
6. category_summary_report.csv
7. category_summary_report.json
8. classifier_output/classifier_output.json
9. classifier_output/classifier_output.csv
10. output_document_inactive_people.csv
11. output_document_inactive_people.json
12. Marketing suppression deliverable ({BusinessUnit}_NoLongerThere_{date}.csv)
13. output_document_alternate_contacts.csv
14. output_document_alternate_contacts.json
15. output_document_inactive_new_org.csv
16. output_document_inactive_new_org.json
17. output_document_undeliverables.csv
18. output_document_undeliverables.json
19. output_document_inactive_no_cupola_match.csv
20. output_document_inactive_no_cupola_match.json
21. cupola_audit_log.csv
22. cupola_audit_log.json
23. cupola_audit_log_rollback_plan.csv
24. output_document_multipub_audit.csv
25. output_document_multipub_audit.json
26. output_document_email_update_requests.csv
27. output_document_email_update_requests.json
28. action_log.log
29. batch_report.html
30. batch_report.pptx
31. output_document_human_review.csv / .json
32. impact_report.txt / .json

1. run.log#

Purpose#

The primary runtime log file for the entire pipeline execution. Captures every log message emitted by any Python logger during the run at the DEBUG level and above. This is the most granular diagnostic artifact and is the first place to look when troubleshooting unexpected behavior, errors, or performance issues.

Generation Conditions#

Always generated. Created at the start of every run via setup_logging() in logging_config.py.

Format#

Plain text. Each line follows the enhanced logging format:

{timestamp} - [{correlation_id}] - {logger_name} - {level} - {message}

Field Descriptions#

Notes#

• The file handler is set to DEBUG level, which is more verbose than the console handler (set to INFO). This means the file will contain detailed diagnostic information not shown in the terminal.
• Noisy third-party libraries (httpx, urllib3, msal) are suppressed to WARNING level to keep the log focused on application-level events.

2. stage_execution.log#

Purpose#

A structured, stage-by-stage execution log that tracks the pipeline's progression through its major processing phases. Unlike run.log which captures every message, this file is organized into discrete stage sections with JSON-encoded data blocks, making it ideal for programmatic post-run analysis and pipeline health monitoring.

Generation Conditions#

Always generated. Created at run start by the StageLogger class. The final summary section is written when the pipeline completes (normal or early exit).

Format#

Plain text with embedded JSON blocks. The file is divided into:

1. A header section
2. One section per pipeline stage
3. A final summary section

Structure#

Header#

====================================================================================================
AUTORESPONDER PROCESS - STAGE EXECUTION LOG
Run Started: {ISO 8601 timestamp}
====================================================================================================

Per-Stage Section#

Each stage that executes during the pipeline gets its own section:

====================================================================================================
STAGE: {stage_name}
Timestamp: {ISO 8601 timestamp}
----------------------------------------------------------------------------------------------------
STAGE_DATA (JSON):
{JSON object with stage metadata, timing, and data}

SUMMARY:
  Duration: {X.XX}ms ({X.XX}s)
  Status: {completed|failed|skipped}
  Emails Processed: {count}      (if applicable)
  Error: {error message}          (if applicable)
  Details: {JSON details}         (if applicable)
====================================================================================================

STAGE_DATA JSON Fields#

Final Summary Section#

FINAL SUMMARY

Contains a SUMMARY_DATA (JSON) block and a HUMAN-READABLE SUMMARY.

3. processing_report.log#

Purpose#

The primary human-readable processing report. Provides a comprehensive, formatted text summary of every email that was processed, including the contact lookup results, LLM classification, determination, Multipub validation, standard actions, executed actions, and final outcome. This is the main report for operational review of a batch run.

Generation Conditions#

Generated when at least one email is processed. Not generated if the pipeline finds zero emails in Step 1 (early exit).

Format#

Plain text, structured with fixed-width formatting and separator lines. The report has four major sections: Header, Per-Email Details, Summary, and Output Document Lists.

Sections#

Header#

• Generated: Timestamp when the report was generated (format: YYYY-MM-DD HH:MM:SS {timezone})
• Mode: The run mode — DRY-RUN (all connections mocked), READ-ONLY (live reads, write operations MOCKED), or LIVE
• Run window: Start time through end time with total duration in seconds
• Emails: Total number of emails processed in this run

Per-Email Block (repeated for each email)#

Each email gets a detailed block with these subsections:

Email Identification:

• Email ID: The unique message identifier from the email system (typically the database Id column from the Hodor dmorders_thompson table)
• From: The sender's display name and email address in format Name <email@domain.com>
• Subject: The full email subject line
• Received: The date and time the email was received
• Inbox: Which inbox/account the email was fetched from (e.g., energy@thompson.com, grants@thompson.com)
• Body: A preview of the email body text, truncated to 500 characters with total character count shown if truncated. Newlines are collapsed to spaces for readability.

Contact Lookup:

• Contact: Whether the contact was found and in how many systems, with the list of systems. Systems that were mocked in the current run are annotated with (mock). The count is broken down into live vs. mock counts.
• Sources (all queried connectors): A comma-separated list of all backend connectors that were queried during contact lookup (e.g., cupola, hodor, multipub, salesforce), regardless of whether they returned results.
• Sources used (by field): A semicolon-separated list showing which backend system provided each specific data field. Format: field_name:source_system (e.g., person_name:cupola; org_name:hodor; cupola_org_id:cupola).

LLM Classification (if classification was performed):

• Initial Category: The category assigned by the first-pass classification LLM agent before QA review. Only shown if QA correction was applied.
• Final Category: The category after QA agent review. Annotated with (QA corrected) if the QA agent changed the initial classification.
• LLM Category: Shown when no QA correction was applied — the single category from classification.
• QA Explanation: The QA agent's reasoning for why it changed (or confirmed) the classification.
• Person Status: The employment/organizational status of the person as determined by the LLM (e.g., left_company, retired, deceased, active, on_leave).
• Email Status: The status of the email address itself as determined by the LLM (e.g., valid, invalid, bounced, changed).

Determination:

• Determination: The final determination type in uppercase. Possible values: INACTIVE (person has left/retired/deceased — mark inactive across systems), ACTIVE (person is still active — ensure records are current), REPLACEMENT (a replacement contact was identified — mark original inactive and add replacement), TITLE_UPDATE (person's title has changed), EMAIL_UPDATE (person's email address has changed), UNKNOWN (not relevant, spam, or unclassifiable — no action needed).
• Confidence: The confidence score for the determination, displayed as a percentage (e.g., 95%).
• Source Email: The email address extracted from the auto-response body that was used as the basis for contact lookup (may differ from the sender email when the bounced email references a different address).
• New Email: A new email address for the person, extracted from the auto-response (relevant for EMAIL_UPDATE and some REPLACEMENT scenarios).
• Replacement: The name and email of the replacement contact in format Name <email>. If multiple replacements were identified, each is numbered (Replacement 1, Replacement 2, etc.).
• Repl. Title: The job title of the replacement contact, if provided.
• Personal Email: A retired/personal email address for the person (e.g., when someone leaves a company and provides their personal email).
• Long-term Leave: Displayed as Yes when the person is identified as being on extended leave rather than having permanently departed.
• Reasoning: The LLM's reasoning/notes explaining why this determination was made.

Multipub Subscription Validation (if validation was performed):

• Subscriber: The Multipub subscriber number and how it was matched (e.g., by email, by name). Shows Not found in Multipub if no subscriber record was located.
• Active Subs: Whether active subscriptions were found, with the count of active orders.
• Expired (12mo): Whether recently expired subscriptions (within 12 months) were found, with order count.
• Single-Issue: Whether recent single-issue purchases were found, with order count.
• Subscriptions: No relevant subscription activity — shown when none of the above subscription types were found.
• Review Flag: The reason the record was flagged for manual review (e.g., active subscriptions found for an inactive person).
• DEFERRED: Indicates that the inactive marking was HALTED because the person has active subscriptions in Multipub. These records require manual review before proceeding.

Standard Actions: A numbered list describing what actions WOULD be performed in a live run for this determination type, regardless of the current run mode. This serves as documentation of the expected workflow. Actions reference specific systems (Cupola, Hodor, Multipub, Salesforce) and note which are mocked in the current run.

Actions Executed: A list of every action that was actually executed (or mocked) during this run. Each action shows:

• [ OK] or [FAIL] status indicator
• The system name (e.g., cupola, hodor, salesforce)
• The operation performed (e.g., mark_inactive, add_contact, update_email)
• Detail text explaining the specific action taken

The section header varies by mode: (ALL MOCKED - dry-run), (writes MOCKED - read-only mode), or no annotation in live mode.

Outcome:

• Outcome: The final processing status. Possible values: SUCCESS (all actions completed successfully), FAILED (one or more actions failed), SKIPPED NO CONTACT (contact not found in any system — no actions to take), SKIPPED UNKNOWN (determination was unknown — no actions needed), ERROR (an unexpected error occurred during processing), DEFERRED MULTIPUB (processing halted due to active Multipub subscriptions requiring manual review), PENDING (processing not yet complete — should not appear in final reports).
• Reason: Explanation for why processing was skipped, if applicable.
• Error: The error message if the status is ERROR or FAILED.
• Duration: Processing time for this individual email in milliseconds.

Summary Section#

Aggregated counts across all emails in the batch:

• Total Emails: Total number of emails processed
• Successful: Count of emails with success status
• Failed: Count with failed status
• Skipped (no contact): Count with skipped_no_contact status
• Skipped (unknown det): Count with skipped_unknown status
• Errors: Count with error status
• LLM Category Breakdown: Count of emails per LLM classification category
• Determination Breakdown: Count of emails per determination type
• QA Corrections: Number of emails where the QA agent changed the initial classification
• Multipub Validation: Counts for validated, active subscriptions found, recently expired, recent single-issue, and deferred (halted)
• Action Totals: Total actions executed, succeeded, and failed
• Output Document Lists: Record counts for Inactive People, Alternate Contacts, and Inactive at New Org lists
• Total Run Duration: Wall-clock time for the entire batch run in seconds

Output Document Lists (appended if data exists)#

Detailed listings for three output document types. See the individual output document file descriptions below for field details.

Output replay (regenerated/)#

The output replay utility (auto-responder-replay-output / scripts/replay_output.py) re-runs the shared batch pipeline (pipeline/batch_processor.py) for emails extracted from an existing processing_reports/run_* folder. Regenerated files are written only under run_*/regenerated/; originals in the run root are never overwritten.

After replay, regenerated/replay_verification.json summarizes per-file comparison (match, diff, error, or skipped) against the source artifact. Volatile LLM fields (confidence, QA explanation, timestamps) are ignored by default.

v1 scope: output_document_*, processing_report_*, category_summary_report, classifier output, cupola audit, impact report, and batch report (full-run). Notification CSVs (Hodor import, Tarun undetermined, Multipub follow-up) are deferred to v1.1.

4. processing_report.json#

Purpose#

A JSON companion to the human-readable processing report (.log). Contains the same data in a machine-parseable format suitable for programmatic consumption, integration with dashboards, or post-run analysis scripts.

Generation Conditions#

Generated whenever processing_report.log is generated (when at least one email is processed).

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

Record Fields (each object in records array)#

Email Identification Fields#

Contact Lookup Fields#

Determination Fields#

Multipub Validation Fields#

Raw LLM Output Fields#

Actions Fields#

Outcome Fields#

5. processing_report_master.csv and processing_report_ip4.csv#

Each run emits two CSV companions from ReportGenerator.write_report: processing_report_master.csv (full column ledger) and processing_report_ip4.csv (IP4-facing subset, fixed column order — 2026-05-03).

processing_report_master.csv#

Purpose#

Spreadsheet-compatible export with one row per processed email and the complete flattened column set (ReportGenerator._CSV_COLUMNS). Use this file for internal analysis, Client Services run reports, and audit.

Generation Conditions#

Generated whenever processing_report.log is generated.

Format#

CSV with UTF-8 BOM encoding (utf-8-sig for Excel compatibility). All fields are quoted (QUOTE_ALL). Newlines within field values are replaced with spaces to prevent row splitting.

Column Reference#

processing_report_ip4.csv#

Purpose#

Filtered export for Sai Teja / IP4: only rows that need manual Cupola follow-up under the agreed LLM categories, with a fixed 23-column layout so templates and macros do not drift (ReportGenerator._CSV_COLUMNS_IP4).

Generation Conditions#

Written together with the master CSV whenever processing_report.log is generated.

Row filter#

Only emails whose Final LLM Category (or, if empty, Initial LLM Category) normalizes to one of: Out of Office, Retired, Deceased, Left Company, Changed Email (ReportGenerator._IP4_ACTIONABLE_CATEGORIES). All other categories are excluded from this file (they still appear on the master CSV and in processing_report.json).

Format#

Same as the master CSV: UTF-8 BOM, QUOTE_ALL, newline sanitation.

Column Reference (fixed order — 23 columns)#

6. category_summary_report.csv#

Purpose#

A consolidated summary that groups all processed emails into five main business categories. This report collapses the granular LLM categories into broader groups for high-level analysis and reporting to stakeholders who need to understand the distribution of auto-response types without granular detail.

Generation Conditions#

Generated when at least one email is processed. Not generated if the records list is empty.

Format#

CSV with UTF-8 BOM encoding. All fields are quoted. Rows are ordered by category in a fixed sequence: Undeliverable, Left Company / Retired / Deceased, Out of Office, Changed Email, Other.

Category Mapping#

Column Reference#

7. category_summary_report.json#

Purpose#

JSON companion to the category summary CSV. Provides the same grouped data in a machine-readable format with records organized under their respective category keys.

Generation Conditions#

Generated alongside category_summary_report.csv.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

8. classifier_output/classifier_output.json#

Purpose#

The raw, unprocessed output from the LLM classification and QA agents for every email that went through classification. This file preserves the full agent responses before any post-processing, mapping, or interpretation by the pipeline. It serves as the primary audit trail for LLM decision-making and is essential for debugging classification issues, evaluating LLM accuracy, and tuning prompts.

Generation Conditions#

Generated only when at least one email went through LLM classification (i.e., at least one record has raw_classification_result or raw_qa_result populated). Created in a classifier_output/ subdirectory within the run folder.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

Record Fields#

9. classifier_output/classifier_output.csv#

Purpose#

A tabular/spreadsheet-friendly view of the LLM classification and QA outputs. Flattens the raw agent responses into discrete columns for side-by-side comparison of initial classification vs. QA review results, and includes the Determination the pipeline derived from the QA-final category (see column 6 below).

Note on categories: The classification and QA agents are expected to assign a single label from the nine LLM categories. If the model returns a compound string (for example comma-separated labels), the pipeline normalizes it to one canonical category using a fixed severity priority before mapping to actions, and the CSV reflects that normalized value in Initial Category and Final Category.

Generation Conditions#

Generated alongside classifier_output.json.

Format#

CSV with UTF-8 BOM encoding. All fields quoted.

Column Reference#

10. output_document_inactive_people.csv#

Purpose#

A business deliverable listing all people determined to be INACTIVE (left company, retired, deceased) along with the specific actions taken or planned across each backend system. This document is used by operations teams to verify that inactive contacts have been properly removed or suppressed across CUPOLA, HODOR, SFMC, and Multipub. It also provides the sales team with active subscription information so they can follow up on transferring subscriptions. Not emailed on N04. The marketing team receives the slimmer SFMC import file(s) described in Marketing suppression deliverable (*_NoLongerThere_*.csv) via notify_marketing_suppression.

Generation Conditions#

Generated only when at least one inactive person record exists in the output document collector.

Format#

CSV with UTF-8 BOM encoding. All fields quoted.

Column Reference#

11. output_document_inactive_people.json#

Purpose#

JSON companion to the inactive people CSV. Contains the same data in structured format for programmatic consumption.

Generation Conditions#

Generated alongside the CSV when inactive person records exist.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

Marketing suppression deliverable ({BusinessUnit}_NoLongerThere_{YYYY-MM-DD}.csv)#

Purpose#

SFMC-ready suppression import file(s) for the marketing team (notification catalog N04) — this is the production suppression path. Derived from the same InactivePersonRecord rows as output_document_inactive_people.csv, but only the email list and three import columns — no Cupola/Hodor/Multipub detail. Live SFMC REST upsert during processing is not in production; see MARKETING_SUPPRESSION.html.

Generation Conditions#

Written in the same pass as output_document_inactive_people.csv when at least one inactive person record exists. Implemented in marketing_suppression_deliverable.write_marketing_suppression_deliverables. One file per business-unit label (sorted by label); emails are deduped case-insensitively within each file.

Format#

CSV with UTF-8 BOM encoding. All fields quoted. Filename pattern: {BusinessUnit}_NoLongerThere_{YYYY-MM-DD}.csv where YYYY-MM-DD is parsed from run_{date}_{time} on the run folder, or today if the pattern does not match. Today resolve_business_unit_label maps every inbox to Marketing.

Column Reference#

Notification#

Notifier.notify_marketing_suppression discovers files with glob *_NoLongerThere_*.csv and attaches them only. output_document_inactive_people.csv is not attached. See NOTIFICATIONS_CATALOG — N04 and the detailed guide MARKETING_SUPPRESSION.html.

12. output_document_alternate_contacts.csv#

Purpose#

A business deliverable consolidating all replacement/alternate contacts identified from auto-response emails. When an inactive person's auto-response mentions a replacement (e.g., "Please contact Jane Doe at jane@company.com instead"), the replacement's information is captured here with planned actions for adding or updating them across CUPOLA, HODOR, and Multipub.

Generation Conditions#

Generated only when at least one alternate contact record exists in the output document collector.

Format#

CSV with UTF-8 encoding, minimal quoting.

Column Reference#

13. output_document_alternate_contacts.json#

Purpose#

JSON companion to the alternate contacts CSV. Contains the same data in structured format.

Generation Conditions#

Generated alongside the CSV when alternate contact records exist.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

14. output_document_inactive_new_org.csv#

Purpose#

A business deliverable tracking inactive people who have moved to a new organization. When an auto-response indicates someone has left for a different company (e.g., "I have moved to XYZ Corp"), this document captures the new organization details and records the planned actions for potentially adding or updating them in CUPOLA and HODOR at their new organization.

Generation Conditions#

Generated only when at least one inactive-at-new-org record exists in the output document collector.

Format#

CSV with UTF-8 encoding, minimal quoting.

Column Reference#

15. output_document_inactive_new_org.json#

Purpose#

JSON companion to the inactive-at-new-org CSV. Contains the same data in structured format.

Generation Conditions#

Generated alongside the CSV when inactive-at-new-org records exist.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

16. output_document_undeliverables.csv#

Purpose#

A business deliverable listing all emails classified as undeliverable — bounce-backs, invalid email addresses, and mail delivery failures. These records represent email addresses that are no longer valid and need to be removed or suppressed across backend systems to maintain data hygiene.

Generation Conditions#

Generated only when at least one undeliverable record exists in the output document collector.

Format#

CSV with UTF-8 BOM encoding. All fields quoted.

Column Reference#

17. output_document_undeliverables.json#

Purpose#

JSON companion to the undeliverables CSV. Contains the same data in structured format.

Generation Conditions#

Generated alongside the CSV when undeliverable records exist.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

18. output_document_inactive_no_cupola_match.csv#

Purpose#

Handoff list for every CUPOLA-undetermined case — inactive (or inactive-stage) determinations with no Cupola match, ACTIVE determinations with no Cupola row (no-auto-add policy), and ACTIVE determinations whose matched Cupola row is inactive (reactivation candidates). Used by IP4 / operations for manual Cupola research or record creation. Delivered via notify_sai_action_items (catalog N05/N06) — To: NOTIFICATION_EMAIL_SAI; global Max + Vish Cc.

Generation Conditions#

Generated when at least one InactiveNoCupolaMatchRecord was collected during the run (OutputDocumentCollector.inactive_no_cupola_match).

Format#

CSV with UTF-8 encoding. Column headers follow the same human-readable style as other output_document_* CSVs.

Column Reference#

Headers match output_document_generator.py (generate_inactive_no_cupola_match_csv).

19. output_document_inactive_no_cupola_match.json#

Purpose#

JSON companion to the IP4 no-Cupola handoff CSV.

Generation Conditions#

Generated alongside the CSV when inactive-no-Cupola-match records exist.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

20. cupola_audit_log.csv#

Purpose#

A dedicated audit trail for all changes made (or planned) in the CUPOLA contact management system during a run. This file documents every status change (marking contacts active/inactive) and every new contact addition, providing a complete record for compliance, rollback, and operational review purposes.

Generation Conditions#

Always generated every run: CupolaAuditLogger.write_audit_log() writes header-only CSV and an empty entries array in JSON when no Cupola actions were logged.

Format#

CSV with UTF-8 BOM encoding. All fields quoted.

Column Reference#

21. cupola_audit_log.json#

Purpose#

JSON companion to the CUPOLA audit CSV. Contains the same data in structured format for programmatic consumption.

Generation Conditions#

Generated alongside the CSV on every run (empty entries when nothing was logged).

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

22. cupola_audit_log_rollback_plan.csv#

Purpose#

A revertible record of every CUPOLA status_change that was actually executed (Auto Applied=Yes) during the run. Generated by CupolaAuditLogger._write_rollback_plan (src/auto_responder/utils/cupola_audit_logger.py) so an operator can roll the batch back with simple SQL if a problem is discovered after the fact.

Generation Conditions#

Generated when at least one audit entry has auto_applied=True and update_succeeded is not False. Not written when:

• the run only emitted recommendations (CUPOLA_AUTOMATIC_UPDATES=false), or
• every auto-applied UPDATE failed, or
• there were no Cupola actions at all.

Format#

CSV with UTF-8 BOM encoding. All fields quoted (csv.DictWriter with QUOTE_ALL).

Column Reference#

Operational notes#

• The plan is written next to cupola_audit_log.csv / .json in the same run directory.
• Rows that hit the MISSING marker should be triaged before running their SQL — the run wrote them without observing the prior state, which usually means a mock or read-only wrapper was active.
• The plan is regenerated per run; older plans are not garbage-collected.

23. output_document_multipub_audit.csv#

Purpose#

Per-row Multipub validation audit for every INACTIVE determination that was checked against the Multipub subscription gate. Written to the run directory for engineers; not emailed (Tarun receives notify_tarun_undetermined_sender_review only). After review, Tarun may post files back through POST /multipub/upload (Yes → notify_multipub_subscriber_followup_from_upload to Angel/Yogesh).

Generation Conditions#

Generated when at least one MultipubAuditRecord was collected during the run (OutputDocumentCollector.multipub_audit). Both deferred and non-deferred inactive paths produce a row when Multipub validation runs.

Format#

CSV with UTF-8 BOM encoding. Booleans rendered as Yes / No (via _sanitize_for_csv). All fields quoted.

Column Reference#

Headers come from OutputDocumentGenerator.generate_multipub_audit_csv (src/auto_responder/utils/output_document_generator.py).

24. output_document_multipub_audit.json#

Purpose#

JSON companion to the Multipub audit CSV — same data, structured for programmatic consumption.

Generation Conditions#

Generated alongside the CSV whenever Multipub audit records exist.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

25. output_document_email_update_requests.csv#

Purpose#

Per-row deliverable for the Changed Email category. Written to the run directory; not bundled into marketing emails (N04 attaches only *_NoLongerThere_*.csv suppression imports from inactive people). Replaces the historical "12Feb-10Mar Email Update Requests" manual export.

Generation Conditions#

Generated by ReportGenerator.write_email_update_requests_deliverable when at least one processed email maps to the Changed Email main category. When zero rows qualify, the file is skipped and a single INFO log line is emitted.

Format#

CSV with UTF-8 BOM encoding. All fields quoted.

Column Reference#

26. output_document_email_update_requests.json#

Purpose#

JSON companion to the email-update-requests CSV.

Generation Conditions#

Generated alongside the CSV when Changed Email rows exist.

Format#

JSON object with UTF-8 encoding, 2-space indentation.

Top-Level Fields#

27. action_log.log#

Purpose#

A verbose execution log that tracks every individual operation (database lookups, updates, notifications, LLM calls) in detail, primarily used during dry-run and read-only modes. This file shows exactly what the system would do (or did do) for each email, including mock operations that simulate real actions. It serves as the definitive record of operational intent and is particularly valuable for validating pipeline behavior before switching to live mode.

Generation Conditions#

Generated when the pipeline runs in dry-run mode or read-only mode. Not generated in full live mode. Created at the start of the run.

Format#

Plain text with timestamped entries.

Structure#

Header#

================================================================================
DRY-RUN EXECUTION LOG
Started: {ISO 8601 timestamp}
================================================================================

Entry Types#

Each entry is timestamped with [HH:MM:SS] in Eastern Time.

Email Processing Start:

[HH:MM:SS] EMAIL PROCESSING: {email_id} from {sender_email}
[HH:MM:SS]   Subject: {subject (truncated to 100 chars)}

Contact Lookup:

[HH:MM:SS] CONTACT LOOKUP: {email}
[HH:MM:SS]   [MOCK] {System}: Found contact {contact_id}
[HH:MM:SS]   [MOCK] {System}: Not found

LLM Classification:

[HH:MM:SS]   LLM Classification: {category} (confidence: {confidence})
[HH:MM:SS]     Extracted new email: {new_email}
[HH:MM:SS]     Extracted alternate contact: {contact_info}
[HH:MM:SS]     Extracted personal email: {personal_email}

Determination:

[HH:MM:SS]   Determination: {determination} (confidence: {score})

Database Updates (mocked):

[HH:MM:SS]   [MOCK] Would {operation} in {System} for {contact_id} ({key=value, ...})

Notifications (mocked):

[HH:MM:SS]   [MOCK] Would send notification: {type}
[HH:MM:SS]     To: {recipient}
[HH:MM:SS]     Subject: {subject}

Action Execution:

[HH:MM:SS] ACTION EXECUTION: Determination={determination} for {email}

Email Completion:

[HH:MM:SS]   Email processing {SUCCESS|FAILED} for {email}

Summary Section (appended at end of run)#

================================================================================
SUMMARY
================================================================================
Total Emails Processed: {count}

Determinations:
  - {type}: {count}

Database Operations (would be performed):
  - {System}: {count} {operation_type}, {count} {operation_type}

Notifications (would be sent):
  - {type}: {count}

LLM Classification Calls: {count}
Execution Duration: {seconds} seconds
Completed: {ISO 8601 timestamp}
================================================================================

Summary Fields#

28. batch_report.html#

Purpose#

A self-contained, visually rich HTML dashboard summarizing the entire batch run. Designed for browser viewing and sharing with stakeholders. Features interactive Plotly charts, KPI cards, per-email detail tables, and links to the output document files. This is the most polished and accessible output artifact, suitable for non-technical audiences.

Generation Conditions#

Generated when at least one email is processed.

Format#

Single HTML file with embedded CSS. Uses the Plotly JavaScript library via CDN (https://cdn.plot.ly/plotly-2.27.0.min.js) for interactive charts and Google Fonts (Outfit, IBM Plex Mono, IBM Plex Sans) for typography. Dark theme (slate/charcoal background with sky-blue and teal accents).

Sections#

Run Overview (KPI Cards)#

Output Document Counts#

Visual Analysis (Interactive Charts)#

In-Depth Analysis (Per-Email Table)#

Output Documents (Links)#

Provides download links (relative file paths) to the three output document pairs:

• Inactive People (CSV · JSON)
• Alternate Contacts (CSV · JSON)
• Inactive at New Org (CSV · JSON)

Note: Undeliverables are generated as separate files but are not linked from the HTML report.

29. batch_report.pptx#

Purpose#

A PowerPoint presentation summarizing the batch run for executive review or team meetings. Contains approximately 10 slides covering KPIs, determination breakdowns, outcome status, LLM category analysis, actions by system, confidence and quality metrics, per-email summary tables, and output document counts.

Generation Conditions#

Generated alongside batch_report.html when at least one email is processed.

Format#

PowerPoint .pptx file generated using the python-pptx library.

Slides#

30. output_document_human_review.csv / .json#

Purpose#

Consolidated Human Review digest introduced by the active-only automation policy. Captures every row that the pipeline refused to act on automatically so IP4 / operations can triage manually. Written by OutputDocumentCollector.add_human_review. Actionable rows ride in notify_sai_action_items; metadata (counts + reason legend) is included in notify_venu_cupola_audit_files.

Generation Conditions#

Generated whenever OutputDocumentCollector.human_review is non-empty. Rows are added by ActionEngine from several handlers:

Format#

CSV with UTF-8 BOM encoding; JSON with 2-space indentation. All fields quoted.

Column Reference (CSV)#

Headers come from output_document_generator.py (generate_human_review_csv). Column titles use spaced words (e.g. Sender Email, Lookup Email).

JSON Top-Level Fields#

31. impact_report.txt / .json#

Purpose#

Per-run headline summary introduced by the active-only automation policy. Produced by utils/impact_report.py and attached inline to Notifier.notify_run_audit_for_ip4 (Sai-only run audit).

Generation Conditions#

Always written at end of run (after the CUPOLA audit logger finishes flushing). The counts are derived from the in-memory CupolaAuditLogger.entries list, so read-only mode and dry-run runs still emit the report (counts are zero when no writes occurred).

Format#

• impact_report.txt — plain text with three labelled counts, one per line.
• impact_report.json — structured object with the same counts plus a timestamp.

Fields#

32. action_items_tracker.csv (cross-run)#

Purpose#

Central queue of one row per action notification email sent in a run (N01–N07). Appended once per run by append_action_items_for_run after all notifications complete. Each row includes ActionItemCount and a per-attachment breakdown in Summary (artifact line counts, not separate tracker rows). Default path: {REPORT_OUTPUT_DIR}/action_items_tracker.csv; override with ACTION_ITEMS_TRACKER_PATH. Post-run completion requests use catalog N12 via auto-responder-request-action-item-confirmation (not appended to this CSV); N12 bodies use collect_action_item_detail_rows for the same counts.

Generation Conditions#

Skipped when the run folder RunId already exists in the tracker (idempotent re-run/resend). New rows are appended with Completed=false for manual spreadsheet triage.

Columns#

Glossary of Systems#

Glossary of Determination Types#

Glossary of Processing Statuses#

Maintaining this document#

Edit docs/DATA_DICTIONARY.html directly. Preview locally from the repo root:

python scripts/serve_data_dictionary.py

Then open http://127.0.0.1:8765/DATA_DICTIONARY.html in a browser.