Business Intelligence Analyst

Job Description

Responsibilities

1. Deterministic Testing & Data Validation

• Validate generative AI tool outputs for structured, rules-based use cases by reconciling results against trusted data sources and established SQL-based metrics.
• Ensure consistency, explainability, and auditability of outputs by confirming alignment with existing data pipelines and query logic
• Expand and maintain test coverage across prioritized use cases to establish a robust, high-confidence baseline for the platform
• Partner with data engineering and analytics teams to identify and resolve discrepancies in underlying data or logic

2. Non-Deterministic Testing & Scenario Evaluation

• Design and execute scenario-based testing for more complex, AI-driven outputs where direct validation is not always possible
• Evaluate results based on intent accuracy, reasonableness, and confidence thresholds rather than exact match validation
• Prioritize testing across higher-risk and high-impact use cases using curated question sets and real-world scenarios
• Identify patterns in output variability and drive iterative refinement to improve reliability and user trust

3. Human-in-the-Loop Review & Continuous Monitoring

• Conduct ongoing review of generative AI tool interactions post-launch, validating outputs and ensuring quality across all user scenarios
• Identify edge cases, inconsistencies, and emerging risks, and escalate findings to product and engineering teams
• Synthesize insights from testing and live usage to inform enhancements, training data improvements, and governance practices
• Serve as an accountable reviewer, providing a critical control point for responsible AI deployment and continuous improvement

Qualifications

Required Skills & Experience

• Robust SQL skills required.
• Robust analytical background with experience in data validation, SQL, and analytics workflows
• Ability to assess outputs both quantitatively (data accuracy) and qualitatively (reasonableness, business context)
• Demonstrated critical thinking and sound judgment, especially in ambiguous or non-deterministic environments
• Experience working with large datasets, reporting tools, or analytics platforms

Preferred Qualifications

• Exposure to AI/ML or generative AI tools and associated testing or validation frameworks
• Experience in scenario-based testing, UAT, or model validation
• Familiarity with financial services, retirement, or plan sponsor analytics

Required Skills: SQL, AI