AI Applied Scientist

About the company Wizard is the top-performing AI Shopping Agent, delivering the best products from across the web with unmatched accuracy, quality, and trust. The company is building the core science infrastructure that defines and measures agent quality, sitting at the intersection of applied ML, evaluation science, and product. About the role Wizard is hiring an AI Applied Scientist as a foundational hire on the science team. The role owns how Wizard measures, understands, and improves the accuracy of its AI agent — defining what "good" looks like, building systems to measure it, and leading the science work to improve it, including fine-tuning LLM judges. This is a fully remote role for US-based candidates and is scoped to grow into broader applied science as the agent expands into recommendations, personalization, ranking, multimodal, and conversational understanding. Responsibilities - Define and evolve accuracy metrics across the full shopping experience (retrieval, ranking, recommendations, outcomes). - Design and run experiments to measure improvements and regressions. - Build and maintain evaluation datasets, benchmarks, and scoring frameworks. - Improve LLM judges that power the evaluation pipeline through prompting, calibration, and fine-tuning. - Translate ambiguous product questions into clear, measurable hypotheses and analysis. - Partner with ML Engineers to validate model changes and guide iteration. - Identify failure modes and edge cases, and drive improvements through data. Requirements - 5+ years in Applied ML, AI Research, or Applied Science; PhD or equivalent depth strongly preferred. - Hands-on experience evaluating modern AI/ML systems (LLMs, agents, ranking, or recommendations). - Direct experience with LLM-based systems: judge models, RAG, prompt engineering, fine-tuning, RLHF. - Strong experimentation foundations: A/B testing, causal inference, statistical rigor. - Proven ability to operate in ambiguity and communicate clearly across technical and product teams. Nice to have - Not specified Benefits & perks - Equity (stock options). - Medical, dental, and vision coverage. - 401(k) plan. - Flexible PTO and company holidays. - Periodic company offsites and team gatherings. Compensation $225,000 - $280,000 USD base salary, varying based on skills, experience, level, and location. Equity included.