LLM / Reasoning Engineer

Position: LLM / Reasoning Engineer

Location: Austin, TX / Charlotte, NC / San Diego, CA / NYC, NY 

Experience: 8 years, with demonstrated impact taking LLM applications into production and making them measurably better over time.

You own the reasoning tier — the layer that turns deterministic detections into defensible dispositions. Success here is measured, not asserted: accuracy against decisions already made by hand, and a rising share of items dispositioned without human effort. 

Responsibilities

·       Design the reasoning tier: structured-output disposition schemas, context-engineered prompts and skills, and a curated exemplar set drawn from the team''s own prior decisions.

·       Build and own the gold set — and measure precision, recall and confidence calibration against it.

·       Work along with the existing domain team for execution.

·       Implement abstention as a first-class output (insufficient evidence must escalate, never guess) and self-consistency sampling for high-stakes items.

·       Mine reasoning traces to extract new deterministic rules, continuously pushing items down from the model tier to the rule tier — driving cost and latency down while consistency rises.

·       Drive building the regression suite for the AI layer so that a prompt, rule or model-version change cannot silently degrade quality.

·       Partner with domain SMEs to convert their verdicts into exemplars, noise filters and rules — closing and compounding the loop.

·       Own model routing (inexpensive models for the mechanical majority, frontier models for the hard tail) and the per-cycle token budget. 

Qualifications

·       Python — production-grade.

·       Deep production LLM application engineering: structured / schema-constrained generation, context engineering, tool use, and evaluation — not prototypes.

·       Evaluation engineering: golden sets, LLM-as-judge (and a clear-eyed view of its limits), calibration, trajectory and trace scoring, regression benchmarking.

·       A track record of making an LLM system measurably better sprint over sprint, with numbers you can quote.

·       RAG and grounding quality — retrieval precision, recall and faithfulness scoring.

·       Sound statistics and ML fundamentals; understands why same-model self-validation has correlated failure modes and designs around it.

·       AWS Bedrock / Claude (or equivalent frontier models) in production.

Working knowledge

·       Code graphs and static analysis; SQL; relational data models.

·       Fine-tuning / PEFT; agentic orchestration frameworks; MLOps.

Alchemy: Transforming Your Professional Vision into Reality

Since our inception in 2013, Alchemy has been dedicated to reshaping organizational performance through innovative IT services. With a vision to empower businesses seeking a transformative edge, we’ve positioned ourselves at the forefront of digitization and software modernization.

Our name reflects our mission: to transmute technology into gold-standard solutions for our esteemed clients. We proudly serve a diverse range of sectors, including IT and ITES, BFSI, Telecom and Media, Automotive, Manufacturing, Energy, Oil and Gas, Real Estate, Retail, Healthcare, and more.

With a global footprint spanning the USA, India, Europe, Canada, Singapore, Japan, and parts of Central and West Africa, we harness a unique blend of competencies, frameworks, and cutting-edge technologies. Together, we drive growth and innovation across industries, helping organizations turn their visions into reality.

Alchemy – Connecting Talent with Opportunities (Diversity, Equity and Inclusion)