Position Description
Individual within this role will be part of the Research Team. This team supports the firm’s research and its team of analysts available to institutional clients through a Research Portal web site and chat tools including AI-driven chat. To make this vast amount of information more accessible, the firm has built a variety of tools including a search engine, a machine-learning-driven search intent classifier, an LLM-driven question-answering system, and an ontology/taxonomy, all supported by reference data about companies, industries, and analysts. Global Research is building a team of SMEs and LLM/NLP experts to support this.

Seeking a candidate with experience as an annotator for AI and NLP-products, including the annotation of complex datasets to train and refine AI models and to support the ongoing operations and maintenance of such models. The Lead Annotator will work a diverse set of products and projects focused on multiple different uses of large language models – Q&A, summarization, red lining etc.

As the Lead Annotator, you will be responsible for developing and overseeing the annotation framework, ensuring high quality labeling and annotations. You will also help design, customize and optimize annotations tools to support the workflows. You will also develop and manage necessary guidance and feedback processes and programs for junior annotators. You will also participate in hands-on annotation work together with the team.
You will be working together with ontologists and data scientists in understanding the firm’s knowledge graphs and NLP/LLM-products and support them with the necessary annotation work and data from annotations. You will help develop the necessary reporting and summarization of the annotation projects for model development and maintenance purposes and will also create the necessary monitoring and reporting to inform management on the progress of the annotation projects and the quality of the AI models.
You have a deep understanding of language models and will stay updated on the developments in the AI and language modeling space, in particular as they pertain to the financial services industries.

Responsibilities:

• Develop and refine annotation framework and guidelines.
• Oversee the annotation of datasets, ensuring accuracy and consistency.
• Implement quality measures and metrics to maintain high standards of data labeling and annotations.
• Participate in hands-on annotation work.
• Design and customize tools to enhance efficiency and ensure quality annotations.
• Train and mentor junior annotators.
• Collaborate with team to align annotation strategies with project goals and product missions.
• Create and maintain consistent and actionable reporting of annotations.
• Monitor and report on the progress of annotation projects.

** Ideal Experience:

• Financial services experience ideal (not required)
• Experience with annotator tools and machine learning ideal (not required)

Required Skills:

• 2 years of proven experience in data annotation and LLMs
• 1 year of proven team management/leadership abilities
• Familiarity with annotation practices, tools and
• Detail-oriented with a critical eye for accuracy
• Excellent communication skills
• Ability to work in a fast-paced environment
• Ability to create and modify data annotation guidance, processes, etc.

Prior annotation experience with some of the big financial data aggregators – AlphAsense, Bloomberg, Factset, Refinitiv ideal

Prior experience working on chatbots or smart search solutions or smart summary type solutions.

Job Type: Temporary

Pay: From $52.00 per hour

Experience level:

• 3 years

Application Question(s):

• Do you have at least 1 year of experience as a Team Lead?
• Do you have at least 2 years of experience as a Lead Annotator?

Ability to Commute:

• Alpharetta, GA 30009 (Required)

Ability to Relocate:

• Alpharetta, GA 30009: Relocate before starting work (Required)

Work Location: Hybrid remote in Alpharetta, GA 30009