comet

An open-source platform for end-to-end evaluation and observability of large language models and AI agents, enabling you to log, trace, and visualize LLM calls across complex GenAI systems, capture context, tool usage, user feedback, and performance signals, and support human-in-the-loop debugging and iterative improvement.

Key features include granular trace logging and visualization, automated evaluation metrics for hallucination, relevance, context accuracy, and safety; SME-assisted annotation workflows, online production evaluations, prompt and agent optimization, versioned experiments, and flexible hosting with easy integration into existing codebases and MLOps pipelines, solving problems of black-box behavior, reproducibility, data drift, and collaboration for data scientists, machine learning engineers, researchers, and operations teams.

comet can be found in LLMOps & Prompt Engineering categories.