Compare GPT-4, Claude & Gemini Responses with Contextual AI's LMUnit Evaluation

PROBLEM

Evaluating and comparing responses from multiple LLMs (OpenAI, Claude, Gemini) can be challenging when done manually.

• Each model produces outputs that differ in clarity, tone, and reasoning structure.
• Traditional evaluation metrics like ROUGE or BLEU fail to capture nuanced quality differences.
• Human evaluations are inconsistent, slow, and difficult to scale.

This workflow automates LLM response quality evaluation using Contextual AI’s LMUnit, a natural language unit testing framework that provides systematic, fine-grained feedback on response clarity and conciseness.

Note: LMUnit offers natural language-based evaluation with a 1–5 scoring scale, enabling consistent and interpretable results across different model outputs.

How it works

• A chat trigger node collects responses from multiple LLMs such as **OpenAI GPT-4.1, Claude 4.5 Sonnet, and Gemini 2.5 Flash.
• Each model receives the same input prompt to ensure fair comparison, which is then aggregated and associated with each test cases
• We use Contextual AI's LMUnit node to evaluate each response using predefined quality criteria:
  • “Is the response clear and easy to understand?” - Clarity
  • “Is the response concise and free from redundancy?” - Conciseness
• LMUnit then produces evaluation scores (1–5) for each test
• Results are aggregated and formatted into a structured summary showing model-wise performance and overall averages.

How to set up

• Create a free Contextual AI account and obtain your CONTEXTUALAI_API_KEY.
• In your n8n instance, add this key as a credential under “Contextual AI.”
• Obtain and add credentials for each model provider you wish to test:
  • OpenAI API Key: platform.openai.com/account/api-keys
  • Anthropic API Key: console.anthropic.com/settings/keys
  • Gemini API Key: ai.google.dev/gemini-api/docs/api-key
• Start sending prompts using chat interface to automatically generate model outputs and evaluations.

How to customize the workflow

• Add more evaluation criteria (e.g., factual accuracy, tone, completeness) in the LMUnit test configuration.
• Include additional LLM providers by duplicating the response generation nodes.
• Adjust thresholds and aggregation logic to suit your evaluation goals.
• Enhance the final summary formatting for dashboards, tables, or JSON exports.
• For detailed API parameters, refer to the LMUnit API reference.
• If you have feedback or need support, please email [email protected].