Async Comparator for Answer Ranking Context

Question

Async Comparator for Answer Ranking
Context
In large language model (LLM) training and evaluation, it is common to use a comparator function that takes two answers and determines which one is better. This binary comparison is widely used in data collection pipelines for ranking, filtering, and preference modeling.
To improve efficiency, especially when comparing multiple unrelated answer pairs, we want to run these comparisons concurrently using Python's asyncio.
This task consists of three parts:
1. Implement an async batch comparator that runs multiple pairwise comparisons concurrently.
2. Use the pairwise comparator to find the best answer out of N.
3. Use the pairwise comparator to find the top-K best answers out of N.

Provided Comparator Function
You are given the following asynchronous function that compares two answers and returns the better one:
import asyncio
import random

async def async_compare(a: str, b: str) -> str:
    """
    Asynchronously compares two answers and returns the better one.
    This function simulates some latency.
    """
    await asyncio.sleep(0.1)  # Simulate latency
    return a if random.random() < 0.5 else b

Task 1: Batch Async Comparison
asy

Accepted Answer

A detailed reference solution is available to Crack ML Interview members.