$ show QwenLM/Qwen-AgentWorld
QwenLM/Qwen-AgentWorld is a research benchmark evaluation toolkit for Language World Models (LWMs). The repository contains the AgentWorldBench evaluation infrastructure: a standalone CLI script (`eval/eval.py`) that orchestrates a two-stage evaluation pipeline — first calling an OpenAI-compatible inference endpoint to generate environment predictions from a world model, then using an LLM judge (e.g., GPT) to score the predictions across five dimensions (format, factuality, consistency, realism, quality). The code covers seven agent domains (MCP, Search, Terminal, SWE, Android, Web, OS) through per-domain prompt templates and task configurations. The repository is not a model serving system or training framework; it is a reproducible evaluation harness published alongside the Qwen-AgentWorld model weights and benchmark dataset (hosted separately on Hugging Face). The core value is in the robust parsing utilities for extracting structured output from LLM responses, including sophisticated handling of think/response tag removal and multi-strategy JSON extraction from judge outputs.
The LLM identified 2 files worth highlighting for reuse:
Implements `parse_judge_output()` and `extract_choice_answer()` — robust multi-strategy parsers for extracting structured scores and choices from LLM judge outputs. Uses cascading extraction methods (markdown code blocks, brace matching, JSON repair) to handle imperfect model outputs. Includes `_remove_thinking_tags()` (re-exported from output_parser) to strip chain-of-thought content. The choice extraction handles \boxed{} patterns and fallback text matching. This module is self-contained, has no internal dependencies beyond `task_configs`, and could be extracted as a standalone library for any project needing to parse structured data from LLM text completions.
Provides `_remove_thinking_tags()` and `parse_model_output()` — utilities for cleaning and extracting structured content from model-generated text with XML-like tags. The `_remove_thinking_tags` function is particularly sophisticated: it handles nested, orphaned, and multiple think/response tag blocks with three-pass processing (paired matching, orphan closing, orphan opening). This is a general-purpose text processing utility that solves a common problem in LLM workflows (separating reasoning from final answer) with edge-case handling that goes far beyond simple regex.