arcprize/arc-agi-3-benchmarking
What It Calls
Calls external APIs
Your API key
Paid
Free tier
The repository's own service
Without a key: Text generation · Image generation · Speech-to-text · Text-to-speech · Embeddings (OpenAI): has a branch that stops with an error without a key.
| Feature | Provider | Use | Default | Alternative |
|---|---|---|---|---|
| Text generation · Image generation · Embeddings | Google Gemini API | Key needed | generativelanguage.googleapis.com |
Other providers |
| Text generation · Image generation · Speech-to-text · Text-to-speech · Embeddings | OpenAI
1 call sites |
Key needed | api.openai.com |
None |
| Text generation | anthropic, gemini, openrouter, xai | Key needed | api.x.ai, openrouter.ai +2 more |
None |
| External API (arcprize.org) | arcprize.org The repository's own service
1 call sites |
Undetermined | arcprize.org |
None |
| Text generation | Anthropic Claude
1 call sites |
Optional | api.anthropic.com |
Other providers |
| Text generation | OpenRouter
1 call sites |
Optional | openrouter.ai |
Other providers |
| Text generation · Image generation | xAI Grok | Optional | api.x.ai |
Other providers |
| API | Credential | Code evidence | Pricing | Pricing page | Checked |
|---|---|---|---|---|---|
| Google Gemini API | Your API key | benchmarking/runtime_adapters.py:413Text generation, image generation, and embeddings are provided via the Google Gemini API, which requires a Google Gemini API key. |
Free tier | Pricing | 2026-10-05 |
| OpenAI | Your API key | benchmarking/chat_streaming.py:71Text and image generation, speech-to-text, text-to-speech, and embeddings are provided via OpenAI; if the call fails, it results in an error. |
Paid | Pricing | 2026-10-05 |
| arcprize.orgThe repository's own service | Credential unclear | main.py:60Not shown to stop without a key or address |
No pricing info | - | - |
| Anthropic Claude | Your API key | benchmarking/runtime_clients.py:67Anthropic Claude can be used for text generation, though other providers may be chosen instead. |
Paid | Pricing | 2026-10-05 |
| OpenRouter | Your API key | benchmarking/runtime_clients.py:56OpenRouter can be used for text generation, though other providers may be chosen instead. |
Free tier | Pricing | 2026-10-05 |
| xAI Grok | Your API key | benchmarking/runtime_clients.py:54xAI Grok can be used for text and image generation, though other providers may be chosen instead. |
Paid | Pricing | 2026-10-05 |
Data sent out: prompt text · 5 provider(s) unconfirmed
Checked scope: Full check — all 61 code and config files (76 files in the repository) · tree not truncated · no submodules
Next
Badge preview
[](https://whatitcalls.reactiveworks.dev/en/arcprize/arc-agi-3-benchmarking)
Shows as “external API · used” in a README.
Was this analysis helpful?
Freshness details
- Checked commit
3239e226352f3daff4e6f4a8fa92cb15d5f5e8b5- Committed
- 2026-10-06 21:37 UTC
- Checked
- 2026-10-07 11:48 UTC
- HEAD last seen
- 2026-10-07 11:47 UTC ·
3239e22(same as the checked commit) - Read depth
- Basic (up to 400 files · 8 MB)
- Scanner · rules
scan-5+verdict-4- Pricing catalog
- version 2
- Correction revision
- 0
- Summary
- Model ollama-cloud/gemma4:31b-cloud · some rule sentences
What is sent out
- Confirmed Prompt text is sent to the Google Gemini API and OpenAI.
benchmarking/chat_streaming.py:73benchmarking/google_continuation.py:58benchmarking/runtime_adapters.py:404 - Unconfirmed Providers whose payload was not found in the code: anthropic, gemini, openrouter, xai, arcprize.org, Anthropic Claude, OpenRouter, xAI Grok
README claims ↔ code
No README claim contradicts the code.
Code quotes
Source files are not stored. Code is linked at the checked commit on GitHub; stored quotes are at most 3 lines per evidence and 6 per file.
Source: arcprize/arc-agi-3-benchmarking · commit 3239e22 · benchmarking/runtime_clients.py
Source: arcprize/arc-agi-3-benchmarking · commit 3239e22 · benchmarking/runtime_adapters.py
L413SDK call
Checked scope details
Files
- In the repository
- 76
- Checked
- 61 (80%)
Manifests
All priority files read
pyproject.toml
Unresolved import paths (0)
None
Excluded files
.md9(none)3.ini1.yaml1lockfile1
- External code not checked
- None
- Limits hit
- None
Correction history
No corrections yet.