Are Open Source Models Teaching to the Test?

We are continuously testing out new models and new version of models to measure their effectiveness for building accurate GUI Agents. The primary capability of the model we need is Computer Use. When we started working on GUIDE last summer, no model was good enough in the OSworld benchmark. Now, according to the published benchmarks, there are multiple models that are better than the human accuracy.

We started testing with OpenAI, Gemini and Anthropic models. Starting in Q1, the Anthropic models - both Opus and Sonnet - started performing really well on our internal benchmarks. In Q2, the latest GPT models from OpenAI also started performing well. Both the OpenAI and Anthropic models were consistent with the benchmarking data. On two different benchmarks, we were able to test Anthropic models against both Google Vertex and Amazon Bedrock. We were also able to test against GPT 5.5. The results for all these runs are captured below. No consistent pattern in terms of speed, except Opus seems to be faster than Sonnet on both GCP and AWS.

Based on these results and alignment with the osworld benchmarks, we tried the Kimi 2.6 model. Unfortunately the agents did not complete the tasks, so we are not able to report performance results. We have reached out to the Kimi/Moonshot team to understand if they have any guidance for us.