Accuracy is the Key to AI Agents
The effectiveness of any AI agent is entirely dependent on its accuracy. Just as any computer program is only as good as the instructions and commands it invokes, the same is true with any AI agent, as it is only as good as the accuracy of its prompts and instructions. With GUIDE, GUI agents are not only quick to create, test, observe, and invoke using natural English language prompts, they can be created with accuracy using GUIDE’s recommended process.
There are 4 techniques to drive accuracy of GUIDE AI GUI agents:
1. Model Choice
2. Prompt engineering
3. Harness engineering
4. Observability
Not all models are equally good for GUI Agents. And the versions of these models, which can change frequently, can also deliver different levels of accuracy. Model Choice thus becomes extremely important to deliver the most accurate GUI Agents. Our experience to date has been that the Opus and Sonnet models after version 4.5 and the GPT model after 5.5 are very capable models for building accurate GUI agents.
Prompt engineering is the practice of designing instructions, context, examples, and constraints to get an AI model to produce more accurate, useful, and reliable outputs. In our no-code build tool, we have tried different ways of specifying prompts for building GUI Agents. We have tried short text prompts, long text prompts, video recordings annotated with text, and screenshots with text. Long text prompts are very accurate, but the quality of the prompt depends on the individual specifying it. Video prompts were easy to record, but did not result in accurate prompts. We have determined that the most repeatable and simple way to create accurate prompts is using screenshots and text.
Harness engineering is a key tool in creating accurate GUI Agents. The harness interacts with the model and can determine if the agent is following the detailed prompt. Harness stops execution as soon as there is divergence between screenshot and prompt. It is thus the guardrail and when necessary, the “emergency brake” to help keep GUI Agents operating within their intended purpose.
Observability: Once GUI Agents are deployed in production, it is important to track their actions continuously. As soon as a prompt is observed to deviate from its desired execution, the agent builder team should be alerted, and then they should be provided logs, screenshots, and the thought process of the model to rapidly debug the issue. Without keen observability, GUI agent builders are essentially “operating in the dark”.
Nobody disputes the power and potential of AI to automate workflows within enterprises. What can come under scrutiny and even some skepticism is the accuracy of the AI agents that provide the instructions and guidance to execute those workflows. The above process, including the four key elements of model choice, prompt engineering, harness engineering, and observability, serves as an excellent way to create AI GUI agents that operate with precise accuracy from the start, and can maintain accuracy even as enterprise workflows and AI foundation models evolve over time.
