Agent Name: Geotab Ace
Goal: Deliver customized fleet data insights using natural language
Version: 0.6.0-rc7 on production
Base Model: Gemini-1.5-pro (Official model card)
Developed By: Geotab AI Engineering team
Date of Release: February 2025
How can I use this product? Ace is embedded in MyGeotab. Also, as an end user you just have to:
Geotab Ace is a Generative AI (GenAI) assistant that enables users to interact with complex fleet datasets through natural conversation, democratizing data access by removing technical barriers.
Target users: Fleet managers with access to MyGeotab.
Out-of-scope uses: This system will only answer questions and give information related to fleet management.
Supported languages: All languages supported by Gemini (list).
Purpose-built for fleet management: Integrated with the MyGeotab platform, Ace allows fleet managers to access performance data and actionable insights through simple conversations.
Automated SQL generation and data retrieval: Ace translates natural language questions into precise SQL queries for Google BigQuery, delivering customized information that meets each user's specific needs.
Context-aware retrieval: The agent prioritizes retrieving relevant, up-to-date information directly from the user's fleet database rather than relying solely on pre-trained knowledge.
Query processing: When a question is received from the user, it passes through the embedding model to generate a vector representation. This vector is used to search the Vector Database for a semantically similar question, its SQL query, interpretation, and category. If the user's question is irrelevant to fleet management, the large language model does not generate a SQL query.
Context Enhancement: The system combines the original question with the relevant retrieved examples to create enriched context. This combined information is then sent to the large language model (LLM) for SQL query generation.
Iterative Refinement: The generated SQL query created by the LLM is sent to Google Big Query (GBQ) for execution. If an error occurs, the system re-prompts the LLM with the error information and attempts again, continuing this process until successful data retrieval occurs.
Logging: All interactions, queries, and system operations excluding the customer's data are recorded in the logging database for monitoring, debugging, and analytical purposes.
AI Foundation: The agent is built around Google's Gemini 1.5 Pro AI, which processes natural language and converts it to SQL queries. Rather than modifying the AI model itself, the system leverages it within a Retrieval Augmented Generation (RAG) framework to enhance performance and accuracy.
Vector Database: Contains 343 frequently used fleet management questions with corresponding fields: Question text, Reference SQL query, Interpretation details, and Category classification
Embedding Model: A fine-tuned version of OpenAI's text-embedding-002 converts questions into vector representations, enabling semantic matching between user queries and the sample database.
Google Big Query: Serves as the primary data repository for all fleet information. The system sends generated SQL queries to Big Query, which executes them and returns relevant fleet data.
Firestore: Functions as the conversation storage system, maintaining records of user questions within each session, generated SQL queries, and query execution results
Data as a Service (DaaS): Geotab's DaaS platform provides the distributed infrastructure backbone for the system. This enables the application to operate across multiple computing environments while maintaining consistency and performance.
GenAI Gateway: Geotab's centralized Generative AI platform provides standardized API access to various Large Language Models (LLMs). This gateway facilitates seamless integration of AI capabilities into Geotab's product ecosystem and development workflow.
Orchestration Service: Coordinates the overall workflow between system components, ensuring proper sequencing of operations.
Geotab's data that Ace can access includes:
Deployment Environment: Cloud-based, DaaS
Updates and Maintenance Plan: Weekly on Thursdays
Recommended Hardware: 4 vCPUs, 14GB RAM
Scaling Support: Ace supports horizontal scaling with Kubernetes clusters
Data Access: Currently, Ace does not have access to geospatial data.
Knowledge Constraints: Ace is intentionally constrained to content relevant to fleet management.
Chat History: Currently, chat history is not available for the user to view.
Response Speed: The average response time for a predefined question is about 10 seconds. This increases to 36 seconds for questions that are not predefined.
UI Limit: Text input cannot exceed 500 characters for one message.
The agent generates its chain-of-thought, reasoning process, and SQL query to the user. This documentation and the Ace Whitepaper are efforts towards Geotab's commitment to transparency in its responsible AI practices.
Guardrails have been added to restrict irrelevant topics to fleet management and enforce compliance. The RAG-based architecture grounds the responses as much as possible to prevent hallucination. Responses are filtered to reduce harmful outputs, using Gemini's built-in safety filters. Audit logs track and review generated responses for quality control.
Ace, as an agent, is responsible for generating and executing queries. Although users receive answers based on their own data, Ace does not have access to it. Results are stored in a bucket with access granted solely to the user interface for display to the customer. User data is not sent to any third party large language model, and is exclusively handled within Geotab's systems.
This metric indicates how closely the LLM-generated SQL query aligns with the expected query.
Evaluates the performance of our SQL generation by comparing the results of the generated queries against the expected query results. The score considers both the accuracy of the data retrieved (matching rows and columns) and the completeness of the results (missed or additional rows).
Shows how accurately the LLM-generated SQL queries match the correct database tables. It directly measures the SQL's success in targeting the correct tables based on the input query.
Tracks the number of SQL queries the LLM generated before producing a final result. A higher number of attempts could indicate difficulty in understanding or constructing the correct query.
Feedback Mechanism: Users can provide feedback on the Ace interface using a thumbs-up or thumbs-down rating system. The AI Engineering group performs a weekly manual analysis of the feedback.