If 2023 was the year of generative AI-powered chatbots and search, 2024 was all about AI agents. What started from Devin earlier this year grew into a full-blown phenomenon, offering enterprises and individuals a way to transform how they work at different levels, from programming and development to personal tasks such as planning and booking tickets for a holiday.Among these wide-ranging applications, we also saw the rise of data agents this year — AI-powered agents that handle different types of tasks across the data infrastructure stack. Some did basic data integration work while others handled downstream tasks, such as analysis and management in the pipeline, making things simpler and easier for enterprise users. The benefits were improved efficiency and cost savings, leading many to wonder: How will things change for data teams in the years to come?Gen AI agents took over data tasksWhile agentic capabilities have been around for some time, allowing enterprises to automate certain basic tasks, the rise of generative AI has taken things entirely to the next level.With gen AI’s natural language processing and tool use capabilities, agents can go beyond simple reasoning and answering to actually planning multi-step actions, independently interacting with digital systems to complete actions while collaborating with other agents and people at the same time. They also learn to improve their performance over time.Cognition AI’s Devin was the first major agentic offering, enabling engineering operations at scale. Then, bigger players began providing more targeted enterprise and personal agents powered by their models. In a conversation with VentureBeat earlier this year, Google Cloud’s Gerrit Kazmaier said he heard from customers that their data practitioners constantly faced challenges including automating manual work for data teams, reducing the cycle time of data pipelines and analysis and simplifying data management. Essentially, the teams were not short on ideas on how they could create value from their data, but they lacked the time to execute those ideas.To fix this, Kazmaier explained, Google revamped BigQuery, its core data infrastructure offering, with Gemini AI. The resulting agentic capabilities not only provide enterprises the ability to discover, cleanse and prepare data for downstream applications — breaking down data silos and ensuring quality and consistency — but also support pipeline management and analysis, freeing up teams to focus on higher-value tasks. Numerous enterprises today use Gemini’s agentic capabilities in BigQuery, including fintech company Julo, which tapped Gemini’s ability to understand complex data structures to automate its query generation process. Japanese IT firm Unerry also uses Gemini SQL generation capabilities in BigQuery to help its data teams deliver insight more quickly.But discovering, preparing and assisting with analysis was just the beginning. As the underlying models evolved, even granular data operations — pioneered by startups specializing in their respective domains — were targeted with deeper agent-driven automation.For instance, AirByte and Fastn made headlines in the data integration category. The former launched an assistant that created data connectors from an API documentation link in seconds. Meanwhile, the latter enhanced its broader application development offering with agents that generated enterprise-grade APIs — whether it’s for reading or writing information on any topic — using just a natural language description. San Francisco-based Altimate AI, for its part, targeted different data operations including documentation, testing and transformations, with a new DataMates tech, which used agentic AI to pull context from the entire data stack. Other startups, including Redbird and RapidCanvas, worked in the same direction, claiming to offer AI agents that can handle up to 90% of data tasks required in AI and analytics pipelines.
About the Author
