Data streaming platform provider Confluent, now part of IBM, has launched a suite of new capabilities across Confluent Intelligence and Confluent Cloud aimed at simplifying the development and deployment of real-time artificial intelligence applications. These updates target the critical bottleneck that often stalls AI projects: the data layer. By integrating agent-powered workflows, automated data protection, and private cloud connectivity, Confluent aims to provide a robust foundation for building production-ready AI systems that can handle live data streams securely and efficiently.
According to a McKinsey report cited in the announcement, eight out of ten companies identify data limitations as a major obstacle to scaling agentic AI. This resonates with industry experiences where security teams block data from entering AI pipelines due to exposure risks, and developers spend significant time switching between tools to inspect and manage data streams. Confluent’s latest offerings directly address these pain points by embedding intelligence and governance directly into the data streaming layer.
Agent-Powered Workflows and MCP Integration
A centerpiece of the release is the introduction of a fully managed Model Context Protocol (MCP) server and Agent Skills. These tools allow artificial intelligence to manage streaming operations using natural language commands. The MCP server acts as a control plane, enabling developers to build, manage, and debug streaming pipelines through conversational interfaces. Agent Skills add a layer of pre-defined best practices and workflows, ensuring that operations are executed consistently and in line with organizational standards. Together, they enable developers to create and continuously improve real-time applications using AI-powered tools, effectively integrating streaming into modern, agent-driven development workflows. This capability is being made generally available for Confluent Cloud customers.
The agent-powered approach represents a paradigm shift from traditional manual configuration and scripting. Instead of writing complex code for every streaming pipeline adjustment, developers can issue commands like "set up a real-time anomaly detection pipeline for website traffic" and the system will handle the intricacies of data ingestion, transformation, and alerting. This dramatically reduces the time from concept to deployment and lowers the barrier for teams without deep streaming expertise.
Automated Data Privacy with PII Redaction
Data privacy remains a paramount concern, especially in regulated industries such as financial services, healthcare, and insurance. Confluent is addressing this with a new built-in machine learning function for personally identifiable information (PII) detection and redaction. This function operates directly within Flink SQL, the stream processing language, without requiring custom code, external services, or moving data to a separate warehouse first. The capability runs inline on the data stream, redacting PII before it reaches AI applications or downstream systems. This early-access feature for Confluent Intelligence unlocks more AI use cases across highly regulated environments by ensuring compliance with data protection regulations like GDPR, HIPAA, and CCPA.
To further enhance security, Confluent now supports Azure Private Link for private connectivity. This ensures that AI workloads stay off the public internet when calling external models or querying external tables. Flink jobs can securely connect to Azure-hosted services such as Azure OpenAI, Azure SQL, and Cosmos DB over Microsoft’s private backbone. This private connectivity is now generally available on Confluent Cloud, allowing enterprises to maintain strict data residency and security policies while leveraging cloud-based AI services.
Unified Engineering Workflows with dbt Integration
Data engineers often use standard frameworks to build and manage pipelines. Confluent is bridging the gap between batch and streaming by introducing a free, open-source dbt adapter that integrates Flink SQL on Confluent Cloud into the popular dbt tool. Teams can now define, test, and deploy streaming pipelines using the same dbt commands and project structures they rely on for batch processing. This lowers the barrier to Flink adoption and extends existing data workflows into real-time use cases without requiring additional tooling or training. The dbt adapter is generally available on Confluent Cloud.
Additionally, Confluent now supports TimesFM models for robust anomaly detection, as well as models from Anthropic and Fireworks AI. These models can be used directly in Flink stream processing workflows, enabling developers to build sophisticated real-time AI applications that detect anomalies, generate predictions, or classify data on the fly. This model integration streamlines the process of incorporating machine learning into streaming data pipelines, further accelerating time to value.
The combination of these updates—agent-powered workflows, automated PII redaction, private connectivity, dbt integration, and expanded model support—positions Confluent as a comprehensive platform for real-time AI. By embedding security and governance directly into the data streaming layer, the company aims to reduce the friction that often prevents AI projects from moving from concept to production. Sean Falconer, head of AI at Confluent, emphasized that most AI projects fail because the data layer breaks down, despite having the models and mandate. By fixing the data streaming foundation, Confluent hopes to enable teams to ship secure, production-ready AI applications faster and more reliably.
The streaming layer becomes the nervous system of AI, continuously processing both historical and real-time data to provide trusted context for intelligent applications. As enterprises increasingly adopt agentic AI architectures, where autonomous agents make decisions based on live data, having a robust, governed, and easily managed streaming infrastructure becomes critical. Confluent’s latest offerings aim to deliver exactly that, removing barriers to scale while maintaining enterprise-grade compliance and security. With these capabilities, developers can focus on building innovative AI features rather than wrestling with data pipeline complexities.
Source: Computerweekly News