Digital Plumber

This episode directly addresses the intersection of observability, automation, and networking in AI data centers—a growing operational challenge as organizations scale GPU clusters from thousands to tens of thousands of devices. With AI data centers linking massive GPU pools, observability becomes a fundamental blocker to reliable automation. The discussion explores how practitioners are solving this problem in production, including tooling choices, architectural patterns, and the relationship between telemetry collection and automation reliability. For NetDevOps and SRE practitioners, this captures real production insights into why traditional network automation tools struggle at AI infrastructure scale and what observability-first automation looks like in practice.

Modern hybrid networks require unified observability addressing visibility gaps across multi-cloud, SD-WAN and internet paths through continuous active testing and device telemetry in a single operational view. The 2026 industry trend is unified observability with user experience as the primary metric and infrastructure providing diagnostic context. Modern platforms must correlate network and application worlds instantly, providing insights like 'The Zoom call quality dropped because the QoS tag was stripped at the edge router.' For network operations practitioners, outage detection now requires understanding dependencies between user-facing services and underlying network path behavior—a capability demanding integrated synthetic testing, flow analysis, and service telemetry in a single correlated data stream rather than separate tools. This shift eliminates the traditional split between performance monitoring and digital experience monitoring, unifying them into a single correlation layer where every alert carries full network topology and service dependency context.

Broadcom will showcase Ethernet switches, NICs, PCIe components and integrated optics at the October 12-15 OCP Global Summit in San Jose, featuring Tomahawk 6, Tomahawk Ultra, and Jericho 4 Ethernet switches, Thor Ultra 800G and Thor 2 400G AI Ethernet NICs, and the third-generation TH6-Davisson Co-Packaged Optics (CPO) portfolio. The TH6-Davisson delivers approximately 70% reduction in optical interconnect power consumption relative to traditional pluggables—more than 3.5 times lower, resulting in significant TCO and sustainability wins for hyperscale and AI data centers. Improved link stability eliminates manufacturing and test variability typical of pluggable transceivers, reducing flap events which is pivotal for long-duration AI training runs. CPO adoption remains supply-constrained but is no longer research-phase: total shipment volume is expected to range from 10-15k units in 2026, with CPO switches now in production from NVIDIA, Broadcom, and others. For network architects, this marks the transition point where CPO payoff calculations move from theoretical to deployment-ready, especially for high-density scale-out clusters above 10k GPUs where power and link stability directly impact training economics.

Arista is introducing Etherlink SU-144, an Ethernet-based scale-up architecture that expands unified computing domains up to 144 accelerators (XPUs) in a single-hop network and scales to 1,024 XPUs through cross-rack topologies. This massive densification translates to nearly a 50% reduction in the overall data center footprint, resulting in significant time and financial savings. This effort is engineered in partnership with AMD, Arm, Broadcom, d-Matrix, Meta, Microsoft, and Qualcomm through the Ethernet for Scale-up Networks (ESUN) initiative. Scale-up networking—where GPUs share memory and ultra-low latency is required—was historically InfiniBand territory. Arista's multi-vendor ESUN backing signals that Ethernet with RoCEv2 has crossed the threshold from scale-out novelty to scale-up standard, which materially impacts both fabric vendor selection and network engineer skill requirements going forward.

Broadcom announced it will present innovations in scale-up, scale-out, and scale-across AI networking at the 2026 OCP Global Summit running October 12-15 in San Jose, featuring Ethernet switches, NICs, PCIe components and integrated optics to power Open Rack Version 3 (ORV3) solutions for scaling AI infrastructure. Broadcom's contributions to OCP include ORV3 rack-based designs and advances in ORW (wide) racks and new wide co-packaged optics that reflect collaboration solving the bandwidth and efficiency challenges facing AI data centers at scale. The emphasis on Open Rack integration signals that CPO is no longer a point product but a foundational building block for standardized AI cluster design. This is particularly relevant for practitioners building or upgrading multi-vendor fabrics, as ORV3 CPO specs establish interoperability boundaries and eliminate custom integration work that previously locked customers into single vendors.

More than 86,644 Fortinet FortiGate firewalls across 194 countries have been compromised in a credential-harvesting campaign so aggressive that some victim organizations found themselves locked entirely out of their own systems. Hackers gain access to exposed endpoints by using previously leaked credentials or logins obtained from infostealer logs, credential stuffing, and password spraying attacks, then extract additional authentication data from compromised devices and use a distributed GPU cluster running Hashcat and Hashtopolis to crack offline the stolen password hashes. Attackers create admin accounts that were never on the device and sometimes delete or change the original ones, so legitimate owners lose access, and the intrusion chain has been seen as an entry point for ransomware affiliates linked to INC/Lynx and Payload. Federal authorities are urging organizations to terminate active administrative and VPN sessions, reset credentials, enforce phishing-resistant multi-factor authentication, and restrict management access using trusted host lists. This represents a credential-first attack strategy that bypasses patch management entirely.

Cisco and SonicWall spent the first two weeks of October 2026 issuing emergency patches for authentication-bypass flaws scoring near-max CVSS—Cisco's Catalyst SD-WAN Manager and SonicWall's CVE-2026-102255 in SMA 1000. Both enable unauthenticated remote code execution or full system control from the internet perimeter. Network edge appliances show up often in exploited-vulnerability advisories because they are built to be reachable from the public internet, sit in front of everything else on the network, and patching them requires maintenance windows IT teams schedule weeks or months out; detecting post-exploitation activity—reverse shells, unexpected outbound connections from a VPN appliance—often matters as much as preventing the initial breach. SonicWall's advisory carefully scopes damage to SMA 1000 Workplace portal only, not SMA 100 series or SSL-VPN on FireWalls—a distinction that matters for IT teams trying to triage exposure quickly across a mixed fleet.

Modern MSSP operations require multi-tenant isolation with per-tenant reporting and white-label options, noise reduction through behavioral baselines and telemetry correlation into incidents rather than events, response automation with 100+ out-of-the-box playbooks and automated containment in under 90 seconds, and GenAI-assisted playbook generation from plain language in about 30 seconds. Cisco's September 2026 Splunk announcements illustrate this direction, expanding AI-powered security capabilities across network, cloud, application, and identity data with specialized agents supporting threat hunting, investigation, detection engineering, response, and governance. A Palo Alto Networks case study showed analysts now spend 70% of their time on proactive threat hunting rather than reactive triage after implementing AI automation, reshaping SOC staffing models and shifting network threat operations toward higher-value investigation work rather than alert fatigue management.

At WebexOne 2026 (Oct 7), Cisco unveiled Claude Managed Agents as persistent participants in Webex spaces, not just chatbots answering questions. The system allows multi-step workflows: an analytics agent pulls usage numbers when @mentioned, then a presentation agent drops those figures into PowerPoint live, all within the Webex conversation context. The key operational insight is governance: Cisco implemented layered governance so IT can control which agents join which spaces, what tools they can invoke, and how their decisions are audited. The architecture runs on Anthropic's Claude Managed Agents API, with agents operating on separate servers from sandboxes and credentials held in vaults. Cisco also integrated MCP (Model Context Protocol) with OpenAI's dot agents, signaling vendor-neutral tooling. For infrastructure teams, this demonstrates how MCP becomes the connective tissue between agents and operational systems—whether collaboration, observability, or network management. The announcement showed that long-running agents can work overnight and coordinate handoffs between specialized agents without human involvement. Most detailed features ship early 2027, but the architecture is production-ready now.

The Oct 9 release adds multiagent_20261001 workflows allowing agents to plan and execute many sub-agents in phases, combining results at end—useful for tasks like document review or incident orchestration. Workflows run in background and tracked via workflow_run events. Operationally, the governance model is designed for infrastructure: credentials never exposed to agent sandbox, stored in separate vault with scoped access; administrative actions logged for audit; two-person approval on Anthropic's side for sensitive operations. Cloud environments with limited networking now scope allowed_hosts to web_search and web_fetch tools, preventing exfiltration. For ops teams building agents on Claude, this release signal is that stateful multi-step workflows with proper isolation and auditability are now table stakes. The Oct 7 release also restricted networking in managed agents by default (allowed_hosts allowlisting), a security stance important for operations environments. Together, these changes position Claude Managed Agents as production-grade for infrastructure use cases where audit and compliance matter.

The post walks through six rules for avoiding common agent automation failure modes. Central pattern: agents don't see raw credentials. Instead, MCP servers act as proxies—agents call MCP tools, proxy finds vault credential matching the server's URL and swaps it in as the request leaves sandbox. For shell-based access (e.g., Slack API via curl), agents only see opaque placeholders like $SLACK_BOT_TOKEN; platform swaps real token for allowed hosts. This isolation pattern solves the operational dilemma: agents need real access to infrastructure, but exposing credentials in context is a breach risk. The guide emphasizes credential scoping (one per source), monitoring agent access, and audit trails. For infrastructure ops, this translates directly: if you're building agents to interact with network APIs, IPAM systems, or monitoring tools, the MCP + vault pattern is the production-safe approach. The post notes that many agent deployments fail because they lose access to a source without anyone noticing—proper credential and source management prevents this. Real-world example: an agent that provisions network connectivity needs access to IPAM, device APIs, and ticketing; MCP servers for each let the agent work without seeing passwords.

The Dialog framework addresses the critical gap: enterprises want agent capabilities but need IT control over what agents can do, who can invoke them, and audit trails. Cisco's approach: agents (Claude, OpenAI dots, or customer-built) operate inside Webex, but all access to external systems flows through MCP integration points that Dialog controls. IT administrators define policies: which agents can call which tools, which users can invoke agents, sensitive operations require human approval. The framework supports layered autonomy—fully autonomous for low-risk actions, human-in-loop for higher-risk changes. For infrastructure ops, this is the governance model that makes agents acceptable in production: agents can diagnose and remediate, but policy gates prevent accidental cascading changes or unauthorized access. Dialog enters beta early 2027, but the architecture shows that IT governance isn't bolted on after—it's native to how agents execute. This matters for NetOps teams considering agentic platforms: governance should be part of the agent framework, not a separate compliance layer.

Aggregate Performance Reporting (APRF) evolved from an individual submission (draft-brotman-aggregate-performance-reporting) into formal IETF MAILMAINT working-group document stream. The October 9 publication represents a critical gate: WG has accepted responsibility for the specification, enabling community review, wording refinement, and resolution of open questions before Standards Track advancement. Version 00 remains early working text; the draft expires April 12, 2027, and may be revised, replaced, or abandoned. The intended status is Standards Track. The specification introduces JSON structures for streaming email authentication and deliverability metrics (DKIM, SPF, DMARC aggregate reports, reputation signals) over DNS. This standardizes what vendors currently implement ad-hoc, reducing fragmentation and enabling mail operators to ingest telemetry from multiple sources into unified monitoring without custom parsing. Practical implication: teams should not yet deploy production DNS records or build fixed parsers around current syntax, as the format will evolve during WG review.

Claude Haiku 5.5 targets high-volume, latency-sensitive production tasks: classification, extraction, routing, live customer support, and coding subagents. It is the first Haiku-class model with adjustable reasoning effort, allowing operators to trade off response quality, latency, and token consumption in a single call. The effort parameter defaults to medium; low effort is tuned for simple, routine classification; higher settings (high/xhigh/max) justify ambiguous extraction and tool-planning steps. Context window expands 5x from Haiku 4.5 (200K→1M tokens); max output also doubles (64K→128K). However, the updated tokenizer inflates token counts by approximately 30% for the same input text, offsetting the per-token price reduction ($0.10 input vs. $1.00 for Haiku 4.5). Pricing bifurcates at 100K-token prompt length: short prompts stay cheap; prompts exceeding 100K tokens jump to $0.50 input/$2.50 output. Available on Claude API, Amazon Bedrock, Google Cloud, Microsoft Foundry, and Claude Platform on AWS. Knowledge cutoff: June 2026.