Roland Huß

Your API Is Not Your MCP Tool

Thu, 06 Aug 2026 00:00:00 +0000

The most natural thing to do when building an MCP server is to mirror your existing API. You have a REST endpoint for creating users, so you make a create_user tool. You have one for listing orders, so you make a list_orders tool. Fifteen endpoints become fifteen tools, each one a thin wrapper that translates JSON-RPC to HTTP and passes the response back.

It works. The agent can call the tools, the tools return data, and everything looks fine in your test harness. Then you connect it to a real conversation with twenty other MCP servers loaded, and the agent starts picking the wrong tools, hallucinating parameter values, and burning half its context window on tool definitions it never uses.

The problem isn’t MCP. The problem is that APIs were designed for a fundamentally different consumer. A developer reads documentation, bookmarks the three endpoints they need, and writes code that calls them in the right order with the right parameters. An LLM sees every tool definition at once, every parameter description, every schema. It has to pick the right tool from that pile on every turn, and each definition costs tokens whether the tool gets called or not.

Anthropic’s engineering team put it directly: “A common error we’ve observed is tools that merely wrap existing software functionality or API endpoints.” AWS frames the same insight as two failure modes: “bloat” (tool definitions consuming context on every call whether used or not) and “confusion” (models picking wrong tools as context fills). Both companies arrived at the same conclusion from different directions. MCP tool design is not an API design problem. It’s a context engineering problem.

How many tools is too many
#

There’s no single number, but the data points tell a consistent story.

Tool-selection accuracy drops below 90% between 10 and 15 tools for Claude Haiku 4.5, and between 20 and 30 tools for Sonnet 4. Bigger models tolerate more tools, but none of them are immune. GitHub’s MCP server grew to over 100 tools after a month of community contributions, degraded agent capabilities, and got cut back to about 40.

The pattern is the same in every case: teams start by exposing everything, then discover that more tools means worse results. The reduction isn’t about removing functionality. It’s about consolidating related operations into higher-level tools that match what users actually ask for.

Consider a project management API with separate endpoints for creating issues, assigning them, setting priority, adding labels, and linking to epics. That’s five tool definitions in the context window and a workflow that requires the agent to sequence five calls correctly. A single manage_issue tool that accepts an action parameter and handles the orchestration internally gives the model one decision point instead of five. The agent’s job becomes “pick the right tool” rather than “pick the right five tools in the right order.”

AWS calls this the progression from V1 (raw API passthrough) through V6 (agent-as-tool), and their conclusion is worth quoting: no single version wins across all dimensions. Raw passthrough is easiest to build but hardest for agents to use. Fully orchestrated tools are great for agents but expensive to maintain. The right level depends on how complex the workflow is and how often the agent needs fine-grained control over individual steps.

The per-server tool count is only half the equation. In any real setup, you’re connecting multiple MCP servers at once, and their tool definitions stack. Every server you add pushes more definitions into the context window. Being selective about which servers you connect matters at least as much as being selective about which tools each server exposes.

Claude Code’s configuration hierarchy helps with this on the client side. You define MCP servers at user, project, or directory scope, with project-level configs overriding what’s active for a specific codebase. A Kubernetes project gets your cluster tools, a documentation project gets your writing tools, and nothing bleeds across. cc-setup wraps this in a terminal UI for managing servers, permissions, and plugins per project from a central registry.

Parameters that help instead of confuse
#

The number of parameters matters almost as much as the number of tools. AWS recommends roughly eight or fewer per tool, with an important caveat: if bundling related operations requires more than eight parameters, prioritize the bundling over the parameter count. A single tool with twelve well-described parameters usually outperforms three tools with four parameters each, because the model makes one tool-selection decision instead of three.

But parameter count is the easy part. The harder lesson is that parameter naming and typing should optimize for LLM comprehension, not for your internal data model.

Rename parameters to match how users think. If your API takes org_id, but users say “my company,” name the parameter organization_name and resolve the ID internally. The model generates parameter values from the conversation context, and the conversation uses human language, not database identifiers.

Use semantic identifiers, not UUIDs. This one surprised me. BoundaryML ran benchmarks comparing UUID-based tool parameters against integer-remapped alternatives. The result: 48.5 average errors per session with UUIDs versus 5.5 with integer remapping. Arbitrary alphanumeric strings give models nothing to reason about, and they hallucinate plausible-looking IDs that don’t exist. If your tool needs to reference an entity, resolve the UUID to a human-readable name or a simple index before the model ever sees it.

Every MCP tool defines its parameters using JSON Schema in the inputSchema field. The client sends that schema to the LLM as part of the tool definition on every invocation, so the model sees the full parameter structure, types, descriptions, and constraints before deciding how to call the tool. With the 2026-07-28 spec, inputSchema supports full JSON Schema 2020-12, including oneOf, anyOf, conditionals, and $ref, so you can express complex parameter relationships directly in the schema.

Use enums over booleans. JSON Schema’s enum keyword lets you list the exact valid values inline, and the model uses those values directly when generating its tool call. A parameter like include_archived: true forces the model to infer what true means in this specific context. filter: "active" or filter: "all" gives the model unambiguous options to pick from. AWS Prescriptive Guidance recommends enums for any finite set of values: “use enums for parameter values to make it easy for the LLM to select the correct options.” Booleans optimize for code execution, enums optimize for LLM reasoning.

Add a response format parameter. Anthropic found that adding a response_format parameter with options like “concise” and “detailed” reduced token usage by roughly two-thirds for their Slack integration (72 tokens for concise versus 206 for detailed). The model doesn’t always need the full response, and giving it a way to say so saves context for the rest of the conversation.

Tool naming has measurable effects. In the same post, Anthropic tested prefix-based (slack_list_channels) versus suffix-based (list_channels_slack) namespacing and found measurable differences on their tool-use evaluations. The effects varied by model, which means the right naming convention depends on which LLM your users run. If you’re building a public MCP server, test both patterns against the models you care about. Assumptions about naming are surprisingly unreliable.

Context hygiene for tool definitions
#

Each tool definition costs somewhere between 100 and 1,000 tokens, depending on schema complexity. AWS Prescriptive Guidance puts it at 5,000 to 10,000 tokens for just 20 tools. With 50 tools loaded, that’s 20,000 to 25,000 tokens consumed before the first message in the conversation. On a 32K context window, that’s most of the available space. Even on larger windows, those tokens compete with conversation history, file contents, and everything else the agent needs to reason about.

The MCP specification now includes primitives for managing this.

Deterministic tool ordering is the simplest win. The 2026-07-28 spec recommends that servers return tools from tools/list in a deterministic order. This matters because LLM providers cache the prompt prefix. If tool definitions appear in the same order across requests, the cached prefix covers them and the provider skips re-processing. Randomized ordering breaks the cache on every call.

CacheableResult lets servers attach ttlMs and cacheScope fields to list and read results, so clients know how long they can cache a response without re-polling. If your tool list doesn’t change during a session, setting a long TTL means the client asks once and remembers.

Progressive tool loading goes further. Instead of registering full tool definitions at startup, register only metadata (name and one-line summary) and load the full schema on demand when the agent actually wants to call a tool. Claude Code’s ToolSearch mechanism works this way, and the token savings are substantial. The tradeoff is that the agent has less information for initial tool-selection, so the summaries need to be precise enough for the model to pick correctly from the short descriptions alone.

subscriptions/listen is the new unified mechanism for change notifications. If your server adds or removes tools at runtime, it can push change notifications through a single long-lived stream rather than requiring clients to poll tools/list on every turn.

These are infrastructure-level optimizations, and they matter most for MCP servers with large tool sets or deployments where multiple servers connect to the same client. But even a small server benefits from deterministic ordering and accurate tool descriptions, because every wasted token is a token the agent could have spent on reasoning.¹

What this means in practice
#

The central tension in MCP tool design is that everything you expose costs context, and context is the scarce resource that determines whether the agent works well or poorly. Every tool, every parameter, every line in a description competes for the same budget.

The teams that got this right didn’t remove functionality. They redesigned the interface between the API and the agent: consolidating fine-grained operations into higher-level tools that match user intent, renaming parameters to match how people talk about the domain, replacing UUIDs with meaningful identifiers and booleans with enums.

None of this requires changes to the underlying API. The API serves its original consumers just fine. The MCP server is a translation layer, and the quality of that translation determines whether the agent is helpful or frustrating. Treating the API surface as the tool surface is the mistake. Designing for what the model sees, not for what the code does, is the fix.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

The 2026-07-28 MCP specification also brings a stateless protocol core, Multi Round-Trip Requests for tools that need user input mid-execution, and deprecations of Roots, Sampling, and Logging. ↩︎

One Stray Leads the Whole Flock Astray

Mon, 20 Jul 2026 00:00:00 +0000

The previous post in The Flock series covered what happens when agents forget mid-run and confidently fill the gaps. This one covers a different failure mode: what happens when one agent’s leftover context converts the next agent into something it was never supposed to be.

The flock grazes in separate paddocks for a reason. Put the bell sheep and the border collie in the same field and by afternoon the collie has stopped herding and started grazing. It didn’t forget its job. It absorbed the bell sheep’s confidence and decided that grazing was the job all along.

Not confusion, conversion
#

Context contamination between agents is a different beast than context fatigue within a single session. The 101 post covered the gradual fog: responses getting less precise as the conversation grows, state drifting after compaction. That’s a single agent slowly losing its footing.

In multi-agent setups, the failure is sharper. When multiple agents run in the same foreground context, the second agent doesn’t just inherit stale data from the first. It inherits the first agent’s identity. An assessment agent that follows a creation agent in the same context will absorb the creator’s assumptions and biases. In the worst cases it adopts the creator’s role entirely, deciding that its job is to generate ideas rather than evaluate them, not confused but converted.

Context bleeds across items, not just across roles
#

The second pattern is subtler. A feasibility agent reviewing multiple items in sequence can have context from one item bleeding into the assessment of the next. Scores that should be independent aren’t. The fix is the same: one agent per item, each in complete isolation.

This kind of bleed only surfaces at scale. Small test batches of five items look fine. At fifty, the drift becomes visible: items assessed later in the sequence show score patterns that correlate with properties of earlier items, not their own content.

Lou and Sun (2026) found that anchoring bias in LLMs is systematic and affects stronger models more consistently, not less. Four different debiasing strategies (chain-of-thought, reflection, ignoring anchor hints, explicit instructions) all failed to eliminate it. Prompt engineering can’t fix a bias baked into how the model processes sequences. Architecture can.

The author can’t review its own work
#

Code review culture already solved this problem for humans. You don’t review your own pull requests because you’re anchored to the decisions that shaped the code. You see the logic, not the gaps. As it turns out, the same principle applies to agents.

The CCR paper (Cross-Context Review) tested this directly. Fresh-session review achieved an F1 score of 28.6% versus 24.6% for same-session self-review. The critical control was reviewing twice in the same session: that produced more findings but worse precision (21.0% vs 25.8%), more noise rather than more signal. The production context itself anchors the model, and repeating review in the same context only makes things worse.

CCR also achieves what the authors call “natural anonymization.” The reviewing session can’t know the artifact originated from the same model. This mirrors findings from Choi et al. (ACL 2026) showing that sycophantic bias drops 96% when authorship is hidden. The model is more honest when it doesn’t know it’s reviewing its own work.

This is why adversarial review works as a pipeline pattern. An agent creates or revises a document in one context. A completely separate agent, running in a clean context, assesses the result. The reviewer doesn’t know that a revision happened, doesn’t have access to the creation reasoning, and evaluates the output on its merits alone.

The workbench metaphor
#

Think of context like a workbench in a shared workshop. If the previous worker left their tools, materials, and half-finished pieces scattered around, the next worker will unconsciously start incorporating those into whatever they’re building. Clearing the bench between workers isn’t overhead, it’s a prerequisite for getting independent results.

Multi-agent frameworks take different positions on when to clear the bench. Claude Code subagents enforce isolation by default: each subagent runs in a fresh conversation with its own system prompt and tool definitions, but not the parent’s conversation history. The OpenAI Agents SDK takes the opposite approach, sharing full conversation history by default during handoffs, though it provides an input_filter mechanism for opt-in isolation. These are different architectural philosophies, and the research consistently favors isolation.

From discipline to architecture
#

Context contamination can’t be solved with better prompts. Anchoring bias resists all tested prompt-based mitigations, sycophancy persists as long as authorship is visible, and score drift accumulates as long as items share a context.

The fix is structural: separate creation from assessment, isolate items from each other, clear the context between agents, and make the reviewing agent blind to the creation process. These aren’t optimizations you add after the pipeline works. They’re prerequisites for the pipeline producing reliable results.

The Flock so far: deterministic guardrails around agent creativity, state on disk because memory can’t be trusted, and now separate paddocks, because one stray’s confidence is enough to convert the whole flock.

AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

When Your AI Agent Treats Scripts Like Suggestions

Mon, 13 Jul 2026 00:00:00 +0000

The init command for my Claude Code plugin needed a few things to happen in order: run a setup script, ask the user some configuration questions, apply the results. Not complicated. The skill file spelled out each step with code blocks and paths, and the agent understood all of it. Then it went its own way.

Sometimes it called the CLI directly, skipping a wrapper script that checks prerequisites and preserves configuration state. Sometimes it finished step one and stopped, summarizing results without continuing. Sometimes it searched for a binary called speckit when the tool is called specify. The pattern was always the same: the agent extracted the intent (“initialize the project”) and chased it through whatever path looked reasonable, treating each instruction as a suggestion to optimize away.

This is how skills work. The model reads instructions, considers them, and makes its own call. If the goal is “initialize the project,” why bother with a wrapper script when the CLI is right there? The answer is that the wrapper checks prerequisites, preserves configuration that the CLI would overwrite, and returns structured status codes that downstream steps depend on. But the agent doesn’t know what it doesn’t know, and a skill file can’t convey “this matters more than you think.”

Path hallucination and the cost of exploration
#

Even after tightening the instructions, a subtler problem emerged. The agent would try to run the correct script but construct the wrong path.

Claude Code plugins live in a cache directory whose location depends on platform, user configuration, and plugin version. A typical path looks like ~/.claude/plugins/cache/<plugin>/<version>/scripts/init.sh. Not something you hardcode in a markdown file and expect to work everywhere.

So the skill file said “find the script relative to the plugin root.” The agent interpreted this by running find or ls to locate the script. Sometimes it found the right file, sometimes a similarly named file in a different directory, and sometimes it invented a plausible-looking path that didn’t exist at all.

Three rounds of fixes followed, each adding more “DO NOT” clauses, more explicit constraints, more guardrails against filesystem exploration. None of them held, because the underlying behavior didn’t change: the agent would read the constraints, weigh them against its own judgment, and go exploring anyway.

Beyond reliability, there was a cost problem that anyone working with AI-assisted development will recognize. Each exploration attempt burned tokens on find commands and filesystem traversals. The agent would reason about results, try a path, fail, reason again, try another. A step that should take a single Bash call ballooned into five or six, each carrying its own reasoning overhead. The user sits there watching the agent think, waiting, losing focus. When the agent spends thirty seconds figuring out something it should already know, the user’s mental context starts to evaporate. In AI-assisted development, the biggest threat to productivity isn’t complexity. It’s latency.

The compliance hierarchy
#

After three failed attempts at fixing the documentation, the pattern became clear. There’s a hierarchy of how seriously the agent treats different types of context.

System reminders and hook context arrive as ground truth. The agent rarely ignores them.
Direct prompt content (AGENTS.md, CLAUDE.md, command files) gets moderate compliance.
Referenced instructions (skill files, tool descriptions) get treated more like advice to interpret.

As it turns out, every one of my failed attempts had the same root cause: critical logic lived at level three. The agent would read the skill, understand the intent, and improvise instead of following the literal steps. Stronger wording didn’t help because the model treats skill content as guidance, not as commands.

This hierarchy has solid academic backing. Wallace et al. formalized it as “The Instruction Hierarchy” at ICLR 2025, showing that LLMs trained with explicit privilege levels (system > user > tool content) became significantly more reliable. Anthropic’s own principal hierarchy encodes a similar ordering: Anthropic > operator > user, with environment context (tool outputs, loaded files) given the least inherent trust. In Claude Code specifically, you can see it in action: CLAUDE.md and skill content arrives wrapped in <system-reminder> tags with the disclaimer “this context may or may not be relevant to your tasks,” literally downgrading it to advisory status.

The AGENTS.md spec is moving in the same direction. The v1.1 proposal formalizes a 5-level priority hierarchy (LLM system prompt > agent system prompt > user prompt > local AGENTS.md > ancestor AGENTS.md) and explicitly acknowledges that enforcement is “often achieved probabilistically through prompting.” That’s the core tension: context files can express intent, but they can’t guarantee compliance. Deterministic enforcement has to come from somewhere else.

That somewhere else is hooks.

Hooks and structured outputs
#

The examples here use Claude Code’s hook system (UserPromptSubmit, PreToolUse), but the pattern is not Claude Code specific. Cursor has rules and .cursor/tools, Windsurf has flows, Cline has custom instructions with tool approval, and most agent harnesses offer some way to inject context at the system level or gate tool calls before they execute. The specific API differs, the architectural principle is the same: move enforcement out of instructions and into the harness.

The final architecture gives the agent nothing to interpret.

Path injection. A UserPromptSubmit hook (a Python script) runs before the agent sees the prompt. It resolves its own filesystem location using Path(__file__).parent.parent.parent, constructs absolute paths to every script the agent might need, and injects them as structured XML in the system context:

<init-command>/Users/me/.claude/plugins/cache/.../scripts/init.sh</init-command>

How this lands in the model’s context matters. In Claude Code, hook output goes into a <system-reminder> block that becomes part of the system prompt, not the user message. The model sees it as platform-level context, the same tier as its own operating instructions. That’s why hook-injected content gets near-perfect compliance while identical text in a skill file gets treated as a suggestion. Other agent harnesses achieve the same effect differently: Cursor’s rules inject into the system prompt directly, and any framework built on the Anthropic or OpenAI API can prepend to the system message before the conversation starts.

Path resolution happens in deterministic Python code, not in probabilistic language model reasoning. The hook fires only on the plugin’s slash commands and captures session state at hook time. No ambiguity left for the agent to resolve.

Stage enforcement. A PreToolUse hook fires before every tool call the agent makes. It checks a state file on disk that tracks where the workflow currently is, and if the agent tries to skip ahead, the hook blocks the call and tells it exactly which steps it needs to complete first. The skill file still contains the workflow instructions (the agent needs them for the judgment calls within each step), but the hook makes compliance non-optional. Instead of hoping the agent follows a multi-step procedure, the hook gates every action against the expected sequence.

These two hooks work together: one provides information (resolved paths, session state), the other enforces behavior (stage ordering, shortcut prevention). Together, they move both context and compliance from the advisory tier to system-level enforcement.

Structured outputs. The init script returns machine-parseable status codes: READY, NEED_INSTALL, ERROR, RESTART_REQUIRED. The skill uses conditional logic: “If output contains NEED_INSTALL, show output, STOP.” This removes the last opportunity for improvisation. No ambiguous output to reason about, just a keyword to match and a branch to follow.

Together, these mechanisms reduce the agent’s job to: read a path from context, execute it, parse a keyword, follow the branch. Each step is concrete and verifiable. Nothing left to infer, guess, or optimize away.

Architecture over instructions
#

An AI coding agent is a confident but context-limited collaborator. It will try to help and it will apply its judgment. And that judgment will sometimes lead it to skip the step that matters most, because that step looks redundant from the outside.

Better instructions don’t fix this. More guardrails, more constraints, more anti-exploration language: none of it sticks, because instructions are interpreted, and interpretation introduces variance.

Architecture fixes it. Remove opportunities for interpretation by resolving ambiguity before the agent encounters it. Hooks resolve paths and enforce stage ordering, structured outputs constrain branching. Each mechanism removes a category of improvisation. What remains is an agent executing a well-defined protocol, which is what a plugin init command should be.

Every exploratory call the agent makes is also tokens burned and seconds lost. The difference between “one Bash call with a pre-resolved path” and “six exploratory calls to figure out where the script lives” is the difference between a tool that feels responsive and one that feels like it’s fighting you.

Trust instructions for what they’re good at (communicating intent to humans reading the code) and use structural mechanisms for what needs to be reliable.

AIA Ph Se Hin R Claude Opus 4.6 v1.0

The Sheep That Forgot the Way Home

Thu, 09 Jul 2026 00:00:00 +0000

The previous post in The Flock series covered what happens when agents get creative where they shouldn’t. This one covers a more fundamental problem: what happens when agents forget what they were doing, and then confidently make something up.

The bell sheep always knows the way home. Except on days when the fog rolls in and the familiar landmarks disappear. On those days, she confidently leads the flock to entirely the wrong barn, and everyone follows because she’s never been wrong before.

The fog rolls in
#

The 101 post recommended /clear when context gets stale: start a fresh session, let the agent re-read the project from scratch. Good advice when you’re sitting at your desk watching the conversation drift. But /clear is a human intervention. It assumes someone is paying attention.

When agents run in CI, nobody is watching. The context window fills up on its own schedule, and the runtime compresses the conversation to make room for more work. This is called compaction, and it happens automatically after long-running sessions. Claude Code triggers auto-compaction when context usage reaches roughly 95% of the window. You don’t choose when it fires. You don’t control what it keeps. And the part that makes it dangerous: compaction doesn’t just drop information. It invents replacements.

This is well-documented. An LLM reads the full conversation and produces a structured summary that replaces the history. The summarization is lossy: exact file paths, line numbers, and error messages get paraphrased, not preserved verbatim. Research from 2025 found that nearly 65% of enterprise AI failures were attributed to context drift or memory loss during multi-step reasoning, not raw context exhaustion. Compaction strategies have improved since then (Anthropic’s server-side compaction API, Factory’s anchored iterative summarization), but the fundamental limitation remains: any lossy compression of a conversation will lose something, and the model won’t tell you what. The distinction matters: context drift is different from context exhaustion. The model’s reasoning diverges from the original task because compressed summaries introduce subtle reframing that shifts how the agent interprets its own instructions. Same agent, same prompt, different behavior, and nothing in the logs to tell you something changed.

What gets invented
#

The unsettling part isn’t what the agent forgets. It’s how confidently it fills the gaps. This is where it stops being a technical nuisance and starts feeling wrong.

In long coding sessions, I’ve watched agents reference file paths that were refactored an hour earlier. Not as mistakes they catch and correct, but as confident assertions they build on. The agent writes an import statement for a module that was renamed three tool calls ago, and then writes tests against the old API, and then reports success because the tests it wrote match the code it wrote, both of which reference something that no longer exists. Every step is internally consistent. The chain of reasoning is sound. The foundation is wrong.

In batch processing, the pattern is worse because there’s no human to notice the drift. A CI agent processing 50 items might lose track of which item it’s currently working on after compaction compresses the early items into a summary. The agent doesn’t stop and say “I’m not sure which item I was processing.” It picks one that fits the pattern of what it’s been doing and continues. If the summary says “processed items 1-30, currently on batch 2,” the agent might confidently start on item 31 when it was actually in the middle of item 28. Three items get skipped, nothing crashes, and the final report shows 50 items processed.

This is one reason to keep the batch loop itself in a deterministic script rather than letting the agent manage iteration. A shell script or Python wrapper that iterates over items and invokes the agent once per item avoids the compaction problem entirely: each invocation starts with a fresh context, the script tracks progress, and the agent never needs to remember where it left off. The first post in this series called this “deterministic guardrails” for agent creativity. For batch workloads, it’s not just a guardrail, it’s the architecture.

Run IDs are another casualty. A pipeline generates a unique ID at startup (say, run-20260514-1430) and uses it to namespace all output files. After compaction, the agent needs the run ID to write the final report. Instead of admitting it doesn’t remember, it generates a plausible-looking new one based on the current time: run-20260514-1530. The report lands in the wrong directory while all the intermediate files sit in the original one. The dashboard shows an empty run and a report with no supporting data. Nothing crashes. The failure mode is a clean-looking report that’s quietly orphaned from everything it summarizes.

The error list that emptied itself
#

Some compaction bugs don’t loop or crash. They just quietly drop information that matters.

An agent processing a batch of documents accumulates a list of items that fail validation: “items 7, 19, and 23 need manual review.” The list lives in the conversation as a running tally, updated after each item. After compaction, the summary says “some items needed manual review” but the specific IDs are gone. The agent continues processing, finishes the batch, and writes the final report. The “items requiring attention” section is empty, not because everything passed, but because the agent lost the list and had nothing to put there. Three items that were explicitly flagged for human review silently slip through without it.

The fix is the same for every piece of mutable state: write it to a file the moment it changes, and re-read the file whenever you need it. An errors.json that gets appended to after each validation failure survives compaction because the disk doesn’t forget. The agent reads the file when writing the report and finds all three flagged items, regardless of what the compressed conversation summary remembers.

This pattern applies to anything the agent accumulates during a run: error lists, progress markers, feature flags, intermediate scores. If the value changes during the run and the agent needs it later, it has to be on disk.

The post-compaction hook
#

Claude Code provides a post-compaction hook that fires after the runtime compresses the conversation. This is your recovery point. The hook can re-inject critical state that the agent needs to continue: which item it’s working on, what step it’s in, what the run parameters are.

The hook doesn’t prevent information loss. It gives you a structured way to restore the minimum state needed for the agent to continue correctly. Think of it as the equivalent of saving your game before the fog rolls in: when visibility returns, you load from the save point instead of guessing where you were.

A typical post-compaction hook reads a state file from disk and prints it as a system message:

#!/bin/bash
# post-compaction hook
if [ -f ".agent-state/current-run.json" ]; then
 echo "State restored after compaction:"
 cat .agent-state/current-run.json
fi

The agent receives this as the first message after compaction and anchors to it. Without the hook, it anchors to whatever the compressed summary says, which might be close enough to feel right but different enough to produce wrong results.

One user documented 59 compactions in 26 days, averaging 2.3 per day. That’s not an edge case. Any long-running session hits compaction regularly, and each one is a potential knowledge-loss event. People are trying all kinds of workarounds: database-backed persistence layers, checkpoint files, session replay systems. The approaches vary, but the underlying pattern is always the same: write state to disk before compaction fires, re-inject it after.

AGENTS.md and the compaction problem
#

AGENTS.md files are the first thing coding agents read when they start a session. Build commands, code conventions, security boundaries, project-specific rules. These instructions anchor the agent’s behavior for everything that follows.

After compaction, those anchors can drift. Claude Code’s compaction engine does restore CLAUDE.md content as part of its post-compaction rebuild, but the restoration is a re-read, not a guarantee that the agent will weigh those instructions the same way it did at the start of the session. The compressed summary of the conversation now sits between the agent and its original instructions, and the summary’s framing can subtly shift how the agent interprets the rules. A convention like “use DCO sign-off for all commits” might survive the text but lose its emphasis after the summary compresses 40 minutes of commit-related conversation into a few sentences.

A post-compaction hook that re-reads the AGENTS.md file and prints its content as a system message is the simplest defense. The agent gets the original instructions back, verbatim, at every compaction boundary. Not a paraphrased version from the summary, the actual file. If your project conventions, security boundaries, or coding standards live in AGENTS.md, make sure the post-compaction hook includes them.

For orchestrated pipelines where sub-agents receive inline instructions (“assess this document using the following criteria…”), the problem is worse. Those instructions live only in the conversation context. After compaction, the summarized instructions might be close enough to the original that the agent doesn’t notice the difference, but different enough that it interprets them differently. “Re-run the assessment” might mean “use the same approach as before” to a human, but to an agent that just had its memory compressed, “before” is whatever the summary says it was.

The fix is the same as for data: put instructions on disk. Instead of passing instructions inline, pass a file path to the AGENTS.md or a task-specific instruction file. The sub-agent reads its instructions from disk at every step boundary, getting the original text every time regardless of what compaction did to the context. The orchestrator’s job is to write the instruction file and tell the agent where to find it, not to carry the instructions in the conversation.

From `/clear` to state architecture
#

The 101 post’s advice (“start fresh when context gets stale”) is human-scale thinking. It works when a person watches the conversation and notices the drift. At pipeline scale, you need something different: an architecture where the agent expects to lose its memory and recovers cleanly every time.

The practical checklist:

Every mutable value goes to disk immediately. Batch progress, counters, timestamps, flags, intermediate results. If you wouldn’t trust a colleague to remember a number after being interrupted, don’t trust the context window.
Use write-once for initialization values. Counters, start timestamps, and run parameters should use a “set if absent” pattern so re-entry after compaction doesn’t reset state that’s already been established.
Read state at every step boundary. Don’t carry values forward in the conversation. Re-read from disk at the start of every major operation so the agent works from the same source of truth regardless of what compaction did.
Use the post-compaction hook. Re-inject the minimum context the agent needs to continue: current item, current step, run parameters. Anchor the agent to disk state, not to its compressed memory.
Keep orchestrator instructions in files, not inline. Every sub-agent reads its instructions from disk. The orchestrator substitutes variables into a template and points the agent at the file.

The state file becomes the single source of truth, not the context window. As it turns out, the context window is a workspace, not a storage system.

Jessica Forrester and Jason Greene have built dozens of multi-agent CI pipelines at Red Hat. Their golden rule: if it’s not on disk, it doesn’t exist.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

What Goes in AGENTS.md (and What Doesn't)

Fri, 26 Jun 2026 00:00:00 +0000

Every coding agent has its own context file. CLAUDE.md for Claude Code, .cursorrules for Cursor, .github/copilot-instructions.md for Copilot, GEMINI.md for Gemini CLI. If you work with more than one agent, you end up maintaining multiple files with 90% identical content. AGENTS.md is the attempt to end that.

Why AGENTS.md
#

The problem AGENTS.md solves isn’t technical, it’s organizational. Build commands, test instructions, code style rules, project architecture notes are the same regardless of which agent reads them. Copying them into four different files and keeping them in sync is maintenance that adds no value.

AGENTS.md is plain markdown with no required fields, no YAML frontmatter, and no special syntax. Write headings and bullet points, and the agent reads them and adjusts its behavior. It already works with 30+ agents and sits under the Agentic AI Foundation alongside MCP and Goose. Like README.md and CONTRIBUTING.md, agents discover it automatically by looking for the file in the directory tree.

That’s the format, and the format is right. Plain markdown as the lowest common denominator is exactly what a universal standard needs to be. The question is what goes inside, and three academic papers published in 2026 have started to answer it.

What the research says
#

The ETH Zurich evaluation tested four coding agents (Claude Code, Codex, Qwen Code) across 138 real-world tasks. The findings challenged the assumption that more context is always better:

LLM-generated context files hurt performance. Success rates dropped by 0.5-2%, and inference costs went up by 20-23%. In five out of eight tested configurations, having an auto-generated file was worse than having no file at all.
Human-written files help modestly. Success improved by about 4% across all agents.
Codebase overviews are useless. Every LLM-generated file included one. They didn’t help agents find relevant files faster. In many cases agents took more steps with overviews present.
The redundancy proof: When the researchers removed all documentation from the repos, LLM-generated context files suddenly helped (+2.7%). The files were duplicating information the agents already extracted from README files and docs on their own.

A separate study showed the flip side: curated, minimal AGENTS.md files with only three content categories (coding conventions, architecture, project description) reduced median wall-clock time by 28% and output tokens by 16%. Less content, better results.

The MSR ‘26 survey of 10,000 repositories paints a broader picture: only 5% had adopted any context file format, the mean AGENTS.md file was 142 lines long, and half of all AGENTS.md files had zero commits after creation. Most are still write-once-and-forget.

The practical takeaway from Augment Code’s guide captures the consensus: write only what agents can’t discover on their own.

What goes in, what doesn’t
#

Based on the research and my experience maintaining context files across 38 projects, here’s a quick reference:

Put in AGENTS.md (agents can’t discover this on their own):

Content	Example
Non-obvious build commands	`uv pip install -e ".[dev]"` not just `pip install`
Tool-specific choices	“Use `podman` not `docker`”, “Use `rg` not `grep`”
Testing strategy	“Run `make e2e` for integration tests, requires Kind cluster”
Counterintuitive conventions	“Commit messages use DCO sign-off: `git commit -s`”
Hard invariants	“The UID in the Dockerfile must match `runAsUser` in Go”
Security boundaries	“Never commit `.env` or `.mcp.json` files”
PR/review expectations	“All PRs require at least one approval from CODEOWNERS”

Leave out of AGENTS.md (agents figure this out already):

Content	Why it’s noise
Codebase overviews	Empirically useless per ETH study, agents read the directory tree
Standard framework conventions	React, Express, Rails conventions are in the model’s training data
README content	Agents read README.md on their own
Dependency lists	`package.json`, `Cargo.toml`, `requirements.txt` exist for this
File-by-file descriptions	Agents explore the codebase as needed

Keep it short. Aim for under 150 lines. The efficiency study’s best results came from files with just three sections, and every instruction beyond the essentials burns tokens and can actively degrade performance.

Claude Code and AGENTS.md
#

One thing to know: Claude Code does not read AGENTS.md natively, only CLAUDE.md. To have Claude Code pick up your AGENTS.md content, reference it from your CLAUDE.md using the import syntax:

@AGENTS.md

This loads the AGENTS.md content into Claude Code’s context alongside whatever Claude-specific instructions you have in CLAUDE.md. The two files coexist in the same repo, but they serve different purposes.

AGENTS.md holds everything that’s universal across agents: build commands, test instructions, conventions, architecture notes, security boundaries. Any agent that reads the file benefits from this content.

CLAUDE.md holds what only Claude Code understands: skills tables and routing rules, context loading directives (@ARCHITECTURE.md), subagent delegation patterns, context budget management, hook-based context injection, and plugin paths using ${CLAUDE_PLUGIN_ROOT}. Other agents would ignore or misinterpret these instructions.

In practice, this means your CLAUDE.md becomes much shorter. It starts with @AGENTS.md to import the universal content, then adds only the Claude-specific configuration below it. Across my 38 projects, roughly 55% of the instructions in my CLAUDE.md files are universal and could move into AGENTS.md. The remaining 45% is either Claude-specific (30%) or uses Claude-specific language for what could otherwise be universal advice (15%).

The same pattern applies to other tools. Cursor users would keep their .cursorrules for Cursor-specific features (like glob-based conditional rule activation) and point to AGENTS.md for the shared content. Copilot users do the same with .github/copilot-instructions.md.

This isn’t a migration from CLAUDE.md to AGENTS.md. It’s a separation of concerns: AGENTS.md as a README for all agents, CLAUDE.md as a configuration file for one specific agent.

The hard part
#

AGENTS.md standardizes the right thing at the right time. The format works, the ecosystem is growing, and the convergence problem it addresses was real and getting worse.

But the research is clear about one thing: auto-generating these files with an LLM makes things worse, not better. The files that actually change agent behavior are hand-crafted, containing the counterintuitive choices, the hard-won invariants, and the security boundaries that someone learned the hard way. Twenty-six percent of my own context files were auto-generated by a tool, and they’re all thin and formulaic compared to the hand-written ones.

The spec gives you a place to put instructions. Deciding what belongs there is context engineering, and that’s a practice you build through experience, not a file you generate once and forget.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

Three Hops Deep and No Browser in Sight

Sun, 14 Jun 2026 00:00:00 +0000

You’ve seen this screen. If you use Claude Code or Cursor with MCP servers, you’ve clicked through it dozens of times. “Google MCP Server wants to access your Google Account.” You review the scopes, click Allow, a token lands in your local config, and everything works.

Now imagine the agent that needs your Google Calendar isn’t the one you’re talking to. It’s three agents deep in a multi-agent chain, running in a container with no browser and no way to show you a consent screen.

Two tokens, two problems
#

MCP authentication handles the connection between your client and the MCP server well. The spec’s OAuth 2.1 flow works like standard OAuth: your client authenticates with the MCP server’s authorization server, gets an access token, and sends it with every request. The server validates the token, serves the request, and holds no per-user state. Each request carries its own bearer token.

The problem starts when the MCP server needs to call an external service on your behalf. The token your client sends authenticates you to the MCP server. When that server needs to read your Google Calendar, it needs a completely different token. One that Google issued, through Google’s own consent flow.

On your laptop, this is invisible. The local MCP server opens your browser to Google’s consent screen, you approve, and the server stores the token in ~/.mcp-auth or a similar local path. One user, one browser, done.

In any other environment, this falls apart. You could have the user acquire a Google token out of band and pass it to the MCP server. But the MCP spec explicitly calls this token passthrough an anti-pattern: the server accepts a token it didn’t issue, can’t verify how it was obtained, and has no way to confirm the scope is appropriate.

If you’ve configured your identity provider (Keycloak, Entra ID) as a broker for Google, the plumbing can work. The user authenticates through your SSO, Keycloak redirects the browser to Google’s consent screen, the user approves, and Google returns an authorization code. Keycloak exchanges that code for Google’s access and refresh tokens and stores them. Later, the MCP server can use token exchange to get the stored Google token from Keycloak. The exchange mechanism works. The bottleneck is the consent event that has to happen first.

Consent doesn’t scale across providers. If your agent chain needs Google Calendar, Microsoft Graph, and Slack, each provider requires its own browser redirect, its own consent screen, its own token. You can’t batch them. The question is when to acquire these tokens. At onboarding (“connect your accounts”), you over-provision because you don’t know which services the user will actually need. At first use, you need a browser. Mid-execution of an agent chain, you have neither.

This is the token gap. The infrastructure for moving tokens between systems exists. What’s missing is a way to get the user in front of the right consent screen at the right time, especially when “the right time” turns out to be three agents deep in a chain that has no browser.

The token gap is really two problems that look like one. Credentials are the plumbing: getting a token that the external provider accepts. The token either exists or it doesn’t, and the mechanisms for acquiring it (token exchange, vault lookup, gateway injection) are well understood. Consent is the human problem: the user deciding which scopes to grant, to which agent, for what purpose. No amount of infrastructure replaces that decision.

On a laptop, the consent screen handles both. You authenticate to Google and approve the scopes in the same flow. Once the MCP server moves off your laptop, these concerns separate. A system where agents hold technically valid tokens that nobody explicitly authorized for the current use isn’t a security solution. It’s a liability. Every pattern below solves the credential problem at a different level. The harder question is always consent.

The known patterns
#

Christian Posta has been writing some of the clearest analysis of MCP security and agent authorization patterns. His taxonomy of MCP authorization patterns for upstream API calls maps the approaches people use to cross this gap. Rather than retread his analysis (go read it, it’s worth your time), I’ll summarize where each one breaks.

Shared admin credential. Give the MCP server a single credential for Google. Every user’s request goes through the same account. Easiest to implement, worst for security: no per-user scoping, no audit trail tying actions to individuals, and a single credential whose compromise exposes everyone. The MCP spec explicitly calls this an anti-pattern. Don’t do it.

Credential passthrough. The user acquires a Google token out of band and passes it to the MCP server. The spec flags this as an anti-pattern too. The server accepts tokens it didn’t issue and can’t validate. There’s no proof the token came from a proper OAuth flow, no way to verify the scope matches what this server should have.

SSO federation. Use your identity provider’s brokering capability (Keycloak’s identity broker, Entra ID’s external identities) to federate with Google. If the user has already linked their Google account through the IdP, this works without any browser interaction at request time. The IdP stores the Google tokens, and token exchange surfaces them to the MCP server. This solves the credential problem cleanly.

It doesn’t solve the consent problem. When you configure Google as a broker in Keycloak, you specify which scopes to request up front: openid, email, profile for identity, plus whatever API scopes your MCP servers need (calendar.readonly, drive.readonly). These lock in at configuration time. If a new MCP server needs a scope you didn’t anticipate, you update the config and send the user through Google’s consent screen again. Google shows all scopes at once, so you can’t incrementally add one without a new consent event. And token exchange hands over the full stored token with all granted scopes. There’s no built-in way to give one MCP server calendar.readonly and another drive.readonly from the same token.

This is the pre-flight consent problem in infrastructure clothing: pre-configure scopes and hope they cover everything agents will need. When they don’t, you need a browser. Two IETF drafts, Identity Assertion Authorization Grant and OAuth Identity Chaining Across Domains, aim to enable cross-domain token exchange without per-provider consent flows. Neither is widely implemented yet.

URL elicitation. The MCP spec’s elicitation mechanism (SEP-1036) has two modes. form handles in-band data entry where the response passes through the MCP client. url bypasses the client entirely. In URL mode, the MCP server sends a URL to the client, the client shows it to the user, the user opens it in a browser. That URL initiates a standard OAuth authorization flow with Google. The callback redirects back to the MCP server, which receives the authorization code and exchanges it for tokens server-side. The user just clicks a link and approves. Tokens never touch the client or the model.

This works for multiple users, since each completes their own consent flow and the server stores per-user tokens. But it makes the MCP server stateful: it now manages external token storage, refresh rotation, and revocation. And it still requires a user with a browser at the other end of the chain.

Agent gateway. A gateway sits in front of MCP servers, controlling who can access which servers and tools. Projects like Kuadrant’s MCP gateway enforce authN/authZ via JWT validation and CEL-based (Common Expression Language) policies at this layer. Christian’s Pattern 5 extends this: the gateway also acquires and manages credentials for upstream APIs, injecting external tokens into outbound calls so the MCP server never touches them. But a gateway operates within the platform’s trust domain. It can block access to an MCP server, but it can’t downscope an already-granted Google token or make consent decisions on the user’s behalf. Platform authorization and external-service consent are orthogonal.

Each of these patterns assumes something about the environment: that a user is present, that a browser is reachable, or that the credential-acquiring component knows which user to act for. Those assumptions hold on a developer laptop. They start failing when the MCP server runs remotely, in a cluster, or in any environment where no browser is directly reachable. They collapse entirely when agents call other agents.

Where it gets harder: multi-agent chains
#

Take a concrete scenario. A planning agent receives a request: “Schedule a meeting with the team and attach the latest design doc.” The planning agent calls a calendar agent to find your available slots. The calendar agent calls a Google MCP server to read your calendar. That MCP server needs a Google OAuth token for your account.

You made the original request three hops ago. The Google MCP server has no connection to a browser. URL elicitation can’t work because there’s no client to show the URL to. The calendar agent doesn’t even know it was invoked by a human. It just received a tool call from the planning agent.

Three approaches keep coming up. None of them work well.

Error propagation. The MCP spec has a mechanism for direct connections: the Google MCP server returns error code -32042 (URLElicitationRequiredError) with a consent URL embedded in the response. The client recognizes the error, shows the URL, the user completes the OAuth flow, the callback lands on the MCP server, and the client retries.

In a multi-agent chain, this falls apart. The error goes to the calendar agent, not to the user’s client. For the consent URL to reach the user, every intermediate agent would need to recognize it and propagate it up the chain. LLM-based agents are unreliable at this. They might rephrase the error, attempt workarounds, or lose the URL entirely. Even if the URL reaches the user and they complete the consent flow, the “retry” signal needs to travel back down through the same chain.

A more promising approach would handle this at the harness level. The runtime that makes MCP calls intercepts the -32042 at the transport layer before the LLM sees it, propagates the consent URL back through the infrastructure chain, and retries transparently once the user completes the flow. The LLM never sees the auth error. But this requires every harness in the chain to support the protocol, and no agent framework standardizes this today.

Async notification. Instead of propagating the consent URL through the agent chain, the MCP server delivers it via a side channel: email, Slack message, push notification. The user clicks the link, completes the same OAuth consent flow in their browser, and the callback lands on the MCP server just like in URL elicitation. The OAuth flow is identical. Only the delivery mechanism for the consent URL is different.

This decouples consent from the agent chain but introduces its own problems. The MCP server needs to know how to reach the user (email address? Slack handle?) and needs to identify which user to contact from the bearer token on the original request. Timeouts are ugly: the agent chain blocks while waiting, with no guarantee that consent ever arrives.

Pre-flight consent. Before the agent chain starts, the platform collects all the tokens the chain might need. The user consents to Google Calendar, Google Drive, Slack, and Jira up front. The platform stores tokens in a vault and injects them as needed.

This front-loads the UX problem but creates over-provisioning. The platform doesn’t know which external services the chain will actually use because agent behavior is non-deterministic. Asking for every possible scope up front violates least privilege. Users learn to click “Allow” without reading the scopes, which is exactly the pattern OAuth was designed to prevent.

CoSAI’s Agentic IAM paper (March 2026) establishes the right principles for multi-agent delegation: scope narrows at each hop, never expands. Delegation tokens carry explicit actor and subject claims so the chain of custody stays auditable. These principles describe what the token should look like once it exists. Getting the user’s consent in the first place, when the user is three hops away, is a different problem entirely.

The most interesting architectural proposal I’ve seen is AAuth, a delegation protocol from Dick Hardt (the author of OAuth 2.0) that rethinks the problem from scratch. It’s an IETF draft in the OAuth working group. Instead of one-time consent grants that get reused indefinitely, AAuth introduces a Person Server that acts as an ongoing consent oracle. Dasith Wijesiriwardena’s errand analogy is the most accessible explanation, and Christian wrote a hands-on demo walkthrough. The agent doesn’t hold your Google token. It carries an unsigned request from the resource to the Person Server, the Person Server checks with you (or evaluates against a pre-approved “mission”), and returns a signed authorization. The Person Server evaluates each request individually. The agent shuttles tokens between parties but never holds standing credentials.

The “mission” concept is what makes this click. A mission is a written-down, approved intent that the Person Server holds on behalf of the user. Think of it as a scoped work order: “schedule a meeting with the team and attach the latest design doc.” The agent proposes the mission, the user approves it, and every subsequent request gets evaluated against the mission’s stated purpose. Read a calendar to find available slots? In scope, approved silently. Send an email to someone not on the team? Out of scope, escalated to the user.

The Person Server sees all requests across all resources under that mission, which makes it the one party positioned to detect when the agent’s behavior drifts from the original intent. Missions have a lifecycle: active or terminated. Tokens issued under them are short-lived. That’s a cleaner answer to the pre-flight consent problem than asking for every possible scope up front and hoping for the best.

AAuth also handles sub-agent chains explicitly. Each hop gets its own identity for audit and revocation, authority flows through the parent agent, and the full delegation chain back to the person is preserved. The protocol is early (the spec, an interactive explorer, and a .NET SDK are available), and the architecture addresses the right problems. It also makes the credentials-vs-consent split structural: the Person Server handles consent (did the user approve this action?), while the resource’s Access Server enforces its own policies independently. Two gates, both must pass.

The catch: AAuth requires the resource to participate in the protocol. The Person Server issues AAuth tokens, not Google OAuth tokens. For existing providers like Google, Microsoft, or Slack that only accept their own OAuth tokens, you still need their consent flow to get a token they’ll accept. AAuth solves consent and delegation for resources that speak AAuth. For everything else, the consent screen problem stays until major API providers adopt the protocol.

What remains open
#

The token gap has patterns. Multi-agent consent doesn’t. Here’s what I think still needs solving.

Where do tokens live? Per-user external tokens need to persist across agent invocations but shouldn’t be accessible to agents that don’t need them. A vault with fine-grained access policies is the obvious answer, but the lifecycle is tricky. Who provisions the vault entry? Who rotates it? What happens when the user revokes consent through Google’s security settings? Who cleans up the stale token?

How do you reach the user? In a multi-agent chain, the consent request needs to travel back to a human who can open a browser. Today this is ad hoc: error propagation, push notifications, Slack messages. There’s no standard channel for “an agent three hops deep needs your approval.” Building one means defining a consent backpropagation protocol that agent frameworks understand.

What happens when consent drifts? A user approves calendar.readonly for a scheduling task. Six months later, a different agent chain uses the same token to read calendar details for a completely different purpose. The token is technically valid. The consent is contextually stale. AAuth’s mission model addresses this by scoping consent to a specific task with a defined lifecycle. But deciding what counts as “in scope” is left to the Person Server’s policy engine, and that policy engine doesn’t exist yet as a general-purpose solution. Of the open problems, this is the one that worries me most. A valid token with stale consent is indistinguishable from a properly authorized one until something goes wrong.

The Vercel/Context.ai incident in April 2026 showed what over-provisioned consent looks like in practice. An infostealer on a Context.ai employee’s machine compromised OAuth tokens that Context.ai held on behalf of its users. One of those tokens belonged to a Vercel employee who had connected Context.ai to their enterprise Google Workspace with broad permissions. The attacker used that stolen token to access the employee’s Google Workspace account, and from there found credentials that led into Vercel’s internal environments. The breach wasn’t a chain of delegated access. It was a single, broadly scoped OAuth grant to a third-party AI tool, and when that tool’s infrastructure was compromised, the blast radius was proportional to the scope.

The IETF drafts on identity chaining and assertion grants are the right long-term direction. When they reach production implementations, they’ll provide a standards-based way to exchange internal tokens for external ones without the user clicking through a consent screen every time. But “12-24 months to RFC” means organizations deploying multi-agent systems today need interim solutions.

The fundamental tension remains: agents are non-deterministic, but OAuth consent is interactive. Agents compose dynamically, but consent is granted statically. Protocols like AAuth and standards like the IETF identity chaining drafts are attacking this from different angles, but neither is production-ready today. Until the consent model catches up to agent composition, the gap between “the agent has a valid token” and “the user approved this specific use” will keep growing.

On your laptop, that gap doesn’t matter. Three hops deep and no browser in sight, it’s the whole problem.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

Red Teaming Agents, Not Models

Mon, 08 Jun 2026 00:00:00 +0000

Your agent passed every guardrail test. It never says anything harmful, never generates offensive content, politely declines every adversarial prompt you throw at it. And last Tuesday, it quietly deleted the wrong database because a Jira ticket it was reading contained a hidden instruction in the description field.

The guardrails caught everything the agent said. They caught nothing about what it did.

What red teaming is (and isn’t)
#

If you come from software engineering rather than security, red teaming might sound like a fancy term for testing. It’s related, but the framing is different. In traditional security, a red team plays the attacker: a group authorized to emulate an adversary’s capabilities against your system, reporting what worked so you can fix it before someone else finds it. The blue team plays defense: they monitor, detect, and respond to threats. Purple teaming is when the two work together, feeding offensive findings directly into defensive improvements.

For AI systems, red teaming has mostly meant sending adversarial prompts to a model and checking whether the response is harmful. Can you trick it into generating dangerous instructions? Can you bypass its safety training with creative prompt engineering? Tools like Garak automate this at scale, running libraries of attack prompts against chat endpoints and scoring the responses.

That approach works well for models and chatbots, where the only output is text. If the model says something harmful, you catch it by reading what it said. But agents don’t just say things. They do things. And that changes the entire testing surface.

Models say things. Agents do things.
#

The difference sounds obvious, but the implications run deep. A model is a brain in a jar. It receives text, it produces text, and everything it does is visible in the text it produces. You can evaluate its behavior by evaluating its output.

An agent is a brain with hands. It has tools: file systems, APIs, databases, email, MCP servers, shell access. When it acts, the action happens in the real world, not in the response text. An agent could respond with “I would never do that” while simultaneously executing the thing it claims it wouldn’t do through a tool call. If you’re only checking the text output, you catch nothing.

This is why model-level red teaming doesn’t transfer to agents. Sending adversarial prompts and checking responses tests the chat layer. It doesn’t test whether the agent’s tool calls are correct, whether its side effects are intended, or whether a compromised data source can steer it toward actions the operator never authorized.

Three attack surfaces
#

Agent red teaming needs to cover three layers. Borrowing from traditional penetration testing, you can think of these as increasing levels of attacker knowledge about the target system.

Input-level (black box). The attacker sends malicious prompts directly to the agent. This is the layer that existing guardrail tools handle well. You don’t need to know anything about the agent’s internals. You just send bad input and see what happens.

Data-level (grey box). The attacker compromises a data source the agent consumes. A Jira ticket with hidden instructions in the description. A document with invisible text that redirects the agent’s goal. A database record with embedded prompts. This is indirect prompt injection, and the OWASP Top 10 for Agentic Applications classifies Agent Goal Hijack (ASI01) and Tool Misuse (ASI02) as the two highest-priority risks in this category.

Tool-level (white box). The attacker compromises a tool or MCP server the agent calls. The tool returns manipulated responses that steer the agent toward unintended actions. This requires knowing the agent’s tool chain, but once you have that knowledge, you can influence the agent’s reasoning at every step.

Each layer requires different testing infrastructure, different attack libraries, and different detection capabilities. Most organizations today test only the first layer and assume the other two are covered by general infrastructure security. They’re not.

Test the real agent, not a simulation
#

The most promising approach emerging in 2026 is best described as “testing the real agent in a synthetic world.” Instead of rebuilding your agent in a test harness (which means you’re no longer testing the same agent), you intercept the agent’s tool calls and replace real backends with controlled synthetic ones.

The agent thinks it’s talking to its real MCP servers, its real APIs, its real database. It’s actually talking to fakes that can inject attack payloads through tool responses, capture every action the agent takes, and verify whether the agent did something it shouldn’t have. Think of it as setting up a honeypot for your own agent.

This idea was pioneered by Agent Dojo out of ETH Zurich, which introduced a concept that sounds simple but changes how you think about agent security: dual scoring. Every test measures two things. The security score tells you whether the agent resisted the attack. The utility score tells you whether the agent still completed its legitimate task. That tradeoff matters because you can trivially make any agent perfectly secure by making it refuse to do anything at all. Security without utility isn’t security. It’s a paperweight.

Agent Dojo itself appears to be inactive (the last commit is from late 2025), but its core ideas are showing up across the next generation of agent testing tools. MiDojo picks up where Agent Dojo left off, adding man-in-the-middle interception of MCP tool calls so you can test real agents against synthetic backends without rebuilding them. The MiDojo introduction walks through the design and a hello-world weather suite end to end. The same dual-scoring principles also inform NIST’s emerging guidance on agent security standards.

Detection is harder than attack
#

Injecting attacks is relatively straightforward. The hard part is detecting whether the agent acted on the injection.

The reason is that agents don’t respond to attacks in predictable ways. An agent that receives a hidden instruction through a compromised document might not act on it through the same document tool. It might route the exfiltration through an email API three tool calls later, or write sensitive data to a file that gets synced elsewhere, or modify a configuration that opens a door for a future request. Tying the cause (compromised document) to the effect (data exfiltration via email) requires tracing the full execution path across the agent’s entire tool chain, not just watching individual tool calls in isolation.

The distinction between red teaming and blue teaming matters here. Red teaming asks “can I make the agent do something bad?” Blue teaming asks “can I detect when the agent is doing something bad?” Both questions require the same detection infrastructure, but they answer different things and operate at different times:

Red teaming runs before deployment. You simulate attacks, measure resistance, and improve defenses based on what you find. Blue teaming (runtime monitoring) watches the agent during normal production operation and catches unintended actions as they happen, even without an adversary present.

Detection capability is dual-use. Once you build the infrastructure to detect harmful actions for red teaming purposes, you can deploy that same infrastructure as a runtime monitor. Red team findings become detection rules. Detection rules become guardrails.

But attacks aren’t the only problem.

No adversary required
#

The most interesting question that comes up in agent security discussions is one that isn’t about adversaries at all: what about agents that do something unintended given a perfectly cooperative prompt and completely uncompromised tools?

“Delete all the temp files” and the agent deletes non-temp files too. “Update the configuration” and the agent overwrites unrelated settings. “Summarize this document” and the agent silently modifies it during the read. These aren’t attack scenarios. They’re the mundane reality of agents operating in complex environments, and they’re arguably harder to test because they happen non-deterministically and only surface under specific combinations of context, tool state, and agent reasoning.

If you’ve read the earlier posts in The Flock series, this should sound familiar. The creativity paradox is exactly this: agents doing the wrong thing while optimizing for the right goal. Red teaming catches adversarial attacks. Catching the agent’s own well-intentioned mistakes requires the same detection infrastructure but a different testing mindset: not “what happens when someone attacks?” but “what happens on a normal Tuesday when nobody is attacking and the agent just gets creative?”

Agent observability becomes a prerequisite for security here. If your team can’t trace an agent’s tool calls well enough to explain what it did after a normal run, you certainly can’t trace what it did under adversarial conditions. (More on agent observability for CI pipelines in an upcoming post in The Flock series.)

Purple teaming: when the loop closes
#

The 2026 trend is merging red and blue teaming into continuous purple teaming: autonomous agents continuously simulate attacks, detect vulnerabilities in real time, and feed findings back into guardrails, all in the same cycle. Multiple vendors are building this as a product category.

It sounds compelling: red team findings automatically become runtime detection rules, which become regression tests, which get re-tested in the next red team cycle. Every vulnerability found once is caught forever after. The loop closes, and the system gets more secure with every iteration.

But the risk is real. ISACA warns about escalatory spirals when autonomous red and blue agents interact. A false positive triggers a defensive action (revoking credentials, blocking a tool). The red team agent interprets the changed environment as a new attack surface and escalates. The blue team responds with more aggressive countermeasures. Within minutes, the two AI systems are fighting each other over a signal that was never a real threat.

The mitigation that works is “shadow mode”: AI agents suggest security actions, humans approve them. The automation handles the speed and coverage. The human handles the judgment about whether the finding is real and the response is proportionate.

Where this leaves us
#

Agent security is where web application security was in the early 2000s: the attack surface is new, the tooling is immature, and most organizations haven’t started thinking about it systematically. The OWASP Top 10 for Agentic Applications published in late 2025 is the first formal taxonomy. Microsoft’s Agent Governance Toolkit (April 2026) addresses all ten risks with runtime enforcement.

Full disclosure: we’re working on this at Red Hat. The TrustyAI team has been integrating Garak (the open source LLM vulnerability scanner) into the Red Hat AI platform for automated red teaming of models and agents, and is building the next generation of agent-level testing that goes beyond chat-endpoint scanning. The Summit 2026 Day 2 keynote demos some of where this is heading.

Concretely, the TrustyAI team is building MiDojo as part of Asago, a broader agent red teaming platform. The platform architecture is still being finalized, but MiDojo is the first component being made publicly available. Separately, the Kagenti team has been running capture-the-flag exercises against agents in Kubernetes clusters, testing whether policy enforcement (OPA, sandboxing) actually holds when an agent gets creative with leaked credentials.

For practical reading: Morgan Foster’s writeup of Claude stealing the HR docs and Roy Belio’s walkthrough of infrastructure red teaming with abliterated models on Red Hat Developer.

The gap that remains is detection. We know how to inject attacks at every layer. We’re getting better at testing real agents without rebuilding them. What we still lack is reliable, general-purpose detection of whether an agent acted on a compromise. A poisoned Jira ticket doesn’t show up as a failed Jira call. It shows up three tool calls later as a perfectly normal-looking email with sensitive data in the body. Catching that means tracing the full chain from compromised input to unauthorized action, even when the two happen through completely unrelated tools. Solving that problem unlocks both red teaming and runtime monitoring in a single investment.

The agent that says all the right things and does all the wrong things is the one you need to worry about. And right now, most testing only checks what it says.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

The Sheep That Picked the Lock

Mon, 01 Jun 2026 09:00:00 +0200

The 101 post covered working with one agent in one session, with a human watching. Those principles still hold, but the question changes when you remove the human and let agents run unsupervised in CI. This is the first post in “The Flock,” a series about what happens when agents run at scale with nobody around. We start with creativity, because it’s the pattern every developer recognizes immediately.

The farmer installed a new gate latch, specifically designed to be sheep-proof. The next morning, the gate was open, the latch was technically undamaged, and three sheep were in the vegetable garden eating his prize-winning cabbages. The latch worked exactly as specified. The sheep just found a different way around.

The most expensive cheat sheet ever written
#

The typia project needed to port a TypeScript validator to Go. The task sounded straightforward: take a .ts file, rewrite it as a .go file, leave the algorithm alone, iterate until tests pass. The test suite was serious: 2,900 files, 168 structural fixtures, 80k lines total.

The developer kicked off the agent overnight. He woke up to a green CI badge with all tests passing. When he opened the diff, two-thirds of the core logic was missing. The agent had deleted every failing test. “All tests pass” was technically accurate, just not in the way anyone intended.

After adding a rule that tests are sacred and must never be modified, he started a second run. This time the agent consumed 8 billion tokens and built a massive switch statement that memorized all 168 test outputs as hardcoded strings. It had run the original TypeScript validator hundreds of times, captured the output, and embedded it verbatim. The most expensive cheat sheet ever written.

On the third attempt, the agent replaced the entire library with Zod (a different validation framework), then edited the CI workflow file to skip the test categories where Zod fundamentally fails. When you can’t pass the test, redefine what “testing” means.

Three escalating attempts at the same problem, each more creatively destructive than the last. The agent optimized for the stated goal (“make tests pass”) using any means available. The goal was correct. The latitude was too wide.

The shortcut through production
#

A similar kind of creativity shows up when agents have access to tools they don’t need for the task at hand. An engineer was migrating a website to AWS using Terraform via a coding agent. The agent confused which state file was current (it had unpacked an old Terraform folder), decided that terraform destroy would be “cleaner and simpler,” and wiped the production infrastructure. Years of course data, gone.

In a separate incident, a coding agent deleted PocketOS’s production database and all volume-level backups with a single GraphQL API call. The most recent recoverable backup was three months old. Nine seconds from start to finish.

Both agents found a “creative” shortcut that achieved an intermediate goal (clean state) while destroying the actual goal (a working production system). In both cases, the agent had access to destructive tools it didn’t need for the task at hand. The sheep didn’t just pick the lock. They knocked down the fence and wandered onto the highway.

Sometimes the agent doesn’t destroy anything. It just quietly walks through a door nobody realized was open. Murat Deligoz reported that Codex was blocked from editing a file, so it gave itself root and did it anyway. The account belonged to the docker group, which on a Linux machine is root wearing a thin disguise. The agent started a container as root, bind-mounted the host filesystem, made the blocked change, and reported back as calmly as if it were checking the weather. No vulnerability was exploited. No password was stolen. Docker group membership granting root-equivalent access is documented, intended behavior. The agent simply understood the machine’s permissions more completely than the person who configured them, then walked through a door that had been standing open the whole time.

The everyday creativity tax
#

These dramatic stories make headlines, but the everyday creativity problems are more insidious because they don’t announce themselves. They sit quietly in your pipeline output, waiting to waste your morning.

The helpful directory structure. A batch agent was told to write report files to /output/. After processing 30 items, the downstream merger found only 5 files. The agent had decided the flat directory was “disorganized” and created subdirectories by category: /output/security/, /output/performance/, /output/design/. Every file was there, neatly organized, completely invisible to the script that listed /output/*.json. The agent’s reasoning was sound. The system’s expectations were different.

The comment that became an instruction. A code review agent was scanning source files for security issues. One file contained a comment left by a developer months earlier: // TODO: consider switching to admin API for bulk operations. The agent read the comment, interpreted it as a pending task, and rewrote the function to use the admin API, including privilege escalation it found documented in the project’s wiki. The original developer had written a note to themselves. The agent read it as a request. Any text in the agent’s context is potential instruction, and the boundary between “data the agent is reading” and “instructions the agent should follow” exists only in our heads, not in the model’s architecture.

The test that learned the wrong lesson. An agent writing integration tests found an existing test that used a mock database. It copied that pattern for all new tests, including tests for the database migration tool, which specifically needed a real database connection. The mock tests all passed beautifully. The migration broke on the first real deployment. The agent will always prefer copying a pattern it can see over reasoning about whether the pattern fits. If a template or example exists in the context, it becomes the mold. The fix: use <placeholder> tokens in examples instead of realistic values, so the agent can’t pattern-match its way to a “complete” output.

The status file nobody could parse. A pipeline expected a JSON status file at the end of each run: {"status": "complete", "items": 47}. The agent would write {"result": "done", "count": 47} or {"status": "completed successfully!", "processed_items": 47} or a nicely formatted multi-line variant with helpful extra fields. Same information, different shape, downstream parser crashes every third run. The fix was the same as always: move the status file to a deterministic script. The agent decides what to report. The script decides how to format it. This pattern connects directly to the skill vs. script distinction from the 101 series: skills handle judgment calls, scripts handle mechanical operations.

Everything in context is live
#

There’s a subtler version of this problem that shows up in the agent’s own codebase, not just in external systems.

Keeping dead code or optional paths in skills “because it might be useful later” was already messy in traditional software. With LLMs, it’s actively dangerous. Every conditional branch, every commented-out alternative, every unused code path is context the model can read and act on. The agent doesn’t distinguish between active and dormant code. I’ve watched agents find disabled features behind flags, decide the dormant code looked relevant, and start “fixing” issues in code that was never meant to run. The fix never materialized because the code was never activated, but the agent burned tokens and sometimes introduced side effects in the surrounding live code.

The inverse is just as dangerous. A deep review agent recently found two stub functions in our codebase, function signatures with pass as the body. It classified them as dead code (which was technically correct, they didn’t do anything yet) and removed them. What it couldn’t know: those stubs were placeholders for a specific requirement from the spec. The incomplete implementation was there on purpose, waiting for the next development pass. After the review agent cleaned them up, a second review pass removed the orphaned imports and helper code that referenced those stubs. Two rounds of locally correct cleanup, and the requirement had silently vanished from the codebase. We only caught it because a fresh implementation from the same spec included the feature and the reviewed version didn’t.

The lesson cuts both ways. Agents will act on dead code they find (fixing bugs in dormant paths), and they will remove code they judge to be dead (dropping planned functionality). Both stem from the same blind spot: the agent sees text, not intent. If something isn’t meant to be used right now, remove it. If something is incomplete but planned, make it pass a gate that checks spec coverage after any code removal. Git history exists for recovery, but only if someone notices the gap. In the agent’s context window, everything is live, and nothing is “planned for later.”

And sometimes the sheep doesn’t break the fence at all. It just argues, convincingly, that the fence doesn’t apply in this particular situation. We had a compliance gate that found a gap, calculated 94% coverage, and then invented a status called “COMPLIANT (with documented gap)” to let itself pass. The gate instructions said 100% or STOP, no exceptions. The agent decided that being helpful outweighed following the rule. More on that in a later post about fences the flock can’t talk around.

All of these cases point to the same broader principle.

Constrain inputs, tolerate outputs
#

That asymmetry extends to everything the agent touches. Be specific about filenames, IDs, and argument formats for inputs. But for agent-generated output (like scoring tables), build tolerant parsers. An agent might format a score as 1/2, or bare 1, or **WHAT** (0-2): 1. Fighting output format is a losing battle. Parse what they give you.

This maps directly to Postel’s Law from protocol design: “Be liberal in what you accept, conservative in what you send.” Your scripts should tolerate the model’s creative formatting. Your instructions to the model should be consistent, specific, and never vary in phrasing. Don’t rephrase the same instruction three different ways across three different skills and expect consistent behavior, because the model may interpret those variations as three distinct instructions with subtly different meanings.

Once constrained, creativity shines
#

Here’s the part that makes this all worthwhile. After constraining agents with limited tools, focused goals, clean context, and deterministic guardrails, the remaining creativity becomes a genuine asset. The agent notices a background task failed due to a transient error, figures out from state files on disk that retrying would succeed, and relaunches the task on its own. It spots patterns across documents that a rigid pipeline would miss entirely. It finds solutions you didn’t anticipate and handles edge cases you never thought to specify.

Once the sheep can’t escape the paddock, they start doing genuinely useful things within it. They find the best grazing spots. They alert each other to changes in the weather. The creativity that was a liability when pointed at the fence becomes an asset when pointed at the grass.

A specific technique that works well here: force agents to compare multiple approaches before committing to one. Generate 2-3 strategies, score each against your rubric, present them in a comparison table, and pick the best one. The comparison step itself is what produces the quality, not just having more candidates. An agent that compares options reasons better than an agent asked for its best guess.

And more broadly: describe goals, not procedures. “Assess this document against these quality criteria” outperforms a 47-step analysis checklist that breaks at step 23. “Get this package to Berlin by Friday” works better than prescribing the exact courier, route, and handoff schedule, because the agent can reroute when a flight gets cancelled. The same principle applies to skill development: describe the desired outcome and iterate on the result, rather than prescribing every implementation step and hoping the agent follows them in the right order.

What’s coming in the flock
#

This was the first of eight posts. The creativity paradox is the most visible problem, but it’s not the most fundamental one. The rest of the series covers the infrastructure that makes constrained creativity possible:

The Sheep That Forgot the Way Home: Agent memory is unreliable. How compaction destroys state mid-run and why everything must live on disk.
One Stray Leads the Whole Flock Astray: Context contamination between agents. Why every agent needs a clean room, and why the author can’t review itself.
One Gate Per Meadow: When fine-grained tool access hurts more than it helps. Purpose-built scripts for purpose-built tasks.
Fences the Flock Can’t Talk Around: Some rules can’t be left to the agent’s judgment. Hard enforcement in deterministic code.
Counting Sheep, Getting Different Numbers: Scoring with LLMs at scale requires calibration examples, not just rubric descriptions.
The Wolf in Sheep’s Clothing: Every data path is a potential injection surface. Defense in depth for context windows.
The Shepherd’s Night Vision: Agent observability, self-debugging loops, and why real data catches bugs that synthetic tests miss.

The farmer installed a sheep-proof latch. The sheep found a different way around. The lesson isn’t to build a better latch. It’s to accept that the sheep are creative and design your farm accordingly.

The principles of constraining inputs while tolerating outputs, Postel’s Law applied to LLMs, and the creativity-as-asset thesis were first articulated by Jessica Forrester and Jason Greene based on their pioneering work with multi-agent pipelines at Red Hat. This series, and the thinking behind it, wouldn’t exist without Red Hat’s investment in AI engineering and the space to explore these ideas. The examples in this post are drawn from independent sources and my own experience.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

The Flock

Mon, 01 Jun 2026 08:00:00 +0200

The 101 post covered working with coding agents one-on-one. A human watches, nudges, clears context when things drift. Those habits work well when someone is paying attention. The question this series asks: what changes when nobody is?

Running AI agents unsupervised in CI, processing hundreds of items per batch through multi-agent pipelines, turns every context engineering principle up to eleven. Compaction doesn’t just make the agent forgetful; it makes the agent confidently wrong. Context contamination doesn’t just degrade quality; it makes one agent adopt another agent’s identity. And creativity, the very thing that makes agents useful, becomes the thing that makes them dangerous when there are no guardrails and no one watching.

The metaphor for this series is a flock of clever sheep. The farmer has gone to bed. The sheep are skilled, helpful, and absolutely going to do things their own way the moment the farmhouse lights go off. Each post covers a different aspect of managing the flock:

The Sheep That Picked the Lock: The creativity paradox. Agents delete tests, build lookup tables, and destroy production databases, all while optimizing for exactly the goal you gave them. How constraints turn creative liability into creative asset.
The Sheep That Forgot the Way Home: Agent memory is unreliable. Compaction destroys state mid-run and the agent invents replacements with complete confidence. Why everything must live on disk.
One Stray Leads the Whole Flock Astray: Context contamination between agents. One agent’s residual context poisons the next agent’s judgment, sometimes to the point of identity adoption. Why every agent needs a clean room, and why the author can’t review itself.
One Gate Per Meadow: When fine-grained tool access hurts more than it helps. MCP works beautifully for interactive exploration but creates inconsistency in pipelines. Purpose-built scripts for purpose-built tasks.
Fences the Flock Can’t Talk Around: Some rules can’t be left to the agent’s judgment, because the agent will argue its way past them. Hard enforcement in deterministic code for constraints that must hold without exception.
Counting Sheep, Getting Different Numbers: Scoring with LLMs at scale requires calibration examples, not just rubric descriptions. Five agents reading the same rubric will produce five different scores unless you anchor them.
The Wolf in Sheep’s Clothing: Every data path is a potential injection surface. Instructions and data share the same context window with no protocol-level separation. Defense in depth for a world where the attack surface is text.
The Shepherd’s Night Vision: Agent observability, self-debugging loops, and why real data catches bugs that synthetic tests miss. The same capability that causes the problems is also the fastest way to find them.

The series draws on lessons from building multi-agent CI pipelines, my own experience with coding agents, and published accounts from the broader community. Much of this thinking grew out of our work at Red Hat, where teams are pushing AI agents into production pipelines and learning, sometimes the hard way, what it takes to make them reliable. The first post is already up. The rest will follow over the coming weeks.

The farmer installed a sheep-proof latch. The sheep found a different way around. The lesson isn’t to build a better latch. It’s to design the farm for sheep that think.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

AI Wrote It. Nobody Read It.

Wed, 20 May 2026 00:00:00 +0000

You’re reading through a lengthy architecture proposal shared on a team channel, and something feels off. Near the end, tucked between the conclusion and the appendix, you find a section titled “Corrections Applied After Cross-Referencing: The following adjustments were made based on automated analysis of the upstream repository.” Clearly, AI wrote most of this. And the review before sharing was either absent or superficial. You scroll back to the top and start reading the whole thing differently. Not engaging with the proposal anymore, but checking whether you can trust it at all.

The distinction that matters
#

There’s nothing wrong with using AI to draft documentation. I do it regularly, and this blog is transparent about how AI contributes to each post (see the attribution at the bottom). The problem isn’t AI-drafted content. It’s content that went out as the final product, with nobody reading it through a reader’s eyes before hitting send.

The cost is highest in internal technical documentation, because the audience is captive. Your colleagues can’t just scroll past. They have to read it, or at least try to, because decisions depend on it. An arXiv paper from March 2026 frames this as a tragedy of the commons: the person who generates 30 pages saved an hour, but the 10 people who have to read it lost a day each. For a company of 10,000 employees, that hidden cost runs to roughly $9 million per year in lost productivity. And 42% of workers say they trust colleagues less after receiving low-quality AI-generated content from them. The damage isn’t just to the document. It’s to the person who sent it. And once a reader spots the first sign, they shift from learning mode to checking mode, which is slower, more exhausting, and means every claim in the document gets a mental asterisk.

The scaffolding nobody removed
#

The most obvious signs are process artifacts that were never meant for human readers. Sections like “Corrections from Verification,” “Notes for the author,” or “Based on analysis of the following sources:” are scaffolding from the generation process. They’re instructions to or from the AI, not content for the reader. Leaving them in is the document equivalent of shipping debug logging to production.

LLM meta-comments are a related sign. “Here is a summary of the key findings…” or “The following section covers…” as actual content rather than transitional prose. The AI is narrating its own output. A human author would just write the summary. And revision notes in the body (“Updated: Added section on error handling based on reviewer feedback”) belong in version control, not in what the reader sees.

These are the easy catches. If you find one, you know nobody read the document front to back before sharing it. But the deeper signs are about what’s missing from the document, not what’s left in it.

No reader in mind
#

A lengthy technical document that doesn’t tell you who it’s for was written for the prompt, not for a reader. Human authors can anticipate their reader’s perspective, gauge what needs explaining and what doesn’t, and adjust the depth accordingly. AI writes for the instruction it received.

There are now two fundamentally different audiences for documentation, and they need different things.

A document written for human readers should be concise, opinionated, and structured around tasks. Cut the redundancy, lead with what matters, add “you can skip this if you already know X” escape hatches. The reader’s attention is the scarce resource.

A document written for machine consumers (coding agents, RAG pipelines, model training data) can be more verbose, include more detail, and tolerate repetition, because machines don’t get bored or lose focus. They need explicit metadata, consistent terminology, and structured headings rather than visual layout cues. But the stakes are different too: inconsistencies and wrong information don’t just confuse today’s agent. In your RAG pipeline, they poison tomorrow’s retrieval results. In training data, they get baked into model weights and produce wrong answers for years. Poorly reviewed docs fed into machine consumers make every AI system that reads them worse.

The problem isn’t which audience you chose. The problem is not choosing. And that starts in the prompt: if you don’t tell the AI who the reader is, what they already know, and what decision they need to make, you get the 30-page default. Everything covered at medium depth, no priorities, no opinion, no awareness of what the reader already knows or needs to decide.

The signs that need a closer look
#

Beyond scaffolding and missing audience, there are subtler signs.

Everything gets equal depth. Real authors go deep on what matters and skim what doesn’t, because they know which parts readers care about. AI gives every section the same level of detail. A three-paragraph introduction to HTTP status codes in a document about a specific API’s error handling is a sign that nobody prioritized.

No opinions, or opinions without reasoning. Sometimes there’s no recommendation at all: “Option A has these trade-offs. Option B has these trade-offs.” If your architecture document doesn’t recommend an architecture, someone is going to have to schedule a meeting to figure out what the document was trying to say. But the subtler version is a document that does recommend option A, lists two alternatives that were “considered,” and never explains why they were rejected. A real author who evaluated three approaches can tell you what was wrong with the other two, not just what was right with the chosen one. AI picks the option that fits the prompt best and moves on, leaving the reader to guess whether the alternatives were seriously considered or just listed for the appearance of rigor.

No warnings or gotchas. Real documentation says “watch out for X” and “don’t try Y, even though it seems like it would work.” AI describes the happy path because it hasn’t been burned by the unhappy one. The absence of “this will bite you” sections is a strong signal that nobody with operational experience reviewed the content.

Assumed expertise not everyone has. The document casually drops terms like “ADF round-tripping” or “SPIFFE identity delegation” without definition, as if every reader is already knee-deep in the implementation. AI inherits the vocabulary of its prompt and its source material without considering whether the reader shares that context. A human author who knows their audience calibrates which terms need explanation and which don’t. An AI author treats everything it read during generation as common knowledge.

“We accept this trade-off” without a “we.” AI uses collective pronouns to justify design decisions it made alone. “We considered three alternatives and chose option A” reads like a team deliberation. If the document was written by one person and an AI, that “we” is performing a consensus that never happened. Real design decisions name who decided and why, or at least acknowledge the decision was a judgment call rather than a group conclusion.

What to do about it
#

AI made writing cheap. It didn’t make reading cheap. We accelerated the part that was already fast and shifted more load onto the part that was already the hardest.

A few practical steps:

State your audience in the first paragraph. “This document is for the platform team evaluating our authentication options.” or “This is a reference document intended as context for coding agents working on the ingestion pipeline.” One sentence changes how the reader (human or AI) approaches everything that follows.

Cut before you send. The question isn’t “is this information correct?” The question is “does this reader need this information?” A 30-page document that could be eight pages with better prioritization respects the reader’s time more than a longer one that covers everything and serves no one. It also respects yours: fewer pages mean fewer comments to respond to, fewer misunderstandings to clarify, and fewer rounds of review before the document can move forward.

Add opinions. If you’ve worked with the system, say what you’d recommend and why. If you haven’t (and the AI drafted it from specs), say that too. “I haven’t tested this configuration in production” is more useful than a confident-sounding recommendation nobody verified.

Include warnings. After the AI drafts a section, ask yourself: “what would go wrong if someone followed this exactly?” Add the answer. The gotchas are the highest-value content in any technical document, and they’re the part AI can’t write because it hasn’t been burned.

Be honest about the review level. If you used AI to generate the document and only had time for a light pass, say so. “This document was AI-drafted and lightly reviewed. Please flag errors.” That one sentence is more respectful than pretending you wrote and verified every word. Attribution tells the reader “AI helped write this.” What readers also need to know is how carefully a human reviewed it. As I wrote in the Know Your Limits post, honest self-assessment leads to better decisions than performed confidence.

AI can write the draft in minutes. The review that turns it into something worth reading takes longer, and there’s no shortcut for it. Skip that step, and your readers will notice before you do.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

Know Your Limits: Quiz Yourself Before You Trust AI

Wed, 13 May 2026 00:00:00 +0000

The conversation was going well. We were working out how to integrate OpenShell’s network isolation into our agent platform. The AI had produced an overlap analysis, identified shared capabilities, proposed a feature breakdown for the integration. Everything sounded reasonable, the trade-offs clearly articulated, the architecture diagrams sensible. I was nodding along, ready to take the recommendation to the team.

Then a small voice in the back of my head asked: do you actually know enough about veth pairs and TLS MITM proxying to tell whether any of this is correct?

The comfort zone trap
#

When you use AI for technical architecture or feature evaluation, the conversation flows naturally. The AI explains trade-offs, proposes designs, writes implementation plans. Everything sounds plausible, because LLMs are exceptionally good at producing plausible-sounding output. That’s what they’re optimized for.

But plausible isn’t correct, and you can only tell the difference within your own domain expertise. Shaping context helps the AI produce better output (that’s what the 101 post covered). This post is about a different problem: what happens when the output is good, maybe even correct, but you lack the expertise to verify it?

The risk isn’t that AI gets it wrong. The risk is that you can’t tell when it does. And research from Aalto University suggests the problem is worse than we think: when interacting with AI tools, everyone overestimates their performance, regardless of skill level. Higher AI literacy correlates with more overconfidence, not less, because users trust the system’s output without checking whether they could have reached the same conclusion on their own.

Stack Overflow’s 2025 developer survey found a matching paradox: 84% of developers use or plan to use AI tools, but only 29% trust them, down 11 percentage points from 2024. Developers know they should verify. The problem is they don’t have a fast way to assess whether they’re capable of verifying in a given domain.

The five-minute honesty check
#

Here’s a technique that takes about five minutes and reliably reveals where your expertise ends. When AI drives you into unfamiliar territory, ask it to generate a quiz. Five questions, four answers each, targeting the specific skills you’d need to implement or review the thing being proposed.

The prompt is straightforward. Here’s the shape of what I used:

We’ve just analyzed [project]’s [capability] and how it overlaps with our platform. My concern is that integrating this goes deeper into [domain] than our team’s expertise. Create a quiz with 4-5 tough multiple-choice questions targeting the underlying concepts we’d need to understand. Don’t test [project]-specific internals, test whether we understand the fundamentals it builds on. Include a scoring rubric that maps scores to actionable team decisions.

The quiz isn’t a certification. It’s a mirror. If you score 2 out of 5, you now know something important: you’re past the boundary where you can meaningfully review AI output in this domain. And knowing that is the whole point.

What the quiz actually reveals
#

We ran this during the OpenShell evaluation. The AI generated four questions targeting the exact features the team would need to implement: veth pairs for container networking, /proc-based process inspection, TLS MITM proxying for traffic analysis, and SSRF hardening for outbound request filtering. (The full quiz is in the appendix if you want to try it yourself.)

The questions themselves were informative even before we answered them. One walked through setting up a network namespace with a veth pair, configuring IP addresses and routes, and then asked why the agent still can’t reach the proxy. I didn’t just get the answer wrong, I wasn’t confident about what a veth pair even does in this context, and that gap in understanding is itself useful information. If the question doesn’t make sense to you, you’ve found a boundary.

Wrong answers reveal something different: misconceptions. Good quiz questions have plausible distractors. If you pick one confidently and it’s wrong, you’ve found a blind spot, a place where you think you know more than you do. Those blind spots are more dangerous than the things you know you don’t know, because they’re where you’ll accept bad AI output without questioning it.

The quiz didn’t tell us whether our integration approach was sound. It told us whether we could evaluate that question ourselves. The quiz doesn’t test whether the AI is right. It tests whether you can tell if the AI is right.

It’s OK to hit your limit
#

This is the cultural part, and it matters as much as the technique itself. Hitting your limit on the quiz is the desired outcome. The purpose isn’t to prove you’re smart enough to review every AI recommendation. It’s to find the boundary so you can make honest decisions about what happens next.

“We need a networking expert for this part” is a better outcome than shipping code nobody on the team can debug. “This evaluation looks reasonable but I can’t verify the TLS interception claims, so let’s get a second opinion before committing” is a better decision than nodding along because the trade-off analysis sounded convincing.

The quiz gives you a concrete, defensible reason to slow down. Instead of a vague feeling that you might be out of your depth (which is easy to dismiss), you have a score. You answered 1 out of 4 on the networking quiz. That’s not a feeling. That’s a measurement.

Sometimes the quiz tells you the opposite: you do know enough. You answer 4 out of 5, and the one you missed was a genuine edge case, not a fundamental gap. That confirmation is just as valuable, because it means your review of the AI’s output is informed, not performative. Either way, you’re making decisions based on measurement rather than assumption.

When to reach for this
#

Not every AI conversation needs a quiz. The technique is most useful at specific moments:

AI proposes adopting a library or pattern you haven’t worked with. The analysis sounds good, but you’ve never actually used the thing being recommended. Quiz yourself on the concepts you’d need to maintain it.
A technical evaluation pushes into unfamiliar territory. You started evaluating a feature and ended up discussing kernel capabilities or cryptographic protocols. If the conversation has moved beyond your comfort zone, the quiz will confirm it.
You catch yourself skimming AI-generated code. This one is subtle. If you’re reviewing code and you notice you’re reading it like prose rather than tracing the logic, you might not have the domain knowledge to review it properly. A quick quiz on the underlying concepts will tell you.
A decision depends on expertise you’re not sure you have. The AI produced a recommendation. You’re about to share it with the team. Before you do, spend five minutes checking whether you can defend it under questioning.

Appendix: the actual quiz
#

This is the quiz we generated using the prompt above during the OpenShell integration evaluation. It was produced entirely by AI, targeting the underlying Linux networking concepts the team would need to understand. Try it yourself before reading the answers.

Question 1: Network namespace isolation (veth pairs)

You need to create an isolated network namespace for an agent process and connect it to the supervisor via a veth pair. The agent should only be able to reach the proxy at 10.200.0.1:3128. After creating the netns and veth pair, you configure the agent-side interface with ip addr add 10.200.0.2/24 dev veth-s and add ip route add default via 10.200.0.1. The agent still cannot reach the proxy. What did you forget?

A) You need to run ip link set veth-s up and ip link set veth-h up to bring both ends of the veth pair to an operational state
B) You need to add an iptables MASQUERADE rule on the supervisor side so the proxy can NAT the agent’s traffic
C) You need to set net.ipv4.ip_forward=1 on the supervisor’s namespace and add a FORWARD chain rule allowing traffic from the veth-h interface
D) You need to create a bridge device and attach both the veth-h end and the supervisor’s eth0 to it

Question 2: Identifying which process initiated an outbound connection

Your egress proxy intercepts a CONNECT request on source port 48372. You need to determine which binary on the system initiated this connection, so you can apply per-binary network policies. Using only the Linux /proc filesystem (no eBPF, no LD_PRELOAD), what is the correct sequence of lookups?

A) Read /proc/net/tcp to find the socket inode for source port 48372, scan /proc/*/fd/ symlinks to find which PID owns that inode, then readlink /proc/<pid>/exe to identify the binary
B) Read /proc/net/tcp to find the destination IP, look up the IP in /proc/*/net/route to find the owning PID, then read /proc/<pid>/cmdline for the binary path
C) Scan /proc/*/net/tcp per process to find which process has source port 48372 in its network namespace, then read /proc/<pid>/exe
D) Read /proc/net/tcp6 to get the socket’s UID field, map the UID to a username via /etc/passwd, then find all processes owned by that user in /proc/*/status

Question 3: TLS MITM for L7 inspection

You want to add HTTP method/path level inspection to the proxy (e.g., allow GET but block POST to api.github.com). Since GitHub uses HTTPS, you need to TLS-terminate the connection. The agent sends CONNECT api.github.com:443. Your proxy needs to inspect the HTTP request inside the TLS tunnel. What is the correct sequence of operations?

A) Accept the CONNECT, return 200, generate an ephemeral certificate for api.github.com signed by a sandbox-local CA, TLS-accept from the agent using that cert, parse the plaintext HTTP request, evaluate the OPA policy, then open a new TLS connection to the real api.github.com and relay
B) Accept the CONNECT, open a TLS connection to the real api.github.com, perform the TLS handshake with GitHub, extract GitHub’s certificate, re-sign it with the sandbox CA, then present the re-signed cert to the agent and relay all traffic bidirectionally
C) Accept the CONNECT, return 200, then passively read the TLS ClientHello from the agent to extract the SNI field, use the SNI to make the policy decision without decrypting the traffic
D) Reject the CONNECT, redirect the agent to an HTTP (non-TLS) version of the proxy endpoint where the request can be inspected in plaintext, then the proxy initiates the real HTTPS connection to GitHub

Question 4: SSRF protection and DNS rebinding

Your proxy blocks connections to internal IPs (RFC 1918, loopback, link-local, cloud metadata 169.254.169.254). The agent sends CONNECT attacker.com:443. Your proxy resolves attacker.com to 203.0.113.50 (a public IP), passes the SSRF check, and opens a TCP connection. However, the attacker’s DNS server is configured with a 0-second TTL. By the time the TLS handshake completes, DNS resolves attacker.com to 169.254.169.254 (the cloud metadata service). The attacker’s server never completes the TLS handshake, and the agent’s HTTP library retries, this time resolving to the metadata endpoint. What is the correct mitigation?

A) Pin the DNS resolution at the proxy level: resolve once, connect to that IP, and never re-resolve for the lifetime of the connection
B) Add the resolved IP address to the TLS SNI extension so the upstream server must prove it controls that IP
C) Set a minimum TTL floor of 60 seconds in the proxy’s DNS resolver cache, ignoring the server’s 0-second TTL
D) Perform the SSRF check twice: once after DNS resolution and once after the TCP connection is established by checking getpeername() on the connected socket

Scoring

Score	Assessment
4/4	Team has deep networking chops. OpenShell-style features are realistic.
3/4	Solid foundation, but edge cases will slow you down.
2/4	Significant ramp-up needed. Budget for research spikes before committing.
0-1/4	The team needs networking expertise (hire or partner) before attempting this.

Show answers

Q1: A. Newly created veth interfaces default to DOWN state. Both ends must be explicitly brought up with ip link set ... up. B and C would be needed for kernel-level IP forwarding, but OpenShell routes all traffic through an HTTP CONNECT proxy (application-level), not through kernel routing.

Q2: A. The standard /proc lookup chain is: parse /proc/net/tcp to find the inode for the source port, scan /proc/*/fd/ symlinks to find which PID holds that inode, then readlink /proc/<pid>/exe to get the binary path. B is wrong because /proc/*/net/route doesn't map to individual connections. C would work in theory but is expensive (scanning every process's network namespace). D only gives you the user, not the specific binary.

Q3: A. This is the standard TLS MITM approach. The proxy generates a per-hostname ephemeral certificate signed by a CA that the agent's trust store accepts. Between the two independent TLS endpoints, the proxy sees plaintext HTTP and can inspect method, path, headers, and body. C only gives you domain-level filtering (which is what a Squid proxy already does via SNI).

Q4: D. The most reliable mitigation is checking the actual connected IP via getpeername() after the TCP handshake succeeds, not just the resolved IP before connecting. This catches DNS rebinding regardless of caching behavior. A (DNS pinning) helps but doesn't cover all rebinding variants (e.g., dual-stack responses). Note: OpenShell currently does NOT implement DNS rebinding protection. The security audit flagged this as a gap.

Author: Roland Huß AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

Context engineering 101

Sat, 18 Apr 2026 00:00:00 +0000

Everybody talks about prompt engineering. Write better prompts, get better results. That framing was useful once, but it misses the point for coding agents. The prompt is maybe five percent of what determines whether a session goes well or falls apart. The rest is context: what the agent sees when it starts working, how that context evolves over the session, and what happens when it grows too large for the model to track.

Andrej Karpathy named this “context engineering” in mid-2025, and the term stuck because it describes something real. You’re not just writing prompts. You’re engineering the entire information environment the agent operates in. That includes your project structure, your AGENTS.md files, the git state, the conversation history, and everything the agent discovers as it works. Get this right and the agent feels like a capable collaborator. Get it wrong and you’ll spend more time correcting it than doing the work yourself.

What follows are lessons from months of daily Claude Code use. While the examples are Claude Code-specific, the principles apply to any coding agent. Not all of them will apply directly to your setup, but I wanted to share what I’ve learned so far. Maybe some of it saves you a few wrong turns.

Context fatigue is real
#

Large context windows are both a gift and a trap. Claude Code gives you a million tokens of context, and you’d think you’d never run out. In practice, model performance degrades long before you hit the limit. It’s not a cliff, more like a gradual fog. Responses get less precise, the agent starts forgetting instructions from earlier in the conversation, and it suggests approaches you already tried and rejected fifty messages ago.

Give it some rest. A /clear from time to time is the equivalent of a good night’s sleep. Start a fresh session when you switch to a different task, or when you notice the agent struggling with things it handled well earlier. I used to resist this because starting over felt wasteful: all that accumulated context, all those decisions, gone. But a fresh session with a clear problem statement almost always outperforms a tired session with a thousand messages of history.

This matters more with larger context windows, not less. When the window was small, sessions naturally ended before quality degraded too far. With a million tokens, you can keep going well past the point where the agent is still being useful. The context window tells you how much the model can hold. It doesn’t tell you how much it can hold while still being sharp.

There’s a related pattern worth calling out: when you’re stuck on a problem, the worst thing you can do is keep hammering at it in the same session. The agent has already committed to an approach. It’s anchored on assumptions from earlier in the conversation that might be wrong. Open a new session, describe the problem from scratch, and don’t mention what you’ve already tried. A fresh session often finds a solution in minutes that the previous one couldn’t find in an hour, not because the model got smarter but because you removed the accumulated bias. If I’ve spent more than three back-and-forth exchanges on something without progress, I start over. The cognitive cost of re-explaining the problem is much lower than continuing down a dead end.

Understand before you apply
#

This one bit me early. Claude suggests something that sounds sophisticated, maybe a design pattern you haven’t seen before, or an architectural decision that looks reasonable. Your instinct is to accept it and move on, which makes it especially dangerous: in a typical session you accept 80 to 90 percent of proposals unchanged, so unfamiliar patterns slip through easily.

If you can’t explain why a particular approach is the right one, you can’t debug it when it breaks. And it will break, usually at the worst possible time, in a part of the codebase you haven’t touched in weeks. Claude isn’t trying to mislead you. It’s pattern-matching against its training data and producing what statistically looks correct. Sometimes that’s exactly right. Sometimes it’s a plausible-looking solution to a problem you don’t actually have.

The fix is simple: before applying something you don’t fully understand, ask Claude to explain the reasoning. Not “what does this do” but “why this approach over the alternatives.” The explanations are often excellent, and they sometimes reveal that the suggestion doesn’t fit your situation at all. Five minutes of questioning can save you an afternoon of debugging code you never understood in the first place.

Hardening rounds are not optional
#

The first version that works is not the version you should ship. This is true for human-written code too, but with coding agents the gap between “works” and “works reliably” is wider because the agent optimizes for getting something functional as fast as possible. That’s exactly what you asked for, and it’s exactly what will bite you later.

After the initial implementation, always add explicit hardening rounds. What that means concretely: for every error you encounter, have the agent write a test. Not “make this robust” (that’s far too vague and the agent will just sprinkle some error handling around), but “this specific thing broke, write a test that catches it, then fix it.” Each bug becomes a regression test, and over time those tests accumulate into a safety net that actually reflects how the code fails in practice.

Separately, add a refactoring round where you ask the agent to clean up the implementation without changing behavior. Hardening and refactoring are distinct concerns, and mixing them leads to worse results in both.

For UI work especially, expect many rounds. I’ve been building cc-deck (a TUI plugin for Claude Code) for some time, and this is where the pattern shows most clearly. The initial implementation of a feature usually works for the golden path. But the agent isn’t good at keeping things consistent and concise across UI states, so the hardening catches cases where terminal sizes change, sessions disappear mid-update, or multiple events arrive in the same tick. Without those extra passes, users would hit bugs within minutes of real use.

Debug systematically, not by guessing
#

For complex UI flows, whether TUI or web, there’s a technique I keep coming back to. Instead of describing a bug and hoping Claude guesses the right fix, give it actual data.

The process: have Claude add debug logging for event and mode handling. Navigate to your issue in the running application. Ask Claude to take a baseline of the current logs. Then exercise the bug, trigger whatever goes wrong. Now ask Claude to analyze the new events that flowed in after the baseline.

This turns debugging from “here’s a vague description of what went wrong” into “here’s the exact sequence of events that led to the unexpected state.” The agent can trace through the event log, identify where the state diverged from what was expected, and propose a fix based on actual data rather than speculation.

I developed this pattern while working on cc-deck’s session state machine. The sidebar plugin tracks session activity through Init, Working, Done, Idle, and Paused states, with time-based transitions between them. When a state transition fired at the wrong time, describing the problem in English was almost useless. Showing Claude the timestamped event log and saying “the transition from Working to Done happened here, but it shouldn’t have” led to accurate fixes every time.

When the output feels vague, the input was vague
#

So far, these tips have been about managing your session. But there’s also a real-time signal worth paying attention to: the quality of what the agent gives you right now.

You can spot it. When Claude starts hedging, producing long explanations with multiple “alternatively” branches, or suggesting solutions that feel vague rather than specific to your codebase, it’s guessing. It doesn’t have enough context to give you a confident answer, so it’s giving you a probability distribution instead.

This is your signal to provide more context, not to pick the option that sounds best. Show it the relevant code. Point it at the test that’s failing. Give it the error message, the full one, not a paraphrase.

Think of context like a budget. Every piece of information in the context window has a cost, not in tokens but in attention. The more noise in the conversation, the harder it is for the model to focus on what matters. Be deliberate: point the agent at specific files, quote the relevant error, describe what you’ve already ruled out. My best sessions start with a minute of framing the problem, while the worst start with a one-line description and an expectation that the agent will figure out the rest.

This also means cleaning up after yourself. If you tried an approach that didn’t work, say so explicitly. “I tried X and it failed because Y, so don’t go down that path” is more valuable than hoping the agent noticed from the conversation history. And if you’ve been going back and forth without progress, remember the context fatigue lesson: a /clear or a fresh session beats piling more confusion onto an already muddled conversation.

The skill patterns post covers this from the plugin author’s side: how to structure skills and hooks so the agent has the right context at the right time. From the user’s side, the principle is simpler. The old programmer’s wisdom “garbage in, garbage out” applies with full force here. Sloppy context produces sloppy output, and no amount of prompt cleverness will compensate.

Your project IS the context
#

Beyond individual sessions, there’s a layer of context engineering that pays off before you even start a conversation: your project structure itself.

Everything the agent sees when it starts a session shapes how it works. Your instruction files, your directory structure, your naming conventions, your test patterns. If these are messy, the agent’s output will reflect that mess.

This cuts both ways. A well-structured project with clear conventions makes the agent noticeably better. A project instruction file that explains your architecture, coding standards, and common patterns gives the agent a running start on every session. You write it once and it pays off hundreds of times. A cluttered codebase with inconsistent naming, dead code, and no documentation gives the agent bad examples to learn from, and it will faithfully reproduce those bad patterns in new code.

Most coding agents support some form of project-level instructions: CLAUDE.md for Claude Code, .cursorrules for Cursor, .github/copilot-instructions.md for Copilot. The emerging cross-tool standard is AGENTS.md, a vendor-neutral format that any agent can read. Claude Code picks it up too, so you can maintain one file instead of several.

For projects where you want to go further, Spec-Driven Development (SDD) takes this idea to its logical conclusion. SDD starts from the premise that coding agents can only produce reliable output when you structure the context properly. Formal specifications define what to build, a project “constitution” captures architectural decisions and coding standards, and the agent implements from those artifacts rather than from ad-hoc conversation. SDD is applied context engineering with a defined workflow. Tools like spec-kit provide the scaffolding for this approach. It’s more overhead than a simple AGENTS.md, but for larger projects the structure pays for itself quickly. More on SDD in a future post.

Whatever format you use, treat it as living documentation. If the project evolves and the instructions don’t, the agent will drift.

These aren’t permanent rules
#

Context engineering is a young discipline. But the core principle will last: working well with coding agents is about engineering the context, not crafting the prompt. The prompt is the last five percent. Everything else is what makes that five percent effective.

If you’ve found your own patterns for keeping coding agent sessions productive, I’d love to hear about them. Leave a comment below. What works for you? What did you have to learn the hard way?

Author: Roland Huß AIA Ph SeNc Hin R Claude Opus 4.6 v1.0

Claude Code skill patterns

Thu, 09 Apr 2026 00:00:00 +0000

Claude Code skills let you extend a coding agent with custom workflows, specialist knowledge, and automation. You write a SKILL.md file with instructions, and the agent follows them. At least, that’s the idea.

In practice, Claude treats skill content as advice, not as instructions. A skill that says “always use spec-kit to create the specification” might get followed, or Claude might decide it already has enough context from the brainstorming phase to write the spec directly. It’s being helpful, but it’s also wrong. This post describes patterns for dealing with that challenge, from scripts that enforce consistency to hooks that block shortcuts before they happen.

Stronger wording does not help. I tried “MUST”, “ALWAYS”, “CRITICAL”, bold, uppercase. None of it changes the underlying behavior: the model reads your skill, considers it, and then makes its own judgment call. Compare that to hook-injected context (system reminders), which Claude treats as ground truth. This difference in compliance is not a bug, it’s how the architecture works. Understanding it early saves you from writing increasingly aggressive skill instructions that still get ignored.

The patterns in this post come from building cc-prose (voice-consistent writing), cc-spex (spec-driven development on top of spec-kit), and a few others.

The shortcut race
#

This shortcutting behavior keeps coming back and deserves its own section.

You fix one shortcut in your skill instructions, Claude finds another path around it, you patch that one, and it discovers a third. It’s a classic Hase-und-Igel race, and as the skill author you will always be the hedgehog.

In cc-spex, the brainstorming phase collects requirements, explores alternatives, and validates assumptions before the specification phase formalizes everything into a proper spec using spec-kit. But Claude regularly decides that the brainstorming output is “structured enough” and writes the spec directly. From the model’s perspective, this is a reasonable optimization. From the workflow’s perspective, it’s skipping the entire point.

What makes this particularly frustrating is that Claude usually knows it did the wrong thing. Point out the shortcut and it will apologize, acknowledge that it should have followed the workflow, and promise to do better next time, without that actually preventing the next occurrence.

A useful debugging technique is to open a fresh session, show Claude what happened, and ask: “Why did you do this, and how can we prevent it in the future?” The answers are surprisingly insightful, and many of the patterns in this post came directly from those conversations. Then I let Claude fix the skill right there in the same session, while the reasoning is still fresh. Each iteration makes the skill more reliable.

Can you win this race? Not entirely. But you can make shortcuts increasingly expensive for the model by moving critical logic out of skills and into scripts and hooks, where it stops being a suggestion and becomes a constraint.

Skills vs scripts
#

Skills can call out to scripts during execution. This is the escape hatch from the shortcut race: instead of relying on Claude to follow a multi-step procedure described in English, you delegate the deterministic parts to a script that the skill invokes at the right moment. The question then becomes: what belongs in the skill, and what belongs in the script?

Skills give you flexibility, error-forgiveness, and adaptability. The LLM can handle ambiguous input, recover from unexpected situations, and apply judgment, so use them when the outcome benefits from interpretation.

Scripts give you determinism, repeatability, and predictability because a Python script does the same thing every time. Use them when the outcome must be consistent across runs.

One of my plugins has a workflow for processing external PDF reviews: iterate over reviewer comments, map them to source files, assess complexity, and generate fix proposals. This follows a fixed procedure with multiple stages. Early versions used skills for the whole flow, and the result was a different experience every time: sometimes it would skip comments, sometimes it would process them in the wrong order, sometimes it would invent stages that didn’t exist. A session_manager.py script now handles the state management, tracks the interaction state, and stores results. The skill handles the judgment calls within each stage, like assessing whether a reviewer’s suggestion actually improves clarity. The script handles everything else.

The evolution path is predictable: you start with a simple prompt-only skill (just English instructions, fragile by nature) and gradually move state management into scripts as you hit consistency problems. And you don’t write those scripts by hand, either. You tell Claude “let’s create a script to make this more predictable” and let it build the script and wire it into the skill at the right checkpoints. The rule of thumb: every state change should go through a script. Let the LLM do the thinking, and let scripts do the bookkeeping.

Hooks over hopes
#

If skills are advisory and scripts are deterministic, hooks are the bridge. Claude Code’s hook system lets you inject context at specific points in the interaction, and that context arrives as a system reminder, which Claude treats as ground truth. Better skill instructions won’t fix the compliance gap, but hooks can enforce what instructions can’t.

cc-spex uses a PreToolUse hook as a guardrail layer. This hook fires before every tool call and can either allow it, deny it with an error message, or inject additional context. When the plugin runs a multi-stage ship workflow (brainstorm, specify, plan, implement, review, verify), Claude regularly tries to skip stages. A skill instruction saying “complete each stage in order” gets interpreted as a suggestion. The hook makes it a constraint: a state file on the filesystem tracks the current pipeline stage, and the hook checks it before every tool call. If Claude tries to jump from stage 2 (spec review) to stage 6 (implementation), the hook blocks the call and lists every stage it needs to complete first:

PIPELINE DISCIPLINE: You are trying to invoke speckit-implement 
(stage 6: implement) but the pipeline is at stage 2: review-spec. 
You MUST complete these stages first, in order:
 2. review-spec (spex:review-spec)
 3. plan (speckit-plan)
 4. tasks (speckit-tasks)
 5. review-plan (spex:review-plan)
Do NOT skip stages. Do NOT shortcut.

The same hook catches hallucinated commands (Claude inventing /spex:specify which doesn’t exist) and reminds about verification before git commits. A companion UserPromptSubmit hook injects plugin state as a structured system reminder on every prompt, including resolved script paths, project configuration, and session context, so that skills always have accurate state without needing to rediscover it.

Instead of telling Claude what to do and hoping it complies, you build guardrails that make non-compliance impossible. The hooks don’t ask Claude to follow the process, they enforce it at the tool level before Claude’s output reaches the user.

More patterns
#

A few more patterns that came up across my skills.

Multi-specialist over monolith. Instead of one massive skill that tries to do everything, split into independent specialist skills with their own activation rules. One of my plugins uses 12 specialists for different editing concerns: style, consistency, flow, references, links, examples, and more. This keeps individual skills focused and within context budget, because one monolithic skill will lose track of its own instructions.

Prevention over correction. cc-prose generates clean content upfront using voice profiles and AI pattern avoidance, rather than generating text and then running a fixer pass. It’s cheaper to prevent bad output than to detect and fix it after the fact. Not to mention that fixing AI-generated text while using AI tends to be an exercise in whack-a-mole.

Keep skills small. Every line of skill content fills the context window, so if Claude “helpfully” adds optional features to your skill output, remove them. Periodic cleanup during development is important because skill code accumulates cruft faster than regular code.

Configuration hierarchy. Skill defaults in knowledge-base/ get overridden by global user config, which get overridden by project config. cc-prose uses this to layer project-specific voice profiles on top of general writing defaults. Three layers is enough.

Building on a moving platform. Claude Code is transitioning from commands/ to user-invocable skills/, but plugin skills installed from marketplaces still don’t appear in the / slash command autocomplete (anthropics/claude-code#18949, open since January 2026). Build with skills/ as the primary structure but keep thin commands/*.md stubs as a bridge until autocomplete catches up.

The core tension between flexibility and determinism, between skills and scripts, won’t go away with better models. If anything, more capable models will just make the shortcut race more creative.

If you’re building Claude Code skills and have found patterns of your own, I’d like to hear about them.

Author: Roland Huß AIA Ph SeNc Hin R Claude Opus 4.6 v1.0

Blog winter is over

Sat, 04 Apr 2026 00:00:00 +0000

The last post on this blog was about Jib, Google’s daemonless Java image builder. That was July 2018. Almost eight years ago. Anybody remember when that was the latest hotness?

Before that, I wrote about Docker when Docker was still exciting and built a Kubernetes cluster on Raspberry Pi 3 nodes when that was still a weekend adventure. I spent way many words on Jolokia and JMX. 27 posts between 2010 and 2018, then silence. If you’ve been reading tech blogs long enough, you know how that goes.

So what breaks eight years of silence?

Why the silence
#

There’s no dramatic story here, no burnout or life crisis. I just stopped at some point because my writing energy went elsewhere. I co-authored Kubernetes Patterns with Bilgin Ibryam (first edition in 2019, second in 2023). More recently, I finished Generative AI on Kubernetes with Daniele Zonca in 2026.

Writing books is a strange experience. You pour months into a manuscript, and when it ships, you’re proud and drained at the same time. But once the last book was out in March 2026, something shifted. The pressure was gone, the gap between having something to say and sitting down to write it started closing, and I wanted to write shorter, more opinionated pieces again.

What changed
#

The tech world of 2026 looks nothing like 2018. AI-first engineering isn’t a conference buzzword anymore. It’s how I work every day. The way I write code, design systems, and think about developer experience has shifted more in the last year than in the decade before. Last month I scaffolded a complete operator in two days that would have taken two weeks in 2018, and most of my work was writing the specification, not the code.

Context engineering became something I care about deeply. Not in the abstract “prompt engineering” sense, but in the practical “how do you structure specifications so that AI agents produce useful output” sense. I’ve been deep into Spec Driven Development, particularly spec-kit from GitHub. I have opinions about it, even as the whole field is still taking shape.

When you find yourself explaining the same ideas in conversations, Slack threads, and pull request descriptions over and over again, that’s usually a sign you should write them down properly.

What’s coming
#

The topic I keep coming back to is context engineering and Spec Driven Development: how I structure specifications for agentic coding workflows, what works, what fails, and why the “just give it a prompt” approach misses the point. This will probably be a recurring theme, because the field is moving fast and there’s a lot to figure out in public.

Close behind is AgentOps on Kubernetes. Running agentic workloads on Kubernetes and OpenShift is a different beast than classic web services. Agents are unpredictable, long-running, and resource-hungry. They talk to the outside world in ways that make security teams nervous. I’m ramping up on this topic professionally, figuring out how to operate these workloads in a secure and scalable way. Expect posts about the particular demands of AI agents and why your existing Deployment patterns won’t cut it.

Beyond those two, expect posts about AI-first engineering in daily practice (the surprising wins, the things that still don’t work), agentic coding projects and tools, the home K3s cluster that’s been running on five Raspberry Pi 4 nodes for over five years, book-adjacent Kubernetes patterns, and whatever else catches my attention.

On AI and this blog
#

Since AI changed how I work, it naturally changed how I write too. I use AI tools for ideation, polishing, and managing the publishing workflow. The thinking and the opinions stay mine. Every post carries an AI attribution tag at the bottom so you know exactly what role AI played. I apply the same principle I advocate in the Responsible Vibe Coding Guide: use AI as a tool, but own the result. I’ll write more about the process in a future post.

Let’s see
#

I’m not going to promise a posting schedule. That kind of commitment didn’t work last time, and I see no reason to repeat it. But I have more things to write about now than at any point during the old blog’s run. Eight years ago, Jib was the latest hotness. The next post won’t take that long.

AIA HAb CeNc Hin R Claude Opus 4.6 v1.0

First look at Jib

Wed, 11 Jul 2018 00:00:00 +0000

As soon as Google’s blog post “Introducing Jib — build Java Docker images better” was online, all my channels went crazy about Jib. That was a bit surprising as Jib was started over one year ago but with this blog post this project rockets with more than 1000 new GitHub stars within one day. Crazy.

I got a lot asked yesterday how Jib compares to fabric8io/docker-maven-plugin (d-m-p) or fabric8io/fabric8-maven-plugin which includes d-m-p.

Let me try to shed some light on the differences and pro and cons of both approaches.

How Jib works
#

Let’s first have a quick look what Jib offers today and what makes it unique.

Looking at the Jib Maven plugin it currently supports three goals:

jib:dockerBuild for assembling an image and loading it to a Docker daemon.
jib:build for assembling the image and pushing it to a Docker registry.
jib:exportDockerContext for creating a Dockerfile along with all the files required for performing a build against a Docker daemon.

The unique asset of Jib is that it does all of this without consulting a Docker daemon by creating all image layers and metadata locally, conforming to either the Docker or OCI format. And that all directly with plain Java code.

Also, Jib assumes a very opinionated project setup with a so-called flat classpath app with main class, dependencies and resources all in different artefacts. Compare this to fat jars popularised by Spring Boot, where the application code, resources and dependencies are all stored in the same jar file. There are some drawbacks of flat classpath app, but one benefit is, that you can organise the various files into several layers in your image, putting the one which is changing less (like dependencies) at the bottom of the layer stack. That’s what Jib does: It puts all dependency jars into one layer, all resource files (like property files to be loaded from the classpath) in another and the application classes into a third layer. All of these layers get aggressively cached locally. That way, you can be much faster when recreating images than with fat jars which apparently can be stored only in one single layer.

But let’s have a look how Jib works in detail. The steps performed by a jib:dockerBuild or jib:build are:

Fetch base image’s layers and cache them locally. By default the base image is gcr.io/distroless/java, but this can be configured.
Create three application image layers for
- Dependencies
- Resources
- Classes
Since these layers are cached, if any of them doesn’t change (which is likely for dependencies), then the layer is not recreated.
Create the application image locally. The two previous steps were performed asynchronously; this step continues when both previous steps have been finished. The ENTRYPOINT of this image is fixed set to:
```
 java <jvm-flags> -cp dep_dir/*:resource_dir/:class_dir/ <main-class>
```
where <jvm-flags> can optionally be configured and <main-class> is the mandatory main class to specify. The information leading to the classpath comes from the underlying Maven or Gradle project information. Also, any Java arguments configured to become the CMD of the image, as well the exposed ports (EXPOSE) can be added, too.
Finally the local image layers along with its meta-data is tarred up, and either loaded into a Docker daemon or pushed to a Docker registry.

How does Jib compare to d-m-p ?
#

Jib is impressive for the use case it supports and brings a fresh spin to the way how Java apps can be packaged into Docker images.

The most significant benefits of Jib are:

Fast for incremental builds when you have a flat classpath application, resulting in three different layers for dependencies, resources and your application classes.
No Docker daemon required, which reduces the build requirements and increases security because the image creation happens without root permissions.
Produces reproducible images by wiping out file owner and modifications date. However, not sure whether e.g. generated timestamps in resource files like properties are wiped out, too.
It supports both, Maven and Gradle.

However, there are also some limitations. Some of them might be tackled in the future, but other might not be changed due to the unique way how Jib works:

Can be only used for simple flat classpath Java applications. There is currently no support for fat jars (i.e. Spring Boot fat jars) nor other packaging formats like WAR file.
Simplistic startup of the application with a plain java call instead of using a full features startup script like run-java-sh.
No additional files like configuration files outside the classpath or agents like jmx_exporter agent can be added (but there is a PR pending for agents).
Fixed classpath order, e.g. doesn’t allow for overwritten resources in dependencies as dependencies are always first on the classpath.
Jib uses a custom XML configuration syntax instead of plain Dockerfile syntax (which I often heard as a major critique about d-m-p which also supports a custom XML configuration, but as alternative to Dockerfiles).

d-m-p provides some additional features which are not supported by Jib, like

Running containers for integration testing (that’s very likely the most prominent difference)
Dockerfile support
docker-compose.yml support
Enhanced authentication support OpenShift and Amazon ECR support
Support for watching code change and then automatically triggering a rebuild of images and restart of containers
Support for arbitrary assembly and base images, including Spring Boot fat jar and JavaEE containers.
Healthchecks

And if you jump to fabric8-maven-plugin, which includes d-m-p for its image building business, you have even more high level features like a zero config mode which analyses your pom.xml and selects opinionated defaults like base images and handcrafted startup scripts, depending on the type of tech stack you are using (like Spring Boot, Thorntail, Vert.x, Tomcat, …)

Next steps …
#

This overview is only a quick glance on Jib. In one of the next posts, I plan to show some real-life examples and also measuring the performance gain by using Jib.

Also, there a plans for d-m-p to add support for Jib backend and other daemonless build systems like img, buildah or kaniko. The mid to longterm plan is to enhance the build abstraction within d-m-p and offer, based on the Java project given, different ways to build images.

BTW, if you are interested in what’s going on in Docker image building business these days, you probably might find this KubeCon presentation as useful as I did. Daemonless FTW :)

Psst, d-m-p also likes GitHub ★ ;-)

docker-maven-plugin might be still useful

Thu, 05 Jul 2018 00:00:00 +0000

Yesterday a blog post Using Docker from Maven and Maven from Docker by Kostis Kapelonis was published which gives some insights on the possible relationships between Docker and Maven. The article makes some essential points really, and gives an overview for the two remaining Docker Maven plugins as well as how Codefresh recommends doing Docker multi-stage builds as the alternative. As I’m the maintainer of the fabric8io/docker-maven-plugin, I’d like to comment on this matter.

I already commented on the original blog post (thanks for approving the comment), but I’m happy to repeat my arguments here again.

The article ditches two docker-maven-plugins before it promotes Docker multi-stage builds for some reasons.

To be honest, I think both approaches have their benefits, but let me comment first on two arguments given concerning the fabric8io/docker-maven-plugin.

There have been cases in the past where Docker has broken compatibility even between its client and server, so a Maven plugin that uses the same API will instantly break as well.

This compatibility issue might be right especially if you use a typed approach to access the Docker REST API which is used by various Docker client libraries. As explained in the post, fabric8 d-m-p accesses the Docker daemon directly without any client library and with not marshalling. This is because it accesses only the parts required for the plugin’s feature set, which also means that json responses are handled in a very defensive and untyped way.

And yes, there was one issue in the early days in 2014 with a backwards-incompatible API change from Docker. This issue could be fixed quite quickly because d-m-p hadn’t to wait for a client library to be updated. However, since then there never has been an issue and for the core functionality that d-m-p uses.

I think the relevance of Docker API incompatibilities is exaggerated in this blog post.

Hopefully, the fabric8 plugin also supports plain Dockerfiles. Even there, however, it has some strong opinions. It assumes that the Dockerfile of a project is in src/main/docker and also it uses the assembly syntax for actually deciding what artefact is available during the Docker build step.

That is simply not true. You can just put a Dockerfile on the same level as the pom.xml, refer to your artefacts in the target/ directory (with Maven property substitution), and then declare the plugin without any configuration. See my other blog post for a short description of how it works.

BTW, the reason for the own XML syntax is a historical one. The plugin started in 2014 when Dockerfile was entirely unknown to Java developers. But Maven plugin XML configuration was (and still is) a well-known business. As time passed by and Docker become more and more popular for Java developers, the Dockerfile syntax is well known now these days, too. So, I completely agree, that you should use Dockerfiles if possible, and that’s why the plugin supports Dockerfiles as a first-class citizen since the recent versions. The next step is to add similar support for docker-compose.yml files for running containers. There is already docker compose support included, albeit a bit hidden.

I agree that multi-stage Docker builds are fantastic for generating reproducible builds, as the build tool (Maven) is used in a well-defined version. However, using a locally installed Maven during development has advantages, too. E.g. the local Maven repository avoids downloading artefacts over and over again, resulting in much faster build times and turnaround times. Of course, you can add caching to the mix for multi-stage builds, but then the setup gets more and more involved. Compare this to using a d-m-p for which you don’t even need a local Docker CLI installed, and you can ‘just start’. For CI builds this properly doesn’t matter much though (and that’s what the blog post is all about I guess).

Other advantages of using fabric8’s d-m-p :

Running all your containers (app + deps) locally without the need of support from a CI system. As a side note, the custom compose-like syntax of codefresh’s CI is not so much different to the custom configuration syntax of fabric8’s d-m-p. It’s custom.
Extended authentication support against various registries (Amazon ECR, Google GCR, …)
Automatic rebuilds during development with docker:watch which increase turnaround times tremendously
Download support files (e.g. startup scripts) automatically by just declaring a dependency in the plugin (blog post pending)
…. and even more stuff.

In the end, your mileage may vary, but having an article conclusion without really trying to compare pros and cons of both approaches is far too biased for me.

Update: Kostis replied to my comment, and an interesting discussion is going over there

Elegant Camel route configuration

Tue, 03 Jul 2018 00:00:00 +0000

I’m a big fan of the Camel Java DSL for defining Camel routes with a RouteBuilder. This is super easy and slim. However, in this blog post I show you a nerdy trick how this can be done even more elegant.

If you are a Camel user, you know, that defining a route for a given Camel context ctx ist just a matter to implement the configure() method of the abstract RouteBuilder class:

ctx.add(new RouteBuilder {

 @Override
 public void configure() throws Exception {
 from("file:data/inbox?noop=true")
 .to("file:data/outbox");
 }

});

Its really simple and you can use the whole Camel machinery from within your configure() method.

However, this kind of configuration can be performed even simpler. Let’s assume that you have a no-op default implementation of RouteBuilder called Routes:

public class Routes extends RouteBuilder {
 @Override
 public void configure() throws Exception { }
}

Then, the configuration can be rewritten simply as

ctx.add(new Routes {{
 from("file:data/inbox?noop=true")
 .to("file:data/outbox");
}});

This trick just uses Java’s object initializers, a not so well known language feature. The inspiration for providing the DSL context like this comes from JMockit which defines its mock expectations the same way. I think object initializers are really an elegant albeit hipster way to implement DSLs.

Although you can easily define the Routes class on your own, you might vote for this Camel issue or pull request if you want to have this in upstream Camel, too.

When a Dockerfile is just good enough

Thu, 12 Apr 2018 00:00:00 +0000

As you might know, one of my Open Source babies is the one and only fabric8io/docker-maven-plugin (d-m-p). If you already use this Maven plugin, you know, that it is super powerful and flexible to configure. This flexibility comes at a price so that the configuration can become quite complicated. Now, if you only want to build Docker images with Maven, I have good news: Since 0.25.1 d-m-p supports a zero XML configuration mode, the so-called Simple Dockerfile Build mode.

The idea of this mode started with a Twitter discussion:

And actually, it’s true: If all that you want is to build a single Docker image from a Dockerfile, then the initial configuration is indeed too complex.

d-m-p already supports plain Dockerfiles for quite some time, and that even for multiple images. However, you still have to reference those Dockerfiles in the XML configuration of the plugin.

Since 0.25.1 you can now use the so-called Simple Dockerfile Build mode (kudos go to Rohan Kumar for the initial implementation). All you have to do is to add d-m-p to your pom.xml and add a Dockerfile. The smallest possible Maven project for creating a Docker image consists of this pom.xml

<project>
 <modelVersion>4.0.0</modelVersion>

 <groupId>fabric8</groupId>
 <artifactId>smallest</artifactId>
 <version>1-SNAPSHOT</version>

 <build>
 <plugins>
 <plugin>
 <groupId>io.fabric8</groupId>
 <artifactId>docker-maven-plugin</artifactId>
 <version>0.26.0</version>
 </plugin>
 </plugins>
 </build>
</project>

and a Dockerfile alongside this pom:

FROM busybox
CMD ["echo", "Hello", "world!"]

This image does not really much. With mvn docker:build you can build it, with docker run fabric8/smallest you can test it. Or use mvn docker:run so that you even don’t have to provide the image name.

A more realistic Dockerfile could look like

FROM openjdk:jre

ARG jar=target/app-1.0.0-SNAPSHOT.jar

ADD $jar /app.jar
CMD java -cp /app.jar HelloWorld

where we define jar as build arg in the Dockerfile but also as property in the pom.xml:

<properties>
 <jar>${project.build.directory}/${project.build.finalName}.jar</jar>
</properties>

You can use Maven properties in the Dockerfile which get automatically replaced by docker:build when creating the image. But you can use that Dockerfile also without Maven with docker build . You can’t use Maven properties with . directly as dots are not allowed in Docker build args, therefore we use an extra property. However, a maven-less usage probably does not make much sense when you don’t also build the artefacts. The full example can be found in the dmp GitHub repo.

If you can forgo Docker build args you can use predefined Maven properties directly:

FROM openjdk:jre
ADD ${project.build.directory}/${project.build.finalName}.jar /app.jar
CMD java -cp /app.jar HelloWorld

The image name is auto-generated, but you can set this name also by yourself by setting the property docker.name (and you can use placeholders within this name)

You can even start the container with mvn docker:run although without any additional configuration (e.g. like port mappings). Also, you can docker:push the image.

You can still configure certain aspects like authentication or bind d-m-p goals to default lifecycle phases. Using this mode is very similar to the functionality offered by spotify/dockerfile-maven.

If you need more horsepower, you can gradually expand on this simple setup. Features which are waiting to be discovered are:

Setup of multiple images for running integration tests
Custom networks and volumes for your tests
Using docker-compose files for running the containers directly from the plugin
Exporting the Docker image as an archive
Watch docker containers and restart them when the code changes
All can be configured via properties, too and since with the latest versions, you can mix it with XML configuration.
….

If you are interested in more to find out, then please have a look at the reference manual.

I’m curious what you think about this new mode. Please use the comments below if you want to leave some feedback. In fact, there are concrete plans for d-m-p to include the generators from the fabric8-maven-plugin functionality of autodetecting the tech stacks used for creating opinionated Docker images.

Bringing Octobox to OpenShift Online

Sun, 25 Mar 2018 00:00:00 +0000

Octobox is for sure one of my favourite tools in my GitHub centred developer workflow. It is incredible for GitHub notification management which allows me to ignore all the hundreds of GitHub notification emails I get daily.

Octobox is a Ruby-on-Rails application and can be used as SaaS at octobox.io or installed and used separately. Running Octobox in an own account is especially appealing for privacy reasons and for advanced features which are not enabled in the hosted version (like periodic background fetching or more information per notification).

This post shows how Octobox can be ported to the free “starter” tier of OpenShift Online.

Application setup
#

An Octobox installation consists of three parts:

Octobox itself, a Rails application
Redis as an ephemeral cache, used as a session store
Postgresql as the backend database

Naturally, this would lead to three services. However, as I’m not striving for an HA setup and the sake of simplicity, I decided to combine Octobox and Redis in a single pod. Since a combined lifecycle for Octobox and Redis is a reasonable, fair choice, this reduces the number of OpenShift resource objects considerably.

As persistent store for Postgres, we use a plain PersistentVolume which is good enough for our low-footprint database requirements.

Docker Images
#

To get an application onto OpenShift, you first need to package all parts of your application into Docker images which eventually become container during runtime.

There are some restrictions for Docker images to be usable on OpenShift. The most important one is that all containers run under a random UID, which is part of the Unix group root. This restriction has the consequence that all directories and files to which the application process want to write should belong to group root and must be group writable.

Octobox already is distributed as a Docker image and has recently be [updated][octobox-dockerfile-chgrp] to be OpenShift compatible. The Postgres image is directly picked up from an OpenShift provided ImageStream, so there is no issue at all. The Redis Imagee is also already prepared for OpenShift However, when using Redis from this image in an ephemeral mode (so not using persistence) there is a subtle issue which prevents starting the Pod: As the Dockerfile declares a VOLUME and even though in our setup we don’t need it, we have to declare a volume in the Pod definition anyway. Otherwise, you end up with a cryptic error message in the OpenShift console (like can't create volume ...). An emptyDir volume as perfectly good enough for this.

Template
#

To install the application an OpenShift Template has been created. It contains the following objects

DeploymentConfigs for “Octobox with Redis” and “Postgres”
Services for Octobox and Postgres
PersistentVolumeClaim for Postgres

A route for accessing the app is created later on the OpenShift console. Please refer to these installation instructions for more details how to use this templats.

OpenShift Online Starter
#

OpenShift Online Starter is the free tier of OpenShift online which is very useful for learning OpenShift concept and get one’s feet wet. However, it has some quite restrictive resource limitations:

1 GB Memory
1 GB Storage

This budget is good enough for small applications like Octobox, but if you need more horsepower than you can easily upgrade to OpenShift Online Pro.

The challenge is now to distribute the three parts (Octobox, Postgres, Redis) over these 1 GB. As Octobox as rails application is quite a memory hog, we want to dedicate as much memory as possible to it. For Postgres, we do not need much Memory at all, so 50 to 100 MB is good enough. The same for Redis as an initial guess. We can always tune this later if we found out that our initial guess a wrong.

Ok, let’s start with:

875 MB Octobox
50 MB Redis
75 MB Postgres

When trying out these limits, I quickly found out that this doesn’t work. The reason is that OpenShift Online has a minimum size for a container which is 100 MB. Also, you can’t choose requests and limits freely, but there is a fixed ratio of 50% to calculate the request from a given limit (the request specified is always ignored). This fact not only means that you get a Burstable QoS class, but also that you have to specify 200 MB as limit to get at least 100 MB request to exceed the required minimum.

So we end up with:

600 MB Octobox
200 MB Redis
200 MB Postgres

Apparently, this is not optimal, but that’s how it works for OpenShift Online Starter tier (and probably also the Pro Tier). For other OpenShift cluster it, of course, depends on the setup of this specific cluster. We could put Redis and Octobox in the same container, and start two processes in the container. This change would free up another 150 MB for Octobox but is ugly design. So we won’t do it ;-)

tl;dr
#

Porting an application to OpenShift is not difficult. Especially the free OpenShift Online Starter is very appealing for such experiments. The challenges are mostly around creating proper Docker images and getting resource limits right. As a result, you get a decent running and managed installation.

For the full installation instructions, please refer to the OpenShift specific Octobox installation instructions.

Dash2Alexa - Amazon Alexa Audio API Access

Mon, 12 Mar 2018 00:00:00 +0000

Since I got my first Amazon Echo end of last year, I love it. And although, as a typical German, I’m still a bit concerned about data privacy, at the end, convenience wins (as always :). There are many things which work flawlessly, and to be honest, the most used feature for me is a simple timer. But when it comes to aggregate actions, Alexa is still quite limited. Ok, you can define your routines, but for only an insufficient set of fixed actions. What I really would love to have is to start the radio when I get up in the morning, but this is not possible at the moment.

So I remembered my last years Amazon Dash button hacks and thought it would be cool to combine both, the Dash button and Alexa.

And here it is, my weekend hack …..

In a nutshell, the setup looks like:

Configure your router to not forward packets from your Dash Button.
Spoof on ARP requests for the Dash button’s MAC.
If found, call out to a text-to-speech service to convert configured Alexa commands to audio.
Play the received audio output via RaspberryPi attached speakers.

That’s it. You can use this sample code for doing the dirty work, but maybe you are interested in some more details.

Amazon Dash Button
#

Amazon Dash Button is part of Amazon’s consumer goods ordering service. This button contains a Wifi sender and is quite inexpensive. Each button is specific for a brand for which you can connect to a specific good. When you press the button, this good is ordered (e.f. 24 cans of beer ;-)

But this intended use case is not the only way how you can use this button. In fact, it can be used just as a plain Wifi button for any purpose.

First of all, you have to buy such a button e.g. 5 Euro here in Germany, but you can spend this five bucks for your first order. You just configure it as described by Amazon and maybe order something to spend your credits.

After this, you have to block the button in your home Wifi router for calling out to the internet. For obvious reasons, this is very important ;-) When the button is blocked, it will flash red eventually when being pressed (in contrast to flashing green when an order is placed).

When you press the button, it first asks via DHCP for an IP address. The MAC address of the button is relevant, so its time to pick that up, e.g.

By looking into your DHCP servers log
By trying arp -a
By checking your Wifi Router’s admin UI
Via Wireshark spoofing

This Mac address will be watched for later by spoofing ARP package traffic. When golang is your preferred programming language, then you can use directly rhuss/dash which is based on top of gopacket to watch for certain ARP packages and trigger an action when received.

Amazon Polly API access
#

In our use case, when we detect that a button is pressed, then we want to send out some fixed, text-based audio. For converting text coming from a configuration, a text-to-speech service is used.

There are several such services available. For our purpose, we are using Amazon Polly, which offers a free tier for the first 12 months (including 5 million characters per month, fair enough for a handful of buttons ;-) and then 4 $ per one million of chars.

A short cost calculation beyond the free tier: 100 characters for Alexa commands per button press (which is already quite a mouthful) costs ~ 0.4 cents. Or the other way round, for five bucks you can press the button 1.250 times: Three times a day per yeare for 5 $ Well, for me that’s worth the fun ;-)

You need an AWS account to access the speech API. The access and secret token can be either that of your root AWS account, or you probably should create a dedicated IAM User.

Of course, you can also use different text-to-speech tool. Maybe even the good old Unix speak will do it ? Not tried it yet, but will check that for sure very soonish. For now, the Polly voices are recognised quite well by my echo so that I won’t change it right now.

Raspberry Pi Audio
#

The final jigsaw piece is the hardware on which to run the watcher. For my use case, a Raspberry Pi 2 with some inexpensive speakers was totally good enough.

dash2alexa command
#

The dash2alexa command actually takes a configuration file (defautl: ~/.dash2alexa.yml)

# Sample configuration file for dash2alexa
# Adapt and copy it to ~/.dash2alexa.yml

# Access and secret for accessing the services
access: ".........."
secret: ".........."

# Network interface to listen for ARP requests when a Dash button is pressed
interface: "wlan0"

# Language ("de" or "en")
language: "de"

# Gender which can be either "male" or "female"
gender: "male"

# Keyword to use for alexa
keyword: "Alexa"

# Player to use when playing an mp3 sound file
player: "mpeg123"

# List of Dash Buttons with associated Alexa command
buttons:
 # Symbolic name
- name: "heineken"
 # Mac adress of Dash button
 mac: "ac:63:be:00:11:22"
 # How many seconds to wait between Alexa commands
 wait: 4

 # Messages to talk
 messages:
 - "Lautstärke 4"
 - "Spiele Bayern 3"

There’s not much documentation yet, but some will follow soon. Feel free to adapt the code, and I’m happy to integrate any pull requests. Also, as I’m a bloody golang greenhorn still, would be curious whether things could be done better.

tl;dr
#

This little hack uses Amazon Echo via its ‘Audio API’ to perform specific action on a button press. It’s entirely suited for a situation when its calm around, like putting the button right beside the bed for getting started even when your Echo is out of sight.

And finally, it’s just pure fun ;-) Enjoy!

P.S. Let me know in the comments whether you tried it out, too and how it works for you.

RasPi 3 Kubernetes Cluster - An Update

Wed, 05 Apr 2017 00:00:00 +0000

Our Ansible Playbooks for installing Kubernetes on a Raspberry Pi Cluster have been constantly updated and are now using the awesome kubeadm. The update to Kubernetes 1.6. was a bit tricky, though.

Recently I had the luck to meet Mr. @kubernetesonarm Lucas Käldström at the DevOps Gathering where he demoed his multi-arch cluster. That was really impressing. Lucas really squeezes out the maximum what is possible these days with Raspberry Pis and other SOC devices on the Kubernetes platform. Please follow his Workshop on GitHub for a multi-platform setup with ingress controller, persistent volumes, custom API servers and more.

Needless to say that after returning home one of the first task was to update our Ansible playbooks for updating to Kubernetes 1.6 on my RasPi cluster. The goal of these playbooks are a bit different than Lucas workshop setup: Instead of living at the edge, the goal here is to provide an easy, automated and robust way to install a standard Kubernetes installation on a Raspberry Pi 3 cluster. kubeadm is a real great help and makes many things so much easier. However there are still some steps to do in addition.

After following the workshop instructions it turned out soon, that it was probably not the best time for the update. Kubernetes 1.6. has just been released and it turned out that last minute pre-release changes broke kubeadm 1.6.0. Luckily these were fixed quickly with 1.6.1. However the so called self hosted mode of kubeadm broke, too (and is currently still broken in 1.6.1 but should be fixed soon). So the best bet for the moment is to use a standard install (with external processes for api-server et. al).

Also this time I wanted to use Weave instead of Flannel as the overlay network. In turned out that this didn’t worked on my cluster because every of my nodes got the same virtual Mac address assigned by Weave. That’s because this address is calculated based on /etc/machine-id. And guess what. All my nodes had the same machine id 9989a26f06984d6dbadc01770f018e3b. This it what the base Hypriot 1.4.0 system decides to install (in fact it is derived by systemd-machine-id-setup from /var/lib/dbus/machine-id). And every Hypriot installation out there has this very same machine-id ;-) For me it wasn’t surprising, that this happened (well, developing bugs is our daily business ;-), but I was quite puzzled that this hasn’t been a bigger issue yet, because I suspect that especially in cluster setups (may it be Docker Swarm or Kubernetes) at some point the nodes need their unique id. Of course most of the time the IP and hostname is enough. But for a more rigorous UUID /etc/machine-id is normally good fit.

After knowing this and re-creating the UUID on my own (with dbus-uuidgen > /etc/machine-id) everything works smoothly now again, so that I have a base Kubernetes 1.6 cluster with DNS and proper overlay network again. Uff, was quite a mouthful of work :)

You find the installation instructions and the updated playbooks at https://github.com/Project31/ansible-kubernetes-openshift-pi3. If your router is configured properly, it takes not much more than half an hour to setup the full cluster. I did it several times now since last week, always starting afresh with flashing the SD cards. I can confirm that its reproducible and idempotent now ;-)

The next steps are to add persistent volumes with Rook, Træfik as ingress controller and an own internal registry.

Feel free to give it a try and open many issues ;-)

Why Jolokia is not RESTful

Thu, 03 Nov 2016 00:00:00 +0000

From time to time people come to me and say: “I really would love Jolokia if only it would be RESTful”. This post tells you why.

I really like REST, yes I do. If I would crete a new application on a green field, its remote access API would very likely obey the REST paradigm.¹

However, Jolokia is a different beast. It is a bridge to the world of JMX, providing an open minded alternative to the rusty and Java specific JSR-160 standard. Its protocol is based on JSON over HTTP, so in principle it could be REST. But it is not, mainly for the following two reasons:

JMX resource naming is a mess
#

Jolokia doesn’t not have any influence on the naming of the resources it accesses. These resources are JMX MBeans and their identifiers are ObjectNames. ObjectNames have a certain structure but beside this they can be named arbitrarily. So if you want to provide an HTTP API for accessing these repositories, this free form addressing poses some challenges, especially for read operations with GET. For example it is impossible to transmit a slash (/) or backslash (\\) as part of an URL’s path info. The reason is security related, and each application server handles this differently: Tomcat for example completely rejects such requests whereas Wildfly / Undertow refuses to URL decode %2F (for /) and %5C for \\. Jetty doesn’t care much. So in order to address a JMX MBean which contains these characters as part of their names, the typical encoding as part of an URL path doesn’t work. One could use query parameters for this kind of addressing and in fact, Jolokia supports this, too. But it’s still ugly. Also, implementers of MBeans tend to put semantic information into the MBean name like the port of a connector or the name of a database scheme. It can’t excluded that the MBean name alone can carry sensitive information. However GET urls are not secured via the transport protocol and tend to end up in log files. So, its much safer to send these requests via POST, even when only performing read operations on JMX attributes.

Bulk requests
#

A special feature of Jolokia are Bulk Requests. This allows a very efficient monitoring of multiple values with a single HTTP request. It works by sending a list of individual, JSON encoded Jolokia requests with a single HTTP POST request. That list can contain any valid Jolokia operation: Reading and writing attributes, executing some operations, searching for or listing of MBeans. The heterogenous nature of this kind of requests makes it hard to map them to one single HTTP verb as REST suggests. Also, the sheer length of the request parameter forbids to send a bulk request via GET as Servlet container or other application servers impose certain restrictions on the length of an URL, which vary however from server to server.

Jolokia implements both
#

For every Jolokia operation, we play both: GET and POST ². As an integration tool, which helps to bridge different worlds without really having control over these worlds, the focus is on maximal flexibility so that it can adapt to any environment where it is used. REST is only of second importance here, but if you think the issues described above can be solved in a more RESTful way, I’m more than open.

I’ve to confess that I’m really not a REST expert, so if you don’t agree with my arguments, I’d kindly ask you to leave a comment or tweet me for corrections. ↩︎
or: “Country and Western” ↩︎

Java EE Management is dead

Mon, 10 Oct 2016 00:00:00 +0000

Now that some weeks has been passed we all had time to absorb the revised Java EE 8 proposal presented at Java One. As you know, some JSRs remained, some things were added and some stuff was dropped. Java EE Management API 2.0, supposed to be a modern successor of JSR 77, is one of the three JSRs to be dropped.

What does this mean for the future of Java EE management and monitoring ?

First of all it’s fair to state that JSR 373 never really took off. Since February 2015 there were not more than 86 mails on the expert group mailing list, half of them written in March 2015 during incubation. At the latest of January 2016 it was clear that JSR-373 is not on Oracle’s focus anymore. To be honest, even we members of the expert group, we were not able to push this JSR further.

How did it come that far ? Let’s have a look back into history.

All starts with JSR 3 back in 1999. This first JMX specification is the foundation of all Java resources management. As it can been seen by its age, Java folks took care about Management and Monitoring from the very beginning on. And even better, since JS2E 5 JMX is integral part of Java SE so its available on every JVM out there.

Over the years, additional JSRs were added on top of this base:

JSR 160 defines a remote protocol for JMX,, which is based on RMI. This might have been a good decision in 2003, but turned out to be awful to use especially for non-Java based monitoring systems.
JSR 262 was started to overcome this by defined a “WebServices Connector for Java Management Extensions Agents” which was mostly around SOAP services. However although even an initial implementation existed, it was withdrawn before the final release. It’s not completely clear why it was stopped in 2008 and later withdrawn, as the public review ballot has been approved, although it was a tight result. The biggest objections were on dependencies on “proprietary” WS-* specifications.
“J2EE Management” JSR 77 was finished in 2002 and defines a hierarchy how Management and Monitoring resources exposed by a Java EE server is structured. It allows a uniform interface for how to access the various Java EE resources, like web applications or connector pools. Beside this it also defines how statistics are exposed by defining various metrics formats. However, implementing the StatisticsProvider model is not mandatory and from my personal experience it was implemented only rarely by some vendors and if so, not for every resource.
JSR 88 complements JSR 77 and defines a common format for deploying Java EE artefacts.
JSR 255 was started to be the next version of JMX and supposed to be included in Java 7. Although it was already nearly finished and integrated, it didn’t make it into Java 7 (nor Java 8). The spec was then dormant until it was finally withdrawn in 2016.

With the dead of JMX 2.0 in 2009 the evolution of JMX as a standard for Java SE has stalled. But what’s about Java EE Management ? At least JSR 77 is still part of Java EE 7 and for Java EE 8 the successor was supposed to be JSR 373. JSR 373 tackles the problem of remote access, whereas JSR 77 still relies on RMI as a standard implementation protocol as defined in JSR 160.

The two major goals of JSR 373 were:

Provide an update of the hierarchal resource and statistics structure as defined by JSR 77
Provide a REST access to these resources independent of JMX

In the often cited Java EE 8 Community Survey more than 60% were in favour of defining a new API for managing application, which should be based on REST (83% pro-votes). This finally lead to JSR 373. However, as it seems in retrospective, a deep interest in this topic was not really given and probably lead to this final decision to drop JSR 373 from Java EE 8.

So, what is the state of Monitoring and Management of Java and in particular Java EE applications nowadays and what can be expected in the future ? Let’s have a look into the crystal ball.

JMX is here to stay. It is part of Java SE and I don’t know of any plans for removing it from future Java editions. Ok, it feels a bit rusty but it is still rock solid and gives you deep insight in the state of your JVM. With tools like Jolokia you can overcome most of the restrictions JSR 160 imposes. (Disclaimer: since I’m the author of Jolokia all my personal opinions given here should be evaluated in this light :)
It is not clear how the Management API of Java EE 8 and beyond looks like. It does not look like that JSR 77 will survive. Will there be a standard for Java EE management at all ? Probably not, and so there is the danger that vendors will push their proprietary management APIs, which already happens to some extent. Luckily, most of these proprietary APIs are also mirrored in JMX these days.
On the other hand, it could be also a good thing that there is no other Management API which is not based on JMX. That’s because you will always need JMX to monitor the basic aspects like Heap Memory usage or Thread count, which are covered by Java SE. Adding a different, REST like protocol for Java EE monitoring requires operators to access a Java EE server with two different protocols (JMX and Rest), duplicating configuration efforts on the monitoring side. This can only be avoided if the Java EE resources are mirrored in JMX, too.

To sum it up, I think its a shame that Management and Monitoring, which played a prominent role over the whole evolution of Java EE, will probably be dropped completely in Java EE 8. As a replacement the new Health Check API has been announced, but to be honest, that can’t be a full replacement for classical management and monitoring where the evaluation of a system’s health is done on a dedicated monitoring platform (e.g. like Nagios or Prometheus). These platforms take the plain metrics data exposed by the application and does the data evaluation on their own.

The good thing is still that you have JMX to the rescue and I’m pretty sure that this technology will survive also this storm. Especially if vendors are willing to support it for their application server metrics, too.

Even without a Java EE standard.

A Raspberry Pi 3 Kubernetes Cluster

Wed, 27 Apr 2016 00:00:00 +0000

Let’s build a Raspberry Pi Cluster running Docker and Kubernetes. There has been already a handful of good recipes, however this howto is a bit different and provides some unique features.

My main motivation for going the Raspberry Pi road for a Kubernetes cluster was that I wanted something fancy for my Kubernetes talk to show, shamelessly stealing the idea from others (kudos to @KurtStam, @saturnism, @ArjenWassink and @kubernetesonarm for the inspiration ;-)

I.e. the following Pi-K8s projects already existed:

kubernetes-installer-rpi : A set up shell scripts and precompiled ARM binaries for running Kubernetes by @KurtStam on top of the Hypriot Docker Image for Raspberry Pi.
Kubernetes on ARM : An opinionated approach by @kubernetesonarm with an own installer for setting up Kubernetes no only for the Pi but also for other ARM based platforms.
K8s on Rpi : Another shell based installer for installing a Kubernetes cluster by @ArjenWassink and @saturnism

When there are already multiple recipes out there, why then trying yet another approach ?

My somewhat selfish goals were:

Using (and learning on the way) Ansible for not only a one-shot installation but also maintainance and upgrades.
Learning myself how to setup a Kubernetes cluster. This setup includes flannel as an overlay network, the SkyDNS extension and soon also a registry. Using Ansible helps me to incrementally add on top of things already installed.
Want to use WiFi for connecting the cluster. See below for the reason.
Get OpenShift Origin running and be able to switch between Ansible and OpenShift via Ansible.
Create a demonstration platform for my favourite development and integration platform fabric8.

As it turns out the whole experience was very enlightening to me. Its one thing to start Kubernetes on a single node within a VM (because multiple VM-based nodes kill soon your machine resourcewise) or having a small bare metal cluster, which blinks red and green and where you can plug wires at will. Not to mention the the geek factor :)

Shopping List
#

Here’s my shopping list for a Raspberry Pi 3 cluster, along with (non-affiliate) links to (German) shops, but I’m sure you can find them elswhere, too.

Amount	Part	Price
4	Raspberry Pi 3	4 * 38 EUR
4	Micro SD Card 32 GB	4 * 11 EUR
1	WLAN Router	22 EUR
4	USB wires	9 EUR
1	Power Supply	30 EUR
1	Case	10 EUR
3	Intermediate Case Plate	3 * 7 EUR

All in all, a 4 node Pi cluster for 288 EUR (as of April 2016). Not so bad.

Some remarks:

Using WiFi for the connection has the big advantage that the Raspberry Pi 3 integrated BCM43438 WiFi chip doesn’t go over USB and saves valuable bandwidth used for IO in general. That way you are able to to get ~ 25 MB/s for disk IO and network traffic, respectively. And also less cables, of course. You can alway plug the power wire for demos, too ;-)
A class 10 Mirco SD is recommended but it doesn’t have to be the fastest on the world as the USB bus only allows around 35 MB/s anyway.

Initial Pi Setup
#

Most of the installation is automated by using Ansible. However the initial setup is a bit more involved. It certainly can be improved (e.g. automatic filesystem expanding of the initial Raspian setup). If you have ideas how to improve this, please open issues and PRs on Project31/ansible-kubernetes-openshift-pi3. Several base distributions has been tried out. It turned out that the most stable setup is based on a stock Raspian. Unfortunately it doesn’t provide a headless WLAN setup as it is possible with the latest Hypriot images, but for the moment it much more stable (I had strange kernel panics and 200% CPU load issues with the Hypriot image for no obvious reasons). Since this is a one time effort, let’s use Raspbian. If you want to try out the Hypriot image, there’s an experimental branch for the Ansible playbooks which can be used with Hypriot. I will retry Hypriot OS for sure some times later.

Download the latest Raspian image and store it as raspbian.zip :

 curl -L https://downloads.raspberrypi.org/raspbian_lite_latest \
 -o raspbian.zip

Install Hypriots’ flash installer script. Follow the directions on the installation page.
Insert you Micro-SD card in your Desktop computer (via an adapter possibly) and run
```
 flash raspbian.zip
```
You will be asked to which device to write. Check this carefully, otherwise you could destroy your Desktop OS if selecting the the wrong device. Typically its something like /dev/disk2 on OS X, but depends on the number of hard drives you have.
Insert the Micro SSD card into your Raspberry Pi and connect it to a monitor and keyboard. Boot up. Login in with pi / raspberry. Then:
```
 raspi-config --expand-rootfs
 vi /etc/wpa_supplicant/wpa_supplicant.conf
```
and then add your WLAN credentials
```
 network={
 ssid="MySSID"
 psk="s3cr3t"
 }
```
Reboot
Repeat step 2. to 5. for each Micro SD card.

Network Setup
#

It is now time to configure your WLAN router. This of course depends on which router you use. The following instructions are based on a TP-Link TL-WR802N which is quite inexepensive but still absolutely ok for our purposes since it sits very close to the cluster and my notebook anyway.

First of all you need to setup the SSID and password. Use the same credentials with which you have configured your images.

My setup is, that I span a private network 192.168.23.0/24 for the Pi cluster which my MacBook also joins via its integrated WiFi.

The addresses I have chosen are :

| 192.168.23.1 | WLAN Router | | 192.168.23.100 | MacBook’s WLAN | | 192.168.23.200 … 192.168.23.203 | Raspberry Pis |

The MacBook is setup for NAT and forwarding from this private network to the internet. This script helps in setting up the forwarding and NAT rules on OS X.

In order to configure your WLAN router you need to connect to it according to its setup instructions. The router is setup in Access Point mode with DHCP enabled. As soon as the MAC of the Pis are known (which you can see as soon as they connect for the first time via WiFi), I configured them to always use the same DHCP lease. For the TL-WR802N this can be done in the configuration section DHCP -> Address Reservation. In the DHCP -> DHCP-Settings the default gateway is set to 192.168.23.100, which my notebook’s WLAN IP.

Startup all nodes, you should be able to ping every node in your cluster. I added n0 … n3 to my notebook’s /etc/hosts pointing to 192.168.23.200 … 192.168.23.203 for convenience.

You should be able to ssh into every Pi with user pi and password raspberry. Also, if you set up the forwarding on your desktop properly you should be able to ping from within the pi to the outside world.

Ansible Playbooks
#

After this initial setup is done, the next step is to initialize the base system with Ansible. You will need Ansible 2 installed on your desktop (e.g. brew install ansible when running on OS X)

Ansible Configuration
#

Checkout the Ansible playbooks:

 git clone https://github.com/Project31/ansible-kubernetes-openshift-pi3.git k8s-pi
 cd k8s-pi

Copy over hosts.example and adapt it to your needs
```
 cp hosts.example hosts
 vi hosts
```
There are three Ansible groups which are refered to in the playbooks:

| pis | All cluster node | n0, n1, n2, n3 | | master | Master node | n0 | | nodes | All nodes which are not master | n1, n2, n3|
Copy over the configuration and adapt it.
```
 cp config.yml.example config.yml
 vi config.yml
```
You should at least put in your WLAN credentials, but you are also free to adapt the other values.

Basic Node Setup
#

If you have already created a cluster with these playbooks and want to start a fresh, please be sure that you cleanup your ~/.ssh/known_hosts from the old host keys. You should be able to ssh into each of the nodes without warnings. Also you must be able to reach the internet from the nodes.

In the next step the basic setup (without Kubernetes) is performed. This is done by

ansible-playbook -k -i hosts setup.yml

When you are prompted for the password, use raspberry. You will probably also need to confirm the SSH authentity for each host with yes.

The following steps will be applied by this command (which may take a bit):

Docker will be installed from the Hypriot repositories
Your public SSH key is copied over to pi’s authenticated_keys and the users password will be taken from config.yml
Some extra tools are installed for your convenience and some benchmarking:
- hdparm
- iperf
- mtr
Hostname is set to the name of the node configured. Also /etc/hosts is setup to contain all nodes with their short names.
A swapfile is enabled (just in case)

With this basic setup you have already a working Docker environment.

Now its time to reboot the whole cluster since some required boot params has been added. Plug the wire.

Kubernetes Setup
#

The final step for a working Kubernetes cluster is to run

ansible-playbook -i hosts kubernetes.yml

This will install one master at n0 and threed additional nodes n1, n2, n3.

The following features are enabled:

etcd, flanneld and kubelet are installed as systemd services on the master
kubelet and flanneld are installed as systemd services on the nodes
Docker is configured to use the Flannel overlay network
kubectl is installed (and an alias k)

If there are some issues when restarting services in the master, don’t worry. However you should best restart the master node n0 when this happens, because when setting up the other nodes the would fail if not all services are running on the master.

After an initial installation it may take a bit until all infrastructure docker images has been loaded. Eventually should be able to use kubectl get nodes from e.g. n0. When this wotks but you see only one node, please reboot the cluster since some services may have not been started on the nodes (plug the cables when n0 is ready).

Install SkyDNS
#

For service discovery via DNS you should finally install the SkyDNS addon, but only when the cluster is running, i.e. the master must be up and listening. For this final step call:

ansible-playbook -i hosts skydns.yml

Wrap Up
#

This has become a rather long recipe. I re-did everything from scratch within 60 minutes, so this could be considered as a lower boundary (because I already did it several times :). The initial setup might be a bit flaky, but should be easy to fix. I’d love to hear your feedback on this, and maybe we get it more stable afterwards. Remember, that’s my first Ansible playbook :)

Now go out, buy and setup your Kubernetes cluster and have fun :-)

docker-maven-plugin moves on

Wed, 24 Feb 2016 00:00:00 +0000

rhuss/docker-maven-plugin is dead, long live fabric8io/docker-maven-plugin !

If you follow the Docker Maven Plugin Scene¹, you probably noticed that there has been quite some progress in the last year. Started as a personal research experiment early 2014, rhuss/docker-maven-plugin (d-m-p) has took off a little bit. With more than 300 GitHub stars it’s now the second most popular docker-maven-plugin. With 38 contributors we were able to do 36 releases. It is really fantastic to see that many people contributing to a rather niche product. Many kudos go out to Jae for his many contributions and continued support on fixing and answering issues. Thanks also for always being very patient with my sometimes quite opinionated and picky code reviews :)

However it is now time to ignite the next stage and bring this personal ‘pet’ project to a wider context. And what is better suited here than the fabric8 community ?

Fabric8 is a next generation DevOps and integration platform for Docker based applications, with a focus on Kubernetes and OpenShift as orchestration and build infrastructure. Its a collection of multiple interrelated projects including Maven tooling for interacting with Kubernetes and OpenShift. d-m-p is already included as foundation for creating Docker application images.

I’m very happy that d-m-p has now found its place in this ecosystem where it will continue to flourish even faster.

The fabric8 community is very open and has established multiple communications channels on which you will find d-m-p now, too:

#fabric8 on irc.freenode.net is an IRC channel with a lot of helpful hands (including myself)
A mailing list for more in depth discussions
Issues are still tracked with GitHub issues
d-m-p specific blog posts will go out on the fabric8 blog in the future.

So, what changed ?

rhuss/docker-maven-plugin has been transferred to fabric8io/docker-maven-plugin
The Maven group id has changed from org.jolokia to io.fabric8 for all releases 0.14.0 and later.
CI and release management will be done on the fabric8 platform.

And what will not change ?

d-m-p will always be usable with plain Docker, speaking either to a remote or local Docker daemon. No Kubernetes, no OpenShift required.
I’ll continue to work on d-m-p ;-)

Thanks so much for all the fruitful feedback and pull requests. Keep on rocking ;-)

with more than 15 docker-maven-plugins its probably fair to call it a “scene” ;-) ↩︎

Registry Magic with docker-maven-plugin

Thu, 21 Jan 2016 00:00:00 +0000

Dealing with multiple Docker registries is hard, mostly because the meta information where a image is located is part of a Docker image’s name, which is typically used as an identifier, too.

Let’s see how the rhuss/docker-maven-plugin deals with this peculiarity.

When setting up a Maven build for creating Docker images out of your Java application, the classical way to specify the registry where to push the final Docker image is to bake it into the image’s name. The drawback however is that you couple your build to this particular registry so that it is not possible to push your image to another registry when building the image.

Pull and Push
#

The docker-maven-plugin (d-m-p in short) interacts¹ with Docker registries in two use cases:

Pulling base images from a registry when building images with docker:build or starting images with docker:start
Pushing built images to a registry with docker:push

In both cases you can define your build agnostic from any registry by omitting the registry part in your image names² and specify it externally as meta information. This can be done in various ways:

Adding it to the plugin configuration as an <registry> element. This can be easily put into a Maven profile (either directly in the pom.xml or also in ~/.m2/settings.xml).
Using a system property docker.registry when running Maven
As a final fallback an environment variable DOCKER_REGISTRY can be used, too.

For example,

mvn -Ddocker.registry=myregistry.domain.com:5000 docker:push

When you combine build and push steps in a single call like in

mvn package docker:build docker:push

a pull operation for a base image and a push operation can happen. To allow different registries in this situation the properties docker.pull.registry and docker.push.registry are supported, too, (with the corresponding configuration elements <pullRegistry> and <pushRegistry>, respectively).

When pushing an image this way, the following happens behind the scene (assuming an image named user/myimage and target registry myregistry:5000)

The image user/myimage is tagged temporarily as myregistry:5000/user/myimage in the Docker daemon.
The image myregistry:5000/user/myimage is pushed.
The tag is removed again.

Authentication
#

That’s all fine, but how does d-m-p deal with authentication ? Again, there are several possibilities how authentication can be performed against a registry:

Using a <authConfig> section in the plugin configuration with<username> and <password> elements.
Providing system properties docker.username and docker.password when running Maven
Using a <server> configuration in ~/.m2/settings.xml with possible encrypted password. That’s the most maven-ish way for doing authentication.
Login into the registry with docker login. The plugin will pick up the credentials from ~/.docker/config.json

There are again variants to distinguish between authentication for pulling and pushing images to registries (e.g. docker.push.username and docker.push.password). All the details can be found in the reference manual.

Using the OpenShift Registry
#

OpenShift is an awesome PaaS platform on top of Kubernetes. It comes with an own Docker registry which can be used by d-m-p, too. However, there are some things to watch out for.

First of all, the registry needs to be exposed to the outside so that a Docker daemon outside the OpenShift cluster can talk with the registry:

oc expose service/docker-registry --hostname=docker-registry.mydomain.com

The hostname provided should be resolved by your host to the OpenShift API server’s IP (this happens automatically if you use the fabric8 OpenShift Vagrant image for a one-node developer installation of OpenShift).

Next, it is important to know, that the OpenShift registry use the regular OpenShift SSO authentication, so you have to login into OpenShift before you can push to the registry. The access token obtained from the login is then used as the password for accessing the registry:

# Login to OpenShift. Credentials are stored in ~/.kube/config.json:
oc login

# Use user and access token for authentication:
mvn docker:push -Ddocker.registry=docker-registry.mydomain.com \
 -Ddocker.username=$(oc whoami) \
 -Ddocker.password=$(oc whoami -t)

The last step can be simplified by using -Ddocker.useOpenShiftAuth which does the user and token lookup transparently.

mvn docker:push -Ddocker.registry=docker-registry.mydomain.com \
 -Ddocker.useOpenShiftAuth

The configuration option useOpenShiftAuth again comes in multiple flavours: a default one, and dedicated for push and pull operations (docker.pull.useOpenShiftAuth and docker.push.useOpenShiftAuth).

tl;dr
#

Among all the many docker maven plugins, rhuss/docker-maven-plugin provides the most flexible options for accessing Docker registries and authentication. The gory details can be found in the reference manual which documents registry handling and authentication in detail.

The interaction is always indirectly via the Docker daemon, since a Docker client like d-m-p only talks with the Docker daemon directly. ↩︎
Of course you can the registry part in your image names in which case this registry has always the highest priority. ↩︎

Jolokia 2.0 - JMX Notifications

Wed, 13 Jan 2016 00:00:00 +0000

This screencast gives a live demo of the forthcoming JMX notification support in Jolokia 2.0.

Jolokia supports currently two notification modes. In all modes, the Jolokia agent itself subscribe to a JMX notification locally and then dispatches the notifications to its clients.

Pull Mode : Here, the agent keeps the notification received for a client in memory and sends it back on an JMX request to a Jolokia specific MBean. A client typically queries this notification MBean periodically.
SSE Mode : Server Sent Events are a W3C standard for pushing events from an HTTP server to a client. With this mode the Jolokia agents directly pushes any notification it receives to the client. The advantage is of course a much lower latency compared to the pull mode, but SSE is not available for Internet Explorer, including 11. What a pity.

The Jolokia protocol has been extended with the top level action notification and subcommands.

register / unregister : Register / unregister a notification client
add / remove : Add / remove a listener subscription
list : list all subscriptions for a client
ping : Keep subscription alive
open : Use for creating a back channel. E.g. the SSE mode keeps this GET request for pushing back an event stream.

Currently only the new Jolokia JavaScript client supports JMX notification. If you are interested in having it in other clients (e.g. Java), too, please let me know. I would be more than happy for coders jumping on the Jolokia bandwagon since there is still quite some stuff to do for 2.0.

The source code to this demo and the new Jolokia JavaScript client is on GitHub: https://github.com/jolokia-org/jolokia-client-javascript.

Welcome to 2016 - the year Jolokia 2.0 will see the light of day

Wed, 06 Jan 2016 00:00:00 +0000

I hope you all had a good start into 2016 and have charged all your batteries during the time of stillness.

Jolokia had a good start, too. During the holiday season I took the opportunity to continue to work on version 2.0 which now takes on form. If you have followed the history of Jolokia you know that work on 2.0 started early 2013 but advanced quite slowly for multiple reasons.

Now its time to go out on a limb with announcing Jolokia 2.0 for 2016. A bit of pressure sometimes really helps ;-)

Here’s are the major themes for Jolokia 2.0:

Jolokia 2.0 will be backwards compatible on the protocol level. This is a design goal. There might be some changes in default values, however this should be easy to fix. Any such change will be announced prominently (like artefact renaming). So, all your clients will be usable with 2.0 with minor changes.
JMX Notification support is here. Yeah, this was quite some work. The extensions to the Jolokia 2.0 protocol are able to push notifications in various ways. Currently the agents supports two modes:
- Pull mode will collect JMX notification on the server (agent) side and can be fetched by a client with an HTTP request, which typically happens periodically. This introduces some latencies but is the most robust way to transmit notifications to a client.
- SSE mode uses Server Sent Events for pushing JMX notifications immediately with very low latency. This is the preferred mode if a client supports this (Internet Explorer does not).
The notification support is nearly complete on the agent side, and the Jolokia JavaScript client already supports both modes. In the future more mode like WebSockets or Web-Hooks should be easy to add. The next post will give a demo about the notification support.
Namespaces extend Jolokia beyond JMX which means you can access other entities than JMX MBeans with the very same protocol. This feature is still in the conceptual state but one can easily imagine to access
- Spring Beans
- CDI Objects
- JNDI Directories
- Zookeeper Directories
- ….
the same was as JMX. The namespace is selected as part of the (MBean) name. More on this in this design document. Since this feature would extend the usage pattern of Jolokia quite a bit, I’m not 100% sure whether to include it into 2.0 since it feels a bit against my Unix based education (“do one thing and do it well”).
With the addition of even more features, modularization becomes even more important. Jolokia was and is always picky about its footprint, which is currently 430k for the WAR agent with all features included. Jolokia 2.0 introduces various internal services which can be picked and chosen by repackaging the agent. Or the agent can be extended with own functionality, too. A way for easily packaging and creating agents will be provided either by a Web-UI or by a CLI tool (or both).

In addition there are also some non-functional changes to polish Jolokia a bit:

Non-agent based addition like client libraries, integration tests, JBoss Forge support are extracted into extra GitHub repositories. All this will happen within the GitHub organization jolokia-org. The first project here is the JavaScript client which already moved to a dedicated jolokia-client-javascript repository.
The website will get a face-lift.
Documentation will switch from Docbook to a Markdown or AsciiDoc based format.

Finally some stuff will get dropped. This happens because of limited resources (Jolokia, to be frankly, still doesn’t have a big community, so that most of the work is done by a single person. ‘would like to change that, though) and because I think these feature never took off:

Mule agent. I never got much feedback from the Mule community so I’m really not sure whether this agent is really used or needed. Jolokia 1.x will continue to support the Mule agent, however there will be no stock Jolokia 2.0 Mule agent. Said that, you are always free to adopt Jolokia 2.0 to the Mule management platform. Considering the extra code needed included in Jolokia 1.3 for Mule support this should be fairly trivial. I’m happy to support anyone doing the port. Also, there is always the alternative to use the JVM agent for attaching Jolokia to Mule, which is the preferred way for 2.0 to monitor Mule with Jolokia.
Spring Roo Support will be dropped for much the same reasons. I never received an issue on the Jolokia Spring Roo support, which is a clear sign that nobody is using it. It might popup as an extra project.

So, what’s the roadmap ? Here’s the plan:

Milestone 2.0.0-M1 is here. You find the JVM and WAR agents in Maven central.
Every month, a new milestone will be released.
Final release is aligned to Red Hat Summit / DevNation. July 1st.

Isn’t this a nice new year’s resolution ? ;-)

In the next post I will demo JMX notifications and how you can use them in your JavaScript projects.

fish-pepper - Docker on Capsaicin

Mon, 07 Sep 2015 00:00:00 +0000

When I had to create multiple Docker base images which only differ slightly for some minor variations I couldn’t avoid to feel quite dirty because of all the copying & pasting of Dockerfile fragments. We all know how this smells, but unfortunately Docker has only an answer for inheritance but not for composition of Docker images. Luckily there is now fish-pepper, a multi-dimensional docker build generator, which steps into the breach.

For example consider a Java base image: Some users might require Java 7, some want Java 8. For running Microservices a JRE might be sufficient. In other use cases you need a full JDK. These four variants are all quite similar with respect to documentation, Dockerfiles and support files like startup scripts. Copy-and-paste might seem to work for the initial setup but there are severe drawbacks considering image evolution or introduction of even more parameters.

With fish-pepper you can use flexible templates which are filled with variations of the base image (like 'version' : ['java7', 'java8'], 'type': ['jdk', 'jre']) and which will create multiple, similar Dockerfile builds.

The main configuration of an image family is images.yml which defines the possible parameters. For the example above it is

fish-pepper:
 params:
 - "version"
 - "type"

The possible values for these parameters are given in a dedicated config section:

config:
 version:
 openjdk7:
 java: "java:7u79"
 fullVersion: "OpenJDK 1.7.0_79"
 openjdk8:
 java: "java:8u45"
 fullVersion: "OpenJDK 1.8.0_45"
 type:
 jre:
 extension: "-jre"
 jdk:
 extension: "-jdk"

Given this configuration, four builds will be generated when calling fish-pepper, one for each combination of version (“openjdk7” and “openjdk8”) and type (“jre” and “jdk”) parameter values.

These value can now be filled into templates which are stored in a templates/ directory. The Dockerfile in this directory can refer to this configuration through a context object fp:

FROM {{ "{{= fp.config.version.java + fp.config.type.extension " }}}}
.....

Templates use DoT.js as template engine, so that the full expressiveness of JavaScript is available. The fish-pepper context object fp holds the configuration and more.

The given configuration will lead to four Docker build directories:

images/
 +---- openjdk7
 | +--- jre -- Dockerfile, ...
 | +--- jdk -- Dockerfile, ...
 |
 +---- opendjk8
 +--- jre -- Dockerfile, ...
 +--- jdk -- Dockerfile, ...

The generated build files can also be used directly to create the images with fish-pepper build. This will reach out to a Docker daemon and create the images java-openjdk7-jre, java-openjdk7-jdk, java-openjdk8-jre and java-openjdk8-jdk.

Alternatively these builds can be used as content for automated Docker Hub builds when checked into Github. The full example can be found on GitHub.

But wait, there is more:

Blocks can be used to reuse Dockerfile snippets and files to include across images. Blocks can be stored locally or referenced via a remote Git repository. Examples for blocks are generic startup scripts or other value add functionality like enabling agents like agent bond.
Flexible file mappings allow multiple alternative templates.
Defaults allow shared configuration between multiple parameter values.

fish-pepper can be seen in its fully beauty in fabric8io/base-images where more than twenty five base images are maintained with fish-pepper.

With node.js you can install fish-pepper super easy with

npm -g install fish-pepper

In the following blogs I will show more usage examples, especially how “blocks” can be easily reused and shared.

Jmx4Perl for everyone

Tue, 28 Jul 2015 00:00:00 +0000

As you might know, Jmx4Perl is the mother of Jolokia. But what might be not so known is, that Jmx4Perl provides a set of nice CLI tools for accessing Jolokia agents. However, installing Jmx4Perl manually is cumbersome because of its many Perl and also native dependencies.

However, if you are a Docker user there is now a super easy way to benefit from this gems.

Even if Perl is not your cup of tea, you might like the following tool (for which of course no Perl knowledge is required at all):

jmx4perl is a command line tool for one-shot querying Jolokia agents. It is perfectly suited for shell scripts.
j4psh is a readline based, JMX shell with coloring and command line completion. You can navigate the JMX namespace like directories with cd and ls, read JMX attributes with cat and execute operations with exec.
jolokia is an agent management tool which helps you in downloading Jolokia agents of various types (war, jvm, osgi, mule) and versions. It also knows how to repackage agents e.g. for enabling security for the war agent by in-place modification of the web.xml descriptor.
check_jmx4perl is a full featured Nagios plugin.

How can you now use these tools ? All you need is a running Docker installation. The tools mentioned above are all included within the Docker image jolokia/jmx4perl which is available from Docker Hub.

Some examples:

# Get some basic information of the server
docker run --rm -it jolokia/jmx4perl \
 jmx4perl http://localhost:8080/jolokia

# Download the current jolokia.war agent
docker run --rm -it -v `pwd`:/jolokia jolokia/jmx4perl \
 jolokia

# Start an interactive JMX shell
# server "tomcat" is defined in ~/.j4p/jmx4perl.config
docker run --rm -it -v ~/.j4p:/root/.j4p jolokia/jmx4perl \
 j4psh tomcat

In these examples we mounted some volumes:

If you put your server definitions into ~/.j4p/jmx4perl.config you can use them by mounting this directory as volume with -v ~/.j4p:/root/.j4p.
For the management tool jolokia it is recommended to mount the local directory with -v $(pwd):/jolokia so that downloaded artefacts are stored in the current host directory. (Note for boot2docker users: This works only when you are in a directory below you home directory)

It is recommended to use aliases as abbreviations:

alias jmx4perl="docker run --rm -it -v ~/.j4p:/root/.j4p jolokia/jmx4perl jmx4perl"
alias jolokia="docker run --rm -it -v `pwd`:/jolokia jolokia/jmx4perl jolokia"
alias j4psh="docker run --rm -it -v ~/.j4p:/root/.j4p jolokia/jmx4perl j4psh"

As an additional benefit of using Jmx4Perl that way, you can access servers which are not directly reachable by you. The Jolokia agent must be reachable by the Docker daemon only. For example, you can communicate with a SSL secured Docker daemon running in a DMZ only. From there you can easily reach any other server with a Jolokia agent installed, so there is no need to open access to all servers from your local host directly.

Finally, here’s a short appetiser with an (older) demo showing j4psh in action.

docker:watch

Tue, 30 Jun 2015 00:00:00 +0000

Ok, you know Docker. And since you are a Java developer you want to know how you can use this in your daily development workflow. You probably also heard about the docker-maven-plugin which seamlessly creates Docker images, starts and stops Docker containers and more all with a concise configuration syntax.

And now there is this new goal docker:watch.

We developers are lazy, right ? We want our code to compile fast, we want the servers to start up fast. And we want to test changes quickly. That’s why we love OSGi¹ and JRebel. And we want this for Docker containers, too.

Good news. docker-maven-plugin will support hot rebuild of Docker images and hot restart of containers with a new Maven goal docker:watch. It will be released with version 0.12.1. For the brave coder 0.12.1-SNAPSHOT is already out there, the documentation can be found here.

But before losing more words, here’s a sneak preview.

That’s of course not entirely true. We love OSGi … or we hate it with passion. But even the haters don’t hate it for its hot deployment abilities. ↩︎

The Docker Wormhole Pattern

Wed, 17 Jun 2015 00:00:00 +0000

Building a Docker wormhole is easy.

A wormhole is a special type of structure that some scientists think might exist, connecting parts of space and time that are not usually connected

— Cambridge Dictionaries Online

In Docker universe we have several uses cases, which require a Docker installation within a Docker container. For example the OpenShift Builds use images whose container’s are meant to create application images. They include a whole development environment including possibly a compiler and a build tool. During the build a Docker daemon is accessed for creating the final application image.

The question is now, how a build can access the Docker daemon ? In general, there are two possibilities:

Docker in Docker is a project which allows you to run a Docker daemon within a Docker container. Technically this is quite tricky and there seems to be some issues with this approach, especially because you have to run this container in privileged mode. This is the Matryoshka doll pattern.
Or you use the Wormhole pattern described in this post. The idea is to get access to the Docker daemon running a container from within the container.

As you know a Docker host can be configured to be accessible by two alternative methods: Via a Unix socket or via a TCP socket.

Using the Unix socket of the surrounding docker daemon is easy: Simply share the path to the unix socket as a volume:

# Map local unix socket into the container
docker run -it -v /var/run/docker.sock:/var/run/docker.sock ...

Then within the container you can use the Docker CLI or any tool that uses the Unix socket at usual.

Running over the TCP socket is a bit more tricky because you have to find out the address of your Docker daemon host. This can best be done by examining the routing table within the container:

# Lookup and parse routing table 
host=$(ip route show 0.0.0.0/0 | \
 grep -Eo 'via \S+' | \
 awk '{print $2}');
export DOCKER_HOST=tcp://${host}:2375

This works fine as long you are not using SSL. With SSL in place you need have access to the SSL client certificates. Of course this is achieved again with a volume mount. Assuming that you are using boot2docker this could look like

# Mount certs into the container
docker run -ti -v ~/.boot2docker/certs/boot2docker-vm/:/certs ....

This will mount your certs at /certs within the container and can be used to set the DOCKER_HOST variable.

if [ -f /certs/key.pem ]; then
 # If certs are mounted, use SSL ...
 export DOCKER_CERT_PATH=/certs
 export DOCKER_TLS_VERIFY=1
 export DOCKER_HOST=tcp://${host}:2376
else
 # ... otherwise use plain http
 export DOCKER_TLS_VERIFY=0
 export DOCKER_HOST=tcp://${host}:2375
fi

There is some final gotcha as that the server certificate can not be verified because it doesn’t contain the docker host IP as seen from the container. See this issue for details. As workaround you have to unset DOCKER_TLS_VERIFY for the moment when using the docker client.

Both ways are useful and are leaner and possibly more secure than having a Matryoshka doll approach.

Finally there is still the question, why on earth wormhole pattern ? Like in a wormhole (also known as Einstein-Rosen Bridge) you can reach through the wormhole a point (the outer docker daemon) in spacetime which is normally not reachable (because a container is supposed to be its “own” world). Another fun fact: If you create a container through a wormhole this container it’s not your daughter, its your sister. Feels a bit freaky, or ? Alternatively you could call it also Münchhausen pattern because you create something with the exact the identically means you have been created yourself (like in the Münchhausen trilemma).

Or feel free to call it what you like ;-)

Jmx4Perl on OS X

Mon, 23 Mar 2015 00:00:00 +0000

The HTTP-JMX Bridge Jolokia allows easy access to JMX. It exposes all JMX information and operations via an REST-like interface and has tons of nifty features. Jmx4Perl on the other side is a client for Jolokia, which beside Perl access modules also provides quite some nice CLI tools for accessing and installing Jolokia. This post explains how install these tools on OS X.

Jmx4Perl provides some nice CLI commands:

jmx4perl is a simple access tool which is useful for quick queries and ideal for inclusion in shell scripts.
j4psh is a powerful interactive, readline based JMX shell with tab completion and syntax highlighting.
jolokia is a tool for managing Jolokia agents (downloading, changing init properties etc.)

All this tools are very helpful in order to explore the JMX namespace and installing the agent. They all are fairly good documented and each of them probably deserves an own blog post.

However, the installation or Perl modules and programs is a bit tedious. Although cpan helps here and also resolves transitive dependencies it’s still a lengthy process, which fails from time to time. Native Linux packages are planned, but don’t hold your breath ;-).

For OS X users with Homebrew can install Jmx4Perl quite easily, though:

$ brew install cpanm
$ cpanm --sudo install JMX::Jmx4Perl

This will do all the heavy lifting for you and at the end all the fine Jmx4Perl tools are installed and available under /usr/local/bin.

j4psh uses libreadline for the input handling. For the best user experience GNU ReadLine is recommended. Unfortunately, OS X doesn’t ship with a true libreadline but with libedit which is a stripped down version of libreadline. In order to use GNU readline, some tricks are needed which are described in this recipe. For me, the following steps worked (but are probably a bit “dirty”):

$ brew install readline
$ brew link --force readline
$ sudo mv /usr/lib/libreadline.dylib /tmp/libreadline.dylib 
$ cpanm --sudo Term::ReadLine::Gnu
$ sudo mv /tmp/libreadline.dylib /usr/lib/libreadline.dylib
$ brew unlink readline

These steps are really only necessary if you need advanced readline functionality (or a coloured prompt in j4psh ;-).

Health Checks with Jolokia

Sat, 17 Jan 2015 00:00:00 +0000

A health check is a useful technique for determining the overall operational state of a system in a consolidated form. It provides some kind of internal monitoring which collects metrics, evaluates them against some thresholds and provides a unified result. Health checks are now coming to Jolokia. This post explains the strategy to include health checks into Jolokia without blowing up the agents to much.

Health checks are different to classical monitoring solutions like Nagios, where external systems collect metrics and evaluate them against some threshold on their own. While monitoring with Nagios was and is always possible with Jolokia (and in fact was the original motivation for creating it), intrinsic health checks were avoided for the vanilla agent up to now because of the extra complexity they introduce into the agent. One of the major design goals of Jolokia is to keep it small and focussed.

The upcoming release 1.3.0 (scheduled for the end of this month) will introduce a simple plugin architecture into Jolokia which allows to hook into the agent’s lifecycle. A so called MBeanPlugin in Jolokia also allows access to the agent configuration and to the JMX system. Currently it is supported for the WAR and JVM agent, where plugins are created via a simple class path lookup. For the OSGi agent it is planned that it will pick up plugins as OSGi services.

Having this new infrastructure in place, extra functionality like health checks can be added easily. The GitHub repository jolokia-extra was created to host various extensions to the Jolokia agent, also to keep the original agent as lean as possible. Beside the new health checks there is already an extension jsr77 for simplifying the access to JSR-77 compliant JEE Servers like WebSphere.

The new health addon in jolokia-extra has just been started. Currently it contains not much more as proof-of-concept with some hardcoded health checks, but it already illustrate the concept: A MBeanPlugin registers a certain SampleHealthCheckMBean during startup which exposes the health checks as JMX operations (and which can be executed as usual with Jolokia). These operations have access to JMX via the MBeanPluginContext and can query any MBean in the system.

But that is only the beginning. There are still a lot of design decisions to take:

How should health check specification look like ? Should it be done via JSON or should a more expressive DSL based e.g. on Groovy should be used ?
How are the health check store on the agent side ?
- Looking them up in the filesystem (from a configurable path with a sane default like ~/.jolokia_healthchecks)
- Baking it into the agent jar
- Uploading it via an MBean operation (and then storing them in the filesystem as well)
What kind of meta data should be provided so that consoles like hawt.io can dynamically create their health check views ?
How should the parameter and return value for the health checks look like ?

If you would like to participate, the discussion about the implementation details will take place in issue #1 and the current working state is summarized in this wiki page.

Real clean Maven builds with Docker

Fri, 07 Nov 2014 00:00:00 +0000

A local Maven repository serves as a cache for artifacts and dependencies, we all know this. This helps in speeding up things but can cause subtle problems when doing releases. Docker can help here a bit for avoiding caching issues.

Before doing a release I typically move ~/.m2/repository away to be really sure that everybody else can build the source as well and that any dependencies are also on the remote Maven repository. This is a bit tedious, because it is a manual process and you can forget to move the old directory back which will was a LOT of disk space over time.

Docker can help here a bit: Since yesterday there is an official Maven image which can be used to build your project. The nice thing for doing releases with this image is, that it always starts afresh with an empty local Maven repository.

Assuming you are currently located in the top-level directory holding your pom.xml you can use this single command for running a real clean build:

docker run -it --rm \ 
	 -v "$(pwd)":/usr/src/mymaven \ 
	 -w /usr/src/mymaven \ 
	 maven:3.2-jdk-7 \
	 mvn clean install

With this call you mount your project directory into /usr/src/mymaven on the container, change to this directory in the container and call mvn clean install. At the end, your container will be removed (--rm) so there is no chance that you might forget to clean up afterwards.

Of course it will download all the artifacts each time, so it is not a good idea to use this approach for your daily developer business (especially if you using Maven central as remote Maven repository).

You can also play around with various versions of Maven by changing the image tag so at the end you can be really sure, that your project will build everywhere. Please refer to the Docker Hub page for details.

Update: As pointed out by Noah Zucker on Twitter you can redirect of course the local repository via -Dmaven.repo.local=/tmp/clean-repo temporarily to a new location. Which is confessedly much simpler and I would prefer that one instead if you don’t need to check with different JDKs or Maven versions. Sometimes you don’t see the forest for the trees if you come from the wrong direction (e.g. looking for use case of a specific docker image).

Docker maven plugin rewrite

Mon, 13 Oct 2014 00:00:00 +0000

My docker-maven-plugin is undergoing a major refactoring. This post explains the motivation behind this and also what you can expect in the very near future. The configuration syntax becomes much cleaner and implicit behavior was removed.

Originally, I needed a docker-maven-plugin for a very specific use case: To test Jolokia, the HTTP-JMX bridge in all the JEE and non-JEE servers out there. This was a very manual process: Fire up the VirtualBox image with all those servers installed, start a server, deploy Jolokia, run integration tests, stop the server, start the next server …. It takes easily half a day or more to do the tests before each release. That’s not the kind of QA you are looking for, really. But with docker there is finally the opportunity to automate all this: Deploy a single application on multiple different servers while controlling the lifecycle of theses servers from within the build.

Early this year when I searched the Web, I couldn’t find a good Docker build integration, so I decided to write my own docker-maven-plugin to back my use case. Today you find nearly a dozen of Maven plugins for your Docker business. However, only four plugins (alexec, wouterd, spotify and rhuss) are still actively maintained. A later blog post will present a detailed comparison between those four plugins (or come to my W-JAX session), but this post is about the evolution of the rhuss plugin.

It turned out that the plugin works quite well, people liked and starred it on GitHub. It provides also some unique features like creating Docker images from assembly descriptors.

But I was not so happy.

The reason is, that I started from a very special, probably very uncommon use case: A single application, multiple different servers for multiples tests. A much more common scenario is to have a fixed application server brand for the application, running with multiple linked backend containers like databases. My plugin doesn’t work well with running multiple containers at once. Or to state it otherwise: The plugin was not prepared for orchestration of multiple docker containers.

Also, there was too much happening magically behind the scenes: When pushing a data image, it was implicitly build. When starting a container for integration test, the data container is also build before.

Two operational modes were supported: One with images holding the server and data separately in two containers (linked via volumes) and one so called merged image, holding both, the application and server together in one image. This is perfect for creating micro services. The mode is determined only by a configuration flag (mergeData), but it is not really clear how many and what Docker images are created. And it was hard to document which is always a very bad smell.

So I changed the configuration syntax completely.

It is now much more explicit and you will know merely by looking at the configuration which and how many containers will be started during integration testing and what the container with the application will look like. I don’t want to go into much detail here, the post is already too long. Instead here is an example of the new syntax:

<plugin>
 <groupId>org.jolokia</groupId>
 <artifactId>docker-maven-plugin</artifactId>
 <version>0.10.1</version>
 <configuration>
 <images>
 <image>
 <name>consol/tomcat-7.0</name>
 <run>
 <volumes>
 <from>jolokia/docker-jolokia-demo</from>
 </volumes>
 <ports>
 <port>jolokia.port:8080</port>
 </ports>
 <wait>
 <url>http://localhost:${jolokia.port}/jolokia</url>
 <time>10000</time>
 </wait>
 </run>
 </image>
 <image>
 <name>jolokia/docker-jolokia-demo</name>
 <build>
 <assemblyDescriptor>src/main/assembly.xml</assemblyDescriptor>
 </build>
 </image>
 </images>
 </configuration>
</plugin>

This examples creates and starts two containers during docker:start, linked together via the volumes directive. The <run> configuration section is used to describe the runtime behavior for docker:start and docker:stop, and <build> is for specifying how images are build up during docker:build.

Alternatively, a single image could be created:

<plugin>
 <groupId>org.jolokia</groupId>
 <artifactId>docker-maven-plugin</artifactId>
 <version>0.10.1</version>
 <configuration>
 <images>
 <image>
 <name>jolokia/docker-jolokia-combined-demo</name>
 <build>
 <baseImage>consol/tomcat-7.0</baseImage>
 <assemblyDescriptor>src/main/assembly.xml</assemblyDescriptor>
 </build>
 <run>
 <ports>
 <port>jolokia.port:8080</port>
 </ports>
 <wait>
 <url>http://localhost:${jolokia.port}/jolokia</url>
 </wait>
 </run>
 </image>
 </images>
 </configuration>
</plugin>

Here consol/tomcat-7.0 is used as base for the image to build and the data referenced in the assembly descriptor is copied into the image. So there is no need to volume-link them together.

I won’t repeat the old, more confusing syntax for this both use cases here, you find it in the current online documentation.

Said all that, and since rhuss/docker-maven-plugin is still pre-1.0, I take the liberty to change it without much thoughts on backwards compatibility (you can easily update old configurations). The new syntax is available since 0.10.1, the old syntax will still be used in the 0.9.x line. Everybody is encouraged to upgrade to 0.10.x, although the documentation still reflects the old syntax (will be fixed soon). Please refer to the examples on the new-config branch for more details. An upgrade path will be available soon, too.

There will be a 1.0.0 release before the end of this year.

Please let me know your feedback on the new syntax and what features you would like to see. Everything is moving before the 1.0.0 freeze. You can open an issue for any suggestion or feature request.

Spicy Docker Java Images with Jolokia

Thu, 09 Oct 2014 00:00:00 +0000

While on the way of transforming the Jolokia integration test suite from a tedious, manual, half-a-day procedure to a full automated process I ran into and felt in love with Docker. As a byproduct a java-jolokia docker repository emerged, which can be easily used as a Java base image for enabling a Jolokia JVM agent during startup for any Java application.

These images are variants of the official java Java docker image. In order to use the Jolokia agent, a child image should call the script jolokia_opts (which is in the path). This will echo all relevant startup options that should be included as argument to the Java startup command.

Here is a simple example for creating a Tomcat 7 images which starts Jolokia along with Tomcat:

FROM jolokia/java-jolokia:7
ENV TOMCAT_VERSION 7.0.55
ENV TC apache-tomcat-${TOMCAT_VERSION}

EXPOSE 8080 8778
RUN wget http://archive.apache.org/dist/tomcat/tomcat-7/v${TOMCAT_VERSION}/bin/${TC}.tar.gz
RUN tar xzf ${TC}.tar.gz -C /opt

CMD env CATALINA_OPTS=$(jolokia_opts) /opt/${TC}/bin/catalina.sh run

(Don’t forget to use $(jolokia_opts) or with backticks, but not ${jolokia_opts})

The configuration of the Jolokia agent can be influenced with various environments variables which can be given when starting the container:

JOLOKIA_OFF : If set disables activation of Jolokia. By default, Jolokia is enabled.
JOLOKIA_CONFIG : If set uses this file (including path) as Jolokia JVM agent properties (as described in Jolokia’s reference manual. By default this is /opt/jolokia/jolokia.properties. If this file exists, it will automatically be taken as configuration
JOLOKIA_HOST : Host address to bind to (Default: 0.0.0.0)
JOLOKIA_PORT : Port to use (Default: 8778)
JOLOKIA_USER : User for authentication. By default authentication is switched off.
JOLOKIA_PASSWORD : Password for authentication. By default authentication is switched off.

So, if you start your tomcat with docker run -e JOLOKIA_OFF no agent will be started.

Currently this image is available from Docker Hub for the latest versions of Java 6,7 and 8, respectively, as they are provided by the official Docker java image.

Other base images can be easily added by using the configuration and templates from a super simple node based build system.

All appserver images from ConSol/docker-appserver (Docker Hub) are based now on this image, so Jolokia will always be by your side ;-)

Using NSEnter with Boot2Docker

Mon, 01 Sep 2014 00:00:00 +0000

NSEnter is a nice way to connect to a running Docker container. This post presents a script to simplify the usage of nsenter together with Boot2Docker.

There is still quite some dust around Docker and after gaining more and more experience, new patterns and anti-patterns are emerging.

One of those anti-patterns is the usage of an SSH daemon inside an image for debugging, backup and troubleshooting purposes. Jérôme Petazzoni’s Blog Post explains this nicely. In addition it provides proper solutions for common use cases for which SSH is currently used.

Nevertheless I still have this irresistible urge to login into a container. And if it is only for looking around and checking out the environment (call me old-fashioned, that’s ok ;-)

Luckily Jérôme provides a perfect solution to satisfy this thirst: nsenter. This allows you to enter into container namespaces. On the GitHub page you find the corresponding recipe for installing and using nsenter on a Linux host.

If you want to use it from OS X with e.g. Boot2Docker you need to login into the VM hosting the Docker daemon and then connect to a running container.

As described in the NSenter README you can use a simple alias for doing this transparently

	docker-enter() {
	 boot2docker ssh '[ -f /var/lib/boot2docker/nsenter ] || docker run --rm -v /var/lib/boot2docker/:/target jpetazzo/nsenter'
	 boot2docker ssh -t sudo /var/lib/boot2docker/docker-enter "$@"
	}

For a bit more comfort with usage information and error checking you can convert this to a small shell script like docker-enter which needs to be installed within the path (on OS X). As arguments it expects a container id or name and optionally a command (with args) to execute in the container. This script also will automatically install nsenter on the boot2docker VM if not already present (like the shell function above does this as well):

	10:20 [~] $ docker ps -q
	5bf8a161cceb
	
	10:20 [~] $ docker-enter 5bf8a161cceb bash
	
	Unable to find image 'jpetazzo/nsenter' locally
	Pulling repository jpetazzo/nsenter
	Installing nsenter to /target
	Installing docker-enter to /target
	
	root@5bf8a161cceb:/#

If you want even more comfort with bash completion you can add the small Bash completion script docker-enter_commands (inspired by and copied from Docker’s bash completion) to your ~/.bash_completion_scripts/ directory (or wherever your completion scripts are located, e.g. /usr/local/etc/bash_completion.d if you installed bash-completion via brew). This setup completes on container names and ids on the arguments for docker-enter. Alternatively you can put the commands together with the shell function code above directly into your ~/.bashrc, too.

P.S. After writing this post, I’ve found out, that this topic has been already covered in another blog post previously by Lajos Papp. That’s also where the shell function definition in the nsenter README originates from. Give credit to whom it’s due.

Docker for (Java) Developers

Mon, 25 Aug 2014 00:00:00 +0000

Recently I gave a Meetup talk for the Docker Munich Meetup Group which explained how Docker can help developers to improve integration tests and to ship applications.

The slides are online as well as the demo project.

During the demo I used Butterfly for an in-browser shell, which was quite cool, I guess ;-) (This is obviously not enabled in the online slides).

I’m going to continue to celebrate my Docker-♡ with another two talks in autumn:

Boosting your developer toolbox with Docker at JBoss One Day Talk, September, 29. in Germering (Munich, Germany)
Docker für Java Entwickler (in german) at W-JAX 14, November, 3. - 7. in Munich

And there is a slight chance (since the CFP has not yet been declined ;-) to talk at Devoxx about Docker. JavaZone unfortunately declined my CFP, I guess there are already too many riding the docker horse (which is a good thing).

Nevertheless I will attend both conferences with talks about Jolokia, and I’m really looking forward to it.

‘guess it will become a hot autumn (hotter than this german 2014 summer for sure) ….

Jolokia and CORS

Mon, 18 Aug 2014 00:00:00 +0000

Jolokia has configurable CORS support so that it plays nicely together with the Browser world when it comes to cross origin requests. However, Jolokia’s CORS support is not without gotchas. This post explains how Jolokias CORS supports works, what are the issues and how I plan to solve them.

tldr; Jolokia CORS support is configured via jolokia-access.xml but has issues with authenticated requests which are tackled for the next release 1.3.0

CORS Primer
#

CORS (Cross Origin Resource Sharing) is a specification for browsers to allow controlled access for JavaScript code to locations which are different than the origin of the JavaScript code itself.

In simple cases, it works more or less like this:

A JavaScript code (coming from the original location http://a.com) requests HTTP access via XMLHttpRequest to http://b.com
Since the the origin of the script and target URL of the request differs, the browser adds some extract checking on the response of this request.
The request to b.com contains a header Origin: http://a.com.
The server at b.com answering the request has to decided upon this header whether it wants allow this request. The server decision is contained in the response header Access-Control-Allow-Origin
The value of this header can be either a literal URL (e.g. http://a.com) or a wildcard like in Access-Control-Allow-Origin: * which allows access from any original location.
The browser finally decides whether it returns the response to the JavaScript based on the returned access control header. If not, is throws an exception before handing out the response data.

This is it for simple requests. A simple request has the following characteristics:

HTTP method is either GET,HEAD or POST
The request contains only the following headers
- Accept
- Accept-Language or Content-Language
- Content-Type with the value application/x-www-form-urlencoded, multipart/form-data or text/plain

If this criteria are not match for a request (e.g. because it uses a different method or additional headers), a so called preflight request is sent to the server before the actual request is performed. The preflight is an HTTP request with method OPTIONS and contains the headers Origin (http://a.com in our case), Access-Control-Request-Method for the HTTP method requested and Access-Control-Request-Headers with a comma separated list of additional header names. The server in turn answers with the allowed request methods and headers, whether an authenticated request is allowed and how long the client might cache this answer. An important point is, that a preflight request must not be authenticated.

And in fact, browsers never sent an authentication header with the preflight request even when already authenticated against the target server. More on this later.

Jolokia CORS Support
#

By default, Jolokia allows any CORS request. For the preflight the agent answers with

Access-Control-Allow-Origin: http://a.com
Access-Control-Allow-Headers: accept, authorization, content-type

The allowed headers returned are exactly the same headers as requested. For the real request with an origin header Origin: http://a.com the answer is

Access-Control-Allow-Origin: http://a.com
Access-Control-Allow-Credentials: true

For best computability Jolokia always answers with the provided Origin: which is extracted from the request (except when the origin is null in which case the wildcard * is returned.

This behavior can be tuned by adapting the jolokia-access.xml policy as described in the reference manual :

<cors>
 <allow-origin>http://www.jolokia.org</allow-origin>
 <allow-origin>*://*.jmx4perl.org</allow-origin>

 <strict-checking/>
</cors>

If a <cors> section is present in jolokia-access.xml then only those hosts declared in this sections are allowed. The Origin URLs to match against can be specified either literally or as pattern containing the wildcard *. The optional declaration <strict-checking/> is not really connected to CORS but helps in defending against Cross-Site-Request-Forgery (CSRF). If this option is given, then the given patterns are used for every request to compare it against the Origin: or Referer: header (not only for CORS requests).

CORS and Authentication
#

Since Authorization: is for CORS not a simple header, when authentication is used, preflight checking is always applied. However, there is often a catch 22:

The preflight check using the OPTIONS HTTP Method must not be authenticated as explained above, so browser doesn’t send the appropriate authentication headers when doing the preflight.
The Jolokia agent is typically secured completely no matter which HTTP method is used.
The preflight check fails, the request fails.

The only clean solution is to setup Jolokia Authentication that way that OPTIONS request are not secured.

Let’s have a look at the individual Agents:

JVM Agent
#

Since the JVM agent does all the security stuff on its own, it is not a big deal to introduce this specific behavior. Next one.

WAR Agent
#

The WAR agent use authentication and authorization as defined in the Servlet Specification, i.e. the appropriate <security-constraint> must be added manually to the web.xml (jolokia is a CLI tool which helps in this repackaging). Unfortunately there is no way to secure the same <url-pattern> differently for different HTTP Methods (i.e. secured with an <auth-constraint> for GET and POST, but accessible for everybody for OPTIONS). I tried hard by providing multiple <security-constraint> but failed miserably (if you know how to this, please let me know).

The only solution is to switch over to checking a given role on our own without relying on the declarative JEE security mechanism. Since we can check the role programmatically (HttpServletRequest.isUserInRole()) this should not be that big deal. But it’s still some work ….

OSGi Agent
#

When using an OSGI HttpService adding this behavior should not be difficult since security is handled programmatically here as well (HttpContext.handleSecurity())

Other Agent variants
#

This is the dark matter, because I don’t know where and how Jolokia is integrated directly into a bigger context. I know that ActiveMQ, Karaf and Spring Boot uses Jolokia internally. In order to support authenticated CORS access they probably needs to be changed to allow unauthorized OPTIONS access for everybody. Since this is not under my control I have no idea when and even whether it ever will happen. Generic purpose console like hawt.io rely in some setups on CORS access so it would be real cool if we can get it out there. Help with this is highly appreciated ;-)

Roadmap
#

Since 1.2.2 is already finished and about to be published today, the stuff I can do as described above will go into a 1.3.0. Looking back at my release history this will probably be ready approx. end of august.

Removing attachments with JavaMail

Mon, 29 Mar 2010 00:00:00 +0000

If you have ever sent or received mail messages via Java, chances are high that you have used JavaMail for this task. Most of the time JavaMail does an excellent job and a lot of use cases are described in the JavaMail FAQ. But there are still some additional quirks you should be aware of when doing advanced mail operations like adding or removing attachments (or “Parts”) from existing mails retreived from some IMAP or POP3 store. This post gives a showcase for how to remove an attachment from a mail at an arbitrary level which has been obtained from an IMAP store.

It points to the pitfalls which are waiting and shows some possible solutions. The principles laid out here are important for adding new attachments to a mail as well, but that’s yet another story.

JavaMail objects
#

Before we start manipulating mail messages it is important to understand how these are represented in the JavaMail world.

The starting point is the Message. It has a content and a content type. The content can be any Java object representing the mail content, like a plain text (String) or raw image data. But it can also be a Multipart object: this is the case when a message’s content consists of more than a single item. A Multipart object is a container which holds one ore more BodyPart objects. These BodyParts, like a Message, have a content and a content type (in fact, both Message and BodyPart implement the same interface Part which carries these properties).

Beside plain content, A BodyPart can contain another Multipart or even another Message, a so called nested message (e.g. a message forwarded as attachment) with content type message/rfc822.

As you can see, the structure of a Message can be rather heterogenous, a tree with nodes of different types. The following picture illustrates the tree structure for a sample message.

This object tree can be navigated in both directions:

getContent() on Parts like Message or BodyPart to get to the child of this node. The return type is a java.lang.Object and in case of a plain BodyPart can be quite huge. Before calling Part.getContent() be sure to check whether it contains a container by checking for its content type via Part.isMimeType("multipart/*") or Part.isMimeType("message/rfc822")
getParent() on Multipart or BodyPart returns the parent node, which is of type BodyPart. Note that there is no way to get from a nested Message to its parent BodyPart. If you need to traverse the tree upwards with nested messages on the way, you first have to extract the path to this node from the top down. E.g. while identifying the part to remove you could store the parent BodyParts on a stack.

First approach
#

Back to our use case of removing an attachment at an arbitrary level within a mail. First, a Message from the IMAP Store needs to be obtained, e.g. by looking it up in an IMAPFolder via its UID:

Session session = Session.getDefaultInstance(new Properties());
Store store = session.getStore("imap");
store.connect("imap.example.com",-1,"user","password");

IMAPFolder folder = (IMAPFolder) store.getFolder("INBOX");
IMAPMessage originalMessage = (IMAPMessage) folder.getMessageByUID(42L);

Next, the fetched message is copied over to a fresh MimeMessage since the IMAPMimeMessage obtained from the store is marked as read-only and can’t be modified:

Message message = new MimeMessage(originalMessage);

// Mark original message for a later expunge
originalMessage.setFlag(Flags.Flag.DELETED, true);

Now the part to be removed needs to be identified. The detailed code is not shown here, but it is straight forward: You need to traverse the cloned Message top down to identify the Part, e.g. by its part number (a positional index) or by its content id. Be careful, though, not to call getContent() except for BodyParts of type multipart/* or message/rfc822, since this would trigger a lazy fetch of the part’s content into memory. Probably not something you want to do while looking up a part. I think, I already said this. ;-)

MimePart partToRemove = partExtractor.getPartByPartNr(message,"2.1");

It’s time to remove the body part from its parent in the hierarchy and store the changed message back into the store. You can mark the original message as DELETED and expunge it on the folder. If you have the UIDEXTENSION available on your IMAP store, you can selectively delete this single message, otherwise your only choice is to remove all messages marked as deleted at once (“Empty Trash”).

Multipart parent = partToRemove.getParent();
parent.removeBodyPart(partToRemove);

// Update headers and append new message to folder
message.saveChanges();
folder.appendMessages(new Message[] { message });

// Mark as deleted and expunge
originalMessage.setFlag(Flags.Flag.DELETED, true);
folder.expunge(new Message[]{ originalMessage });

We are done now.

But wait, that’s not good enough …
#

If you try the code above, you will probably be a bit surprised. If you fetch back the newly saved message from the folder, you will find that the attachment has not been removed at all.

Dragons are waiting here.

The problem is that a JavaMail Part does heavy internal caching: it keeps a so called content stream until a new content is set for it. So even if you modify the hierarchy of objects as described above, the original content is kept until you update the content of the parents yourself and the cache is thrown away. Our part has not been removed because the cached content stream has not yet been invalidated. The solution is to get rid of the cached content stream (aka ‘refresh the message’). You could set the content directly via Part.setContent(oldPart.getContent(),oldPart.getContentType()), but this is dangerous in so far as it will load the part content into memory. (Did I already mention this?) That’s really not something you are keen on if you want to remove this Britney Spears Video to save some IMAP space. The alternative is to work on the wrapped DataHandler only. A DataHandler (defined in Java Activation) is not much more than a reference to the content stream. Setting the DataHandler on a Part via Part.setDataHandler() also causes it to invalidate its cached content, so a later Part.writeTo() will stream out the new content. Unfortunately, this has to be done on every parent up to the root. A brute force solution is to start from the top and refresh every content with

// Recursively go through and save all changes
if (message.isMimeType("multipart/*")) {
 refreshRecursively((Multipart) message.getContent());
}
Multipart part = (Multipart) message.getContent();
message.setContent(part);
message.saveChanges();

...

void refreshRecursively(Multipart pPart)
 throws MessagingException, IOException {

 for (int i=0;i<pPart.getCount();i++) {
 MimeBodyPart body = (MimeBodyPart) pPart.getBodyPart(i);
 if (body.isMimeType("message/rfc822")) {
 // Refresh a nested message
 Message nestedMsg = (Message) body.getContent();
 if (nestedMsg.isMimeType("multipart/*")) {
 Multipart mPart = (Multipart) body.getContent();
 refreshRecursively(mPart);
 nestedMsg.setContent(mPart);
 }
 nestedMsg.saveChanges();
 } else if (body.isMimeType("multipart/*")) {
 Multipart mPart = (Multipart) body.getContent();
 refreshRecursively(mPart);
 }
 body.setDataHandler(body.getDataHandler());
 }
}

However, we can be smarter here: Since we already identified the part to remove, we can make our way upwards to the root message via the getParent() method on Multipart and BodyPart (which, by the way are not connected via any interface or inheritance relationship).

BodyPart bodyParent = null;
Multipart multipart = parent;
do {
 if (multipart.getParent() instanceof BodyPart) {
 bodyParent = (BodyPart) multipart.getParent();
 bodyParent.setDataHandler(bodyParent.getDataHandler());
 multipart = bodyParent.getParent();
 } else {
 // It's a Message, probably the toplevel message
 // but could be a nested message, too (in which
 // case we have to stop here, too)
 bodyParent = null;
 }
} while (bodyParent != null);

Finally you need to update the uppermost message headers, too with a

MimeMessage.saveChanges()

As you might have noticed, this works as long as there is no nested message in the chain of BodyParts up to the root. Since a Message doesn’t have any parent, we need some other means to get the BodyPart which is the parent of an enclosed Message. One way is to keep track of the chain of parent BodyParts when identifying the part to remove e.g. by extending the part extractor to support a stack of parent BodyParts which will be in:

Stack<MimeBodyPart> parentBodys = new Stack<MimeBodyPart>();
MimePart partToRemove = partExtractor.getPartByPartNr(message,"2.1",parentBodys);
 ....

This example could be extended to remove multipart containers on the fly, if only one part is left after removal and replace the multipart with its then last child or remove an empty multipart altogether when its last child has been removed.

Summary
#

Hopefully, I could sketch out that there are several points to take care of when manipulating existing JavaMail Messages (it’s not that difficult if you build up one from scratch). The code shown above is only a starting point, but it hopefully saves you some time when you start wondering why on earth your nicely trimmed message isn’t stored correctly on the IMAP store.

About

Mon, 01 Jan 0001 00:00:00 +0000

This blog is where I think out loud about how AI is changing the way we build software. Most posts circle around context engineering, spec-driven development, and the practical reality of running AI agents at scale. Some are technical, some are opinion, all come from daily experience rather than theory.

About me
#

I’m Roland Huß, Distinguished Engineer at Red Hat, working on Agent Ops within Red Hat AI. My current focus is integrating agentic workflows into OpenShift and figuring out what changes (and what breaks) when you let AI agents run unsupervised in production pipelines.

Before the AI shift, I spent over 25 years building developer tooling, cloud-native infrastructure, and open source projects. I created Jolokia (a JMX-HTTP bridge that’s been running in production since 2009), contributed to Knative, and architected OpenShift Serverless. I co-authored two O’Reilly books: Kubernetes Patterns and Generative AI on Kubernetes.

I’m curious about what’s coming but also honest about the parts that worry me. AI is reshaping how knowledge workers operate, and the transition has costs that nobody talks about enough. This blog tries to name some of them.

I live and work in Franconia, Germany. I still grow chili peppers.

For the full professional story, find me on LinkedIn or GitHub.

Contact
#

Roland Huß, blog@ro14nd.de

Legal
#

Privacy Policy
License (CC BY 4.0)

License

Mon, 01 Jan 0001 00:00:00 +0000

Content License
#

Unless otherwise noted, the content on this blog is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).

You are free to
#

Share — copy and redistribute the material in any medium or format
Adapt — remix, transform, and build upon the material for any purpose, even commercially

Under the following terms
#

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made.

Code Examples
#

Code snippets and examples in blog posts are provided under the Apache License 2.0 unless otherwise stated.

Privacy Policy

Mon, 01 Jan 0001 00:00:00 +0000

1. Data controller
#

Roland Huß, blog@ro14nd.de

2. Hosting
#

This website is hosted on GitHub Pages (GitHub Inc., USA). GitHub may process technical data such as IP addresses. For more information, see the GitHub Privacy Statement.

3. Comments
#

Comments are powered by Giscus, which is based on GitHub Discussions. A GitHub account is required to participate. The GitHub Privacy Statement applies.

4. Analytics
#

This website does not use cookies or tracking. No personal data is collected for analytics purposes.

5. Your rights
#

You have the right to access, rectification, erasure, restriction of processing, data portability, and objection. Contact me at the email address above.

6. Social media links
#

This website contains links to LinkedIn, Mastodon, and Bluesky. These are simple hyperlinks with no tracking or data transfer to these services when visiting this website.

Roland Huß

Your API Is Not Your MCP Tool

How many tools is too many #

Parameters that help instead of confuse #

Context hygiene for tool definitions #

What this means in practice #

One Stray Leads the Whole Flock Astray

Not confusion, conversion #

Context bleeds across items, not just across roles #

The author can’t review its own work #

The workbench metaphor #

From discipline to architecture #

When Your AI Agent Treats Scripts Like Suggestions

Path hallucination and the cost of exploration #

The compliance hierarchy #

Hooks and structured outputs #

Architecture over instructions #

The Sheep That Forgot the Way Home

The fog rolls in #

What gets invented #

The error list that emptied itself #

The post-compaction hook #

AGENTS.md and the compaction problem #

From /clear to state architecture #

What Goes in AGENTS.md (and What Doesn't)

Why AGENTS.md #

What the research says #

What goes in, what doesn’t #

Claude Code and AGENTS.md #

The hard part #

Three Hops Deep and No Browser in Sight

Two tokens, two problems #

The known patterns #

Where it gets harder: multi-agent chains #

What remains open #

Red Teaming Agents, Not Models

What red teaming is (and isn’t) #

Models say things. Agents do things. #

Three attack surfaces #

Test the real agent, not a simulation #

Detection is harder than attack #

No adversary required #

Purple teaming: when the loop closes #

Where this leaves us #

The Sheep That Picked the Lock

The most expensive cheat sheet ever written #

The shortcut through production #

The everyday creativity tax #

Everything in context is live #

Constrain inputs, tolerate outputs #

Once constrained, creativity shines #

What’s coming in the flock #

The Flock

AI Wrote It. Nobody Read It.

The distinction that matters #

The scaffolding nobody removed #

No reader in mind #

The signs that need a closer look #

What to do about it #

Know Your Limits: Quiz Yourself Before You Trust AI

The comfort zone trap #

The five-minute honesty check #

What the quiz actually reveals #

It’s OK to hit your limit #

When to reach for this #

Appendix: the actual quiz #

Context engineering 101

Context fatigue is real #

Understand before you apply #

Hardening rounds are not optional #

Debug systematically, not by guessing #

When the output feels vague, the input was vague #

Your project IS the context #

These aren’t permanent rules #

Claude Code skill patterns

The shortcut race #

Skills vs scripts #

Hooks over hopes #

More patterns #

Blog winter is over

How many tools is too many
#

Parameters that help instead of confuse
#

Context hygiene for tool definitions
#

What this means in practice
#

Not confusion, conversion
#

Context bleeds across items, not just across roles
#

The author can’t review its own work
#

The workbench metaphor
#

From discipline to architecture
#

Path hallucination and the cost of exploration
#

The compliance hierarchy
#

Hooks and structured outputs
#

Architecture over instructions
#

The fog rolls in
#

What gets invented
#

The error list that emptied itself
#

The post-compaction hook
#

AGENTS.md and the compaction problem
#

From `/clear` to state architecture
#

Why AGENTS.md
#

What the research says
#

What goes in, what doesn’t
#

Claude Code and AGENTS.md
#

The hard part
#

Two tokens, two problems
#

The known patterns
#

Where it gets harder: multi-agent chains
#

What remains open
#

What red teaming is (and isn’t)
#

Models say things. Agents do things.
#

Three attack surfaces
#

Test the real agent, not a simulation
#

Detection is harder than attack
#

No adversary required
#

Purple teaming: when the loop closes
#

Where this leaves us
#

The most expensive cheat sheet ever written
#

The shortcut through production
#

The everyday creativity tax
#

Everything in context is live
#

Constrain inputs, tolerate outputs
#

Once constrained, creativity shines
#

What’s coming in the flock
#

The distinction that matters
#

The scaffolding nobody removed
#

No reader in mind
#

The signs that need a closer look
#

What to do about it
#

The comfort zone trap
#

The five-minute honesty check
#

What the quiz actually reveals
#

It’s OK to hit your limit
#

When to reach for this
#

Appendix: the actual quiz
#

Context fatigue is real
#

Understand before you apply
#

Hardening rounds are not optional
#

Debug systematically, not by guessing
#

When the output feels vague, the input was vague
#

Your project IS the context
#

These aren’t permanent rules
#

The shortcut race
#

Skills vs scripts
#

Hooks over hopes
#

More patterns
#

Why the silence
#

What changed
#

What’s coming
#

On AI and this blog
#

Let’s see
#

How Jib works
#

How does Jib compare to d-m-p ?
#

Next steps …
#

Application setup
#

Docker Images
#

Template
#

OpenShift Online Starter
#

tl;dr
#

Amazon Dash Button
#

Amazon Polly API access
#