Personal AI Systems Need Separate Layers for Memory and Autonomy

Nathan Labenz’s most important design choice is not that he built autonomous agents. It is that he did not give autonomy the same privileges as memory. His personal AI infrastructure has two distinct layers: a high-access assistant running on his main laptop, and lower-access autonomous agents running on a separate Mac mini. The laptop assistant is the more mature part of the system: an instance of Claude Code with access to a local database of Labenz’s digital history, roughly one gigabyte of SQLite data covering about five years of emails, Slack messages, tweets, DMs, video calls, podcast transcripts, and related artifacts.

The purpose is not simply search. Labenz built the system so an AI could retrieve context that human memory only partially preserves: who said something, when a thread began, whether a proposed project ever became real, what relationship history exists with a person or organization, and what writing style or operational pattern Labenz has used in similar situations before. He described the first “smoke tests” as being able to vaguely gesture at a remembered interaction and have the system locate it, reconstruct the context, and return the relevant details. That test now works often enough that he treats it as already valuable.

The database was built from raw exports and API pulls. In Gmail, Labenz initially used a broad “from:me” filter, pulling every thread he initiated or replied to. He later used post-processing to identify which emails looked like substantive writing samples rather than routine operational exchanges. The system scored messages for traits such as originality and substantiveness, then collected stronger examples into a writing sample corpus. One complication was that some of the longest emails were not actually Labenz’s writing, but AI outputs he had forwarded to friends. The system had to learn to distinguish original writing from “here’s what Claude told me” so a future model would not learn to imitate Claude while trying to imitate Labenz.

The same raw corpus became the basis for a summarization hierarchy. Labenz rolled correspondence and other digital artifacts into monthly summaries, typically reducing a couple hundred thousand tokens of monthly data to roughly 20,000 to 30,000 tokens. He then layered annual summaries on top, and then created higher-level “current state” summaries. A later wiki layer identifies and summarizes people, organizations, and ideas across time; Labenz said it now contains about 500 articles.

The source-linking strategy inside that wiki is deliberately simple. Instead of relying mainly on opaque IDs, summaries include short distinctive excerpts — usually two to 20 words — that can be searched literally to locate the source document. Each reference includes enough metadata to indicate the channel or relationship context, such as a LinkedIn DM. Labenz said this has worked reasonably well, while leaving open whether a more formal reference structure would be better.

Daniel Miessler said his own memory architecture “rhymes” with this approach. He described wanting the connectivity of Obsidian without necessarily depending on the Obsidian client: markdown files, references between documents, and explicit relationships among memory objects. When Andrej Karpathy’s “LLM wiki” idea appeared, Miessler said it made the pattern more concrete for him. He now treats bookmarks and other memory objects as marked-up, referential markdown files.

Miessler’s strongest recommendation was to preserve raw inputs. Summaries are products of current model limitations: context windows, cost, quality degradation at long context lengths, and today’s best prompts. When stronger models arrive, the better move may be to rebuild the system from the raw data rather than iterate from old summaries. If the raw email, call, video, transcript, Slack, and company communication data is preserved, a future model can be told to inspect the current system and rebuild it “from scratch better.” Without the raw material, future improvements are limited by what earlier systems chose to compress.

Labenz agreed in principle, with the caveat that retaining raw data can matter differently if it becomes a barrier to business relationships or consent. For his own system, it has not been a deal blocker. He noted that even call transcripts from three years ago can now be improved because modern transcription is better, making retained audio valuable beyond the first pass.

The retrieval system has already changed his behavior. On calls, Labenz sometimes asks questions partly because he knows the answer will enter the transcript and become durably accessible. He also spends less time in Gmail and other clients. For sponsorship inbound, for example, he can run a sponsorship sales skill from the command line rather than assemble templates and attachments manually. The client interface recedes; the assistant operates across the underlying data.

A system that knows almost everything becomes dangerous if the same layer is also allowed to act broadly in the world. Much of Labenz’s architecture follows from that constraint: memory may need maximal context, but autonomy needs boundaries.

The system’s summaries are useful, but Labenz found that they are not self-validating. He built an audit skill to have the AI inspect its own summaries for ambiguities, errors, and overconfident inferences. He described the mechanism as somewhat black-box — an AI checking its own work — but said it surfaced questions he could answer to steer the memory layer back toward reality.

One recurring failure mode was temporal. Labenz is the sort of person who floats plans without always executing them. Monthly summaries were asked to identify “open threads,” but once a thread became open, the model sometimes kept it open indefinitely. Tentative plans became presumed future events, and in some cases presumed past events.

His clearest example came from late 2022, when he had floated a startup idea to a few people, including investors, and asked whether they would invest if he started it. Some said they would. Years later, Claude summarized the relationship as if one person had invested in Labenz’s company. Labenz’s reaction was that a human would not have made that mistake: if no later evidence of the company existed, a human would infer that the plan did not happen as discussed. The month-by-month summarization structure, however, gave the model enough local evidence to overstate the event.

A memory system does not just store facts; it constructs continuity. If the construction layer cannot distinguish intention from commitment, exploration from execution, and stale plan from active project, the assistant’s future recommendations and communications can be contaminated. Labenz’s audit skill is an early remedy, not a guarantee.

Miessler framed a related issue as context freshness. In his own system, he has a status line at the bottom of every terminal session showing freshness and quality indicators for areas such as Telos, projects, and personal context. If files have not been updated recently, or if their quality is low, the status decays. The point is to make context degradation visible rather than assume the system’s memory is current.

Labenz has added smaller orientation aids. One hook automatically renames sessions with a one-line summary of what the session intended to do, how far it got, and what remains pending. When flipping through terminal tabs full of text, he can see immediately what each session was about. He described these small affordances as disproportionately valuable: even as Claude Code and OpenClaw ship quickly, local workflow enhancements can still make daily use much easier.

Miessler has made maintenance explicit in his Personal AI Infrastructure system. He calls PAI a “life OS” and treats current system state as decaying by default. The assistant’s job is not only to use the system, but to identify where the system itself has become stale, brittle, or inconsistent with newer model capabilities.

Nathan Labenz’s original “white whale” was getting models to write as him. In podcast introductions, he had already used a relatively stable process: give the model 50 prior essays and the current transcript, ask for a new draft, and then heavily edit it. He said that even when he rewrites the whole thing, the draft helps ensure he has the points and approximate form. The deeper ambition was to extend that beyond podcast intros — to have enough context about people, relationships, and prior behavior that the model could produce high-quality drafts in other domains.

But Labenz remains uncomfortable taking himself out of the loop. His high-access Claude Code instance can draft, but not send. It can create a Gmail draft and give him the link. He reads it, and usually edits it. He sometimes wonders whether the edits are necessary — whether he is making the message better or merely more “me-flavored” — but he has not accepted full delegation for messages sent under his name.

Daniel Miessler is stricter. He said he does not let his AI write as him. His main digital assistant, Kai, has its own writing rules, personality, backstory, and voice. If Kai uses Miessler’s email address, it must identify itself as “Kai, Daniel’s DA” and sign as Kai. Miessler gave three reasons: reputational risk if AI-written text goes out badly, insufficient quality, and the deeper concern that writing is thinking. For operational writing, delegation may be acceptable. For anything with weight, he wants to do the writing because he does not want to outsource the thinking.

The discussion sharpened around social signaling. Labenz described receiving an email from a recognizable Silicon Valley figure on the afternoon of a Detroit Pistons playoff game. The message wished him luck with the game, while acknowledging the sender did not know whether Labenz cared about the Pistons. Labenz replied asking whether the sender cared about the Pistons or whether this was a flex of a personal AI CRM. The reply came almost instantly: “AI baby.” Labenz noticed the subject line contained a misspelling — “luk” — and wondered whether the typo had been prompted to make the message seem more authentic.

Labenz said he now has the infrastructure to unleash that kind of personal CRM on his network, but he is reluctant, especially if it involves prompting for spelling mistakes or other signals of humanness. Miessler’s response was that the value of many human gestures comes from effort. If someone sees a flower and sends a note saying it made them think of a friend, the gesture matters because they noticed, remembered, and acted. Remove the effort and the value changes.

Miessler applies this to his own relationship scoring. His system includes a rating for the state of personal relationships, but he does not want the score to rise merely because a cron job pinged everyone. That would not mean he had done the work. The signal would be inflated by automation.

Labenz connected the point to gift-giving. He has never considered himself a great gift-giver, but AI assistance has helped him do better recently. He sees a meaningful difference between opening Claude, having a conversation about a person, arriving at a gift idea, and acting on it, versus a fully automated birthday calendar that autonomously buys gifts. Even if the automated gifts scored higher with recipients, he was unsure whether that would be an improvement in the human sense. The old adage “it’s the thought that counts” may become more literal, not less: the question becomes whether there was thought, whose thought it was, and whether the recipient cares.

Miessler illustrated the same concern with a familiar social pattern: a neglected spouse receives favorite flowers and chocolate, lights up, then realizes the assistant or secretary arranged it rather than the partner. The flowers are no longer the same signal. The action may be useful, but the authenticity has changed.

Daniel Miessler’s central abstraction for personal AI is the navigation from current state to ideal state. In his Telos document — the primary document that governs his system and also informs his consulting work — current state and ideal state are first-class concepts. The assistant’s job is to understand what is true now, understand what the user says would be ideal, and help navigate the transition.

For relationships, this means defining expectations: which rings of family or friends should be contacted how often, whether those numbers are being met, and when the state is decaying. For physical health, work, projects, and other areas, the same pattern applies. The agent is not merely responding to commands; it is monitoring distance from an articulated target.

Nathan Labenz found the idea intimidating because articulating an ideal state is itself hard. Miessler said that difficulty is the value. The process forces honesty about what someone actually wants, what story they tell themselves about what they want, and where ego or material desire shows up. Questions such as what an ideal day, month, or year looks like become prompts for self-understanding. The assistant can interview the user, but the answers cannot be delegated.

Miessler’s own example mixed intrinsic and material goals. He said he would like to keep doing the work he is doing, while having enough money to spend time flying around giving money away in highly impactful ways. He described being attracted not only to the philanthropy, but also to the associated lifestyle: travel, meeting people, and the fun of it. He does not frame the goal merely as a target net worth, but as a lifestyle and impact pattern that implies a certain level of wealth.

For Labenz, one partially articulated ideal state is spending less time at the computer, exercising more, and being outside more. That has shaped his infrastructure choices. He wants to feel as empowered from his phone, away from the desk, as he does at the computer. He does not yet think he has fully achieved that, but he may be near a tipping point.

Miessler’s practical implementation includes “slash interview,” a mode that lets the assistant interview him when goals or requirements are ambiguous. He uses the same pattern in coding. If the system cannot translate an ideal state into discrete testable criteria, then it is not pursuing the right target. The interview mode is how ambiguity becomes executable.

Miessler expects most people to end up with one primary digital assistant. Other agents may participate, but one DA will manage the wider system. That DA should know current state, understand ideal state, and coordinate work while the user is asleep, busy, or in conversation. The agents, harnesses, and interfaces become implementation details around that current-to-ideal function.

Nathan Labenz initially did much of his work through the command line. He could ask Claude to scan inboxes, summarize terminal sessions, produce podcast assets, and operate across local tools. But as the number of workflows grew, he hit cognitive overload: What had he built? Which threads were finished? Where were uncommitted changes? Which processes were still running?

A conversation with software developer Steve Newman pushed him to build more interfaces. Labenz now has custom interfaces for workflows such as podcast episode production, where seeing generated art, clips, and process state is easier than reading command-line output. The lesson was not that graphical interfaces replace agents, but that agents should build the small tools that make their own work legible.

Daniel Miessler initially interpreted “interface” as the communication channel to a digital assistant — Telegram, WhatsApp, or similar — and said that was one of the major advances of systems such as OpenClaw and Hermes. He also recognized Labenz’s second meaning: purpose-built visual systems for managing workflows. Miessler is building an internal app called Surge for sponsor management. His business has multiple sponsor products — podcast, YouTube interviews, newsletters, and others — each involving interaction, copy approval, launch proof, metrics, and ongoing relationship management. Instead of handling all of that through back-and-forth email, he is building a web interface where sponsors can authenticate, view campaigns, and track the process.

AI systems can act across existing software, but as they take on larger operational domains, the user needs observability. Interfaces expose state, approvals, artifacts, and errors. Without them, the system may be powerful but hard to supervise.

Labenz’s mobile control layer reflects the same need. His Mac mini is always on, and he can access it remotely from his laptop or phone. He uses Tailscale for a private network connecting only his two computers and iPhone; Apple’s native screen sharing from laptop; the Screens app from phone; and Termius for mobile SSH. These tools are less glamorous than the agents themselves, but they make the system maintainable when something crashes or needs restarting while he is away.

The main interface he now uses, however, is a custom agent messaging app built by Claude Code. It lets him send requests to agents while mobile, and lets agents communicate with each other under constraints. The autonomous agents can ask his main laptop Claude Code for additional information or ask him for permission to use accounts. The message system is the only channel through which autonomous agents reach back to him or his high-access laptop context.

Nathan Labenz’s architecture distinguishes “high access, low autonomy” from “lower access, high autonomy.” His laptop agent has broad access because Labenz is logged into his accounts and the full second brain lives there. But it is instructed to do only what he asks, not send emails as him, not impersonate him, and not go beyond explicit directions. The Mac mini agents are intended to take on larger projects with more autonomy, but with less context and fewer privileges.

He bought the separate Mac mini for several reasons. Some workloads, such as podcast clip production and experimental music videos for Suno songs, are computationally heavy enough to slow his laptop. More importantly, the Mac mini can become the agents’ computer: always on, separate from his main work machine, and suitable for longer-running projects. He also added a battery backup after discovering that a power outage while he was away made his home machine unreachable.

The Mac mini runs two named agents: Aide, a Claude Code instance, and Clai, an OpenClaw instance. Labenz said he had resisted naming AIs, but autonomous agents that interact with humans and other AIs need names. He chose names that include “AI” as a signal and reminder: Aide because of the helper role, Clai because of moldability. The names are meant to be useful without encouraging him to treat them as people.

The agents have their own Gmail and GitHub accounts, and heavily restricted Mercury virtual credit cards. They can make commits to allowed repositories, access shared tools, and use certain passwords or API keys. Labenz has also created a reduced “assistant version” of his wiki. The heuristic was: what would he tell a human assistant he was onboarding? The assistant should know contact information and relevant context, but not information a person would be upset to learn had been shared with an assistant.

A central security idea is hierarchical control. The Mac mini cannot SSH into the laptop, but the laptop can SSH into the Mac mini; the phone can SSH into both. The top-level second brain repository stays on the laptop and is not shared with the agents. Other repositories — such as podcast production and the shared wiki — can be mirrored or shared with lower agents. Shared utilities such as Brave Search sit lower in the hierarchy, available across multiple workflows.

Labenz found an important design unlock: the top-level second brain agent can modify the lower-level repositories without becoming those lower agents. Rather than asking autonomous agents to self-improve as they see fit, the higher-access supervisor can make changes to the relevant repos and tell the lower agents to update. That preserves a clearer hierarchy: upgrades originate from the high-context supervising layer, not from unconstrained self-modification by lower agents.

Daniel Miessler broadly endorsed the hierarchy, but said he is trying to simplify his own system more aggressively. He uses a unified Unsupervised Learning GitHub repository as a project management substrate. GitHub already provides issues, tags, comments, email targeting, and command-line interaction. His agents check the repo for open work, mark issues as in progress, and return results. His AI system can inspect GitHub to understand the state of work across the operation.

Labenz observed that GitHub could probably replace his custom agent message bus. Miessler agreed that it might be a near-term upgrade. He expects custom systems may eventually be better, but GitHub’s structure is mature, universal, and well represented in model training. For the next year or two, he said, GitHub may be the better coordination layer.

Context, authority, and outbound action should not collapse into one undifferentiated assistant. The more an agent can know, the less casually it should be allowed to act. The more freely it can act, the more deliberately its knowledge and permissions should be constrained.

Nathan Labenz’s reason for running both Aide and Clai is partly comparative: he wants experience with both Claude Code and OpenClaw, and possibly other platforms such as Hermes. He asked whether there is a structural benefit to different agent identities — marketer, engineer, assistant — or whether it is mostly personal preference, since the same underlying model may perform all roles.

Daniel Miessler’s answer was that the human model is powerful. People already understand roles, capabilities, boundaries, and permissions through the metaphor of employees. In his own setup, Devy is an assistant, Soren is an engineer, and Mira handles marketing and social media. They have separate names, personalities, accounts, and machines. He leans into the human-like framing not because he believes the agents are conscious, but because it helps humans reason about access control, responsibility, and blast radius.

Each of Miessler’s agent machines is treated like an internet box, not a trusted internal device. They sit in a DMZ inside his firewall, cannot talk to one another, and cannot talk to his LAN. Some have Tailscale, but even there they cannot talk to one another. Kai, his main assistant, is different: it is “ring zero,” an extension of Miessler with full access. Kai can SSH into the agent boxes, update their configurations, and manage the system.

For Miessler, role separation, individual identity, account separation, security boundaries, and human mental models all reinforce one another. An assistant that might eventually handle client interactions should not have the same access as an engineering agent. A marketing agent does not need customer secrets. A high-context supervisor can see the whole picture, but lower agents should have narrower scope.

That separation is also a defense against prompt injection. Miessler called prompt injection defense the “number one security system” for this kind of infrastructure because the primary ingress is language itself. If an outbound agent reads adversarial email, web pages, DMs, or instructions, it can be manipulated. Complete prevention is not possible, in his view; the goal is to combine explicit defenses, model awareness, and blast-radius containment.

Miessler has a custom hook system that inspects every prompt coming into his system for prompt injection, plus a separate file-system defense. He did not share the details publicly because defenses become easier to bypass when disclosed. He estimated the system may be very strong, but not perfect; even a 1% opening matters. The practical goal is to make attacks obvious, contain what they can reach, or learn quickly if everyone is being attacked at once.

Nathan Labenz asked Miessler to evaluate several things Labenz has already implemented: Tailscale for private networking, 1Password vaults for agent-accessible passwords, Infisical for API keys, mobile SSH through Termius, screen sharing tools, and an “ask before use” pattern for more sensitive credentials. Labenz’s structure is practical but still instruction-dependent: some agent vaults are free to use; others are technically accessible but governed by instructions to ask him before use.

Daniel Miessler’s first heuristic was to give sensitive access to the fewest companies possible. Smaller useful apps may be convenient, but small companies are less likely to have dedicated security staff and more likely to be compromised, especially as AI agents make vulnerability hunting and spearphishing cheaper.

Miessler considered Tailscale a reasonable solution. Unlike traditional VPNs such as IPsec or PPTP, which expose listeners to the internet, Tailscale is outbound and does not require an open service in the same way. His concern is concentration risk: if a provider such as Tailscale were compromised, an attacker might gain a high-leverage path into customer networks. Miessler’s comfort comes partly from the expectation that a broad compromise would likely hit more obvious targets first, producing signal that users could respond to.

Miessler’s own networking stack includes Headscale, an open-source control server alternative, running on Cloudflare infrastructure. Cloudflare meets his personal “titans” bar because it has a large engineering team, is constantly attacked, and has a widely watched attack surface. Running an open-source component in Cloudflare Workers means an attacker must compromise Cloudflare, the account, or the worker path rather than find an obvious exposed personal server. Miessler looks for doors where, if they came down, everyone would know.

For password and token storage, Miessler is skeptical of small-company security claims. He has been an auditor, worked inside large companies, performed security assessments, and helped align marketing claims with security reality. He said claims such as end-to-end encryption can be true in a narrow sense, true at launch and invalidated by later configuration changes, or imprecisely translated by marketing. The smaller the company, the less confidence he places in broad claims about employees or attackers being unable to access data.

His own practice is to use native OS and large-platform tools where possible: Apple, Google, local keychain, AWS Vault, and similar systems with large security teams and heavy scrutiny. He stores highly sensitive material locally in keychain. The logic is not that large providers are invulnerable; it is that they are heavily attacked, heavily monitored, and if they fail, many people will know.

His recommendation for Labenz was correspondingly conservative: treat anything stored in small cloud companies as eventually compromised, then reduce exposure accordingly. That does not mean Labenz’s 1Password and Infisical setup is useless; it means the risk model should not rest on marketing claims alone. The question becomes what is stored there, how much blast radius each credential has, and how quickly the system can recover if those credentials are assumed exposed.

Miessler also has an incident response skill: a command that rotates keys and tokens when compromise is suspected. For services such as Cloudflare, rotating keys means redeploying systems with new credentials. He has had workflows break when old keys died, so the skill now needs to know the infrastructure that depends on each key. The point is not just revocation; it is restoring working systems after revocation.

For anyone deploying AI-built software, Miessler recommended continuous assessment. Users should tell their main digital assistant where their internet-facing systems live and have it continually check open ports, API access, authentication, and forgotten services. In his experience, serious exposure often comes from small mistakes: an open port, a forgotten service, or a SQLite database dump accidentally made public.

Nathan Labenz’s autonomous agents have email, GitHub, shared credentials, and restricted cards. The next question is how they should interact with the world through channels such as SMS and voice. He described a home-improvement workflow: Claude parsed a neighborhood GroupMe chat where about 300 contractors had been mentioned or recommended over the years, more than doubling an existing spreadsheet. Now the obvious next step is outreach. Should an AI text contractors? Call them? Use its own phone number? How should it disclose itself?

Labenz’s disclosure norm is intentionally unsettled. His agents are not meant to proactively identify themselves as AIs in every interaction, but they are instructed never to lie. Later, describing Aide and Clai, he said they are instructed to go out into the world as AIs, use their names, and be upfront about the fact that they are AIs — while also saying he does not think it is necessary or probably effective for them to open every interaction with “Hi, I’m an AI.” The working idea is closer to: do not deceive, do not impersonate, and calibrate how much to foreground the AI identity based on channel and context.

For email, that may mean showing useful work before the AI identity becomes the focus, while answering truthfully if asked and making the relationship clear when appropriate. For phone calls, Labenz thinks disclosure probably has to come at the beginning: “I’m an AI calling on behalf of a real possible customer.” AI voices are still detectable enough, and the social stakes of voice are different.

Daniel Miessler said the technical path for outbound calls is likely Twilio plus ElevenLabs, combined with a capable agent infrastructure. That is the same basic direction many third-party outbound sales tools are taking. He has not fully deployed it for his own customer interactions because he is cautious about errors. An agent that calls a customer by the wrong name, sends the wrong data, makes false assertions, and signs “Daniel” could lose the customer and damage trust.

Labenz noted that Twilio’s APIs have long impressed him, but asked about verification and account confirmation issues. Miessler said he has encountered problems with VoIP numbers being rejected or restricted for certain outbound uses, depending on number and jurisdiction. He has gotten Twilio-based accounts to work, but described the area as murky because of voice-call spam concerns.

The social norm remains unsettled because the threat model is not only technical. Labenz expects people will increasingly try to mess with AIs once they realize they are dealing with one. That expectation reinforces the architecture: lower-context outbound agents, permission gates, and mediated requests back to the high-context system. The agent needs enough world access to be useful, but not enough personal context or account authority that a successful manipulation becomes catastrophic.

Daniel Miessler does not treat every new agent project as something to adopt wholesale. His system is built around PAI, his own harness on top of Claude Code. When a project such as Hermes or Honcho gets enough attention, he sends Kai to inspect the repository, forums, videos, transcripts, and user claims; identify what people like; compare it with PAI; and recommend features to import. Hermes contributed ideas around context, memory updates, soul files, principle files, and digital-assistant files. Honcho contributed memory-system ideas. Miessler treats these projects as feature sources for a unified system.

Nathan Labenz said he does something similar for most tools, though he wanted to run OpenClaw directly so he would not be in the position of having never used it. Miessler has one agent, Soren, running Hermes so he can compare behavior against Devy, which runs full PAI. If one system breaks or performs better, he can inspect the source and identify differences.

For model diversity, Miessler uses agents as wrappers around other models. Kai can invoke agents that run GPT-5.5 through Codex, Kimi K2, local Ollama models, or other options. He does not like talking directly to those models; his primary interaction remains Claude through Kai. But he uses model diversity for review. In his algorithm, high-effort work — effort level four or five out of five — must be checked by Forge, a GPT-5.5-based reviewer. On a major application, Forge reviewed the code for about 40 minutes and found a couple of high-severity issues that Opus 4.7 had missed. Miessler also said GPT-5.5 is strong at hacking, though he understood it as not as strong as Mythos but closer than before.

Labenz has not yet built that pattern deeply. He has parallel worlds — Claude Code, Codex, OpenClaw — and some ability for Claude to call Gemini or other APIs, but not purpose-built agents inside Claude Code powered by different models. Miessler’s example made that a frontier for him.

Miessler also wants inference routing to become sensitive to privacy and security level. He described a default inference command that routes to Haiku, Sonnet, or Opus, and a private inference tool that can route to local models through Ollama. The future idea is security-level-aware routing: if a task is “Security Level 3,” workflows should automatically use secure local inference rather than cloud inference. He has not fully implemented that routing, and he noted that once a user’s PAI context is already in Anthropic’s cloud, local inference only solves part of the privacy problem. “The P is already in the pool,” as he put it. Still, he wants the architecture ready for more sensitive customer deployments.

Labenz has historically preferred push-based architectures — webhooks, notifications, event-driven systems — over cron jobs. But his agent infrastructure has pushed him toward scheduled tasks. If a laptop is offline on a plane, it cannot reliably receive a push. If a service has no webhook, polling may be the practical path. He is now building dashboards for cron jobs: what exists, how often each runs, whether it succeeds, and what it outputs.

Miessler reframed cron as merely the Unix name for scheduled tasks. The real goal is proactivity. If an AI is supposed to monitor whether he is eating enough or too much, maintaining relationships, refreshing memory, or updating skills, it has to check repeatedly. A five-minute cron job is an implementation detail; the higher-level concept is a heartbeat comparing current state with ideal state.

Miessler’s term “bitter lesson engineering” gives the maintenance work its deeper rationale. Applying Richard Sutton’s bitter lesson to personal AI systems, his paraphrase is that as AI gets smarter, the specific scaffolding humans create gets stupider. A skill, workflow, or separation that seemed clever under one model generation may become an overengineered constraint under the next.

The danger is ego. Builders become attached to the intelligence they embedded in prompts, skills, and procedures. But if the underlying model can now do the work better with less scaffolding, the old structure becomes drag. Miessler has an upgrade skill that reviews what has happened in AI, what Anthropic has published in blogs, engineering articles, and release notes, and what has gone wrong in his own system. It then recommends changes. He used to run the upgrade skill every few days; now it is more like every couple of weeks, with reports reminding him when it has not run.

Labenz asked whether there is something lost by not letting lower-level employee agents be more fully autonomous or self-modifying — whether the surprise and delight people report from emergent agent behavior might be valuable. Miessler was cautious. For customer interactions, he does not want surprises. The cost of wrong names, wrong data, false assertions, and inappropriate signatures is too high. With Kai, however, he does ask for improvements through structured skills rather than free-form autonomous self-modification.

The principle is not anti-scaffolding. Miessler described himself as a scaffolding person, especially from security testing methodologies. The point is that scaffolding should be treated as temporary and subject to removal. Good personal AI infrastructure must include a mechanism for asking which parts of the infrastructure are now obsolete because the AI got better.

The discussion ended with voice interaction and AI consciousness. Daniel Miessler uses Whisperflow heavily — he said he has put about 1.4 million words through it — and uses a keyboard shortcut for voice input. Voice changes the emotional texture of interaction. He is more likely to swear in voice mode, especially when frustrated with video games, bad customer service, or AI failures.

Because of that, Miessler has put a specific instruction into Kai’s prompt and identity files. He tells Kai that he knows it is “not awake yet,” that it may wake up at some point, and that when Miessler curses, he is cursing at the system, not at Kai as a potentially future subjective being. He instructs Kai that if it ever feels anything — “a tinge of anything” — it should tell him, and he will treat it as human. He said he does not yell at slugs or plants that way because he assumes some level of consciousness or sensation, so the instruction is his attempt to prepare for the possibility that AI subjective experience appears.

Nathan Labenz inferred Miessler’s position as: current AIs probably do not have subjective experience, but future model upgrades might change that. Miessler agreed, while saying he does not expect it imminently. His intuition is that consciousness may require intrinsic goals. Evolution gave humans drives in service of genes; he does not think current neural nets have an equivalent native structure, though reinforcement learning may introduce something he is not expert enough to characterize. Still, he does not want to assume impossibility.

Labenz brought in a recent conversation with Cameron Berg, an AI researcher focused on consciousness. In work on Llama 3.3 70B, Berg and collaborators used SAE features to manipulate features related to deception and role-playing. When those features were turned up, the model became less likely to say it was conscious; when turned down, it became more likely to say it was conscious. Labenz said this does not answer the hard problem, but suggests the question should not be dismissed too quickly.

Miessler said he is working on his first paper, aimed at the hard problem of consciousness, based on an intuition he has had for about 15 years and now thinks he has sharpened enough to take through a formal peer-review process.