Part I: Foundations · Why Autonomous Agents Need Architecture Before LLMs
An autonomous agent does not become safe merely because its language model is impressive. In a manufacturing plant, safety comes from knowing what the agent may see, recommend, change, stop, and never touch.
That distinction matters because the word autonomous hides several very different behaviours. An agent that reads an alarm is not the same as one that opens a maintenance request. An agent that prepares a production sequence is not the same as one that publishes it. An agent that recommends a shutdown is not the same as one that initiates it.
Architecture is therefore not a technical precondition that can be added later. It is the operating boundary that makes autonomy governable.
The situation people actually experience
Picture a late shift in an integrated steel plant. A vibration signal appears on a finishing-line motor. The signal is not yet a confirmed failure. The operator has heard a similar sound before, but the maintenance history shows several possible causes. Production is carrying a priority customer order, and stopping the line would disrupt a sequence that has been prepared over several hours.
An AI agent could help in many ways. It could gather the last maintenance records, compare the signal with previous incidents, retrieve the operating procedure, identify the orders currently on the line, and prepare a recommendation. It might even suggest a controlled inspection during the next safe pause.
But it should not automatically assume that it may stop the line.
The agent may inspect alarms, retrieve procedures, prepare a work order, and recommend a shutdown. Those are four different authorities. Treating them as one capability is how a useful assistant quietly becomes an unsafe actor.
The people closest to the event understand this instinctively. The operator wants useful information without being overruled by an unseen score. The shift in charge wants to know what the agent has checked and what it has not checked. Maintenance wants a clear evidence trail. Production planning wants to understand which promises could be affected. Safety wants assurance that no software shortcut has bypassed a control boundary.
The architecture must respect all of these needs before the agent is allowed to act.
The central argument
Autonomous manufacturing agents should be designed as bounded participants in an operating system, not as clever replacements for the operating system.
The first design question is not, “Which large language model should we use?” It is:
“What is this agent allowed to do, under which conditions, with which evidence, and who remains accountable when the situation changes?”
An answer should make at least six things explicit:
- Purpose: the recurring decision the agent is intended to support.
- Perception: the signals, records, documents, and conversations it may inspect.
- Reasoning: the assumptions, rules, models, and constraints it may use.
- Action: the tools it may call and the changes it may make.
- Authority: the human approvals, limits, and escalation paths around those actions.
- Learning: how outcomes, errors, and overrides will change future behaviour.
Without these boundaries, an agent can appear capable while its actual responsibility remains unclear. That is a dangerous combination in any environment where quality, safety, production continuity, and customer commitments are connected.
Autonomy is a ladder of authority
It is useful to separate the levels of autonomy rather than treat autonomy as a yes-or-no property.
Observe
The agent reads approved signals and assembles relevant information. It does not make recommendations or change records. This level is useful for discovering whether the agent can interpret the plant correctly.
Explain
The agent identifies what changed, why the change may matter, and which evidence supports the interpretation. It should also show uncertainty and missing information.
Prepare
The agent creates a draft work order, recovery plan, schedule option, inspection request, or customer update. A person reviews and submits it.
Recommend
The agent compares alternatives and proposes one. It does not execute. The recommendation must show consequences, assumptions, and the person who is expected to decide.
Approve
The agent may complete a bounded action after a named person confirms it. Approval should not become a ceremonial click; the reviewer must have enough time, context, and authority to disagree.
Execute
The agent performs an action within a tightly defined boundary. The action must be observable, reversible where possible, and subject to a stop mechanism.
The appropriate level depends on consequence, reversibility, confidence, evidence quality, and the time available for intervention. A low-risk administrative action may be executable. A safety-related control or customer commitment may require a much higher threshold.
Why the language model is not the architecture
A language model may be excellent at summarising documents, interpreting free text, or proposing a sequence of steps. It does not automatically know whether a tool call is permitted, whether a status is authoritative, whether a production order is commercially sensitive, or whether a physical action is safe.
Those are architecture questions.
The model may say, “Stop the line immediately,” because that appears to be the safest response to an abnormal signal. But the actual procedure may require a controlled deceleration, a specific inspection, or a supervisor’s confirmation. The model’s linguistic confidence does not create operational authority.
Similarly, an agent may retrieve a maintenance procedure that is technically valid but obsolete for the asset configuration currently in use. It may see a work order as open even though the technician has already completed the physical repair. It may interpret a planned capacity slot as available even though a quality hold has made the material unsuitable.
A reliable architecture makes the model’s role narrower than the total operating decision. It gives the model access to the information and tools it needs, while preventing it from silently expanding its own authority.
The boundary map every agent should have
Before development begins, create a boundary map. It should answer the following questions.
What may the agent read?
List the approved sources: sensor signals, asset history, production state, quality status, work instructions, order commitments, maintenance records, and relevant operator notes. Specify whether the source is authoritative, advisory, delayed, or incomplete.
What may the agent infer?
Distinguish safe interpretation from operational judgement. The agent may infer that a signal resembles a previous incident. It may not infer that a safety interlock can be bypassed or that a customer will accept a substitute unless that authority and evidence are explicitly defined.
What may the agent write?
Writing a draft comment, creating a proposed work order, changing a priority flag, and publishing a production sequence are not equivalent actions. Each requires its own permission.
What may the agent execute?
Tool access should be narrow and purpose-specific. An agent that can create a draft should not automatically be able to approve it. An agent that can query a schedule should not automatically be able to change the schedule.
What must cause escalation?
Escalation conditions may include low confidence, conflicting sources, safety-relevant signals, quality uncertainty, missing approval, irreversible action, customer-contract exposure, or a recommendation outside the agent’s tested operating range.
How can the action be stopped or reversed?
Every executable action needs a visible stop path. Where reversal is impossible, the authority threshold should be higher and the action should remain human-controlled until the organisation has strong evidence.
Architecture without abstraction
A practical agent architecture can be described in layers.
The source layer
ERP, MES, APS, quality systems, maintenance platforms, warehouse systems, sensors, documents, and communication channels provide events and context. The agent should not treat every source as equally reliable. Freshness, ownership, and status semantics must be visible.
The identity and policy layer
The system must know who is asking, which agent is acting, which plant or line is in scope, and which policies apply. A recommendation for one product family or site may not be valid for another.
The context layer
This layer assembles the situation around the decision: current conditions, affected orders, dependencies, recent interventions, quality status, maintenance history, and remaining alternatives.
The reasoning layer
Rules, models, retrieval, simulation, and language reasoning may all contribute. Their role should be visible enough for a reviewer to understand the recommendation.
The tool layer
Tools should have narrow contracts. “Create draft work order” is safer than “manage maintenance.” “Prepare schedule option” is safer than “change schedule.” Narrow tools make permissions and audits easier.
The control and learning layer
Audit logs, approvals, escalation, rollback, monitoring, incident review, outcome capture, and override recording complete the architecture. Without them, the organisation cannot tell whether the agent helped, caused delay, or simply moved responsibility out of sight.
What failure looks like
When an agent fails, avoid saying only that “the AI was wrong.” Diagnose the boundary that failed.
Perception failure
The agent did not receive the relevant signal, received it too late, or treated an advisory source as authoritative.
Context failure
The agent detected a plant condition but did not connect it to the affected product, order, quality requirement, or customer commitment.
Permission failure
The agent had more access than the decision required, or the organisation had not defined which role could approve the action.
Reasoning failure
The agent proposed an option that violated a constraint, misunderstood a local rule, or treated a technically possible action as operationally acceptable.
Escalation failure
The agent encountered uncertainty but continued instead of asking for help, or escalated so frequently that people learned to ignore it.
Execution failure
The recommendation was accepted, but the action did not reach the actual workflow, operator, machine, supplier, or customer conversation.
Learning failure
The organisation did not record what happened, why a person overrode the agent, or whether the result was better. The same failure then returns as a new surprise.
This diagnosis turns agent development into operational learning instead of a contest between human confidence and model confidence.
From prediction to controlled action
Prediction can be an important beginning. It may identify an abnormal equipment pattern, a likely quality concern, a capacity conflict, or a sequence at risk. But prediction has no operational value until it changes what someone can do.
For an autonomous-agent use case, ask four questions:
- What decision does the prediction unlock?
- How much time remains before the choice becomes less useful?
- What options are safe and feasible?
- Who must be able to challenge or stop the action?
If the agent cannot answer these questions, it may still be a useful monitoring capability, but it is not ready for autonomous action.
A practical adoption sequence
Days 1–30: Define the boundary
Choose one recurring decision with a clear owner and visible consequence. Interview the people who make, review, execute, and live with that decision. Record what the agent may read, what it must not infer, which options exist, and which authority each option requires.
Days 31–60: Build a recommendation workspace
Connect the minimum evidence required. Let the agent gather context, retrieve procedures, compare options, and explain uncertainty. Keep actions in draft form. Measure whether users understand the recommendation and whether the evidence arrives early enough.
Days 61–90: Run shadow mode
Allow the agent to prepare recommendations without execution. Compare them with actual decisions, overrides, and outcomes. Deliberately test missing data, conflicting statuses, unusual shifts, stale procedures, and attempts to exceed permissions.
After 90 days: Earn bounded authority
Select only actions that are narrow, observable, reversible where possible, and supported by evidence. Give the agent the smallest authority that creates value. Review incidents and near misses before expanding the boundary.
Questions for leaders and architects
- What is the exact decision this agent is meant to support?
- Which data may it read, and which source is authoritative?
- What is the smallest action that would create value?
- Which actions are impossible to reverse?
- What does the agent do when evidence conflicts?
- How much time does a human have to intervene?
- Who can suspend the agent immediately?
- How will an override be explained and learned from?
- What evidence would justify increasing authority?
- What must remain outside the agent’s scope permanently?
Conclusion: intelligence needs a place to stand
An autonomous agent is not made responsible by adding a human name to an approval field. Responsibility emerges from the relationship between capability, boundary, authority, evidence, and consequence.
The model may help interpret a maintenance signal, summarise a quality concern, or prepare a recovery option. Architecture determines whether that help reaches the right person, uses the right evidence, respects the right limits, and leaves behind a record that can be examined.
The most mature manufacturing organisations will not be those that give agents the largest number of permissions. They will be those that know precisely where authority creates value, where uncertainty requires human judgement, and where an action must remain deliberately bounded.
The question is not whether the agent can act. It is whether the organisation can explain, limit, observe, and reverse its action.
That is why architecture must come before the LLM.
Disclaimer
Industry situations in this chapter are composite illustrations unless explicitly attributed to a public source. They are not claims about any particular company, plant, vendor, or incident. Implementations must be validated against local safety, quality, cybersecurity, regulatory, contractual, labour, and data-governance requirements.
#AgenticAI #AIArchitecture #BoundedAutonomy #ManufacturingAI #AIGovernance #IndustrialAutomation #HumanOversight #SmartManufacturing #ResponsibleAI #DigitalTransformation
