Merge pull request 'docs: document the repo as a multi-site baseline generator (README + AGENTS.md)' (#3) from docs/readme-agents into main

Reviewed-on: #3

AGENTS.md

@@ -0,0 +1,127 @@
+# AGENTS.md — machine-facing rules for this repo
+
+**Read this before touching anything.** `README.md` explains the project to a human;
+this file states the rules an automated agent must follow. Where the two seem to
+disagree, this file wins for process questions.
+
+---
+
+## 1. What this repo is
+
+It produces **clean, deployable code baselines** for the Cadline web sites hosted on
+`tanis.cadline.hu`. A baseline serves two purposes at once:
+
+1. the **known-good reference** the HIDS baseline-diff compares the live docroot against;
+2. the **source of truth for deployment** — the tree that will be pushed to the server,
+   replacing direct editing on the live host.
+
+The repo stores baseline **constituents plus a generator**, never the deployed tree.
+`build-baseline.sh` assembles `out/` deterministically; `out/` is git-ignored.
+
+## 2. One site per branch
+
+There is **no per-site repo**. Sites and CMS versions are separated by **git branch**:
+
+| Branch | Site | CMS core |
+|---|---|---|
+| `main` | www.archline.hu | Joomla 3.8.11 |
+| `site/archlinexp.eu` | www.archlinexp.eu | Joomla 3.9.2 |
+
+`build-baseline.sh` is **site-agnostic**: it walks `core/`, `packages/`, `deployed/`,
+`cadline/`, `overrides/` and contains no site-specific logic. A new site means a new
+branch with different *content*, **not** a modified script. If you find yourself editing
+the generator to special-case a site, stop — that is the wrong branch of the design.
+
+Site branches are **parallel and long-lived**. They do **not** merge into `main`.
+Only repo-wide things (this file, `README.md`, the generator, `lib/`) belong on `main`.
+
+## 3. The cardinal rule: never build a baseline from an infected tree
+
+Every one of these sites is or was compromised. A baseline built by copying the live
+docroot **would cement the malware into the reference and into every future deploy**.
+
+Therefore:
+
+- Take the CMS core from the **official upstream package**, never from live.
+- Take the site's own code from the **developer source repo**
+  (`git.cadline.hu/cadline/websource`, per-site subdirectory), never from live.
+- Only for extensions with **no obtainable upstream** may files come from live — and
+  then they are *vetted*, not trusted.
+- **The build output must be double-checked before it is used.** This is not optional:
+  the archline baseline's first candidate contained 34 malware files, and only the
+  double-check caught them.
+
+## 4. Mandatory verification (do not skip, do not weaken)
+
+After every build, run all of these against `out/`:
+
+1. **YARA** with the operator's ruleset. Expected: **zero** hits.
+2. **IOC paths**: not one entry of `malware-iocs.paths` may exist under `out/`.
+3. **Disguise sweep**: no `*.php.json`, `*.phar*`, `*.phtml*`, `_h3x_*`; no `.json` file
+   containing a PHP open tag.
+4. **Structural check** — the rule that catches what content matching misses:
+
+   > An extra file on a **CMS core component path** that is **not** a template override
+   > (`templates/<tpl>/html/...`) is **not a module**. It is a leftover or a plant.
+
+   Diff `out/` against the vanilla package: for every path under a core component that
+   the vanilla does not have, decide explicitly. This rule found a file-write webshell
+   (`components/com_mailto/mail.php`) and an Akeeba Kickstart leftover carrying a
+   password — **both invisible to YARA**.
+5. **Core version**: assert the built `libraries/src/Version.php` is the intended release.
+
+A contaminated baseline must **never** be shipped. If verification fails, fix the
+constituents and rebuild — do not filter the symptom out of `out/`.
+
+## 5. Repository boundary (hard rule)
+
+This repo lives on **`git.cadline.hu`** — the customer's Gitea, a different server from
+the operator's. Therefore:
+
+- **Never reference the operator's infrastructure here.** No operator issue URLs, no
+  operator hostnames, no `Addresses-Issue:` trailers pointing at operator trackers —
+  not in files, not in commit messages, not in branch names.
+- Issue tracking for this work lives on the operator side and is **not** linked from
+  here. Describe *what* and *why* in the commit message itself instead.
+- An `@illusion.hu` e-mail address in a commit trailer (authorship attribution) is the
+  one accepted exception.
+
+## 6. Commits and branches
+
+- Conventional Commits, **English**, subject ≤72 chars, imperative mood.
+- Trailer: `Assisted-by: claude-code@<model>` on AI-assisted commits.
+- **No** `Addresses-Issue:` / `Closes` (see §5).
+- Never force-push a branch someone else may be building on; on your own fresh branch it
+  is fine (use `--force-with-lease`).
+- **Never push to `main`.** Open a pull request; a human merges.
+- **Never commit a secret.** These trees legitimately contain third-party code with
+  embedded credentials — if you must refer to one, cite `file:line` and the *type*, never
+  the value. Redaction applies to commit messages, docs and any output you produce.
+
+## 7. Layer semantics (what goes where)
+
+| Directory | Holds | Source of truth |
+|---|---|---|
+| `core/` | CMS official package, deployed form, installer removed | upstream vendor |
+| `packages/<slug>/` | extensions with obtainable upstream; modified ones authored as two commits (pristine → delta) so `git diff` documents our change | upstream + our patch |
+| `deployed/<slug>/` | extensions with **no** obtainable upstream: vetted live files | live (vetted) |
+| `cadline/<slug>/` | the site's own code | developer source repo |
+| `overrides/` | the site's modifications to core files, **on the current core's base** | developer source |
+| `malware-iocs.paths` | paths excluded from the build (droppers, leftovers) | this repo |
+
+`overrides/` rsyncs **last**, so it wins over `core/`.
+
+**Do not take a customized core file from live.** The updater skips customized files, so
+the live copy silently stays on an old base (the eu site ran a 3.9.2 version string over
+pre-3.8.13 `com_users` code). Take vanilla for the version you target, then apply only
+the genuine delta.
+
+## 8. Definition of done
+
+A baseline branch is done when:
+
+- `build-baseline.sh` runs unmodified and produces `out/`;
+- all five checks in §4 pass, and the numbers are recorded in the commit message;
+- every layer's provenance is traceable (which file came from upstream, from the
+  developer source, or vetted from live);
+- no secret value appears anywhere in the diff or the message.

README.md

@@ -1,52 +1,81 @@
-# www.archline.hu — clean baseline (constituents + generator)
+# Cadline web baselines — constituents + generator
 
-This repo produces the **clean code baseline** of the www.archline.hu docroot
+This repo produces the **clean code baselines** of the Cadline web sites hosted on
-(Cadline, Joomla 3.8.11, host `tanis.cadline.hu`) — the post-compromise, malware-free,
+`tanis.cadline.hu` — malware-free, faithful reproductions of what each site's source
-faithful reproduction of the live server's source files.
+*should* be, assembled from clean sources rather than copied from the live server.
 
-The baseline is the **input** to the HIDS baseline-diff verifier: a per-vhost,
+A baseline has **two jobs**:
-known-good reference tree the verifier compares the live docroot against
+
-(`baseline-match / data / malware / residual`). This repo only *produces* that tree;
+1. **Known-good reference.** The HIDS baseline-diff compares the live docroot against it
-the verification runs on the HIDS side.
+   and classifies every file (`baseline-match / data / malware / residual`).
+2. **Deployment source of truth.** The baseline is what gets deployed to the server, so
+   that code reaches production through git instead of being edited directly on the live
+   host. Because the live tree is regenerated from the baseline, cleanup becomes a
+   by-product of deployment: anything not in the source disappears.
+
+The repo stores the baseline **constituents plus a generator** — never the deployed tree.
 
 ---
 
-## Building the baseline (read this first)
+## Sites and branches
 
-The repo does **not** contain the deployed docroot — it contains the baseline
+There is **no separate repo per site**. Sites (and CMS versions) live on **branches**:
-*constituents* plus a generator script. Run the script to produce the deployed tree:
+
+| Branch | Site | CMS core | Own code source |
+|---|---|---|---|
+| `main` | www.archline.hu | Joomla 3.8.11 | reconstructed here (no usable developer source) |
+| `site/archlinexp.eu` | www.archlinexp.eu | Joomla 3.9.2 | `cadline/websource` → `www.archlinexp.eu/` |
+
+`build-baseline.sh` is **site-agnostic** — it walks the constituent directories and has
+no site-specific logic. A new site is a new branch with different *content*, not a new
+script and not a new repo. Site branches are parallel and long-lived; they do not merge
+into `main`. Only repo-wide files (this README, `AGENTS.md`, the generator, `lib/`) live
+on `main` for every branch to inherit.
+
+Automated agents: read [`AGENTS.md`](AGENTS.md) first — it carries the rules
+(verification, repo boundary, layer semantics) in machine-facing form.
+
+---
+
+## Building a baseline (read this first)
 
 ```sh
-# 1. Clone WITH submodules (common_cadline_libraries + maintenance are submodules):
+# 1. Clone, and check out the branch of the site you want:
 git clone --recurse-submodules ssh://git@git.cadline.hu:22222/cadline_web/www_archline_hu.git
 cd www_archline_hu
-#    (if you already cloned without --recurse-submodules:)
+git checkout site/archlinexp.eu       # or stay on main for www.archline.hu
+#    (if you cloned without --recurse-submodules and the branch uses submodules:)
 git submodule update --init
 
-# 2. Generate the baseline:
+# 2. Generate:
-./build-baseline.sh            # → ./out/
+./build-baseline.sh                   # → ./out/
 
-# 3. Use ./out/ as the per-vhost vanilla_ref (the known-good reference tree).
+# 3. Use ./out/ as that site's known-good reference / deploy source.
 ```
 
-`out/` is the deployed **code** docroot (~23.5k files). It is git-ignored and fully
+`out/` is the deployed **code** docroot. It is git-ignored and fully regenerated on every
-regenerated on every run (the script is idempotent — it `rm -rf out/` first). Point the
+run (the script `rm -rf out/` first, so it is idempotent).
-verifier's baseline-diff at `out/`, or copy it to wherever the verifier expects the ref.
 
-**Prerequisites:** `bash`, `rsync`, `git` (for submodules), `find`, `sed`. No PHP, no
+**Prerequisites:** `bash`, `rsync`, `git`, `find`, `sed`. No PHP, no build toolchain, no
-build toolchain, no network at build time (submodules are already cloned).
+network at build time. `lib/jmap.py` (python3) is an *authoring* aid used when preparing a
-`lib/jmap.py` (python3) is a *maintenance* tool used only when authoring a package's
+package's pristine commit — **`build-baseline.sh` never invokes it**.
-pristine commit — **it is not invoked by `build-baseline.sh`**.
 
 **What `out/` contains / excludes:**
-- **Contains:** Joomla 3.8.11 core + every installed extension (110 components, ~98
+
-  modules, 164 plugins, 8 templates) + the Cadline-own code — all as deployed files.
+- **Contains:** the CMS core + every installed extension + the Cadline-own code, as
-- **Excludes:** malware/IOCs (see [`malware-iocs.md`](malware-iocs.md)) and runtime
+  deployed files.
-  **data** (`cache/`, `tmp/`, `logs/`, `images/`, `public/`, media data). The result is
+- **Excludes:** malware and known planted paths (`malware-iocs.paths`), and runtime
-  code only.
+  **data** (`cache/`, `tmp/`, `logs/`, `images/`, `public/`, media). The result is code
-- Byte-matches the live docroot for all legit code; it is intentionally **clean** on the
+  only — which is also why a deploy must never wipe the data directories.
-  4 files the attacker infected (3 trojanized core files + `shaper_helix3/index.php`),
+
-  so the verifier flags those on the live server. Verified: non-malware code residual = 0.
+### Verify before you trust it
+
+The build output is **not** trusted until checked. This is not ceremony: the first
+archline baseline candidate contained 34 malware files, and only the check caught them;
+on the eu site the check caught a file-write webshell and a leftover Akeeba restoration
+file that carried a password — **neither of which YARA flagged**. See `AGENTS.md` §4 for
+the full checklist (YARA, IOC paths, disguise sweep, the structural core-namespace rule,
+core version).
 
 ---
 
@@ -54,49 +83,77 @@ pristine commit — **it is not invoked by `build-baseline.sh`**.
 
 | Path | What | How `build-baseline.sh` uses it |
 |---|---|---|
-| `core/` | Joomla 3.8.11 official package (deployed form) | rsync → `out/`, minus demo assets + data dirs |
+| `core/` | CMS official package (deployed form, installer removed) | rsync → `out/`, minus demo assets + data dirs |
-| `packages/<slug>/` | 5 upstream-obtainable extensions, deployed-layout, reconciled to live bytes. Modified ones are two commits (pristine upstream → live delta) so `git log`/`git diff` shows exactly how we differ from upstream. | rsync → `out/` |
+| `packages/<slug>/` | extensions with an obtainable upstream, deployed-layout. Modified ones are authored as two commits (pristine upstream → our delta) so `git log`/`git diff` shows exactly how we differ. | rsync → `out/` |
-| `deployed/<slug>/` | ~55 no-source extensions: vetted live files (deployed layout). `_shared/` holds cross-extension language/vendor/manifests. | rsync → `out/` |
+| `deployed/<slug>/` | extensions with **no** obtainable upstream: vetted live files, deployed layout. | rsync → `out/` |
-| `cadline/<slug>/` | Cadline own-code. `cadline/backend/common_cadline_libraries` and `…/maintenance` are **git submodules** of the developer repos (cadcommonlib / maintenance). | rsync → `out/` (`.git` excluded) |
+| `cadline/<slug>/` | Cadline own-code. On `main` this includes git submodules of the developer repos (cadcommonlib / maintenance). | rsync → `out/` (`.git` excluded) |
-| `overrides/` | Cadline core modifications (live versions overlaid on the package core) + a few package-missing core files. | rsync → `out/` (last, so it wins) |
+| `overrides/` | the site's modifications to core files, on the current core's base. Rsynced **last**, so it wins. | rsync → `out/` |
-| `lib/jmap.py` | Generic Joomla manifest→deployed-path mapper (authoring aid only). | not used at build time |
+| `malware-iocs.paths` | machine-readable exclusion list (droppers, leftovers) | build step 6 |
+| `lib/jmap.py` | Joomla manifest→deployed-path mapper (authoring aid only) | not used at build time |
 
 Details: [`RUNBOOK.md`](RUNBOOK.md) (recipe + module inventory),
 [`PROVENANCE.md`](PROVENANCE.md) (per-extension origins),
-[`malware-iocs.md`](malware-iocs.md) (excluded IOC set).
+[`malware-iocs.md`](malware-iocs.md) (the excluded IOC set, `main`).
 
 ---
 
-## Updating the baseline
+## Why customized core files never come from live
 
-- **Developer-controlled code** (`common_cadline_libraries`, `maintenance`): the
+A CMS updater skips files the site has customized. The version string moves forward; the
-  developer pushes to the cadcommonlib / maintenance repos, then:
+customized files silently stay on the old base. The eu site demonstrated this: it
+reported Joomla 3.9.2 while its `com_users` was still pre-3.8.13 code — 31 stale files.
+
+So: take **vanilla for the version you target**, then apply only the genuine delta into
+`overrides/`. Never lift the customized file from live — that would preserve the stale
+(and possibly vulnerable) base forever.
+
+---
+
+## This branch: www.archline.hu (`main`)
+
+Joomla 3.8.11, host `tanis.cadline.hu`. `out/` ≈ 23.5k files. It byte-matches the live
+docroot for all legit code and is intentionally **clean** on the 4 files the attacker
+infected (3 trojanized core files + `shaper_helix3/index.php`), so the verifier flags
+those on the live server. Verified: non-malware code residual = 0.
+
+The own-code here is **reconstructed in this repo**, because the developer source
+(`cadline/websource`) does not contain this site — its `sub.archline.hu` directory is an
+older snapshot missing ~630 files and the third-party module layer entirely.
+
+### Updating
+
+- **Developer-controlled code** (`common_cadline_libraries`, `maintenance`): the developer
+  pushes to the cadcommonlib / maintenance repos, then:
   ```sh
   git submodule update --remote     # pull the developer's latest
   ./build-baseline.sh               # regenerate out/
   git add -A && git commit          # record the new submodule pointers
   ```
-- **Everything else** (Joomla core EOL, 3rd-party extensions): effectively frozen; if a
+- **Everything else** (EOL core, 3rd-party extensions): effectively frozen. If a legit
-  legit change ever lands, re-sync the affected `deployed/<slug>/` or `overrides/` from
+  change lands, re-sync the affected `deployed/<slug>/` or `overrides/` and commit.
-  the live docroot and commit.
 
 **Known divergence (heads-up for the verifier):** `common_cadline_libraries` and
-`maintenance` currently take the *authoritative* submodule HEAD content, which differs
+`maintenance` take the *authoritative* submodule HEAD content, which differs from the
-from the current live docroot (mixed CRLF/LF + a content drift on
+current live docroot (mixed CRLF/LF + a content drift on `crm_hardlock.class.php` /
-`crm_hardlock.class.php` / `POfile.php`) because live was not last deployed from those
+`POfile.php`) because live was not last deployed from those HEADs. The verifier flags
-HEADs. The verifier will flag this until live is re-deployed from the submodules or the
+this until live is re-deployed or the drift is reconciled.
-drift is reconciled. Line-ending normalization, if wanted, is a verifier-side choice.
+
+## This branch: www.archlinexp.eu (`site/archlinexp.eu`)
+
+Joomla 3.9.2. `out/` ≈ 12.3k files. Built from clean sources end to end: vanilla 3.9.2
+core + the developer source's own code + a vetted module layer, with only 5 genuine core
+modifications in `overrides/`. The developer source for this site is **complete and
+clean** (its `com_users` is already on the 3.9.2 base), which is why this site — not
+archline.hu — was the first to get a fully source-built baseline.
 
 ---
 
 ## Authority
 
-The live docroot (RO mirror) is the source of truth for reproduction; the developer git
+- The **upstream vendor package** is authoritative for the CMS core.
-repos are authoritative for the code they own. The earlier single-commit "monolith"
+- The **developer source repo** is authoritative for the code it owns.
-baseline is an unvalidated helper only (it dropped extension manifests and clean core
+- The **live docroot** (read-only mirror) is authoritative only for reproduction fidelity
-files and kept installer junk) — it is not a target.
+  and for extensions that have no obtainable upstream — and even then the files are
-
+  vetted, never trusted.
-## References
+- The earlier single-commit "monolith" baseline is an unvalidated helper only (it dropped
-
+  extension manifests and clean core files and kept installer junk) — not a target.
-- The archline.hu 2026 security audit and incident analysis (operator-held).
-- The excluded malware set is documented in [`malware-iocs.md`](malware-iocs.md).