From 6f66bfbf84c305e977a39114df7629efb87f6b2e Mon Sep 17 00:00:00 2001 From: Mateusz Srebrny Date: Mon, 1 Dec 2025 15:36:51 +0100 Subject: [PATCH 1/6] RTS-234 chungus LLM policy --- README.md | 2 +- agreements.md | 31 +++++++++++++++++++++++++++++++ docs/Code_Review.md | 6 ++++++ 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c8bb9df..9a7ff40 100644 --- a/README.md +++ b/README.md @@ -151,7 +151,7 @@ We employ several techniques to output a lot of value in a unit of time: While some software houses lose \~30% of their performance (due to miscommunications, etc.), we stay way below 1% (the exact number is hard to measure when the amount of waste is so low). As mentioned above, we tend to pause work on a ticket when we are not sure, choosing a small delay on a particular task over potentially having to discard work due to a bad assumption. Our clients prefer it this way. 10. Use modern IDEs. Pycharm, Cursor, Windsurf or VisualStudio with a LLM plugin. Some people are trying to use VIM with plugins for LLMs, but nowadays it's mostly Cursor/Windsurf/Pycharm. -11. Use LLMs (ChatGPT, GitHub Copilot, Claude etc) to speed up the work on the code, though watch every single byte of the diff like it's been written by a party you shouldn't trust. +11. Use LLMs to speed up coding, but we still own QA: read every byte and aim for our normal quality. For small, contained “one‑shot” changes we may relax QA—and even skip review—if we save the prompt/spec so future bugs can be fixed smoothly. See the LLM‑assisted coding agreement (agreements.md#llm-assisted-coding). # Code Review diff --git a/agreements.md b/agreements.md index 8741688..1c8fe3a 100644 --- a/agreements.md +++ b/agreements.md @@ -166,6 +166,37 @@ That's one of the reasons we have someone in that role for every project. This also allows for other solutions such as, for example, splitting the cost of a fix between two clients, discounts etc - something you couldn't do on your own. Fortunately, with a dedicated client contact person, you don't have to! +## LLM-Assisted Coding + +Motivation + +- LLMs let us move faster (including during reviews), but cumulative low-quality changes can erode long-term maintainability — one of our trademarks. +- Responsibility remains human: both the author and the reviewer own the quality of what ships, regardless of tooling. +- We may intentionally accept slightly lower quality for one‑shot, low‑risk changes to gain speed, but we do so when it is clearly safe and we document the prompt/spec. +- There’s a trade‑off: more features with less quality vs fewer features with higher quality. Over time, lower quality makes teams ship fewer features due to maintenance drag; we optimize for sustained maintainability while using one‑shots to keep speed where risk is minimal. +- Practical observation: there’s a limit to how much an LLM can safely rewrite in a day without agreed QA. + +Scope and rules + +- Default path: normal QA and review. One‑shot changes (small, independently judgeable changes) may be merged with relaxed quality and may skip review if all of the following hold: + - The change is low‑risk, contained, and has minimal blast radius; the author performs a basic functional check. + - The final “prompt/spec” is saved in the repository as markdown and kept up to date when the code changes. + - The conversation history may be condensed into a short “final prompt/summary” rather than stored verbatim. + - PR/commit mentions that the change is LLM‑assisted and links to the prompt/spec. +- Larger/core contributions done with LLM assistance require a case‑by‑case, agreed QA/release plan. If we must merge a larger LLM change before full QA, do it consciously with that plan in place. Do not merge low‑quality core code without such agreement. +- Use case‑by‑case judgement for one‑shots; if uncertain, take the normal QA/review path. + +Author responsibilities + +- Read every byte of generated code, functionally test it, and prepare the change for regular review (structure the diff, write tests/docs where applicable). +- Include a link to the prompt/spec in the PR or commit description. Do not include secrets or client‑sensitive data in prompts. + +Review expectations + +- Normal review applies; the reviewer may use or not use LLM tooling. +- Verify that the prompt/spec exists, is up to date, and meaningfully reflects the change. +- Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in prompts/specs. + ## Fast track decisions via Slack instead of standard Sociocracy approach At Reef Technologies, we mostly make decisions in our weekly Sociocracy meetings. diff --git a/docs/Code_Review.md b/docs/Code_Review.md index a86ee81..f395515 100644 --- a/docs/Code_Review.md +++ b/docs/Code_Review.md @@ -12,6 +12,9 @@ As with every policy, if something is bad and you care about it enough, feel fre and the governance process, operating in front of you and with you, will decide whether to change (most likely yes). +Responsibility remains with humans: authors and reviewers are accountable for code quality whether or not LLMs were used. See the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). +Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (prompt/spec saved, low blast radius, basic functional check). Otherwise, request review as usual. + ## When/how to request a code review 1. If you are a developer, always perform basic functional testing (manually!) of your code and self-review your PR @@ -31,6 +34,7 @@ and the governance process, operating in front of you and with you, will decide commits instead, like “Fix XYZ”, “Add tests”.
Note.This is a limitation of GitHub and a primary motivator to consider moving development off to gitlab / gerrit, though as of writing this document we have not decided to switch.
+6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding) and link to the saved prompt/spec. If you’re using the LLM one‑shot relaxation, say so explicitly; those relaxed changes should stand alone, not be tucked into larger PRs. Authors must read and functionally verify all generated code before requesting review; if merging without review under the one‑shot rule, mention “llm‑one‑shotted” and the prompt/spec path in the commit message instead. ## How to review @@ -101,6 +105,8 @@ Note: not all of these apply to all PRs, of course. 12. Performance / memory considerations. Will it OOM in a corner case after the change? 13. Self-healing. If a (network, HDD) device or a part of the system breaks temporarily, will the rest of the system recover automatically or will it require manual intervention? +14. For LLM‑assisted changes: is the prompt/spec saved, linked in the PR, and kept up to date? +15. Maintainability not degraded (structure, naming, tests, docs); no secrets in prompt/specs. ## How to resolve the reviewer’s comments From 56b8ae9869c4f6257cb8e09c545c1824dbff4937 Mon Sep 17 00:00:00 2001 From: Mateusz Srebrny Date: Mon, 1 Dec 2025 15:57:59 +0100 Subject: [PATCH 2/6] line breaking --- README.md | 4 +++- agreements.md | 35 ++++++++++++++++++++++++----------- docs/Code_Review.md | 12 +++++++++--- 3 files changed, 36 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 9a7ff40..fc29ca4 100644 --- a/README.md +++ b/README.md @@ -151,7 +151,9 @@ We employ several techniques to output a lot of value in a unit of time: While some software houses lose \~30% of their performance (due to miscommunications, etc.), we stay way below 1% (the exact number is hard to measure when the amount of waste is so low). As mentioned above, we tend to pause work on a ticket when we are not sure, choosing a small delay on a particular task over potentially having to discard work due to a bad assumption. Our clients prefer it this way. 10. Use modern IDEs. Pycharm, Cursor, Windsurf or VisualStudio with a LLM plugin. Some people are trying to use VIM with plugins for LLMs, but nowadays it's mostly Cursor/Windsurf/Pycharm. -11. Use LLMs to speed up coding, but we still own QA: read every byte and aim for our normal quality. For small, contained “one‑shot” changes we may relax QA—and even skip review—if we save the prompt/spec so future bugs can be fixed smoothly. See the LLM‑assisted coding agreement (agreements.md#llm-assisted-coding). +11. Use LLMs to speed up coding, but we still own QA: read every byte and aim for our normal quality. For small, + contained “one‑shot” changes we may relax QA—and even skip review—if we save the prompt/spec so future bugs can be + fixed smoothly. See the [LLM‑assisted coding agreement](agreements.md#llm-assisted-coding). # Code Review diff --git a/agreements.md b/agreements.md index 1c8fe3a..5628a68 100644 --- a/agreements.md +++ b/agreements.md @@ -170,32 +170,42 @@ Fortunately, with a dedicated client contact person, you don't have to! Motivation -- LLMs let us move faster (including during reviews), but cumulative low-quality changes can erode long-term maintainability — one of our trademarks. +- LLMs let us move faster (including during reviews), but cumulative low-quality changes can erode long-term + maintainability — one of our trademarks. - Responsibility remains human: both the author and the reviewer own the quality of what ships, regardless of tooling. -- We may intentionally accept slightly lower quality for one‑shot, low‑risk changes to gain speed, but we do so when it is clearly safe and we document the prompt/spec. -- There’s a trade‑off: more features with less quality vs fewer features with higher quality. Over time, lower quality makes teams ship fewer features due to maintenance drag; we optimize for sustained maintainability while using one‑shots to keep speed where risk is minimal. +- We may intentionally accept slightly lower quality for one‑shot, low‑risk changes to gain speed, but we do so when it is + clearly safe and we document the prompt/spec. +- There’s a trade‑off: more features with less quality vs fewer features with higher quality. Over time, lower quality + makes teams ship fewer features due to maintenance drag; we optimize for sustained maintainability while using + one‑shots to keep speed where risk is minimal. - Practical observation: there’s a limit to how much an LLM can safely rewrite in a day without agreed QA. Scope and rules -- Default path: normal QA and review. One‑shot changes (small, independently judgeable changes) may be merged with relaxed quality and may skip review if all of the following hold: +- Default path: normal QA and review. One‑shot changes (small, independently judgeable changes) may be merged with + relaxed quality and may skip review if all of the following hold: - The change is low‑risk, contained, and has minimal blast radius; the author performs a basic functional check. - The final “prompt/spec” is saved in the repository as markdown and kept up to date when the code changes. - The conversation history may be condensed into a short “final prompt/summary” rather than stored verbatim. - PR/commit mentions that the change is LLM‑assisted and links to the prompt/spec. -- Larger/core contributions done with LLM assistance require a case‑by‑case, agreed QA/release plan. If we must merge a larger LLM change before full QA, do it consciously with that plan in place. Do not merge low‑quality core code without such agreement. +- Larger/core contributions done with LLM assistance require a case‑by‑case, agreed QA/release plan. If we must merge a + larger LLM change before full QA, do it consciously with that plan in place. Do not merge low‑quality core code + without such agreement. - Use case‑by‑case judgement for one‑shots; if uncertain, take the normal QA/review path. Author responsibilities -- Read every byte of generated code, functionally test it, and prepare the change for regular review (structure the diff, write tests/docs where applicable). -- Include a link to the prompt/spec in the PR or commit description. Do not include secrets or client‑sensitive data in prompts. +- Read every byte of generated code, functionally test it, and prepare the change for regular review (structure the diff, + write tests/docs where applicable). +- Include a link to the prompt/spec in the PR or commit description. Do not include secrets or client‑sensitive data in + prompts. Review expectations - Normal review applies; the reviewer may use or not use LLM tooling. - Verify that the prompt/spec exists, is up to date, and meaningfully reflects the change. -- Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in prompts/specs. +- Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in + prompts/specs. ## Fast track decisions via Slack instead of standard Sociocracy approach @@ -271,11 +281,14 @@ See the new [Equipment Funding Policy](https://github.com/reef-technologies/hand ## LLM tool use & funding -Each Reef Technologies senior staff member is eligible to receive company-funded LLM service subscription (e.g. GitHub Copilot, ChatGPT Plus). -The subscription is paid by Reef Technologies for entire time it is being used it for any tasks related to Reef Technologies projects. +Each Reef Technologies senior staff member is eligible to receive company-funded LLM service subscription (e.g. GitHub +Copilot, ChatGPT Plus). +The subscription is paid by Reef Technologies for entire time it is being used it for any tasks related to Reef +Technologies projects. Please make sure the subscription is terminated if you don't need it anymore to prevent license waste. -Please note, while our default contract with clients allows use of LLMs, make sure to check with a project manager before using LLM on any particular project. +Please note, while our default contract with clients allows use of LLMs, make sure to check with a project manager +before using LLM on any particular project. As of mid 2023 every client has approved LLMs except one that is still considering it. As for work done for Reef Technologies internal projects, you are free to use LLMs as you see fit. In all cases you are responsible for quality of your work, regardless if LLM was used to produce it or not. diff --git a/docs/Code_Review.md b/docs/Code_Review.md index f395515..19ec92e 100644 --- a/docs/Code_Review.md +++ b/docs/Code_Review.md @@ -12,8 +12,10 @@ As with every policy, if something is bad and you care about it enough, feel fre and the governance process, operating in front of you and with you, will decide whether to change (most likely yes). -Responsibility remains with humans: authors and reviewers are accountable for code quality whether or not LLMs were used. See the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). -Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (prompt/spec saved, low blast radius, basic functional check). Otherwise, request review as usual. +Responsibility remains with humans: authors and reviewers are accountable for code quality whether or not LLMs were +used. See the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). +Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (prompt/spec saved, low +blast radius, basic functional check). Otherwise, request review as usual. ## When/how to request a code review @@ -34,7 +36,11 @@ Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreeme commits instead, like “Fix XYZ”, “Add tests”.
Note.This is a limitation of GitHub and a primary motivator to consider moving development off to gitlab / gerrit, though as of writing this document we have not decided to switch.
-6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding) and link to the saved prompt/spec. If you’re using the LLM one‑shot relaxation, say so explicitly; those relaxed changes should stand alone, not be tucked into larger PRs. Authors must read and functionally verify all generated code before requesting review; if merging without review under the one‑shot rule, mention “llm‑one‑shotted” and the prompt/spec path in the commit message instead. +6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding) and link to the saved prompt/spec. + If you’re using the LLM one‑shot relaxation, say so explicitly; those relaxed changes should stand alone, not be + tucked into larger PRs. Authors must read and functionally verify all generated code before requesting review; if + merging without review under the one‑shot rule, mention “llm‑one‑shotted” and the prompt/spec path in the commit + message instead. ## How to review From b012502b95de0132ffd6d29933eef9311d72b5eb Mon Sep 17 00:00:00 2001 From: Mateusz Srebrny Date: Thu, 4 Dec 2025 09:55:02 +0100 Subject: [PATCH 3/6] review fixes --- agreements.md | 18 +++++++++++------- docs/Code_Review.md | 6 +++--- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/agreements.md b/agreements.md index 5628a68..2502afe 100644 --- a/agreements.md +++ b/agreements.md @@ -184,10 +184,12 @@ Scope and rules - Default path: normal QA and review. One‑shot changes (small, independently judgeable changes) may be merged with relaxed quality and may skip review if all of the following hold: - - The change is low‑risk, contained, and has minimal blast radius; the author performs a basic functional check. - - The final “prompt/spec” is saved in the repository as markdown and kept up to date when the code changes. - - The conversation history may be condensed into a short “final prompt/summary” rather than stored verbatim. - - PR/commit mentions that the change is LLM‑assisted and links to the prompt/spec. + - The change is low‑risk, contained, and has minimal blast radius. + - The author performs a basic functional check. + - The “final prompt/spec” is saved in the repo as markdown (distilled ask + key constraints/acceptance criteria; the + chat can be condensed to a short summary) and kept up to date when the code changes. + - The one‑shot relaxation is mentioned explicitly in the PR/commit. + - The change is kept standalone (not tucked into a larger PR). - Larger/core contributions done with LLM assistance require a case‑by‑case, agreed QA/release plan. If we must merge a larger LLM change before full QA, do it consciously with that plan in place. Do not merge low‑quality core code without such agreement. @@ -197,13 +199,15 @@ Author responsibilities - Read every byte of generated code, functionally test it, and prepare the change for regular review (structure the diff, write tests/docs where applicable). -- Include a link to the prompt/spec in the PR or commit description. Do not include secrets or client‑sensitive data in - prompts. +- When using the one‑shot relaxation, save the prompt/spec near the change (even if you refactored the generated code). + For LLM‑assisted work that goes through normal QA/review, keeping a prompt/spec is optional. Do not include secrets or + client‑sensitive data in prompts. Review expectations - Normal review applies; the reviewer may use or not use LLM tooling. -- Verify that the prompt/spec exists, is up to date, and meaningfully reflects the change. +- If a one‑shot change is being reviewed anyway, verify that the prompt/spec exists, is up to date, and meaningfully + reflects the change. - Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in prompts/specs. diff --git a/docs/Code_Review.md b/docs/Code_Review.md index 19ec92e..a56214a 100644 --- a/docs/Code_Review.md +++ b/docs/Code_Review.md @@ -36,8 +36,8 @@ blast radius, basic functional check). Otherwise, request review as usual. commits instead, like “Fix XYZ”, “Add tests”.
Note.This is a limitation of GitHub and a primary motivator to consider moving development off to gitlab / gerrit, though as of writing this document we have not decided to switch.
-6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding) and link to the saved prompt/spec. - If you’re using the LLM one‑shot relaxation, say so explicitly; those relaxed changes should stand alone, not be +6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). If you’re using the LLM one‑shot + relaxation, save the prompt/spec in the repo and say so explicitly; those relaxed changes should stand alone, not be tucked into larger PRs. Authors must read and functionally verify all generated code before requesting review; if merging without review under the one‑shot rule, mention “llm‑one‑shotted” and the prompt/spec path in the commit message instead. @@ -111,7 +111,7 @@ Note: not all of these apply to all PRs, of course. 12. Performance / memory considerations. Will it OOM in a corner case after the change? 13. Self-healing. If a (network, HDD) device or a part of the system breaks temporarily, will the rest of the system recover automatically or will it require manual intervention? -14. For LLM‑assisted changes: is the prompt/spec saved, linked in the PR, and kept up to date? +14. If a one‑shot LLM change is being reviewed: is the prompt/spec saved in the repo and kept up to date? 15. Maintainability not degraded (structure, naming, tests, docs); no secrets in prompt/specs. ## How to resolve the reviewer’s comments From 9ee1f114dd9f71e6180fea4244ce86bac813f321 Mon Sep 17 00:00:00 2001 From: Mateusz Srebrny Date: Wed, 17 Dec 2025 09:21:14 +0100 Subject: [PATCH 4/6] relaxing prompt saving (and smaller review comments applied) --- README.md | 4 ++-- agreements.md | 23 ++++++++++++----------- docs/Code_Review.md | 15 +++++++-------- 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 041ce2f..1011090 100644 --- a/README.md +++ b/README.md @@ -154,8 +154,8 @@ We employ several techniques to output a lot of value in a unit of time: As mentioned above, we tend to pause work on a ticket when we are not sure, choosing a small delay on a particular task over potentially having to discard work due to a bad assumption. Our clients prefer it this way. 10. Use modern IDEs. Pycharm, Cursor, Windsurf or VisualStudio with a LLM plugin. Some people are trying to use VIM with plugins for LLMs, but nowadays it's mostly Cursor/Windsurf/Pycharm. 11. Use LLMs to speed up coding, but we still own QA: read every byte and aim for our normal quality. For small, - contained “one‑shot” changes we may relax QA—and even skip review—if we save the prompt/spec so future bugs can be - fixed smoothly. See the [LLM‑assisted coding agreement](agreements.md#llm-assisted-coding). + contained “one‑shot” changes we may relax QA—and even skip review—if the change is low‑risk and self‑explanatory (or + briefly noted for future maintainers). See the [LLM‑assisted coding agreement](agreements.md#llm-assisted-coding). # Code Review diff --git a/agreements.md b/agreements.md index 98420ab..2a19d85 100644 --- a/agreements.md +++ b/agreements.md @@ -174,21 +174,23 @@ Motivation maintainability — one of our trademarks. - Responsibility remains human: both the author and the reviewer own the quality of what ships, regardless of tooling. - We may intentionally accept slightly lower quality for one‑shot, low‑risk changes to gain speed, but we do so when it is - clearly safe and we document the prompt/spec. + clearly safe. - There’s a trade‑off: more features with less quality vs fewer features with higher quality. Over time, lower quality makes teams ship fewer features due to maintenance drag; we optimize for sustained maintainability while using one‑shots to keep speed where risk is minimal. - Practical observation: there’s a limit to how much an LLM can safely rewrite in a day without agreed QA. +**Never include secrets or client‑sensitive data in prompts or any saved notes.** + Scope and rules - Default path: normal QA and review. One‑shot changes (small, independently judgeable changes) may be merged with relaxed quality and may skip review if all of the following hold: - The change is low‑risk, contained, and has minimal blast radius. - The author performs a basic functional check. - - The “final prompt/spec” is saved in the repo as markdown (distilled ask + key constraints/acceptance criteria; the - chat can be condensed to a short summary) and kept up to date when the code changes. - - The one‑shot relaxation is mentioned explicitly in the PR/commit. + - The change is either self‑explanatory on its own, or (if not obvious) accompanied by a short note capturing the + ask/purpose/constraints so the next person can pick it up without guesswork. + - The one‑shot relaxation is mentioned explicitly in the PR/commit (e.g. include `llm-one-shot`). - The change is kept standalone (not tucked into a larger PR). - Larger/core contributions done with LLM assistance require a case‑by‑case, agreed QA/release plan. If we must merge a larger LLM change before full QA, do it consciously with that plan in place. Do not merge low‑quality core code @@ -199,17 +201,16 @@ Author responsibilities - Read every byte of generated code, functionally test it, and prepare the change for regular review (structure the diff, write tests/docs where applicable). -- When using the one‑shot relaxation, save the prompt/spec near the change (even if you refactored the generated code). - For LLM‑assisted work that goes through normal QA/review, keeping a prompt/spec is optional. Do not include secrets or - client‑sensitive data in prompts. +- When using the one‑shot relaxation, make the change pick‑up‑able: if it is not trivial/self‑explanatory, add a short + note (or code comment) capturing the ask/purpose/constraints. If it is obvious, no note is needed. Review expectations - Normal review applies; the reviewer may use or not use LLM tooling. -- If a one‑shot change is being reviewed anyway, verify that the prompt/spec exists, is up to date, and meaningfully - reflects the change. -- Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in - prompts/specs. +- If a one‑shot change is being reviewed anyway, confirm it is low‑risk/contained and either self‑explanatory or + accompanied by a brief note as above. +- Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in any notes or + prompts. ## Fast track decisions via Slack instead of standard Sociocracy approach diff --git a/docs/Code_Review.md b/docs/Code_Review.md index a56214a..ec46b9c 100644 --- a/docs/Code_Review.md +++ b/docs/Code_Review.md @@ -14,8 +14,8 @@ and the governance process, operating in front of you and with you, will decide Responsibility remains with humans: authors and reviewers are accountable for code quality whether or not LLMs were used. See the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). -Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (prompt/spec saved, low -blast radius, basic functional check). Otherwise, request review as usual. +Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (low blast radius, basic +functional check, self‑explanatory or briefly noted). Otherwise, request review as usual. ## When/how to request a code review @@ -37,10 +37,9 @@ blast radius, basic functional check). Otherwise, request review as usual. primary motivator to consider moving development off to gitlab / gerrit, though as of writing this document we have not decided to switch. 6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). If you’re using the LLM one‑shot - relaxation, save the prompt/spec in the repo and say so explicitly; those relaxed changes should stand alone, not be - tucked into larger PRs. Authors must read and functionally verify all generated code before requesting review; if - merging without review under the one‑shot rule, mention “llm‑one‑shotted” and the prompt/spec path in the commit - message instead. + relaxation, keep the change standalone (not tucked into larger PRs), make it self‑explanatory or add a brief note, + read/functionally verify all generated code before requesting review or merging, and mention `llm-one-shot` in the + PR/commit. ## How to review @@ -111,8 +110,8 @@ Note: not all of these apply to all PRs, of course. 12. Performance / memory considerations. Will it OOM in a corner case after the change? 13. Self-healing. If a (network, HDD) device or a part of the system breaks temporarily, will the rest of the system recover automatically or will it require manual intervention? -14. If a one‑shot LLM change is being reviewed: is the prompt/spec saved in the repo and kept up to date? -15. Maintainability not degraded (structure, naming, tests, docs); no secrets in prompt/specs. +14. If a one‑shot LLM change is being reviewed: is it low‑risk/contained and either self‑explanatory or briefly noted? +15. Maintainability not degraded (structure, naming, tests, docs); no secrets in notes/prompts. ## How to resolve the reviewer’s comments From 9b7c8331670cefbcee4212dcc5493ba01a852fb2 Mon Sep 17 00:00:00 2001 From: Mateusz Srebrny Date: Thu, 18 Dec 2025 13:13:01 +0100 Subject: [PATCH 5/6] note in the code or accompanying md file as agreed on s3 --- README.md | 2 +- agreements.md | 8 +++++--- docs/Code_Review.md | 6 ++++-- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1011090..4dbe734 100644 --- a/README.md +++ b/README.md @@ -155,7 +155,7 @@ We employ several techniques to output a lot of value in a unit of time: 10. Use modern IDEs. Pycharm, Cursor, Windsurf or VisualStudio with a LLM plugin. Some people are trying to use VIM with plugins for LLMs, but nowadays it's mostly Cursor/Windsurf/Pycharm. 11. Use LLMs to speed up coding, but we still own QA: read every byte and aim for our normal quality. For small, contained “one‑shot” changes we may relax QA—and even skip review—if the change is low‑risk and self‑explanatory (or - briefly noted for future maintainers). See the [LLM‑assisted coding agreement](agreements.md#llm-assisted-coding). + briefly noted in the repository for future maintainers). See the [LLM‑assisted coding agreement](agreements.md#llm-assisted-coding). # Code Review diff --git a/agreements.md b/agreements.md index 2a19d85..b8b16e9 100644 --- a/agreements.md +++ b/agreements.md @@ -189,7 +189,8 @@ Scope and rules - The change is low‑risk, contained, and has minimal blast radius. - The author performs a basic functional check. - The change is either self‑explanatory on its own, or (if not obvious) accompanied by a short note capturing the - ask/purpose/constraints so the next person can pick it up without guesswork. + ask/purpose/constraints so the next person can pick it up without guesswork. The note must live in the repository + (in the code or an accompanying `.md`). A PR/commit message alone is not sufficient. - The one‑shot relaxation is mentioned explicitly in the PR/commit (e.g. include `llm-one-shot`). - The change is kept standalone (not tucked into a larger PR). - Larger/core contributions done with LLM assistance require a case‑by‑case, agreed QA/release plan. If we must merge a @@ -202,13 +203,14 @@ Author responsibilities - Read every byte of generated code, functionally test it, and prepare the change for regular review (structure the diff, write tests/docs where applicable). - When using the one‑shot relaxation, make the change pick‑up‑able: if it is not trivial/self‑explanatory, add a short - note (or code comment) capturing the ask/purpose/constraints. If it is obvious, no note is needed. + note capturing the ask/purpose/constraints in the repository (in the code or an accompanying `.md`). If it is obvious, + no note is needed. Do not rely on PR/commit messages as the only place the intent is recorded. Review expectations - Normal review applies; the reviewer may use or not use LLM tooling. - If a one‑shot change is being reviewed anyway, confirm it is low‑risk/contained and either self‑explanatory or - accompanied by a brief note as above. + accompanied by a brief note in the repository as above (not only in the PR/commit message). - Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in any notes or prompts. diff --git a/docs/Code_Review.md b/docs/Code_Review.md index ec46b9c..e1e21ea 100644 --- a/docs/Code_Review.md +++ b/docs/Code_Review.md @@ -15,7 +15,7 @@ and the governance process, operating in front of you and with you, will decide Responsibility remains with humans: authors and reviewers are accountable for code quality whether or not LLMs were used. See the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (low blast radius, basic -functional check, self‑explanatory or briefly noted). Otherwise, request review as usual. +functional check, self‑explanatory or briefly noted in the repository). Otherwise, request review as usual. ## When/how to request a code review @@ -38,6 +38,7 @@ functional check, self‑explanatory or briefly noted). Otherwise, request revie not decided to switch. 6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). If you’re using the LLM one‑shot relaxation, keep the change standalone (not tucked into larger PRs), make it self‑explanatory or add a brief note, + ensure the note lives in the repository (in the code or an accompanying `.md`, not only in the PR/commit message), read/functionally verify all generated code before requesting review or merging, and mention `llm-one-shot` in the PR/commit. @@ -110,7 +111,8 @@ Note: not all of these apply to all PRs, of course. 12. Performance / memory considerations. Will it OOM in a corner case after the change? 13. Self-healing. If a (network, HDD) device or a part of the system breaks temporarily, will the rest of the system recover automatically or will it require manual intervention? -14. If a one‑shot LLM change is being reviewed: is it low‑risk/contained and either self‑explanatory or briefly noted? +14. If a one‑shot LLM change is being reviewed: is it low‑risk/contained and either self‑explanatory or briefly noted in + the repository? 15. Maintainability not degraded (structure, naming, tests, docs); no secrets in notes/prompts. ## How to resolve the reviewer’s comments From fc1a545a244b9d50156d1d9b8b3613fa735491c7 Mon Sep 17 00:00:00 2001 From: Mateusz Srebrny Date: Thu, 18 Dec 2025 13:22:51 +0100 Subject: [PATCH 6/6] more precise one-shot definition --- agreements.md | 6 ++++-- docs/Code_Review.md | 5 +++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/agreements.md b/agreements.md index b8b16e9..b1acb7f 100644 --- a/agreements.md +++ b/agreements.md @@ -184,8 +184,10 @@ Motivation Scope and rules -- Default path: normal QA and review. One‑shot changes (small, independently judgeable changes) may be merged with - relaxed quality and may skip review if all of the following hold: +- Default path: normal QA and review. +- One‑shot changes (small, well‑separated changes with limited implications on the rest of the codebase, independently + judgeable, and reasonably redoable within one day) may be merged with relaxed quality and may skip review if all of + the following hold: - The change is low‑risk, contained, and has minimal blast radius. - The author performs a basic functional check. - The change is either self‑explanatory on its own, or (if not obvious) accompanied by a short note capturing the diff --git a/docs/Code_Review.md b/docs/Code_Review.md index e1e21ea..75e45db 100644 --- a/docs/Code_Review.md +++ b/docs/Code_Review.md @@ -14,8 +14,9 @@ and the governance process, operating in front of you and with you, will decide Responsibility remains with humans: authors and reviewers are accountable for code quality whether or not LLMs were used. See the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding). -Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (low blast radius, basic -functional check, self‑explanatory or briefly noted in the repository). Otherwise, request review as usual. +Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (well‑separated with limited +implications and low blast radius, basic functional check, self‑explanatory or briefly noted in the repository). +Otherwise, request review as usual. ## When/how to request a code review