Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions source/best-practice.rst
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@ solutions:
* Contain too many known “bad” words, e.g. “low interest rate loans”
* Has “.exe” attachments

* An AI approach that learns from a corpus of previous emails to read an emails
content, and classifies it as phishing, or not phishing.
* An AI approach that learns from a corpus of previous emails to read an email's
content, and classify it as phishing, or not phishing.

The respective advantages of each approach are:

Expand Down Expand Up @@ -197,7 +197,7 @@ natural part of using the guide. Our previous effort is not wasted, but informs
our current and future decisions.

We can break AI approaches into 3 categories. We provide a more detailed
explanation of these terms in the [appendix]. However, briefly:
explanation of these terms in the [appendix - WHERE?]. However, briefly:

* In supervised learning approaches we are learning by example. We have a set of
input data points with corresponding known output data points (labels) for
Expand Down Expand Up @@ -247,13 +247,13 @@ We will not cover again the work of general goal setting here, which we assume
should be familiar to any competent engineer. However, over other engineering
projects AI and Machine learning projects have several uncommon considerations.

The first is that AI and Machine Learning algorithms are often stochastic in
One such consideration is that AI and Machine Learning algorithms are often stochastic in
nature, behaving in a non-deterministic and statistical way. All goals and KPIs
must be aware of this. For example, for a face detection software it is
appropriate to set goals such as “95% of this set of faces can be recognised”
rather than “the software must recognise all these specific people”.

The second is that human interpretability is a difficult topic in AI and Machine
A second consideration is that human interpretability is a difficult topic in AI and Machine
Learning, and it may be difficult to define why the software is doing the things
or making the decisions that it makes. We will cover this topic (and its
mitigations in more detail) in later sections, but it is important to be aware
Expand Down Expand Up @@ -492,6 +492,8 @@ a high AI Risk category, and be subject to extra requirements.
:ref:`Version Control, CI/CD for Data<How do I collect data for my AI/ML project?>`
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

IS "CI/CD" REALLY THE APPROPRIATE TERMINOLOGY HERE?

Any electronic systems engineer should be familiar with version control. These
ideas are just as important in developing AI applications as any other software
product. In this step we explore:
Expand Down Expand Up @@ -526,7 +528,7 @@ Examples
:ref:`Documentation<How do I collect data for my AI/ML project?>`
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Another task that any electronic systems engineer should be familiar with is
Another task that any software engineer should be familiar with is
documentation. As with version control, good documentation is just as important,
or perhaps even more so, as any other software development project. In this step
we will explore:
Expand Down Expand Up @@ -824,7 +826,7 @@ wealth of resources elsewhere (imputation), but we note that the extent to which
any of these strategies is plausible depends on how the missingness in our data
has come about. There are three ways the data might be missing:

* Missing totally at random (MCAR) - the data is missing completely randomly
* Missing completely at random (MCAR) - the data is missing completely randomly
with no pattern at all
* Missing random (MAR) - the data is missing randomly, but in a way that is
explained by the data you are using to predict things
Expand Down
2 changes: 2 additions & 0 deletions source/definitions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
Application Specific Requirements
==================================

IS "APPLICATION SPECIFIC" THE PHRASE? IF SO, PERHAPS CLARIFY WHY

.. _AI Risk:

AI Risk
Expand Down
63 changes: 32 additions & 31 deletions source/using-the-guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,31 +2,31 @@

.. _Using this Guide:

Using this guide: The 5 questions
Using this guide: The five questions
=================================

This guide will help you answer 5 questions step by step:
This guide will help you answer five questions step by step:

* Should I use AI/ML?
* How do I define my AI/ML project
* How do I collect data?
* How do I train my AI/ML application?
* How do I deploy my AI/ML application?
1. Should I use AI/ML?
2. How do I define my AI/ML project
3. How do I collect data?
4. How do I train my AI/ML application?
5. How do I deploy my AI/ML application?

Answering these questions will take you from conceptualizing your
AI application through to deploying it in the real world.

These 5 questions are broken into smaller questions, requirements, and tasks to
These five questions are broken into subsidiary questions, requirements, and tasks to
be completed. We set out the question in the order they should be considered.
We strongly recommend you don't skip ahead, although from time to time you may
want to go back and review your answers to earlier questions.

These smaller questions, requirements and tasks for each of the 5 main questions
These smaller questions, requirements and tasks for each of the five main questions
are recorded in a table, showing what the activity is, and where you should
record the documentary evidence of that activity being completed. Some of the
entries may be optional, depending on earlier answers. Once all the required
smaller questions, requirements, and tasks are addressed, you will have your
answer to the main question, and can move on to the next of these 5 questions.
subsidiary questions, requirements, and tasks are addressed, you will have your
answer to the main question, and can move on to the next of these five questions.

+----------------------------+--------------------------------------+----------+
| Question/Task/Requirement | Evidence | Done |
Expand All @@ -41,12 +41,12 @@ answer to the main question, and can move on to the next of these 5 questions.

.. _Should I Use AI/ML?:

Should I use AI/ML?
-------------------
Question 1: Should I use AI/ML?
-------------------------------

Developing an AI application can present significant challenges. Collection of
data, testing and validation are challenges. As seen in the previous section,
many applications of AI will come with special requirements that can be a
data, testing and validation are challenges.
Many applications of AI will come with special requirements that can be a
challenge in themselves. To address this, the very first step in this guide is
to be able answer the following: should I use AI to solve my problem? We break
this problem down into two parts:
Expand All @@ -55,10 +55,9 @@ this problem down into two parts:
approaches?
* Does the problem touch on any application or data areas that may effectively
prohibit development?

* What is the risk that the project falls into the unacceptable AI Risk
category?
* What is the risk that the project makes use of problematic data?
* What is the risk that the project falls into the :ref:`unacceptable AI Risk
category<Unacceptable AI Risk>`?
* What is the risk that the project makes use of :ref:`problematic data<Problematic Data Risk>`?

+--------------------------------------------------------+----------+----------+
| Requirement | Evidence | Complete |
Expand All @@ -72,15 +71,15 @@ this problem down into two parts:

.. _How do I define my AI project?:

How do I define my AI project?
------------------------------
Question 2: How do I define my AI project?
------------------------------------------

In the previous step we worked on an engineering case for AI in our project. In
this step, we will take the first step in realizing our project by setting the
goals and bounds of the project. There are three steps to complete in this
section:

* Defining goals and metrics for success
* Defining goals and metrics for success (KPIs)
* Defining limitations and boundaries on the project
* Completing a risk assessment of the project as a whole

Expand All @@ -89,23 +88,25 @@ section:
+========================================================+==========+==========+
| :ref:`Establish Goals and KPIs` | - | |
+--------------------------------------------------------+----------+----------+
| LIMITATIONS AND BOUNDARIES ON THE PROJECT? | - | |
+--------------------------------------------------------+----------+----------+
| :ref:`Risk Assessment` | - | |
+--------------------------------------------------------+----------+----------+

.. _How do I collect data for my AI/ML project?:

How do I collect data for my AI/ML project?
-------------------------------------------
Question 3: How do I collect data for my AI/ML project?
-------------------------------------------------------

In the previous step, we defined the scope of our project. In this step, we move
on to the first part of the practical engineering of our AI project: collecting
the data. AI/ML applications are, at their core, data driven. At some level,
data will need to be collected, or at the very least, processed. It’s very
important to get this right, as the strength of the data will have a strong
effect on the efficacy of training and deploying our AI application. We set out
impact on the efficacy of training and deploying our AI application. We set out
a number of steps for this section, but our primary challenges are:

* Making sure the data we’re is collecting useful, truthful, and effective data
* Making sure the data we’re is collecting useful, truthful, and effective
* Making sure we transform our raw data into a form that can effectively
utilized by AI/ML algorithms
* Making sure our infrastructure for collection, storage, and access is
Expand Down Expand Up @@ -134,8 +135,8 @@ a number of steps for this section, but our primary challenges are:

.. _How do I train my AI/ML application?:

How do I train my AI/ML application?
------------------------------------
Question 4: How do I train my AI/ML application?
------------------------------------------------

In the previous step, we collected the data for our AI project. In this step,
we will make use of it by using it to train an AI algorithm of our choice to
Expand Down Expand Up @@ -169,16 +170,16 @@ steps for this section, but our primary challenges are:

.. _How do I deploy my AI application?:

How do I deploy my AI application?
----------------------------------
Question 5: How do I deploy my AI application?
----------------------------------------------

After training our AI application, we can finally deploy it and (hopefully)
achieve the goals set out in our previous steps. This step will likely represent
a welcome return to familiarity for the professional engineer, as the process
for deploying an AI application is fairly similar to that of deploying any other
software application. Our process will proceed broadly in three steps:

* Preparing our trained the model for a live environment
* Preparing our trained model for a live environment
* Engineering a process for deployment and model updating
* Setting up continuous monitoring for our model

Expand Down