Building a testing abstraction on top of innmind/operating-system

[Baptouuuu]

Situation

Over the past months/years the ecosystem has been transformed to help build innmind/actors.

The main transformation was to improve handling errors in a type safe way. This resulted in throwing exceptions being replaced by returning Attempt<T> (example).

This change throughout the ecosystem raised an issue: maintainability. Many packages use interfaces to define public contracts. This mean that any change to these interfaces require to publish a new major version of the package. Even to add a new feature that could be backward compatible it requires a new major version.

With the current number of packages this takes a lot of time.

That's why in the last wave of major versions many interfaces have been removed. This allows to better control what can be done and safely add new features in minor versions.

However packages higher in the dependency graph relied on these interfaces to create mocks in their tests. This means there are hundreds of tests that needs to be rewritten. But with the new APIs it takes a lot of effort to rewrite them.

Bringing back interfaces is out of the question, these mocks are also problematic as it leads to brittle tests as it hides bugs and hinders refactoring.

This leads to the next major project of this ecosystem: building an abstraction to test code relying on innmind/operating-system.

Goal

On top of helping rewrite the tests relying on mocks, this abstraction will help better test the implementation of innmind/actors.

To do this, the abstraction will need to hijack (in a type safe way) the innmind/operating-system API to change its implementation. This will be most likely be done via its Config system.

The APIs needed to build this abstraction will need to be agnostic of this testing context.

This restriction is meant to lay the ground work for other systems hijacking this operating-system, such as a profiling abstraction (something like Apple's Instruments app).

Strategies

This testing abstraction will work in 2 modes:

• sequential
• reactive

The ability to mix both is yet to be determined.

Sequential

This is an approach similar to mocks, where you define the precise order of calls that needs to happen.

This mode is mainly here to ease rewriting existing tests. But it should also be helpful to test small apps with a straightforward logic.

Note

Seems the sequential API could be built on top the reactive one.

Reactive

This will help take full advantage of innmind/black-box and Property Based Testing to test complex systems. The main one being innmind/actors.

By not specifying the order of operations it can address both testing synchronous and asynchronous code.

Since the ecosystem is built around immutable and declarative structures it will allow to build these reactive tests as test oracles that can be reused across projects.

For example, a web API could provide a test oracle of its API as innmind/framework middleware. And an app that needs to test against this API could mock the calls and route them through the middleware.

This makes sure the mocks will always behave the same way as the real API.

API Draft

This section is here to help draft this abstraction API.

This graph represent the extent of components that needs to expose testing APIs.

flowchart LR
    os["OS"]
    fs["Filesystem"]
    watch["Watch"]
    process["Process"]
    signals["Signals"]
    halt["Halt"]
    clock["Clock"]
    sockets["Sockets"]
    ports["Ports"]
    status["Status"]
    control["Control"]
    remote["Remote"]
    ssh["SSH"]
    http["HTTP"]
    sql["SQL"]
    os --> status
    os --> ports
    os --> sockets
    os --> clock
    os --> fs
    os --> process
    fs --> watch
    process --> signals
    process --> halt
    os --> remote
    remote --> ssh
    remote --> http
    remote --> sql
    os --> control
    ssh -. same as .-> control

Loading

The API to compose the different systems is yet to be determined.

Design to split what is state from what is communications and the interactions:

flowchart TD
    Machine --> State
    Machine --> Communications
    State --> Clock
    State --> Filesystem
    Filesystem --> FSState
    Filesystem --> Watch
    Watch -- Affect --> Clock
    Watch --> FSState
    State --> Processes
    Processes -- Has access --> FSState
    State --> SQL
    SQL --> DB1
    SQL --> DB2
    DB1 --> Table1
    DB1 --> Table2
    DB2 --> TableX
    Communications --> HTTP
    Communications --> Sockets
    Communications --> Ports
    Communications --> Signals
    Sender(("Sender")) --> Signals
    HTTP --> HTTP_DAS@{shape: das, label: "Reactor"}
    Sockets --> Sockets_DAS@{shape: das, label: "Reactor"}
    Ports --> Ports_DAS@{shape: das, label: "Reactor"}

Loading

Usage

$config = Innmind\Testing\Config::new()
    ->startClockAt($pointInTime)
    ->handleExecutable(
        'bin (ie: ls, dot, etc...', 
        static fn(Command $command, ProcessBuilder $builder) => $builder->success(),
    )
    ->handleSQLQuery(
        static fn(Query $query, Database $db): Sequence<Row> => $db
            ->table('name')
            ->select(),
    )
    ->sockets(
        static fn($sockets) => $sockets 
            ->listen(
                '/path/to/socket',
                static fn($tbd) => $tbd,
            )
            ->connect(
                '/path/to/socket',
                static fn($tbd) => $tbd,
            ),
    )
    ->ports(
        static fn($ports) => $ports 
            ->listen(
                8080,
                static fn($tbd) => $tbd,
            )
            ->connectTo(
                $ip,
                8080,
                static fn($tbd) => $tbd,
            ),
    )
    ->http(
        static fn($config) => $config
            ->host(
                'url authority A',
                static fn(ServerRequest $request) => $tbd,
            ),
    )
    ->ssh(
        'url',
        static fn($config) => $config->handleExecutable(
            'bin',
            static fn(Command $command, ProcessBuilder $builder) => $builder->success(),
        ),
    );
$os = $os->map($config);
$config->signal(Signal::terminate);

An abstraction should be built on top of this to give access to the new $os to the callables in order for reactors to affect the machine.

As for the filesystem it should emulate a concrete one and affect the filesystem watch. Processes should rely on the configured $os to access the filesystem if needed to only expose one API to end users.

---

Ideas

HTTP

use Innmind\Testing\Remote\Http;

$transport = Http::sequential(/* instance of Innmind\BlackBox\Runner\Assert */)
    ->expect(
        fn(Request $request): void => $assert,
        fn(Request $request, Respond $respond): Respond => $respond
            ->response(Response::of(...$args))
            ->connectionFailure(string $reason)
            ->failure(string $reason)
            ->malformedResponse() // specify content ?
    )
    ->expect($next, $request);
$transport = Http::reactive(
    fn(ServerRequest $request): Attempt<Response> => $response,
);

For the reactive approach simply defining a callable allows to build multiple abstractions. Like one built on top of innmind/router or another via innmind/framework. It exposes a ServerRequest to allow to reuse these packages, this means it will need to transform a Request to a ServerRequest (this may need to use innmind/http-parser).

$transport = Http::reactive(
    fn(ServerRequest $request) => Router::of(
        Pipe::new()
            ->get()
            ->endpoint('/')
            ->handle($handler),
    )($request),
);
// or
$transport = Http::reactive(
    fn(ServerRequest $request) => Application::http($os, $env)
        ->map(new Middleware)
        ->run($request),
);

This could further be simplified by creating abstractions on top of innmind/router and innmind/framework. The latter is a good candidate, as the $os could require hijacking its implementation as well and $env could benefit a fluent API to define pairs.

Http::reactive()
    ->host(
        'sub.example.com',
        fn($builder) => $builder->app(new Middleware)
        // or
        fn($builder) => $builder->router($component)
    )
    ->host('another', callable);

Clock

Using the forzen clock provided by innmind/time-continuum should be enough.

[Baptouuuu]

In order to have an API uniformity to how this package will be built, these packages will need to be updated to use the strategy pattern:

• innmind/file-watch
• innmind/filesystem (this one may be tricky in order to still be compatible with innmind/s3)
• innmind/http-transport
• innmind/io
• innmind/server-control
• innmind/server-status
• innmind/time-continuum
• innmind/time-warp
• formal/access-layer

The strategy pattern here intend to make the interface @internal and use named constructors.

Since these changes will affect the innmind/foundation and need a new major release, it would be interesting to cleanup:

• innmind/media-type
• innmind/url

The main change is to make the constructors private, and fix problems described in issues.

[Baptouuuu]

Might as well create a new major of innmind/immutable to remove deprecated code and simplify a few method outputs 🤔

[Baptouuuu]

The project has been bootstrapped at https://github.com/Innmind/testing

[Baptouuuu]

Turns out this is called Deterministic Simulation Testing https://notes.eatonphil.com/2024-08-20-deterministic-simulation-testing.html

Maybe the package should reflect that name.

[Baptouuuu]

While working on all these packages it showed that a new package may be needed: innmind/mutable. It would contain mutable data structure such as Queue, Stack or Ring.

It would rely on innmind/immutable for its API. For example Queue::pull() would return a Maybe<T> as the queue can be empty.

[Baptouuuu]

Set and Map are also needed.