Enable Automation style tests of v2 applications - Fluent assertions

[tznind]

Feature

Enable easy end to end testing of v2 applications. Initially for just ourselves but later perhaps our users too.

• Standalone project with only dependency on Terminal.Gui
• Not tied to xunit / nunit for maximum flexibility
• Test both drivers simultaneously

    [Theory]
    [InlineData (V2TestDriver.V2Win)]
    [InlineData(V2TestDriver.V2Net)]
    public void GuiTestContext_StartsAndStopsWithoutError (V2TestDriver d)
    {
        using var context = With.A<Window> (40, 10,d);

• Screenshot and Logging support

using GuiTestContext c = With.A<Window> (40, 10,d)
                             .WithContextMenu (ctx, menuItems)
                             .ScreenShot ("Before open menu", _out)

                             // Click in main area inside border
                             .RightClick (1, 1)
                             .ScreenShot ("After open menu", _out)
                             .Down ()
                             .Down ()
                             .Down ()
                             .Right()
                             .ScreenShot ("After open submenu", _out)
                             .Down ()
                             .Enter ()
                             .ScreenShot ("Menu should be closed after selecting", _out)
                             .Stop ()
                             .WriteOutLogs (_out);

    Assert.True (clicked);

Example Output

Before open menu:
┌──────────────────────────────────────┐
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
└──────────────────────────────────────┘

After open menu:
┌──────────────────────────────────────┐
│                                      │
│┌────────┐                            │
││ One    │                            │
││ Two    │                            │
││ Three  │                            │
││ Four  ►│                            │
││ Five   │                            │
││ Six    │                            │
└└────────┘────────────────────────────┘

After open submenu:
┌──────────────────────────────────────┐
│                                      │
│┌────────┐┌───────────┐               │
││ One    ││ SubMenu1  │               │
││ Two    ││ SubMenu2  │               │
││ Three  ││ SubMenu3  │               │
││ Four  ►││ SubMenu4  │               │
││ Five   ││ SubMenu5  │               │
││ Six    ││ SubMenu6  │               │
└└────────┘│ SubMenu7  │───────────────┘

Menu should be closed after selecting:
┌──────────────────────────────────────┐
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
│                                      │
└──────────────────────────────────────┘

Main Loop Coordinator booting...
Main Loop Coordinator booting complete
[ConfigurationManager] [Load] reset = False
[ConfigurationManager] [Reset] _allConfigProperties = 
[ConfigurationManager] [Initialize]   Class: TestConfiguration
[ConfigurationManager] [Initialize]   Class: AppSettingsTestClass
[ConfigurationManager] [Initialize]   Class: UICatalogApp
[ConfigurationManager] [Initialize]   Class: ConfigurationEditor
[ConfigurationManager] [Initialize]   Class: Application
[ConfigurationManager] [Initialize]   Class: ConfigurationManager
[ConfigurationManager] [Initialize]   Class: ThemeManager
[ConfigurationManager] [Initialize]   Class: Color
[ConfigurationManager] [Initialize]   Class: Colors
[ConfigurationManager] [Initialize]   Class: Glyphs
[ConfigurationManager] [Initialize]   Class: FileDialogStyle
[ConfigurationManager] [Initialize]   Class: Key
[ConfigurationManager] [Initialize]   Class: Button
[ConfigurationManager] [Initialize]   Class: CheckBox
[ConfigurationManager] [Initialize]   Class: Dialog
[ConfigurationManager] [Initialize]   Class: FileDialog
[ConfigurationManager] [Initialize]   Class: FrameView
[ConfigurationManager] [Initialize]   Class: ContextMenu
[ConfigurationManager] [Initialize]   Class: MessageBox
[ConfigurationManager] [Initialize]   Class: NerdFonts
[ConfigurationManager] [Initialize]   Class: Window
[SettingsScope] [Update] Read from "resource://[Terminal.Gui]/Terminal.Gui.Resources.config.json"
[SettingsScope] [Update] Read from "ConfigurationManager.RuntimeConfig"
[SettingsScope] [Update] "./.tui/config.json" does not exist.
[SettingsScope] [Update] "C:\Users\44777/.tui/config.json" does not exist.
[SettingsScope] [Update] "./.tui/ReSharperTestRunner.config.json" does not exist.
[SettingsScope] [Update] "C:\Users\44777/.tui/ReSharperTestRunner.config.json" does not exist.
Run 'Window(){X=0,Y=0,Width=0,Height=0}'
[MainLoop] [AnySubViewsNeedDrawn] Window triggered redraw (NeedsDraw=True NeedsLayout=True) 
Console size changes from '{Width=0, Height=0}' to {Width=40, Height=10}
[InputProcessor] [OnMouseEvent] Mouse Interpreter raising Button3Pressed, ReportMousePosition
[InputProcessor] [OnMouseEvent] Mouse Interpreter raising Button3Released, ReportMousePosition
[MouseInterpreter] [RaiseClick] Raising click event:Button3Clicked at screen {X=1,Y=1}
[InputProcessor] [OnMouseEvent] Mouse Interpreter raising Button3Clicked
[MainLoop] [AnySubViewsNeedDrawn] Window triggered redraw (NeedsDraw=True NeedsLayout=True) 
[AnsiResponseParser] [AppendOutput] AnsiResponseParser releasing '\0'
[InputProcessor] [OnKeyDown] InputProcessor raised CursorDown
[MainLoop] [AnySubViewsNeedDrawn] Menu triggered redraw (NeedsDraw=True NeedsLayout=False) 
[AnsiResponseParser] [AppendOutput] AnsiResponseParser releasing '\0'
[InputProcessor] [OnKeyDown] InputProcessor raised CursorDown
[MainLoop] [AnySubViewsNeedDrawn] Menu triggered redraw (NeedsDraw=True NeedsLayout=False) 
[AnsiResponseParser] [AppendOutput] AnsiResponseParser releasing '\0'
[InputProcessor] [OnKeyDown] InputProcessor raised CursorDown
[MainLoop] [AnySubViewsNeedDrawn] Menu triggered redraw (NeedsDraw=True NeedsLayout=False) 
[AnsiResponseParser] [AppendOutput] AnsiResponseParser releasing '\0'
[InputProcessor] [OnKeyDown] InputProcessor raised CursorRight
[MainLoop] [AnySubViewsNeedDrawn] Window triggered redraw (NeedsDraw=True NeedsLayout=True) 
[AnsiResponseParser] [AppendOutput] AnsiResponseParser releasing '\0'
[InputProcessor] [OnKeyDown] InputProcessor raised CursorDown
[MainLoop] [AnySubViewsNeedDrawn] Menu triggered redraw (NeedsDraw=True NeedsLayout=False) 
[AnsiResponseParser] [AppendOutput] AnsiResponseParser releasing '
'
[InputProcessor] [OnKeyDown] InputProcessor raised Enter
[MainLoop] [AnySubViewsNeedDrawn] Window triggered redraw (NeedsDraw=True NeedsLayout=True) 
RequestStop ''
Input loop exited cleanly

Goals

Perform 'automation' style tests using black box input/output.

• Design from ground up to prevent tests ever hanging (i.e. with timeouts etc baked in)
• Simplify API so user doesn't need to know as much about the internals of Terminal.Gui to test things (e.g. no need to call NewKeyEvent etc)
• Traceability (comprehensive logging and 'screenshoting' to assist debugging failure states)
• v2 drivers only but ideally support net and win
• Input events supplied at raw 'console' level i.e. going through whole application/driver subsystem.

Examples

Started exploring a way to empower our users and ourselves to write cleaner tests around full Terminal.Gui application loops.

    [Fact]
    public void TestWindowsResize ()
    {
        var lbl = new Label ()
                                {
                                    Width = Dim.Fill ()
                                };
        using var c = With.A<Window> (40, 10)
                          .Add (lbl )
                          .Assert (lbl.Frame.Width.Should().Be(38)) // Window has 2 border
                          .ResizeConsole (20,20)
                          .Assert (lbl.Frame.Width.Should ().Be (18))
                          .Stop ();

    }

Also

    [Fact]
    public void ContextMenu_Click ()
    {
        var clicked = false;

        var ctx = new ContextMenu ();
        var menuItems = new MenuBarItem (
                                         [
                                             new ("_New File", string.Empty, () => { clicked = true; })
                                         ]
                                        );

        using var c = With.A<Window> (40, 10)
                          .WithContextMenu(ctx,menuItems)
                          // Click in main area inside border
                          .RightClick(1,1)
                          .ScreenShot ("After open menu",_out)
                          .LeftClick (3, 3)
                          .Stop ();
        Assert.True (clicked);
    }

Produces

  Standard Output: 
After open menu:
┌──────────────────────────────────────┐
│                                      │
│┌───────────┐                         │
││ New File  │                         │
│└───────────┘                         │
│                                      │
│                                      │
│                                      │
│                                      │
└──────────────────────────────────────┘

Class Diagram

Fixes

• Fixes #_____

Proposed Changes/Todos

• Todo 1

Pull Request checklist:

• I've named my PR in the form of "Fixes #issue. Terse description."
• My code follows the style guidelines of Terminal.Gui - if you use Visual Studio, hit CTRL-K-D to automatically reformat your files before committing.
• My code follows the Terminal.Gui library design guidelines
• I ran dotnet test before commit
• I have made corresponding changes to the API documentation (using /// style comments)
• My changes generate no new warnings
• I have checked my code and corrected any poor grammar or misspellings
• I conducted basic QA to assure all features are working

[TheTonttu]

FYI there are some license related controversy surrounding Fluent Assertions. 7.x was the last Apache licensed version before the library switched to custom hybrid license. It doesn't really affect this project but just something to keep in mind if you're planning on relying on it extensively.

[tznind]

Interesting, in that case maybe I will just switch to Action and let caller decide. Thanks for the heads up!

[tznind]

I've chopped out fluent assertions so we can stick with MIT license and renamed the new project FluentTesting instead to avoid any confusion.

[tznind]

Ok I think this is good as a first cut!

Let me know what you think, to start with this can just be for internal testing. Later on we can probably setup CI to release it as a new nuget package to enable our users to test their applications.

Currently the faked input support is:

• Cursor keys
• Mouse left/right click

We can add other things later on.