3.3.1. Automating testing

The example in Section 3.3 is a fairly common framework, and it makes sense to automate it using TCL procedure. For example the test name and command line could be passed as arguments to the procedure as follows.

proc runmytest { test_name command_line } {
    global verbose

    # When verbose, tell the user what we are running
    if { $verbose > 1 } {
        send_user "starting $command_line\n"
    }
    
    # Run the program.
    spawn $command_line
    
    expect {
        # Check for any warning messages in the output first
        Warning {
        fail "$test_name: warning: $expect_out(buffer)"
        }
    
        # Check for any error messages
        ERROR {
        fail "$test_name: error: $expect_out(buffer)"
        }
    
        # The string indicating successful completion
        "Test complete" {
             pass "$test_name\n"
        }
    
        # EOF and timeout only come after everything else. EOF must be an error.
        eof {
        fail "$test_name: EOF\n"
        }
    
        # Timeout requires inspection to determine the cause of failure.
        timeout {
            unresolved "$test_name: timeout"
        }
    }
}

The code is identical, except we must note from within the procedure that verbose is a global variable.

Our series of tests could then just be as follows

# Timeout reduced to 3 seconds
set timeout 3

runmytest "Simple test" "or32-elf-sim -f default.cfg test-prog.or32"
runmytest "Harder test" "or32-elf-sim -f default.cfg test-prog2.or32"
runmytest "Hardest test" "or32-elf-sim -f default.cfg test-prog3.or32"

The correct place for a procedure shared amongst a number of tests like this is the tool configuration file (see Section 2.4).