PowerShell Scripts for Testing

NEW POST: Test results in XML format?

Kibi78704 — Thu, 05 Jun 2008 15:36:05 GMT

Let's take the second approach, where I post the code to the forum and you commit it. That seems like the easiest at this point. It will take me a while to get to this, since I have several other tasks on my plate that take precedence. :) I was hoping that someone had already done the work.

BTW - I am using the PsExpect library on a daily basis, and am very, very happy with it. It is now a standard module in all of my automated tests. Thanks!

NEW POST: Tutorial (French language)

ageras — Thu, 05 Jun 2008 01:35:25 GMT

Merci!

Batchman wrote:
I've created a tutorial (French language):
ftp://ftp-developpez.com/laurent-dardenne/articles/Windows/PowerShell/TestsUnitairesSousPowerShell/fichiers/Tests-unitaires-sous-PowerShell.pdf

NEW POST: Tutorial

ageras — Thu, 05 Jun 2008 01:34:03 GMT

thanks very much! I will!

NEW POST: Test results in XML format?

ageras — Thu, 05 Jun 2008 01:32:27 GMT

Kibi78704 wrote:
I've been using this library since roughly January, and have had great success with it.

I find that I am going to have to extend the library to send output not only to both the console and to a log file, but also to an XML file so that eventually we can store the test results in a custom database. If anyone else has already done this, I'd appreciate an update to the code base. If not, does anyone else have need for XML output? If so, I will submit my modifications here; otherwise, I'll keep them for just my company's use.

Great to hear that you've had some success with it. I have heard that requirement from one other group, albeit for different reasons - they wanted to transform it into HTML and automatically post it to Sharepoint (that's what they do with their Quick Test Pro test results). I can add you as a contributor if you're interested, otherwise just post the code in the forum and I'll commit it on your behalf. Whatever is easier for you.

NEW POST: Test results in XML format?

Kibi78704 — Wed, 04 Jun 2008 15:23:55 GMT

I've been using this library since roughly January, and have had great success with it.

I find that I am going to have to extend the library to send output not only to both the console and to a log file, but also to an XML file so that eventually we can store the test results in a custom database. If anyone else has already done this, I'd appreciate an update to the code base. If not, does anyone else have need for XML output? If so, I will submit my modifications here; otherwise, I'll keep them for just my company's use.

NEW POST: Tutorial (French language)

Batchman — Thu, 29 May 2008 13:33:25 GMT

I've created a tutorial (French language):
ftp://ftp-developpez.com/laurent-dardenne/articles/Windows/PowerShell/TestsUnitairesSousPowerShell/fichiers/Tests-unitaires-sous-PowerShell.pdf

CREATED ISSUE: Bug in AssertThrows ?

Batchman — Sat, 10 May 2008 14:36:53 GMT

Hello,
no entry is created in $Assertions for the second use case, it this a bug or not ?
Thanks.

# -------- begin code
$Assertions.Clear()
$msg = "Required."

#First
Function Test-Parameter($Required=$(throw $msg)) { write "..." }
$sbTest = { Test-Parameter Test}
AssertThrows System.Management.Automation.RuntimeException $msg $sbTest "With parameter"
AssertThrows System.Management.Automation.RuntimeException $msg $sbTest "With parameter" -Intent $Intention.ShouldFail

#Second
Function Test-Parameter($Required=$(throw $msg)) { write "fait qq chose"|out-null}
#no assertion
AssertThrows System.Management.Automation.RuntimeException $msg $sbTest "With parameter"
#No assertion
AssertThrows System.Management.Automation.RuntimeException $msg $sbTest "With parameter" -Intent $Intention.ShouldFail

$sbTest = { Test-Parameter }
AssertThrows System.Management.Automation.RuntimeException $msg $sbTest "With no parameter"
RaiseAssertions

COMMENTED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

Kibi78704 — Fri, 18 Jan 2008 20:41:56 GMT

I am verifying a series of SQL query results, and need to assert that the results are equal. At the moment, I am just comparing the count of rows in the same table in two separate database instances. I can easily get the results instantiated in objects of type System.Data.DataSet, but am having an interesting time trying to compare them.

I am using a customized version of Lee Holmes' Invoke-SqlCommand.ps1 (see http://www.leeholmes.com/blog/InteractingWithSQLDatabasesInPowerShellInvokeSqlCommand.aspx) to run my queries, but the basic design is the same. The query results are returned in a couple of instances of System.Data.DataSet called $expDataset and $actDataset.

I tested with 3 sets of values. Below are code excerpts and a copy of the displayed results.

*************************

*Code excerpt:*

write-host " "
write-host "*****"
write-host " "
$expDataset.Tables | Select-Object -Expand Rows -Last 1
$actDataset.Tables | Select-Object -Expand Rows -Last 1
AssertEqual $expDataset $actDataset -Label $label -Intent $intent
AssertEqual $expDataset.Tables $actDataset.Tables -Label $label -Intent $intent
AssertEqual $expDataset.Tables[0] $actDataset.Tables[0] -Label $label -Intent $intent
AssertEqual $expDataset.Tables[0].Table $actDataset.Tables[0].Table -Label $label -Intent $intent
AssertEqual $expDataset.Tables[0].Table.Row $actDataset.Tables[0].Table.Row -Label $label -Intent $intent
RaiseAssertions

*************************

*Display:*

PS C:\Global\SPi> C:\global\SPi\TestAssertEqual-TableCount.ps1

Security Warning
Run only scripts that you trust. While scripts from the Internet can be useful, this script can potentially harm your
computer. Do you want to run C:\global\psexpect\Ps1TestLib\src\TestLibConstants.ps1?
[D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): r

Security Warning
Run only scripts that you trust. While scripts from the Internet can be useful, this script can potentially harm your
computer. Do you want to run C:\global\psexpect\Ps1TestLib\src\TestLib.ps1?
[D] Do not run [R] Run once [S] Suspend [?] Help (default is "D"): r

*****

Column1
-------
45
45
1/17/2008 2:45:06 PM,SHOULDPASS,FAILED,TC-Process-test,System.Data.DataSet expected but was System.Data.DataSet
1/17/2008 2:45:06 PM,SHOULDPASS,FAILED,TC-Process-test,System.Data.DataTableCollection expected but was System.Data.Data
TableCollection
1/17/2008 2:45:06 PM,SHOULDPASS,FAILED,TC-Process-test,Table expected but was Table
1/17/2008 2:45:06 PM,SHOULDPASS,PASSED,TC-Process-test
1/17/2008 2:45:06 PM,SHOULDPASS,PASSED,TC-Process-test

*****

156067
156224
1/17/2008 2:45:06 PM,SHOULDFAIL,FAILED,TC-ProcessLog-test,System.Data.DataSet expected but was System.Data.DataSet
1/17/2008 2:45:06 PM,SHOULDFAIL,FAILED,TC-ProcessLog-test,System.Data.DataTableCollection expected but was System.Data.D
ataTableCollection
1/17/2008 2:45:06 PM,SHOULDFAIL,FAILED,TC-ProcessLog-test,Table expected but was Table
1/17/2008 2:45:07 PM,SHOULDFAIL,PASSED,TC-ProcessLog-test,Expected 'FAILED' but was 'PASSED'.
1/17/2008 2:45:07 PM,SHOULDFAIL,PASSED,TC-ProcessLog-test,Expected 'FAILED' but was 'PASSED'.

*****

8904
8905
1/17/2008 2:45:07 PM,SHOULDFAIL,FAILED,TC-Error-test,System.Data.DataSet expected but was System.Data.DataSet
1/17/2008 2:45:07 PM,SHOULDFAIL,FAILED,TC-Error-test,System.Data.DataTableCollection expected but was System.Data.DataTa
bleCollection
1/17/2008 2:45:07 PM,SHOULDFAIL,FAILED,TC-Error-test,Table expected but was Table
1/17/2008 2:45:07 PM,SHOULDFAIL,PASSED,TC-Error-test,Expected 'FAILED' but was 'PASSED'.
1/17/2008 2:45:07 PM,SHOULDFAIL,PASSED,TC-Error-test,Expected 'FAILED' but was 'PASSED'.

PS C:\Global\SPi>
Comments: ** Comment from web user: Kibi78704 **

.NET & PowerShell are new to me, so this is all probably just a newbie mistake on my part. I'll check out your suggestions. Thanks for the pointers!

BTW - the only reason I was deep-checking the entire data structure was because I was not getting the results I expected. I'd love to be able to do one and only one assert on the *appropriate* object for the verification.

COMMENTED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

Kibi78704 — Fri, 18 Jan 2008 20:34:14 GMT

I believe that I am not using System.Data.Dataset correctly, and that is the actual problem, rather than your code.

COMMENTED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

ageras — Fri, 18 Jan 2008 18:04:20 GMT

Sorry - some background - sp_table_validation is a SQL Server stored procedure.

COMMENTED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

ageras — Fri, 18 Jan 2008 18:02:31 GMT

Yes, it looks like a framework feature and how they have implemented the .Equals method on those objects. You might be able to write a function for getting the CRC for the contents and use that as the basis for comparison instead of deep-checking the entire data structure in your test code. Look at the help documentation for sp_table_validation for an example of what I mean - it generates a CRC for an entire table.

COMMENTED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

Kibi78704 — Fri, 18 Jan 2008 17:33:15 GMT

I followed your suggestion. It looks like the problem is deeper than your code.

*****************************

Code Excerpt:

write-host " "
write-host "*****"
write-host " "
write-host "expQuery = $expQuery"
$expDataset.Tables | Select-Object -Expand Rows
write-host "actQuery = $actQuery"
$actDataset.Tables | Select-Object -Expand Rows
if ($actDataset.Tables -eq $expDataset.Tables) {write-host "actDataset.Tables = expDataset.Tables"}
else {write-host "actDataset.Tables != expDataset.Tables"}
AssertEqual $expDataset $actDataset -Label $label -Intent $intent
if ($actDataset -eq $expDataset) {write-host "actDataset = expDataset"}
else {write-host "actDataset != expDataset"}
AssertEqual $expDataset.Tables $actDataset.Tables -Label $label -Intent $intent
if ($actDataset.Tables -eq $expDataset.Tables) {write-host "actDataset.Tables = expDataset.Tables"}
else{write-host "actDataset.Tables != expDataset.Tables"}
AssertEqual $expDataset.Tables[0] $actDataset.Tables[0] -Label $label -Intent $intent
if ($actDataset.Tables[0] -eq $expDataset.Tables[0]) {write-host "actDataset.Tables[0] = expDataset.Tables[0]"}
else {write-host "actDataset.Tables[0] != expDataset.Tables[0]"}
AssertEqual $expDataset.Tables[0].Table $actDataset.Tables[0].Table -Label $label -Intent $intent
if ($actDataset.Tables[0].Table -eq $expDataset.Tables[0].Table) {write-host "actDataset.Tables[0].Table = expDataset.Tables[0].Table"}
else {write-host "actDataset.Tables[0].Table != expDataset.Tables[0].Table"}
AssertEqual $expDataset.Tables[0].Table.Row $actDataset.Tables[0].Table.Row -Label $label -Intent $intent
if ($actDataset.Tables[0].Table.Row -eq $expDataset.Tables[0].Table.Row) {write-host "actDataset.Tables[0].Table.Row = expDataset.Tables[0].Table.Row"}
else {write-host "actDataset.Tables[0].Table.Row != expDataset.Tables[0].Table.Row"}
RaiseAssertions

*****************************

Output to Console:

*****

expQuery = select count(*) from maintenance44.dbo.Process
45
actQuery = select count(*) from maintenance56.dbo.Process
45
actDataset.Tables != expDataset.Tables
actDataset != expDataset
actDataset.Tables != expDataset.Tables
actDataset.Tables[0] != expDataset.Tables[0]
actDataset.Tables[0].Table = expDataset.Tables[0].Table
actDataset.Tables[0].Table.Row = expDataset.Tables[0].Table.Row
1/18/2008 11:12:46 AM,SHOULDPASS,FAILED,TC-Process-test,System.Data.DataSet expected but was System.Data.DataSet
1/18/2008 11:12:46 AM,SHOULDPASS,FAILED,TC-Process-test,System.Data.DataTableCollection expected but was System.Data.DataTableCollection
1/18/2008 11:12:46 AM,SHOULDPASS,FAILED,TC-Process-test,Table expected but was Table
1/18/2008 11:12:47 AM,SHOULDPASS,PASSED,TC-Process-test
1/18/2008 11:12:47 AM,SHOULDPASS,PASSED,TC-Process-test

*****

expQuery = select count(*) from maintenance44.dbo.ProcessLog
156067
actQuery = select count(*) from maintenance56.dbo.ProcessLog
156224
actDataset.Tables != expDataset.Tables
actDataset != expDataset
actDataset.Tables != expDataset.Tables
actDataset.Tables[0] != expDataset.Tables[0]
actDataset.Tables[0].Table = expDataset.Tables[0].Table
actDataset.Tables[0].Table.Row = expDataset.Tables[0].Table.Row
1/18/2008 11:12:47 AM,SHOULDFAIL,FAILED,TC-ProcessLog-Should display on console in yellow-test,System.Data.DataSet expected but was System.Data.DataSet
1/18/2008 11:12:47 AM,SHOULDFAIL,FAILED,TC-ProcessLog-Should display on console in yellow-test,System.Data.DataTableCollection expected but was System.Data.DataTableCollection
1/18/2008 11:12:47 AM,SHOULDFAIL,FAILED,TC-ProcessLog-Should display on console in yellow-test,Table expected but was Table
1/18/2008 11:12:47 AM,SHOULDFAIL,PASSED,TC-ProcessLog-Should display on console in yellow-test,Expected 'FAILED' but was 'PASSED'.
1/18/2008 11:12:47 AM,SHOULDFAIL,PASSED,TC-ProcessLog-Should display on console in yellow-test,Expected 'FAILED' but was 'PASSED'.

*****

COMMENTED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

ageras — Fri, 18 Jan 2008 01:16:08 GMT

I'll work up my own example first - but one thing you can try is to use the -eq operator with the comparisons that unexpectedly fail - and see if it reports an equality.

COMMENTED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

Kibi78704 — Thu, 17 Jan 2008 22:31:36 GMT

I'll be happy to send you my code, if it will help. I've used your TestLib.ps1 & TestLibConstants.ps1 paradigm for creating my own library & constants.

CREATED ISSUE: AssertEqual is not playing nicely with System.Data.DataSet

Kibi78704 — Thu, 17 Jan 2008 21:01:50 GMT

UPDATED RELEASE: 0.4 Alpha (Jan 08, 2008)

Tue, 08 Jan 2008 21:14:39 GMT

Includes functions and aliases for them so that you can write test scripts in the 'given-when-then' behaviour-driven development (BDD) style. This time the source code for the sample applications is included, not just the Powershell script.

The BDD example is in samples\Test-VcsBehave.ps1.

UPDATED RELEASE: 0.4 Alpha (Jan 08, 2008)

Tue, 08 Jan 2008 21:13:45 GMT

Includes aliases so that you can write test scripts in the 'given-when-then' behaviour-driven development (BDD) style. This time the source code for the sample applications is included, not just the Powershell script.

The BDD example is in samples\Test-VcsBehave.ps1.

UPDATED RELEASE: 0.4 Alpha

Tue, 08 Jan 2008 21:13:22 GMT

UPDATED WIKI: user guide

ageras — Tue, 08 Jan 2008 21:03:55 GMT

PSExpect - PowerShell Scripts for Testing

Installation

The PowerShell Scripts for Testing is a function library that implements the familiar xUnit-style assert* statements in PowerShell, Microsoft's new scripting language and command shell.

To be able to use the function library, you must have the following on your computer:

Microsoft .NET Framework 2.0
Microsoft PowerShell 1.0

To install the function library, follow these steps:

Obtain the source code from the PowerShell Scripts for Testing site http://www.codeplex.com/psexpect
Before un-zipping the downloaded source code, right-click on the .zip archive file and view the file properties.

If you see a message that says, "This file came from another computer and might be blocked to help protect this computer"
then click the 'Unblock' button beside the message. This will cause Powershell to treat the scripts as local when
you go to run them. Note - do this BEFORE unzipping the files to save yourself some time.

Unzip the library files into a known location on your hard drive, something like c:\global\psexpect (does not have

to be in your path).

Ensure that you have set the execution policy on your machine to match your expectations. For example, if you

want to run local scripts that are unsigned (as the scripts are provided in PSExpect) then change the execution policy
to 'Unrestricted'. The commands you will need are get-executionpolicy and set-executionpolicy. Remember - only do
this if you understand what it means. See http://technet.microsoft.com/en-us/library/bb978644.aspx?wt.slv=RightRail for
more information.

Run the provided library test script by first navigating to the folder it is located in

set-location <<install-dir>>\Ps1TestLib\src

replacing <<install-dir>> with the location where you unzipped the downloaded files, and then running the script
using the command,

.\TestAll.ps1

Note: Alternatively, you can run the script by specifying it's full path at the Powershell command prompt.

Confirm that the script ran. There is a mix of passing and failing test conditions. All of the test cases that have labels ending with "-test" should be marked as having PASSED. In addition, you should see a file created in the same directory that contains log entries - one per assert statement - that are a record of the test run.

Note: Test conditions that fail are presented in RED in the console and ones that pass are presented in GREEN. You will also see YELLOW in the console output to represent tests that are designed to fail. You will see some errors thrown during the tests for AssertThrows that you can safely ignore that contain the text,

'Get-WmiObject : Command execution stopped because the shell variable "ErrorActionPreference" is set to Stop: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)'.

These errors are expected, getting the error does not mean that you have to make changes to your Powershell environment or to the shell you are executing the tests in.

Note: The last couple of assertions test the Windows service pack level and the Windows installation date. These are there for illustration purposes on using Windows Management Instrumentation (WMI) from Powershell. Running WMI easily is one of Powershell's strengths and I wanted to demonstrate it to the masses.

If these assertions FAIL on your machine, you can adjust them test itself to make it pass. For instance, if you are running Windows XP SP2, then the service level test should specify '2' as the expected result. If you're running Vista (as of Jan. 1, 2008) then the service pack level would be '0'. You can modify the installation date test in a similar way to see if it passes or fails.

Use any of the provided library test scripts TestAssert*.ps1 as an example for getting your own test script started, or see the samples in the samples folder.
Also included in the /doc folder is a syntax highlighting file for EditPadPro (http://www.editpadpro.com/).

Usage Guide

There are some unique features in the PowerShell Scripts for Testing functions that distinguish it from other testing frameworks such as xUnit. In the following text, 'test condition' refers to any Assert* statement.

First, when the test script runs and evaluates a test condition, there is a log entry created in the file referred to by the variable $LogFileName. You can modify the default simply by re-assigning the value of this variable at any time in your script. The log file is intended to provide a digital record of a test run. You can eliminate this feature by setting $LogFileName = $null.

Second, every test condition can be supplied with a "label" parameter - this label parameter appears in the log entry every time the test condition is evaluated. The label is intended to provide a trace back into your test script so that you can quickly identify a failing test condition.

The label for the test condition can also be used to limit the test conditions that run. The variable $TestConditionFilter is a regular expression that defaults to match every label - you can set it to any valid regular expression. If the test condition label matches the regular expression in that variable, then it will be evaluated.

Similarly, the variable $RaiseAssertionFilter limits the test condition results that are displayed to the console during a test run. It defaults so that every test condition result is displayed, but you can change that by setting this variable to any valid regular expression. This filter applies to the combination of the result of the test condition as well as the label so that you can raise only failed test results with a particular label.

Displays results from only test conditions that failed
$RaiseAssertionFilter = $FailPrefix

Displays results from failed test conditions that have the word 'test' in the label
$RaiseAssertionFilter = $FailPrefix + "*test"

Ideally the rest of the library is straight-forward to use based on your experience with NUnit or some other xUnit testing framework.

FAQ

Why are there so many functions when I can just use AssertBlock for everything?
This is something that Jeffrey Snover pointed out when I first asked about unit testing in PowerShell (I had originally suggested cmdlets with an 'assert' verb. He even wrote an AssertBlock function to demonstrate it. My own answer was that the tester's intentions were clearer if aptly-named functions were used. I asked around and found another good answer to come from Brian Marick - that the other functions are there so that the error message was more specific if there was a failing assertion. My advice - make your intentions clear so that other testers don't have to evaluate a script block in their heads.

Why not just provide a wrapper around NUnit?
I almost did this, but opted for a simpler implementation that has no external (at least external to PowerShell) dependencies. A dependency on NUnit might be a barrier to running smoke tests on production servers.

Why the log file?
As a tester, I need a record of tests that are run - whether they pass or fail - for audit and for other test effectiveness studies.

Why the labels?
At higher test levels, it becomes less and less convenient to maintain a "one test method, one test assertion" philosophy that would enable you to trace exactly to the test condition that failed. There are almost always multiple test conditions for each user story at the acceptance test level, for example, and this means many test assertions within a single test script. The label is there to regain the traceability we typically would encode within xUnit as part of automated unit testing.

Why is everything global?
I'm not sure this is the right thing to do, yet. Answering this question is on our to-do list. We felt it was more important to get something out there and try it than to worry about the scope question.

UPDATED WIKI: Home

ageras — Tue, 08 Jan 2008 21:00:56 GMT

PowerShell Scripts for Testing
Microsoft PowerShell is Microsoft's new scripting language and shell first implemented and released in late 2006 in preparation for the Exchange Server 2007 release. Its initial audience seems to be systems administrators, and many of the built-in cmdlets (command-lets) focus on making calls to infrastructure-oriented features such as Windows Management Instrumentation (WMI) and Active Directory (ADSI).

This project envisions an expanded purpose for PowerShell - testing. With simple supporting functions, we believe that PowerShell could be used for application testing and infrastructure testing. In fact, it is well-suited for such purposes given its support for control structures, exception handling, and access to .NET system classes.

All that is missing is a set of xUnit-style assert* functions. Given this library, developers that are used to NUnit or other testing tools that use the assert* metaphor can adapt quickly to writing tests in PowerShell. Some of the testing that we envision this could be used for:

GUI bypass testing - write workflow scenarios that are either data- or keyword-driven that call your business services layer directly
Behaviour Driven Development - write test cases in the 'given-when-then' style of writing functional test scripts
GUI testing - adapt the workflow scenarios from GUI bypass testing to your GUI - PowerShell can access .NET classes from both custom or system namespaces
deployment testing - create a script that verifies your release went as expected - check running processes, services, database metadata, database content, application file versions, configuration file contents, etc.
infrastructure testing - create a script that verifies the hardware, operating system, running services, running processes, etc. match your expectations
... (the list goes on and on)

There is a companion blog for PSExpect here: http://testfirst.spaces.live.com

Project Charter User Guide Data-driven Testing Using Excel