Using Compare Suite to run automated tests

We have recently published videos where RoutineBot does an automatic check of “BSC Designer” software:

What was left behind the scenes is Compare Suite, which was used to compare actual and expected results, and the batch files that were used to automate the scripts.  We’ve gotten some questions asking to share the full testing technology used. In this article you will find a script code that use RoutineBot and Compare Suite for test automation. It’s not the work of the art, but it does what it needs to do.

Preparations for testing

Before we will go into the specific details here are general requirements that we had for an automated testing:

  • It should be 100% automated, without user iteration; all tests should be executed via a single batch file.
  • All test scripts should take into account that something might go wrong; in this case a test script needs to log the situation appropriately and terminate the tested program so that the next test could be started without problems.
  • We need to compare expected results to actual results; if the results don’t match 100%, then we need to be able to find out where the problem was; we will use Compare Suite for that.
  • We will need RoutineBot to test parts of the GUI that we cannot test by just passing appropriate parameters via a command line.

Test scripts organization

Each test script is an independent test module. It is located in a separate folder and can be started independently from other scripts. To make it easier we’ve called the folders  test-001, test-002 etc…

Folder with GUI test scripts

There is a “Run All Tests.bat” file that prints test names and calls test scripts one by one:

echo Test1: KPI Math

call test-002\run_test.cmd

echo Test2: Examples

call test-003\run_test.cmd

As you might have noticed, all CMD files that start the script have the same name – “run_test.cmd,” the difference is only in the name of the folder. It was done intentionally to make the process of test scripts replicating easy. If you need to create a new script, just make a copy of the script folder and change “call test-003\run_test.cmd” to “call test-004\run_test.cmd” and that’s it.

Test script organization

Let’s take the testing process of BSC Designer as an example. BSC Designer supports command line interface, so testing some of the functions will be easy.

We will need to:

  1. Open (via command line interface) in BSC Designer a project file that we want to use for a test
  2. Generate an HTML report for this project file using a command line interface of BSC Designer
  3. Compare the resulting HTML report with an HTML report that was generated and checked manually

Here is the body of the run_test.cmd script, please see the comments to get an idea about why we need certain parts:

rem we want to see just an output that we echo
@echo off
set bsc_designer_folder=C:\Program Files (x86)\BSC Designer\

rem We get in MyPath the current folder of the script
rem and in bsc_project we have the full path to the project that we want to use for a test

set “MyPath=%~dpnx0″ & call set “MyPath=%%MyPath:\%~nx0=%%”
set bsc_project=”%MyPath%\02-kpis-math.bsc”

rem Now we need to delete the results of all test if any
del /Q /S “%MyPath%\cs_reports\*.*” >> log.txt
del /Q /S “%MyPath%\actual_results\*.*” >> log.txt
del /Q /S “%MyPath%\actual_results\bsc-report.files\*.*” >> log.txt

rem If this parameter is set then we are not actually testing, but using this script to create files that will be treated as a correct result
IF “%1″==”prepare_correct_results” GOTO prepare_correct

rem We run BSC Designer software passing project name as a parameter
rem the resulted report will be generated in “actual_results” folder

“%bsc_designer_folder%bsc_designer.exe” %bsc_project% -html -export “%MyPath%\actual_results\bsc-report.html” -overwrite >> log.txt

:compare
rem Now, we run Compare Suite to compare files from “actual_results” folder and “correct_results” folder

“C:\Program Files (x86)\Compare Suite\comparesuite.exe” “%MyPath%\actual_results\bsc-report.html” “%MyPath%\correct_results\bsc-report.html” -r:”%MyPath%\cs_reports\cs_report.htm” -AllowMultipleInstance  >> log.txt
:checks
rem We do a kind of self-test, that allows one to track situations when a CS report was not created for some reason 
find  “STRING_DOES_NOT_EXIST” %MyPath%\cs_reports\cs_report.htm >> log.txt
if %errorlevel% equ 1 goto test0_notfound
echo ….. SELF-TEST FAILED
goto all_done

:test0_notfound
rem echo Test %CurrDirName% ….. SELF-TEST PASSED
goto test0_done
:test0_done
find  “File comparison Report” %MyPath%\cs_reports\cs_report.htm >> log.txt
if %errorlevel% equ 1 goto test1_notfound
goto test1_done
:test1_notfound
echo ….. SELF-TEST FAILED
goto all_done

:test1_done
rem So, if a self-test was passed, it means that a cs_report file exists and contains some data about the difference between expected and actual results
rem Now, we will try to find the string in the report that corresponds to the Similarity, % record.


rem if we find there 100, then two files are equal and the test was passed.

find  “<td class=””val””>100</td>” %MyPath%\cs_reports\cs_report.htm >> log.txt
if %errorlevel% equ 1 goto test2_notfound
echo ….. +
goto test2_done
:test2_notfound
echo ….. FAILED
goto test2_done

:test2_done
goto all_done

:prepare_correct

rem We need this part to be able to prepare correct results in the beginning
echo Preparing correct results
del /Q /S “%MyPath%\correct_results\*.*” >> log.txt
del /Q /S “%MyPath%\correct_results\bsc-report.files\*.*” >> log.txt
“%bsc_designer_folder%bsc_designer.exe” %bsc_project% -html -export “%MyPath%\correct_results\bsc-report.html” -overwrite >> log.txt
echo Finished preparing
pause

:all_done

If the test was passed correctly, then you’ll see in the batch output “….. +” if not you’ll see “….. FAILED”

Running test with RoutineBot

Now, let me show you how to execute the test with RoutineBot. Instead of running BSC Designer in a command line mode we will need to start a RoutineBot script:

set script_name=script.rbp

set “MyPath=%~dpnx0″ & call set “MyPath=%%MyPath:\%~nx0=%%”

“C:\Program Files (x86)\RoutineBot\Interpreter.exe” “%MyPath%\%script_name%”  -Warnings “ON” -LogFile

“C:\Tmp\bsc_designer_tests\_log.txt”
xcopy “P:\Project\Projects\BSC_Designer\tests\z_rb_logs\*.*” “%MyPath%\actual_results\*.*” /e /i /h >> log.txt

As you can see, RoutineBot script needs to write its log into C:\Tmp\bsc_designer_tests\_log.txt. Then, we follow the same algorithm, we have the log for the correct results and we have the log for actual results. We pass these two files to Compare Suite (here are the details about Compare Suite’s command line parameters).

Running all test

Each test was written as a separate module. You can always run a selected test script or run all of them using run_test.cmd file.

Hopefully your test log will look something like this:

The results of batch GUI script execution

If not, you can always check cs_reports folder where we saved the results of the Compare Suite comparison to find out what worked wrong.

Feel free to share your thoughts and experience in the comments. 

Posted in Tests
Compare Suite Video Introduction
Download free 30-day trial version.

Where Compare Suite is used:

Compare Suite Features:

Compare FileCompare documents to find differences regardless of the format: MS Word, MS Excel, MS PowerPoint, PDF.

Compare FoldersCompare two local or remote folders to find and synchronize any changes made.

Compare by KeywordsUse the keywords comparison to analyze non-related documents created from different formats.

Compare ArchiveArchive comparison: compare Rar and Zip archive files without unpacking / unzipping.

COmpare Web-PageCompare web pages as code with HTML tags or as text as seen in a web browser.

Compare ImagesCompare two images of various formats to find all possible differences, even slight ones.

Compare MultimediaMultimedia comparison. Compare information from multimedia and graphic formats.

Ignore WordsIgnore words. Compare Suite can ignore certain keywords or strings during comparison.

Integrate with DMSDMS Integration. Easily integrate Compare Suite into 3rd party document management systems.

Highlight SyntaxSyntax highlighting for .pas, .asm., cpp, PHP, XML, bat, css, dpk, xsl files supported.

Compare FTPCompare files located on FTP as easily as on hard drives or local area networks.

Compare Command LineUse command line to automate comparisons and integrate it with 3rd party tools.

Compare UnicodeUNICODE is fully supported: open and compare documents in any language or encoding.


Review compared documents:

Compare ReportCreate a fully customizable comparison report on two files or folders.

Compare CommentsWrite your own comments on changes you or your colleagues make, improving collaboration.

Audit DocumentUse document audit to accept or decline changes made in plain text files.

What is Compare Suite?

  • It is a file compare tool
  • In office Compare Suite helps to compare word files, Excel spreadsheets, web-pages
  • Software developers and web-masters use Compare Suite to do version compare, approve changes in documents or compare certain file types like XML, PHP, C-Sharp
  • Office managers use Compare Suite to diff PDF files, compare invoice files
  • Compare Suite also make it much easier synchronization of directories