| 1 | Automated Testing System {#dev_guides__tests} |
| 2 | ====================================== |
| 3 | |
| 4 | @section testmanual_1 Introduction |
| 5 | |
| 6 | This document provides overview and practical guidelines for work with OCCT automatic testing system. |
| 7 | Reading this section *Introduction* should be sufficient for OCCT developers to use the test system |
| 8 | to control non-regression of the modifications they implement in OCCT. Other sections provide |
| 9 | more in-depth description of the test system, required for modifying the tests and adding new test cases. |
| 10 | |
| 11 | @subsection testmanual_1_1 Basic Information |
| 12 | |
| 13 | OCCT automatic testing system is organized around DRAW Test Harness [1], |
| 14 | a console application based on Tcl (a scripting language) interpreter extended by OCCT-related commands. |
| 15 | Standard OCCT tests are included with OCCT sources and are located in subdirectory *tests* |
| 16 | of the OCCT root folder. Other test folders can be included in the scope of the test system, |
| 17 | e.g. for testing applications based on OCCT. |
| 18 | Logically the tests are organized in three levels: |
| 19 | |
| 20 | * Group: group of related test grids, usually relating to some part of OCCT functionality (e.g. blend) |
| 21 | * Grid: set of test cases within a group, usually aimed at testing some particular aspect or mode of execution of the relevant functionality (e.g. buildevol) |
| 22 | * Test case: script implementing individual test (e.g. K4) |
| 23 | |
| 24 | Some tests involve data files (typically CAD models) which are located separately |
| 25 | and are not included with OCCT code. The archive with publicly available |
| 26 | test data files should be downloaded and installed independently on OCCT code from dev.opencascade.org. |
| 27 | |
| 28 | @subsection testmanual_1_2 Intended Use of Automatic Tests |
| 29 | |
| 30 | Each modification made in OCCT code must be checked for non-regression |
| 31 | by running the whole set of tests. The developer who does the modification |
| 32 | is responsible for running and ensuring non-regression on the tests that are available to him. |
| 33 | Note that many tests are based on data files that are confidential and thus available only at OPEN CASCADE. |
| 34 | Thus official certification testing of the changes before integration to master branch |
| 35 | of official OCCT Git repository (and finally to the official release) is performed by OPEN CASCADE in any case. |
| 36 | |
| 37 | Each new non-trivial modification (improvement, bug fix, new feature) in OCCT |
| 38 | should be accompanied by a relevant test case suitable for verifying that modification. |
| 39 | This test case is to be added by developer who provides the modification. |
| 40 | If a modification affects result of some test case(s), |
| 41 | either the modification should be corrected (if it causes regression) |
| 42 | or affected test cases should be updated to account for the modification. |
| 43 | |
| 44 | The modifications made in the OCCT code and related test scripts |
| 45 | should be included in the same integration to master branch. |
| 46 | |
| 47 | @subsection testmanual_1_3 Quick Start |
| 48 | |
| 49 | @subsubsection testmanual_1_3_1 Setup |
| 50 | |
| 51 | Before running tests, make sure to define environment variable CSF_TestDataPath |
| 52 | pointing to the directory containing test data files. |
| 53 | (The publicly available data files can be downloaded |
| 54 | from http://dev.opencascade.org separately from OCCT code.) |
| 55 | The recommended way for that is adding a file *DrawInitAppli* |
| 56 | in the directory which is current at the moment of starting DRAWEXE (normally it is $CASROOT). |
| 57 | This file is evaluated automatically at the DRAW start. Example: |
| 58 | |
| 59 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 60 | set env(CSF_TestDataPath) d:/occt/tests_data |
| 61 | return ;# this is to avoid an echo of the last command above in cout |
| 62 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 63 | |
| 64 | All tests are run from DRAW command prompt, thus first run draw.tcl or draw.sh to start DRAW. |
| 65 | |
| 66 | @subsubsection testmanual_1_3_2 Running Tests |
| 67 | |
| 68 | To run all tests, type command *testgrid* followed by path |
| 69 | to the new directory where results will be saved. |
| 70 | It is recommended that this directory should be new or empty; |
| 71 | use option –overwrite to allow writing logs in existing non-empty directory. |
| 72 | |
| 73 | Example: |
| 74 | |
| 75 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 76 | Draw[]> testgrid d:/occt/results-2012-02-27 |
| 77 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 78 | |
| 79 | If empty string is given as log directory name, the name will be generated automatically |
| 80 | using current date and time, prefixed by *results_*. Example: |
| 81 | |
| 82 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 83 | Draw[]> testgrid {} |
| 84 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 85 | |
| 86 | For running only some group or a grid of tests, |
| 87 | give additional arguments indicating group and (if needed) grid. Example: |
| 88 | |
| 89 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 90 | Draw[]> testgrid d:/occt/results-2012-02-28 blend simple |
| 91 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 92 | |
| 93 | As the tests progress, the result of each test case is reported. |
| 94 | At the end of the log summary of test cases is output, |
| 95 | including list of detected regressions and improvements, if any. Example: |
| 96 | |
| 97 | |
| 98 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 99 | Tests summary |
| 100 | |
| 101 | CASE 3rdparty export A1: OK |
| 102 | ... |
| 103 | CASE pipe standard B1: BAD (known problem) |
| 104 | CASE pipe standard C1: OK |
| 105 | No regressions |
| 106 | Total cases: 208 BAD, 31 SKIPPED, 3 IMPROVEMENT, 1791 OK |
| 107 | Elapsed time: 1 Hours 14 Minutes 33.7384512019 Seconds |
| 108 | Detailed logs are saved in D:/occt/results_2012-06-04T0919 |
| 109 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 110 | |
| 111 | The tests are considered as non-regressive if only OK, BAD (i.e. known problem), |
| 112 | and SKIPPED (i.e. not executed, e.g. because of lack of data file) statuses are reported. |
| 113 | See <a href="#testmanual_3_4">Grid’s *cases.list* file</a> chapter for details. |
| 114 | |
| 115 | The detailed logs of running tests are saved in the specified directory and its sub-directories. |
| 116 | Cumulative HTML report summary.html provides links to reports on each test case. |
| 117 | An additional report TESTS-summary.xml is output in JUnit-style XML format |
| 118 | that can be used for integration with Jenkins or other continuous integration system. |
| 119 | Type *help testgrid* in DRAW prompt to get help on additional options supported by testgrid command. |
| 120 | |
| 121 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 122 | Draw[3]> help testgrid |
| 123 | testgrid: Run all tests, or specified group, or one grid |
| 124 | Use: testgrid logdir [group [grid]] [options...] |
| 125 | Allowed options are: |
| 126 | -verbose {0-2}: verbose level, 0 by default, can be set to 1 or 2 |
| 127 | -parallel N: run in parallel with up to N processes (default 0) |
| 128 | -refresh N: save summary logs every N seconds (default 60) |
| 129 | -overwrite: force writing logs in existing non-empty directory |
| 130 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 131 | |
| 132 | @subsubsection testmanual_1_3_3 Running Single Test |
| 133 | |
| 134 | To run single test, type command *test*’ followed by names of group, grid, and test case. |
| 135 | |
| 136 | Example: |
| 137 | |
| 138 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 139 | Draw[1]> test blend simple A1 |
| 140 | CASE blend simple A1: OK |
| 141 | Draw[2]> |
| 142 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 143 | |
| 144 | Note that normally intermediate output of the script is not shown. |
| 145 | To see intermediate commands and their output, type command *decho on* |
| 146 | before running the test case. (Type ‘decho off’ to disable echoing when not needed.) |
| 147 | The detailed log of the test can also be obtained after the test execution by command *dlog get*. |
| 148 | |
| 149 | @section testmanual_2 Organization of Test Scripts |
| 150 | |
| 151 | @subsection testmanual_2_1 General Layout |
| 152 | |
| 153 | Standard OCCT tests are located in subdirectory tests of the OCCT root folder ($CASROOT). |
| 154 | Additional test folders can be added to the test system |
| 155 | by defining environment variable CSF_TestScriptsPath. |
| 156 | This should be list of paths separated by semicolons (*;*) on Windows |
| 157 | or colons (*:*) on Linux or Mac. Upon DRAW launch, |
| 158 | path to tests sub-folder of OCCT is added at the end of this variable automatically. |
| 159 | Each test folder is expected to contain: |
| 160 | |
| 161 | * Optional file parse.rules defining patterns for interpretation of test results, common for all groups in this folder |
| 162 | * One or several test group directories. |
| 163 | |
| 164 | Each group directory contains: |
| 165 | |
| 166 | * File grids.list that identifies this test group and defines list of test grids in it. |
| 167 | * Test grids (sub-directories), each containing set of scripts for test cases, and optional files cases.list, parse.rules, begin, and end. |
| 168 | * Optional sub-directory data |
| 169 | * Optional file parse.rules |
| 170 | * Optional files begin and end |
| 171 | |
| 172 | By convention, names of test groups, grids, and cases should contain no spaces and be lowercase. |
| 173 | Names begin, end, data, parse.rules, grids.list, cases.list are reserved. |
| 174 | General layout of test scripts is shown on Figure 1. |
| 175 | |
| 176 | ![](/devs_guide/tests/images/tests_image001.png "") |
| 177 | |
| 178 | Figure 1. Layout of tests folder |
| 179 | |
| 180 | @subsection testmanual_2_2 Test Groups |
| 181 | |
| 182 | @subsubsection testmanual_2_2_1 Group Names |
| 183 | |
| 184 | Test folder usually contains several directories representing test groups (Group 1, Group N). |
| 185 | Each directory contains test grids for certain OCCT functionality. |
| 186 | The name of directory corresponds to this functionality. |
| 187 | Example: |
| 188 | |
| 189 | @verbatim |
| 190 | caf |
| 191 | mesh |
| 192 | offset |
| 193 | @endverbatim |
| 194 | |
| 195 | @subsubsection testmanual_2_2_2 Group's *grids.list* File |
| 196 | |
| 197 | The test group must contain file *grids.list* file |
| 198 | which defines ordered list of grids in this group in the following format: |
| 199 | |
| 200 | ~~~~~~~~~~~~~~~~~ |
| 201 | 001 gridname1 |
| 202 | 002 gridname2 |
| 203 | ... |
| 204 | NNN gridnameN |
| 205 | ~~~~~~~~~~~~~~~~~ |
| 206 | |
| 207 | Example: |
| 208 | |
| 209 | ~~~~~~~~~~~~~~~~~ |
| 210 | 001 basic |
| 211 | 002 advanced |
| 212 | ~~~~~~~~~~~~~~~~~ |
| 213 | |
| 214 | @subsubsection testmanual_2_2_3 Group's *begin* File |
| 215 | |
| 216 | The file *begin* is a Tcl script. It is executed before every test in current group. |
| 217 | Usually it loads necessary Draw commands, sets common parameters and defines |
| 218 | additional Tcl functions used in test scripts. |
| 219 | Example: |
| 220 | |
| 221 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 222 | pload TOPTEST ;# load topological command |
| 223 | set cpulimit 300 ;# set maximum time allowed for script execution |
| 224 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 225 | |
| 226 | @subsubsection testmanual_2_2_4 Group's *end* File |
| 227 | |
| 228 | The file end is a TCL script. It is executed after every test in current group. |
| 229 | Usually it checks the results of script work, makes a snap-shot |
| 230 | of the viewer and writes *TEST COMPLETED* to the output. |
| 231 | Note: *TEST COMPLETED* string should be presented in output |
| 232 | in order to signal that test is finished without crash. |
| 233 | See <a href="#testmanual_3">Creation And Modification Of Tests</a> chapter for more information. |
| 234 | Example: |
| 235 | |
| 236 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 237 | if { [isdraw result] } { |
| 238 | checkshape result |
| 239 | } else { |
| 240 | puts *Error: The result shape can not be built* |
| 241 | } |
| 242 | puts *TEST COMPLETED* |
| 243 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 244 | |
| 245 | @subsubsection testmanual_2_2_5 Group’s *parse.rules* File |
| 246 | |
| 247 | The test group may contain *parse.rules* file. |
| 248 | This file defines patterns used for analysis of the test execution log |
| 249 | and deciding the status of the test run. |
| 250 | Each line in the file should specify a status (single word), |
| 251 | followed by a regular expression delimited by slashes (*/*) |
| 252 | that will be matched against lines in the test output log to check if it corresponds to this status. |
| 253 | The regular expressions support subset of the Perl re syntax. |
| 254 | The rest of the line can contain a comment message |
| 255 | which will be added to the test report when this status is detected. |
| 256 | Example: |
| 257 | |
| 258 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 259 | FAILED /\b[Ee]xception\b/ exception |
| 260 | FAILED /\bError\b/ error |
| 261 | SKIPPED /Cannot open file for reading/ data file is missing |
| 262 | SKIPPED /Could not read file .*, abandon/ data file is missing |
| 263 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 264 | |
| 265 | Lines starting with a *#* character and blank lines are ignored to allow comments and spacing. |
| 266 | See <a href="#testmanual_2_3_4">Interpretation of test results</a> chapter for details. |
| 267 | |
| 268 | If a line matches several rules, the first one applies. |
| 269 | Rules defined in the grid are checked first, then rules in group, |
| 270 | then rules in the test root directory. This allows defining some rules on the grid level |
| 271 | with status IGNORE to ignore messages that would otherwise be treated as errors due to the group level rules. |
| 272 | Example: |
| 273 | |
| 274 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 275 | FAILED /\bFaulty\b/ bad shape |
| 276 | IGNORE /^Error [23]d = [\d.-]+/ debug output of blend command |
| 277 | IGNORE /^Tcl Exception: tolerance ang : [\d.-]+/ blend failure |
| 278 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 279 | |
| 280 | @subsection testmanual_2_3 Test Grids |
| 281 | |
| 282 | @subsubsection testmanual_2_3_1 Grid Names |
| 283 | |
| 284 | Group folder can have several sub-directories (Grid 1… Grid N) defining test grids. |
| 285 | Each test grid directory contains a set of related test cases. |
| 286 | The name of directory should correspond to its contents. |
| 287 | |
| 288 | Example: |
| 289 | |
| 290 | caf |
| 291 | basic |
| 292 | bugs |
| 293 | presentation |
| 294 | |
| 295 | Where **caf** is the name of test group and *basic*, *bugs*, *presentation*, etc are the names of grids. |
| 296 | |
| 297 | @subsubsection testmanual_2_3_2 Grid’s *begin* File |
| 298 | |
| 299 | The file *begin* is a TCL script. It is executed before every test in current grid. |
| 300 | Usually it sets variables specific for the current grid. |
| 301 | Example: |
| 302 | |
| 303 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 304 | set command bopfuse ;# command tested in this grid |
| 305 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 306 | |
| 307 | @subsubsection testmanual_2_3_3 Grid’s *end* File |
| 308 | |
| 309 | The file *end* is a TCL script. It is executed after every test in current grid. |
| 310 | Usually it executes specific sequence of commands common for all tests in the grid. |
| 311 | Example: |
| 312 | |
| 313 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 314 | vdump $logdir/${casename}.gif ;# makes a snap-shot of AIS viewer |
| 315 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 316 | |
| 317 | @subsubsection testmanual_2_3_4 Grid’s *cases.list* File |
| 318 | |
| 319 | The grid directory can contain an optional file cases.list |
| 320 | defining alternative location of the test cases. |
| 321 | This file should contain singe line defining the relative path to collection of test cases. |
| 322 | |
| 323 | Example: |
| 324 | |
| 325 | ../data/simple |
| 326 | |
| 327 | This option is used for creation of several grids of tests with the same data files |
| 328 | and operations but performed with differing parameters. |
| 329 | The common scripts are usually located place in common |
| 330 | subdirectory of the test group (data/simple as in example). |
| 331 | If cases.list file exists then grid directory should not contain any test cases. |
| 332 | The specific parameters and pre- and post-processing commands |
| 333 | for the tests execution in this grid should be defined in the begin and end files. |
| 334 | |
| 335 | @subsection testmanual_2_4 Test Cases |
| 336 | |
| 337 | The test case is TCL script which performs some operations using DRAW commands |
| 338 | and produces meaningful messages that can be used to check the result for being valid. |
| 339 | Example: |
| 340 | |
| 341 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 342 | pcylinder c1 10 20 ;# create first cylinder |
| 343 | pcylinder c2 5 20 ;# create second cylinder |
| 344 | ttranslate c2 5 0 10 ;# translate second cylinder to x,y,z |
| 345 | bsection result c1 c2 ;# create a section of two cylinders |
| 346 | checksection result ;# will output error message if result is bad |
| 347 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 348 | |
| 349 | The test case can have any name (except reserved names begin, end, data, cases.list, parse.rules). |
| 350 | For systematic grids it is usually a capital English letter followed by a number. |
| 351 | |
| 352 | Example: |
| 353 | |
| 354 | @verbatim |
| 355 | A1 |
| 356 | A2 |
| 357 | B1 |
| 358 | B2 |
| 359 | @endverbatim |
| 360 | |
| 361 | Such naming facilitates compact representation of results |
| 362 | of tests execution in tabular format within HTML reports. |
| 363 | |
| 364 | @subsection testmanual_2_5 Directory *data* |
| 365 | |
| 366 | The test group may contain subdirectory data. |
| 367 | Usually it contains data files used in tests (BREP, IGES, STEP, etc.) |
| 368 | and / or test scripts shared by different test grids |
| 369 | (in subdirectories, see <a href="#testmanual_2_3_4">Grid’s *cases.list* file</a> chapter). |
| 370 | |
| 371 | @section testmanual_3 Creation And Modification Of Tests |
| 372 | |
| 373 | This section describes how to add new tests and update existing ones. |
| 374 | |
| 375 | @subsection testmanual_3_1 Choosing Group, Grid, and Test Case Name |
| 376 | |
| 377 | The new tests are usually added in context of processing some bugs. |
| 378 | Such tests in general should be added to group bugs, in the grid |
| 379 | corresponding to the affected OCCT functionality. |
| 380 | New grids can be added as necessary to contain tests on functionality not yet covered by existing test grids. |
| 381 | The test case name in the bugs group should be prefixed by ID |
| 382 | of the corresponding issue in Mantis (without leading zeroes). |
| 383 | It is recommended to add a suffix providing a hint on the situation being tested. |
| 384 | If more than one test is added for a bug, they should be distinguished by suffixes; |
| 385 | either meaningful or just ordinal numbers. |
| 386 | |
| 387 | Example: |
| 388 | |
| 389 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 390 | 12345_coaxial |
| 391 | 12345_orthogonal_1 |
| 392 | 12345_orthogonal_2 |
| 393 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 394 | |
| 395 | In the case if new test corresponds to functionality for which |
| 396 | specific group of tests exists (e.g. group mesh for BRepMesh issues), |
| 397 | this test can be added (or moved later by OCC team) to this group. |
| 398 | |
| 399 | @subsection testmanual_3_2 Adding Data Files Required for a Test |
| 400 | |
| 401 | It is advisable that tests scripts should be made self-contained whenever possible, |
| 402 | so as to be usable in environments where data files are not available. |
| 403 | For that simple geometric objects and shapes can be created using DRAW commands in the test script itself. |
| 404 | If test requires some data file, it should be put to subdirectory data of the test grid. |
| 405 | Note that when test is integrated to master branch, |
| 406 | OCC team can move data file to data files repository, |
| 407 | so as to keep OCCT sources repository clean from big data files. |
| 408 | When preparing a test script, try to minimize size of involved data model. |
| 409 | For instance, if problem detected on a big shape can be reproduced on a single face |
| 410 | extracted from that shape, use only this face in the test. |
| 411 | |
| 412 | @subsection testmanual_3_3 Implementation of the Script |
| 413 | |
| 414 | Test should run commands necessary to perform the operations being tested, |
| 415 | in a clean DRAW session. This includes loading necessary functionality by *pload* command, |
| 416 | if this is not done by *begin* script. The messages produced by commands in standard output |
| 417 | should include identifiable messages on the discovered problems if any. |
| 418 | Usually the script represents a set of commands that a person would run interactively |
| 419 | to perform the operation and see its results, with additional comments to explain what happens. |
| 420 | Example: |
| 421 | |
| 422 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 423 | # Simple test of fusing box and sphere |
| 424 | box b 10 10 10 |
| 425 | sphere s 5 |
| 426 | bfuse result b s |
| 427 | checkshape result |
| 428 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 429 | |
| 430 | Make sure that file parse.rules in the grid or group directory contains |
| 431 | regular expression to catch possible messages indicating failure of the test. |
| 432 | For instance, for catching errors reported by *checkshape* command |
| 433 | relevant grids define a rule to recognize its report by the word *Faulty*: FAILED /\bFaulty\b/ bad shape |
| 434 | For the messages generated in the script the most natural way is to use the word *Error* in the message. |
| 435 | Example: |
| 436 | |
| 437 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 438 | set expected_length 11 |
| 439 | if { [expr $actual_length - $expected_length] > 0.001 } { |
| 440 | puts *Error: The length of the edge should be $expected_length* |
| 441 | } |
| 442 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 443 | |
| 444 | At the end, the test script should output *TEST COMPLETED* string |
| 445 | to mark successful completion of the script. |
| 446 | This is often done by the end script in the grid. |
| 447 | When test script requires data file, use Tcl procedure *locate_data_file* |
| 448 | to get path to the data file, rather than explicit path. |
| 449 | This will allow easy move of the data file from OCCT repository |
| 450 | to the data files repository without a need to update test script. |
| 451 | Example: |
| 452 | |
| 453 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 454 | stepread [locate_data_file CAROSKI_COUPELLE.step] a * |
| 455 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 456 | |
| 457 | When test needs to produce some snapshots or other artifacts, |
| 458 | use Tcl variable logdir as location where such files should be put. |
| 459 | Command *testgrid* sets this variable to the subdirectory of the results folder |
| 460 | corresponding to the grid. Command *test* sets it to $CASROOT/tmp unless it is already defined. |
| 461 | Use Tcl variable casename to prefix all the files produced by the test. |
| 462 | This variable is set to the name of the test case. |
| 463 | Example: |
| 464 | |
| 465 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 466 | xwd $logdir/${casename}.gif |
| 467 | vdisplay result; vfit |
| 468 | vdump $logdir/${casename}-axo.gif |
| 469 | vfront; vfit |
| 470 | vdump $logdir/${casename}-front.gif |
| 471 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 472 | |
| 473 | could produce: |
| 474 | |
| 475 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 476 | A1.gif |
| 477 | A1-axo.gif |
| 478 | A1-front.gif |
| 479 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 480 | |
| 481 | @subsection testmanual_3_4 Interpretation of Test Results |
| 482 | |
| 483 | The result of the test is evaluated by checking its output against patterns |
| 484 | defined in the files parse.rules of the grid and group. |
| 485 | The OCCT test system recognizes five statuses of the test execution: |
| 486 | |
| 487 | * SKIPPED: reported if line matching SKIPPED pattern is found (prior to any FAILED pattern). This indicates that the test cannot be run in the current environment; most typical case is absence of the required data file. |
| 488 | * FAILED: reported if some line matching pattern with status FAILED is found (unless it is masked by preceding IGNORE pattern or a TODO statement, see below), or if message TEST COMPLETED is not found at the end. This indicates that test produces bad or unexpected result, and usually highlights a regression. |
| 489 | * BAD: reported if test script output contains one or several TODO statements and corresponding number of matching lines in the log. This indicates a known problem (see 3.5). The lines matching TODO statements are not checked against other patterns and thus will not cause a FAILED status. |
| 490 | * IMPROVEMENT: reported if test script output contains TODO statement for which no corresponding line is found. This is possible indication of improvement (known problem disappeared). |
| 491 | * OK: If none of the above statuses have been assigned. This means test passed without problems. |
| 492 | |
| 493 | Other statuses can be specified in the parse.rules files, these will be classified as FAILED. |
| 494 | Before integration of the change to OCCT repository, all tests should return either OK or BAD status. |
| 495 | The new test created for unsolved problem should return BAD. |
| 496 | The new test created for a fixed problem should return FAILED without the fix, and OK with the fix. |
| 497 | |
| 498 | @subsection testmanual_3_5 Marking BAD Cases |
| 499 | |
| 500 | If the test produces invalid result at a certain moment then the corresponding bug |
| 501 | should be created in the OCCT issue tracker [3], and the problem should be marked as TODO in the test script. |
| 502 | The following statement should be added to such test script: |
| 503 | |
| 504 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 505 | puts *TODO BugNumber ListOfPlatforms: RegularExpression* |
| 506 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 507 | |
| 508 | where: |
| 509 | |
| 510 | * BugNumber is an ID of the bug in the tracker. For example: #12345 |
| 511 | * ListOfPlatform is a list of platforms at which the bug is reproduced (e.g. Mandriva2008, Windows or All). |
| 512 | |
| 513 | *Note: the platform name is custom for the OCCT test system;* |
| 514 | *it can be consulted as value of environment variable os_type defined in DRAW.* |
| 515 | |
| 516 | Example: |
| 517 | |
| 518 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 519 | Draw[2]> puts $env(os_type) |
| 520 | windows |
| 521 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 522 | |
| 523 | * RegularExpression is a regular expression which should be matched against the line indicating the problem in the script output. |
| 524 | Example: |
| 525 | |
| 526 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 527 | puts *TODO #22622 Mandriva2008: Abort .* an exception was raised* |
| 528 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 529 | |
| 530 | Parser checks the output of the test and if an output line matches |
| 531 | the RegularExpression then it will be assigned a BAD status instead of FAILED. |
| 532 | For each output line matching to an error expression a separate TODO line |
| 533 | must be added to mark the test as BAD. |
| 534 | If not all the TODO statements are found in the test log, |
| 535 | the test will be considered as possible improvement. |
| 536 | To mark the test as BAD for an incomplete case |
| 537 | (when final TEST COMPLETE message is missing) |
| 538 | the expression *TEST INCOMPLETE* should be used instead of regular expression. |
| 539 | |
| 540 | Example: |
| 541 | |
| 542 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 543 | puts *TODO OCC22817 All: exception.+There are no suitable edges* |
| 544 | puts *TODO OCC22817 All: \\*\\* Exception \\*\\** |
| 545 | puts *TODO OCC22817 All: TEST INCOMPLETE* |
| 546 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 547 | |
| 548 | @section testmanual_4 Extended Use |
| 549 | |
| 550 | @subsection testmanual_4_1 Running Tests on Older Versions of OCCT |
| 551 | |
| 552 | Sometimes it might be necessary to run tests on previous versions of OCCT (up to to 6.5.3) |
| 553 | that do not include this test system. This can be done by adding DRAW configuration file DrawAppliInit |
| 554 | in the directory which is current by the moment of DRAW startup, |
| 555 | to load test commands and define necessary environment. Example |
| 556 | (assume that d:/occt contains up-to-date version of OCCT sources |
| 557 | with tests, and test data archive is unpacked to d:/test-data): |
| 558 | |
| 559 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 560 | set env(CASROOT) d:/occt |
| 561 | set env(CSF_TestScriptsPath) $env(CASROOT)/tests |
| 562 | source $env(CASROOT)/src/DrawResources/TestCommands.tcl |
| 563 | set env(CSF_TestDataPath) d:/test-data |
| 564 | return |
| 565 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 566 | |
| 567 | Note that on older versions of OCCT the tests are run in compatibility mode |
| 568 | and not all output of the test command can be captured; |
| 569 | this can lead to absence of some error messages (can be reported as improvement). |
| 570 | |
| 571 | @subsection testmanual_4_2 Adding Custom Tests |
| 572 | |
| 573 | You can extend the test system by adding your own tests. |
| 574 | For that it is necessary to add paths to the directory where these tests are located, |
| 575 | and additional data directory(ies), to the environment variables CSF_TestScriptsPath and CSF_TestDataPath. |
| 576 | The recommended way for doing this is using DRAW configuration file DrawAppliInit |
| 577 | located in the directory which is current by the moment of DRAW startup. |
| 578 | |
| 579 | Use Tcl command *_path_separator* to insert platform-dependent separator to the path list. |
| 580 | Example: |
| 581 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
| 582 | set env(CSF_TestScriptsPath) \ |
| 583 | $env(TestScriptsPath)[_path_separator]d:/MyOCCTProject/tests |
| 584 | set env(CSF_TestDataPath) \ |
| 585 | d:/occt/test-data[_path_separator]d:/MyOCCTProject/tests |
| 586 | return ;# this is to avoid an echo of the last command above in cout |
| 587 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 588 | |
| 589 | @section testmanual_5 References |
| 590 | |
| 591 | -# DRAW Test Harness User’s Guide |
| 592 | -# Perl regular expressions, http://perldoc.perl.org/perlre.html |
| 593 | -# OCCT MantisBT issue tracker, http://tracker.dev.opencascade.org |