72b7576f |
1 | Automated Testing System {#dev_guides__tests} |
2 | ====================================== |
3 | |
4 | @section testmanual_1 Introduction |
5 | |
6 | This document provides overview and practical guidelines for work with OCCT automatic testing system. |
7 | Reading this section *Introduction* should be sufficient for OCCT developers to use the test system |
8 | to control non-regression of the modifications they implement in OCCT. Other sections provide |
9 | more in-depth description of the test system, required for modifying the tests and adding new test cases. |
10 | |
11 | @subsection testmanual_1_1 Basic Information |
12 | |
13 | OCCT automatic testing system is organized around DRAW Test Harness [1], |
14 | a console application based on Tcl (a scripting language) interpreter extended by OCCT-related commands. |
15 | Standard OCCT tests are included with OCCT sources and are located in subdirectory *tests* |
16 | of the OCCT root folder. Other test folders can be included in the scope of the test system, |
17 | e.g. for testing applications based on OCCT. |
18 | Logically the tests are organized in three levels: |
19 | |
20 | * Group: group of related test grids, usually relating to some part of OCCT functionality (e.g. blend) |
21 | * Grid: set of test cases within a group, usually aimed at testing some particular aspect or mode of execution of the relevant functionality (e.g. buildevol) |
22 | * Test case: script implementing individual test (e.g. K4) |
23 | |
24 | Some tests involve data files (typically CAD models) which are located separately |
25 | and are not included with OCCT code. The archive with publicly available |
26 | test data files should be downloaded and installed independently on OCCT code from dev.opencascade.org. |
27 | |
28 | @subsection testmanual_1_2 Intended Use of Automatic Tests |
29 | |
30 | Each modification made in OCCT code must be checked for non-regression |
31 | by running the whole set of tests. The developer who does the modification |
32 | is responsible for running and ensuring non-regression on the tests that are available to him. |
33 | Note that many tests are based on data files that are confidential and thus available only at OPEN CASCADE. |
34 | Thus official certification testing of the changes before integration to master branch |
35 | of official OCCT Git repository (and finally to the official release) is performed by OPEN CASCADE in any case. |
36 | |
37 | Each new non-trivial modification (improvement, bug fix, new feature) in OCCT |
38 | should be accompanied by a relevant test case suitable for verifying that modification. |
39 | This test case is to be added by developer who provides the modification. |
40 | If a modification affects result of some test case(s), |
41 | either the modification should be corrected (if it causes regression) |
42 | or affected test cases should be updated to account for the modification. |
43 | |
44 | The modifications made in the OCCT code and related test scripts |
45 | should be included in the same integration to master branch. |
46 | |
47 | @subsection testmanual_1_3 Quick Start |
48 | |
49 | @subsubsection testmanual_1_3_1 Setup |
50 | |
51 | Before running tests, make sure to define environment variable CSF_TestDataPath |
52 | pointing to the directory containing test data files. |
53 | (The publicly available data files can be downloaded |
54 | from http://dev.opencascade.org separately from OCCT code.) |
55 | The recommended way for that is adding a file *DrawInitAppli* |
56 | in the directory which is current at the moment of starting DRAWEXE (normally it is $CASROOT). |
57 | This file is evaluated automatically at the DRAW start. Example: |
58 | |
59 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
60 | set env(CSF_TestDataPath) d:/occt/tests_data |
61 | return ;# this is to avoid an echo of the last command above in cout |
62 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
63 | |
64 | All tests are run from DRAW command prompt, thus first run draw.tcl or draw.sh to start DRAW. |
65 | |
66 | @subsubsection testmanual_1_3_2 Running Tests |
67 | |
68 | To run all tests, type command *testgrid* followed by path |
69 | to the new directory where results will be saved. |
70 | It is recommended that this directory should be new or empty; |
71 | use option –overwrite to allow writing logs in existing non-empty directory. |
72 | |
73 | Example: |
74 | |
75 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
76 | Draw[]> testgrid d:/occt/results-2012-02-27 |
77 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
78 | |
79 | If empty string is given as log directory name, the name will be generated automatically |
80 | using current date and time, prefixed by *results_*. Example: |
81 | |
82 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
83 | Draw[]> testgrid {} |
84 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
85 | |
86 | For running only some group or a grid of tests, |
87 | give additional arguments indicating group and (if needed) grid. Example: |
88 | |
89 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
90 | Draw[]> testgrid d:/occt/results-2012-02-28 blend simple |
91 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
92 | |
93 | As the tests progress, the result of each test case is reported. |
94 | At the end of the log summary of test cases is output, |
95 | including list of detected regressions and improvements, if any. Example: |
96 | |
97 | |
98 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
99 | Tests summary |
100 | |
101 | CASE 3rdparty export A1: OK |
102 | ... |
103 | CASE pipe standard B1: BAD (known problem) |
104 | CASE pipe standard C1: OK |
105 | No regressions |
106 | Total cases: 208 BAD, 31 SKIPPED, 3 IMPROVEMENT, 1791 OK |
107 | Elapsed time: 1 Hours 14 Minutes 33.7384512019 Seconds |
108 | Detailed logs are saved in D:/occt/results_2012-06-04T0919 |
109 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
110 | |
111 | The tests are considered as non-regressive if only OK, BAD (i.e. known problem), |
112 | and SKIPPED (i.e. not executed, e.g. because of lack of data file) statuses are reported. |
113 | See <a href="#testmanual_3_4">Grid’s *cases.list* file</a> chapter for details. |
114 | |
115 | The detailed logs of running tests are saved in the specified directory and its sub-directories. |
116 | Cumulative HTML report summary.html provides links to reports on each test case. |
117 | An additional report TESTS-summary.xml is output in JUnit-style XML format |
118 | that can be used for integration with Jenkins or other continuous integration system. |
119 | Type *help testgrid* in DRAW prompt to get help on additional options supported by testgrid command. |
120 | |
121 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
122 | Draw[3]> help testgrid |
123 | testgrid: Run all tests, or specified group, or one grid |
124 | Use: testgrid logdir [group [grid]] [options...] |
125 | Allowed options are: |
126 | -verbose {0-2}: verbose level, 0 by default, can be set to 1 or 2 |
127 | -parallel N: run in parallel with up to N processes (default 0) |
128 | -refresh N: save summary logs every N seconds (default 60) |
129 | -overwrite: force writing logs in existing non-empty directory |
130 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
131 | |
132 | @subsubsection testmanual_1_3_3 Running Single Test |
133 | |
134 | To run single test, type command *test*’ followed by names of group, grid, and test case. |
135 | |
136 | Example: |
137 | |
138 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
139 | Draw[1]> test blend simple A1 |
140 | CASE blend simple A1: OK |
141 | Draw[2]> |
142 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
143 | |
144 | Note that normally intermediate output of the script is not shown. |
145 | To see intermediate commands and their output, type command *decho on* |
146 | before running the test case. (Type ‘decho off’ to disable echoing when not needed.) |
147 | The detailed log of the test can also be obtained after the test execution by command *dlog get*. |
148 | |
149 | @section testmanual_2 Organization of Test Scripts |
150 | |
151 | @subsection testmanual_2_1 General Layout |
152 | |
153 | Standard OCCT tests are located in subdirectory tests of the OCCT root folder ($CASROOT). |
154 | Additional test folders can be added to the test system |
155 | by defining environment variable CSF_TestScriptsPath. |
156 | This should be list of paths separated by semicolons (*;*) on Windows |
157 | or colons (*:*) on Linux or Mac. Upon DRAW launch, |
158 | path to tests sub-folder of OCCT is added at the end of this variable automatically. |
159 | Each test folder is expected to contain: |
160 | |
161 | * Optional file parse.rules defining patterns for interpretation of test results, common for all groups in this folder |
162 | * One or several test group directories. |
163 | |
164 | Each group directory contains: |
165 | |
166 | * File grids.list that identifies this test group and defines list of test grids in it. |
167 | * Test grids (sub-directories), each containing set of scripts for test cases, and optional files cases.list, parse.rules, begin, and end. |
168 | * Optional sub-directory data |
169 | * Optional file parse.rules |
170 | * Optional files begin and end |
171 | |
172 | By convention, names of test groups, grids, and cases should contain no spaces and be lowercase. |
173 | Names begin, end, data, parse.rules, grids.list, cases.list are reserved. |
174 | General layout of test scripts is shown on Figure 1. |
175 | |
176 | ![](/devs_guide/tests/images/tests_image001.png "") |
177 | |
178 | Figure 1. Layout of tests folder |
179 | |
180 | @subsection testmanual_2_2 Test Groups |
181 | |
182 | @subsubsection testmanual_2_2_1 Group Names |
183 | |
184 | Test folder usually contains several directories representing test groups (Group 1, Group N). |
185 | Each directory contains test grids for certain OCCT functionality. |
186 | The name of directory corresponds to this functionality. |
187 | Example: |
188 | |
189 | @verbatim |
190 | caf |
191 | mesh |
192 | offset |
193 | @endverbatim |
194 | |
195 | @subsubsection testmanual_2_2_2 Group's *grids.list* File |
196 | |
197 | The test group must contain file *grids.list* file |
198 | which defines ordered list of grids in this group in the following format: |
199 | |
200 | ~~~~~~~~~~~~~~~~~ |
201 | 001 gridname1 |
202 | 002 gridname2 |
203 | ... |
204 | NNN gridnameN |
205 | ~~~~~~~~~~~~~~~~~ |
206 | |
207 | Example: |
208 | |
209 | ~~~~~~~~~~~~~~~~~ |
210 | 001 basic |
211 | 002 advanced |
212 | ~~~~~~~~~~~~~~~~~ |
213 | |
214 | @subsubsection testmanual_2_2_3 Group's *begin* File |
215 | |
216 | The file *begin* is a Tcl script. It is executed before every test in current group. |
217 | Usually it loads necessary Draw commands, sets common parameters and defines |
218 | additional Tcl functions used in test scripts. |
219 | Example: |
220 | |
221 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
222 | pload TOPTEST ;# load topological command |
223 | set cpulimit 300 ;# set maximum time allowed for script execution |
224 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
225 | |
226 | @subsubsection testmanual_2_2_4 Group's *end* File |
227 | |
228 | The file end is a TCL script. It is executed after every test in current group. |
229 | Usually it checks the results of script work, makes a snap-shot |
230 | of the viewer and writes *TEST COMPLETED* to the output. |
231 | Note: *TEST COMPLETED* string should be presented in output |
232 | in order to signal that test is finished without crash. |
233 | See <a href="#testmanual_3">Creation And Modification Of Tests</a> chapter for more information. |
234 | Example: |
235 | |
236 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
237 | if { [isdraw result] } { |
238 | checkshape result |
239 | } else { |
240 | puts *Error: The result shape can not be built* |
241 | } |
242 | puts *TEST COMPLETED* |
243 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
244 | |
245 | @subsubsection testmanual_2_2_5 Group’s *parse.rules* File |
246 | |
247 | The test group may contain *parse.rules* file. |
248 | This file defines patterns used for analysis of the test execution log |
249 | and deciding the status of the test run. |
250 | Each line in the file should specify a status (single word), |
251 | followed by a regular expression delimited by slashes (*/*) |
252 | that will be matched against lines in the test output log to check if it corresponds to this status. |
253 | The regular expressions support subset of the Perl re syntax. |
254 | The rest of the line can contain a comment message |
255 | which will be added to the test report when this status is detected. |
256 | Example: |
257 | |
258 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
259 | FAILED /\b[Ee]xception\b/ exception |
260 | FAILED /\bError\b/ error |
261 | SKIPPED /Cannot open file for reading/ data file is missing |
262 | SKIPPED /Could not read file .*, abandon/ data file is missing |
263 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
264 | |
265 | Lines starting with a *#* character and blank lines are ignored to allow comments and spacing. |
266 | See <a href="#testmanual_2_3_4">Interpretation of test results</a> chapter for details. |
267 | |
268 | If a line matches several rules, the first one applies. |
269 | Rules defined in the grid are checked first, then rules in group, |
270 | then rules in the test root directory. This allows defining some rules on the grid level |
271 | with status IGNORE to ignore messages that would otherwise be treated as errors due to the group level rules. |
272 | Example: |
273 | |
274 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
275 | FAILED /\bFaulty\b/ bad shape |
276 | IGNORE /^Error [23]d = [\d.-]+/ debug output of blend command |
277 | IGNORE /^Tcl Exception: tolerance ang : [\d.-]+/ blend failure |
278 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
279 | |
280 | @subsection testmanual_2_3 Test Grids |
281 | |
282 | @subsubsection testmanual_2_3_1 Grid Names |
283 | |
284 | Group folder can have several sub-directories (Grid 1… Grid N) defining test grids. |
285 | Each test grid directory contains a set of related test cases. |
286 | The name of directory should correspond to its contents. |
287 | |
288 | Example: |
289 | |
290 | caf |
291 | basic |
292 | bugs |
293 | presentation |
294 | |
295 | Where **caf** is the name of test group and *basic*, *bugs*, *presentation*, etc are the names of grids. |
296 | |
297 | @subsubsection testmanual_2_3_2 Grid’s *begin* File |
298 | |
299 | The file *begin* is a TCL script. It is executed before every test in current grid. |
300 | Usually it sets variables specific for the current grid. |
301 | Example: |
302 | |
303 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
304 | set command bopfuse ;# command tested in this grid |
305 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
306 | |
307 | @subsubsection testmanual_2_3_3 Grid’s *end* File |
308 | |
309 | The file *end* is a TCL script. It is executed after every test in current grid. |
310 | Usually it executes specific sequence of commands common for all tests in the grid. |
311 | Example: |
312 | |
313 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
314 | vdump $logdir/${casename}.gif ;# makes a snap-shot of AIS viewer |
315 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
316 | |
317 | @subsubsection testmanual_2_3_4 Grid’s *cases.list* File |
318 | |
319 | The grid directory can contain an optional file cases.list |
320 | defining alternative location of the test cases. |
321 | This file should contain singe line defining the relative path to collection of test cases. |
322 | |
323 | Example: |
324 | |
325 | ../data/simple |
326 | |
327 | This option is used for creation of several grids of tests with the same data files |
328 | and operations but performed with differing parameters. |
329 | The common scripts are usually located place in common |
330 | subdirectory of the test group (data/simple as in example). |
331 | If cases.list file exists then grid directory should not contain any test cases. |
332 | The specific parameters and pre- and post-processing commands |
333 | for the tests execution in this grid should be defined in the begin and end files. |
334 | |
335 | @subsection testmanual_2_4 Test Cases |
336 | |
337 | The test case is TCL script which performs some operations using DRAW commands |
338 | and produces meaningful messages that can be used to check the result for being valid. |
339 | Example: |
340 | |
341 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
342 | pcylinder c1 10 20 ;# create first cylinder |
343 | pcylinder c2 5 20 ;# create second cylinder |
344 | ttranslate c2 5 0 10 ;# translate second cylinder to x,y,z |
345 | bsection result c1 c2 ;# create a section of two cylinders |
346 | checksection result ;# will output error message if result is bad |
347 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
348 | |
349 | The test case can have any name (except reserved names begin, end, data, cases.list, parse.rules). |
350 | For systematic grids it is usually a capital English letter followed by a number. |
351 | |
352 | Example: |
353 | |
354 | @verbatim |
355 | A1 |
356 | A2 |
357 | B1 |
358 | B2 |
359 | @endverbatim |
360 | |
361 | Such naming facilitates compact representation of results |
362 | of tests execution in tabular format within HTML reports. |
363 | |
364 | @subsection testmanual_2_5 Directory *data* |
365 | |
366 | The test group may contain subdirectory data. |
367 | Usually it contains data files used in tests (BREP, IGES, STEP, etc.) |
368 | and / or test scripts shared by different test grids |
369 | (in subdirectories, see <a href="#testmanual_2_3_4">Grid’s *cases.list* file</a> chapter). |
370 | |
371 | @section testmanual_3 Creation And Modification Of Tests |
372 | |
373 | This section describes how to add new tests and update existing ones. |
374 | |
375 | @subsection testmanual_3_1 Choosing Group, Grid, and Test Case Name |
376 | |
377 | The new tests are usually added in context of processing some bugs. |
378 | Such tests in general should be added to group bugs, in the grid |
379 | corresponding to the affected OCCT functionality. |
380 | New grids can be added as necessary to contain tests on functionality not yet covered by existing test grids. |
381 | The test case name in the bugs group should be prefixed by ID |
382 | of the corresponding issue in Mantis (without leading zeroes). |
383 | It is recommended to add a suffix providing a hint on the situation being tested. |
384 | If more than one test is added for a bug, they should be distinguished by suffixes; |
385 | either meaningful or just ordinal numbers. |
386 | |
387 | Example: |
388 | |
389 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
390 | 12345_coaxial |
391 | 12345_orthogonal_1 |
392 | 12345_orthogonal_2 |
393 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
394 | |
395 | In the case if new test corresponds to functionality for which |
396 | specific group of tests exists (e.g. group mesh for BRepMesh issues), |
397 | this test can be added (or moved later by OCC team) to this group. |
398 | |
399 | @subsection testmanual_3_2 Adding Data Files Required for a Test |
400 | |
401 | It is advisable that tests scripts should be made self-contained whenever possible, |
402 | so as to be usable in environments where data files are not available. |
403 | For that simple geometric objects and shapes can be created using DRAW commands in the test script itself. |
404 | If test requires some data file, it should be put to subdirectory data of the test grid. |
405 | Note that when test is integrated to master branch, |
406 | OCC team can move data file to data files repository, |
407 | so as to keep OCCT sources repository clean from big data files. |
408 | When preparing a test script, try to minimize size of involved data model. |
409 | For instance, if problem detected on a big shape can be reproduced on a single face |
410 | extracted from that shape, use only this face in the test. |
411 | |
412 | @subsection testmanual_3_3 Implementation of the Script |
413 | |
414 | Test should run commands necessary to perform the operations being tested, |
415 | in a clean DRAW session. This includes loading necessary functionality by *pload* command, |
416 | if this is not done by *begin* script. The messages produced by commands in standard output |
417 | should include identifiable messages on the discovered problems if any. |
418 | Usually the script represents a set of commands that a person would run interactively |
419 | to perform the operation and see its results, with additional comments to explain what happens. |
420 | Example: |
421 | |
422 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
423 | # Simple test of fusing box and sphere |
424 | box b 10 10 10 |
425 | sphere s 5 |
426 | bfuse result b s |
427 | checkshape result |
428 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
429 | |
430 | Make sure that file parse.rules in the grid or group directory contains |
431 | regular expression to catch possible messages indicating failure of the test. |
432 | For instance, for catching errors reported by *checkshape* command |
433 | relevant grids define a rule to recognize its report by the word *Faulty*: FAILED /\bFaulty\b/ bad shape |
434 | For the messages generated in the script the most natural way is to use the word *Error* in the message. |
435 | Example: |
436 | |
437 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
438 | set expected_length 11 |
439 | if { [expr $actual_length - $expected_length] > 0.001 } { |
440 | puts *Error: The length of the edge should be $expected_length* |
441 | } |
442 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
443 | |
444 | At the end, the test script should output *TEST COMPLETED* string |
445 | to mark successful completion of the script. |
446 | This is often done by the end script in the grid. |
447 | When test script requires data file, use Tcl procedure *locate_data_file* |
448 | to get path to the data file, rather than explicit path. |
449 | This will allow easy move of the data file from OCCT repository |
450 | to the data files repository without a need to update test script. |
451 | Example: |
452 | |
453 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
454 | stepread [locate_data_file CAROSKI_COUPELLE.step] a * |
455 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
456 | |
457 | When test needs to produce some snapshots or other artifacts, |
458 | use Tcl variable logdir as location where such files should be put. |
459 | Command *testgrid* sets this variable to the subdirectory of the results folder |
460 | corresponding to the grid. Command *test* sets it to $CASROOT/tmp unless it is already defined. |
461 | Use Tcl variable casename to prefix all the files produced by the test. |
462 | This variable is set to the name of the test case. |
463 | Example: |
464 | |
465 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
466 | xwd $logdir/${casename}.gif |
467 | vdisplay result; vfit |
468 | vdump $logdir/${casename}-axo.gif |
469 | vfront; vfit |
470 | vdump $logdir/${casename}-front.gif |
471 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
472 | |
473 | could produce: |
474 | |
475 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
476 | A1.gif |
477 | A1-axo.gif |
478 | A1-front.gif |
479 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
480 | |
481 | @subsection testmanual_3_4 Interpretation of Test Results |
482 | |
483 | The result of the test is evaluated by checking its output against patterns |
484 | defined in the files parse.rules of the grid and group. |
485 | The OCCT test system recognizes five statuses of the test execution: |
486 | |
487 | * SKIPPED: reported if line matching SKIPPED pattern is found (prior to any FAILED pattern). This indicates that the test cannot be run in the current environment; most typical case is absence of the required data file. |
488 | * FAILED: reported if some line matching pattern with status FAILED is found (unless it is masked by preceding IGNORE pattern or a TODO statement, see below), or if message TEST COMPLETED is not found at the end. This indicates that test produces bad or unexpected result, and usually highlights a regression. |
489 | * BAD: reported if test script output contains one or several TODO statements and corresponding number of matching lines in the log. This indicates a known problem (see 3.5). The lines matching TODO statements are not checked against other patterns and thus will not cause a FAILED status. |
490 | * IMPROVEMENT: reported if test script output contains TODO statement for which no corresponding line is found. This is possible indication of improvement (known problem disappeared). |
491 | * OK: If none of the above statuses have been assigned. This means test passed without problems. |
492 | |
493 | Other statuses can be specified in the parse.rules files, these will be classified as FAILED. |
494 | Before integration of the change to OCCT repository, all tests should return either OK or BAD status. |
495 | The new test created for unsolved problem should return BAD. |
496 | The new test created for a fixed problem should return FAILED without the fix, and OK with the fix. |
497 | |
498 | @subsection testmanual_3_5 Marking BAD Cases |
499 | |
500 | If the test produces invalid result at a certain moment then the corresponding bug |
501 | should be created in the OCCT issue tracker [3], and the problem should be marked as TODO in the test script. |
502 | The following statement should be added to such test script: |
503 | |
504 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
505 | puts *TODO BugNumber ListOfPlatforms: RegularExpression* |
506 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
507 | |
508 | where: |
509 | |
510 | * BugNumber is an ID of the bug in the tracker. For example: #12345 |
511 | * ListOfPlatform is a list of platforms at which the bug is reproduced (e.g. Mandriva2008, Windows or All). |
512 | |
513 | *Note: the platform name is custom for the OCCT test system;* |
514 | *it can be consulted as value of environment variable os_type defined in DRAW.* |
515 | |
516 | Example: |
517 | |
518 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
519 | Draw[2]> puts $env(os_type) |
520 | windows |
521 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
522 | |
523 | * RegularExpression is a regular expression which should be matched against the line indicating the problem in the script output. |
524 | Example: |
525 | |
526 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
527 | puts *TODO #22622 Mandriva2008: Abort .* an exception was raised* |
528 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
529 | |
530 | Parser checks the output of the test and if an output line matches |
531 | the RegularExpression then it will be assigned a BAD status instead of FAILED. |
532 | For each output line matching to an error expression a separate TODO line |
533 | must be added to mark the test as BAD. |
534 | If not all the TODO statements are found in the test log, |
535 | the test will be considered as possible improvement. |
536 | To mark the test as BAD for an incomplete case |
537 | (when final TEST COMPLETE message is missing) |
538 | the expression *TEST INCOMPLETE* should be used instead of regular expression. |
539 | |
540 | Example: |
541 | |
542 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
543 | puts *TODO OCC22817 All: exception.+There are no suitable edges* |
544 | puts *TODO OCC22817 All: \\*\\* Exception \\*\\** |
545 | puts *TODO OCC22817 All: TEST INCOMPLETE* |
546 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
547 | |
548 | @section testmanual_4 Extended Use |
549 | |
550 | @subsection testmanual_4_1 Running Tests on Older Versions of OCCT |
551 | |
552 | Sometimes it might be necessary to run tests on previous versions of OCCT (up to to 6.5.3) |
553 | that do not include this test system. This can be done by adding DRAW configuration file DrawAppliInit |
554 | in the directory which is current by the moment of DRAW startup, |
555 | to load test commands and define necessary environment. Example |
556 | (assume that d:/occt contains up-to-date version of OCCT sources |
557 | with tests, and test data archive is unpacked to d:/test-data): |
558 | |
559 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
560 | set env(CASROOT) d:/occt |
561 | set env(CSF_TestScriptsPath) $env(CASROOT)/tests |
562 | source $env(CASROOT)/src/DrawResources/TestCommands.tcl |
563 | set env(CSF_TestDataPath) d:/test-data |
564 | return |
565 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
566 | |
567 | Note that on older versions of OCCT the tests are run in compatibility mode |
568 | and not all output of the test command can be captured; |
569 | this can lead to absence of some error messages (can be reported as improvement). |
570 | |
571 | @subsection testmanual_4_2 Adding Custom Tests |
572 | |
573 | You can extend the test system by adding your own tests. |
574 | For that it is necessary to add paths to the directory where these tests are located, |
575 | and additional data directory(ies), to the environment variables CSF_TestScriptsPath and CSF_TestDataPath. |
576 | The recommended way for doing this is using DRAW configuration file DrawAppliInit |
577 | located in the directory which is current by the moment of DRAW startup. |
578 | |
579 | Use Tcl command *_path_separator* to insert platform-dependent separator to the path list. |
580 | Example: |
581 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl} |
582 | set env(CSF_TestScriptsPath) \ |
583 | $env(TestScriptsPath)[_path_separator]d:/MyOCCTProject/tests |
584 | set env(CSF_TestDataPath) \ |
585 | d:/occt/test-data[_path_separator]d:/MyOCCTProject/tests |
586 | return ;# this is to avoid an echo of the last command above in cout |
587 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
588 | |
589 | @section testmanual_5 References |
590 | |
591 | -# DRAW Test Harness User’s Guide |
592 | -# Perl regular expressions, http://perldoc.perl.org/perlre.html |
593 | -# OCCT MantisBT issue tracker, http://tracker.dev.opencascade.org |