0024427: Wrong section curves
[occt.git] / dox / dev_guides / tests / tests.md
CommitLineData
72b7576f 1 Automated Testing System {#dev_guides__tests}
2======================================
3
e5bd0d98 4@tableofcontents
5
72b7576f 6@section testmanual_1 Introduction
7
8This document provides overview and practical guidelines for work with OCCT automatic testing system.
9Reading this section *Introduction* should be sufficient for OCCT developers to use the test system
10to control non-regression of the modifications they implement in OCCT. Other sections provide
11more in-depth description of the test system, required for modifying the tests and adding new test cases.
12
13@subsection testmanual_1_1 Basic Information
14
e5bd0d98 15OCCT automatic testing system is organized around DRAW Test Harness @ref user_guides__test_harness "DRAW Test Harness",
72b7576f 16a console application based on Tcl (a scripting language) interpreter extended by OCCT-related commands.
e5bd0d98 17
72b7576f 18Standard OCCT tests are included with OCCT sources and are located in subdirectory *tests*
19 of the OCCT root folder. Other test folders can be included in the scope of the test system,
20 e.g. for testing applications based on OCCT.
e5bd0d98 21
72b7576f 22Logically the tests are organized in three levels:
23
24 * Group: group of related test grids, usually relating to some part of OCCT functionality (e.g. blend)
25 * Grid: set of test cases within a group, usually aimed at testing some particular aspect or mode of execution of the relevant functionality (e.g. buildevol)
26 * Test case: script implementing individual test (e.g. K4)
27
28Some tests involve data files (typically CAD models) which are located separately
29 and are not included with OCCT code. The archive with publicly available
30 test data files should be downloaded and installed independently on OCCT code from dev.opencascade.org.
31
32@subsection testmanual_1_2 Intended Use of Automatic Tests
33
34Each modification made in OCCT code must be checked for non-regression
35by running the whole set of tests. The developer who does the modification
36is responsible for running and ensuring non-regression on the tests that are available to him.
37Note that many tests are based on data files that are confidential and thus available only at OPEN CASCADE.
38Thus official certification testing of the changes before integration to master branch
39of official OCCT Git repository (and finally to the official release) is performed by OPEN CASCADE in any case.
40
41Each new non-trivial modification (improvement, bug fix, new feature) in OCCT
42should be accompanied by a relevant test case suitable for verifying that modification.
43This test case is to be added by developer who provides the modification.
44If a modification affects result of some test case(s),
45either the modification should be corrected (if it causes regression)
46or affected test cases should be updated to account for the modification.
47
48The modifications made in the OCCT code and related test scripts
49should be included in the same integration to master branch.
50
51@subsection testmanual_1_3 Quick Start
52
53@subsubsection testmanual_1_3_1 Setup
54
55Before running tests, make sure to define environment variable CSF_TestDataPath
56pointing to the directory containing test data files.
57(The publicly available data files can be downloaded
58from http://dev.opencascade.org separately from OCCT code.)
59The recommended way for that is adding a file *DrawInitAppli*
60in the directory which is current at the moment of starting DRAWEXE (normally it is $CASROOT).
61This file is evaluated automatically at the DRAW start. Example:
62
63~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
64set env(CSF_TestDataPath) d:/occt/tests_data
65return ;# this is to avoid an echo of the last command above in cout
66~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
67
68All tests are run from DRAW command prompt, thus first run draw.tcl or draw.sh to start DRAW.
69
70@subsubsection testmanual_1_3_2 Running Tests
71
72To run all tests, type command *testgrid* followed by path
73to the new directory where results will be saved.
74It is recommended that this directory should be new or empty;
75use option –overwrite to allow writing logs in existing non-empty directory.
76
77Example:
78
79~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
80 Draw[]> testgrid d:/occt/results-2012-02-27
81~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
82
83If empty string is given as log directory name, the name will be generated automatically
84using current date and time, prefixed by *results_*. Example:
85
86~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
87 Draw[]> testgrid {}
88~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
89
90For running only some group or a grid of tests,
91give additional arguments indicating group and (if needed) grid. Example:
92
93~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
94 Draw[]> testgrid d:/occt/results-2012-02-28 blend simple
95~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
96
97As the tests progress, the result of each test case is reported.
98At the end of the log summary of test cases is output,
99including list of detected regressions and improvements, if any. Example:
100
101
102~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
103 Tests summary
104
105 CASE 3rdparty export A1: OK
106 ...
107 CASE pipe standard B1: BAD (known problem)
108 CASE pipe standard C1: OK
109 No regressions
110 Total cases: 208 BAD, 31 SKIPPED, 3 IMPROVEMENT, 1791 OK
111 Elapsed time: 1 Hours 14 Minutes 33.7384512019 Seconds
112 Detailed logs are saved in D:/occt/results_2012-06-04T0919
113~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
114
115The tests are considered as non-regressive if only OK, BAD (i.e. known problem),
116and SKIPPED (i.e. not executed, e.g. because of lack of data file) statuses are reported.
117See <a href="#testmanual_3_4">Grid’s *cases.list* file</a> chapter for details.
118
119The detailed logs of running tests are saved in the specified directory and its sub-directories.
120Cumulative HTML report summary.html provides links to reports on each test case.
121An additional report TESTS-summary.xml is output in JUnit-style XML format
122that can be used for integration with Jenkins or other continuous integration system.
123Type *help testgrid* in DRAW prompt to get help on additional options supported by testgrid command.
124
125~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
126 Draw[3]> help testgrid
127 testgrid: Run all tests, or specified group, or one grid
128 Use: testgrid logdir [group [grid]] [options...]
129 Allowed options are:
130 -verbose {0-2}: verbose level, 0 by default, can be set to 1 or 2
131 -parallel N: run in parallel with up to N processes (default 0)
132 -refresh N: save summary logs every N seconds (default 60)
133 -overwrite: force writing logs in existing non-empty directory
134~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
135
136@subsubsection testmanual_1_3_3 Running Single Test
137
138To run single test, type command *test*’ followed by names of group, grid, and test case.
139
140Example:
141
142~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
143 Draw[1]> test blend simple A1
144 CASE blend simple A1: OK
145 Draw[2]>
146~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
147
148Note that normally intermediate output of the script is not shown.
149To see intermediate commands and their output, type command *decho on*
150before running the test case. (Type ‘decho off’ to disable echoing when not needed.)
151The detailed log of the test can also be obtained after the test execution by command *dlog get*.
152
153@section testmanual_2 Organization of Test Scripts
154
155@subsection testmanual_2_1 General Layout
156
157Standard OCCT tests are located in subdirectory tests of the OCCT root folder ($CASROOT).
158Additional test folders can be added to the test system
159by defining environment variable CSF_TestScriptsPath.
160This should be list of paths separated by semicolons (*;*) on Windows
161or colons (*:*) on Linux or Mac. Upon DRAW launch,
162path to tests sub-folder of OCCT is added at the end of this variable automatically.
163Each test folder is expected to contain:
164
165 * Optional file parse.rules defining patterns for interpretation of test results, common for all groups in this folder
166 * One or several test group directories.
167
168Each group directory contains:
169
170 * File grids.list that identifies this test group and defines list of test grids in it.
171 * Test grids (sub-directories), each containing set of scripts for test cases, and optional files cases.list, parse.rules, begin, and end.
172 * Optional sub-directory data
173 * Optional file parse.rules
174 * Optional files begin and end
175
176By convention, names of test groups, grids, and cases should contain no spaces and be lowercase.
177Names begin, end, data, parse.rules, grids.list, cases.list are reserved.
178General layout of test scripts is shown on Figure 1.
179
dba69de2 180@image html /dev_guides/tests/images/tests_image001.png
181@image latex /dev_guides/tests/images/tests_image001.png
72b7576f 182
183Figure 1. Layout of tests folder
184
185@subsection testmanual_2_2 Test Groups
186
187@subsubsection testmanual_2_2_1 Group Names
188
189Test folder usually contains several directories representing test groups (Group 1, Group N).
190Each directory contains test grids for certain OCCT functionality.
191The name of directory corresponds to this functionality.
192Example:
193
194@verbatim
195 caf
196 mesh
197 offset
198@endverbatim
199
200@subsubsection testmanual_2_2_2 Group's *grids.list* File
201
202The test group must contain file *grids.list* file
203which defines ordered list of grids in this group in the following format:
204
205~~~~~~~~~~~~~~~~~
206001 gridname1
207002 gridname2
208...
209NNN gridnameN
210~~~~~~~~~~~~~~~~~
211
212Example:
213
214~~~~~~~~~~~~~~~~~
215 001 basic
216 002 advanced
217~~~~~~~~~~~~~~~~~
218
219@subsubsection testmanual_2_2_3 Group's *begin* File
220
221The file *begin* is a Tcl script. It is executed before every test in current group.
222Usually it loads necessary Draw commands, sets common parameters and defines
223additional Tcl functions used in test scripts.
224Example:
225
226~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
227 pload TOPTEST ;# load topological command
228 set cpulimit 300 ;# set maximum time allowed for script execution
229~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
230
231@subsubsection testmanual_2_2_4 Group's *end* File
232
233The file end is a TCL script. It is executed after every test in current group.
234Usually it checks the results of script work, makes a snap-shot
235of the viewer and writes *TEST COMPLETED* to the output.
236Note: *TEST COMPLETED* string should be presented in output
237in order to signal that test is finished without crash.
238See <a href="#testmanual_3">Creation And Modification Of Tests</a> chapter for more information.
239Example:
240
241~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
242 if { [isdraw result] } {
243 checkshape result
244 } else {
245 puts *Error: The result shape can not be built*
246 }
247 puts *TEST COMPLETED*
248~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
249
250@subsubsection testmanual_2_2_5 Group’s *parse.rules* File
251
252The test group may contain *parse.rules* file.
253This file defines patterns used for analysis of the test execution log
254and deciding the status of the test run.
255Each line in the file should specify a status (single word),
256followed by a regular expression delimited by slashes (*/*)
257that will be matched against lines in the test output log to check if it corresponds to this status.
258The regular expressions support subset of the Perl re syntax.
259The rest of the line can contain a comment message
260which will be added to the test report when this status is detected.
261Example:
262
263~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
e5bd0d98 264 FAILED /\\b[Ee]xception\\b/ exception
265 FAILED /\\bError\\b/ error
72b7576f 266 SKIPPED /Cannot open file for reading/ data file is missing
267 SKIPPED /Could not read file .*, abandon/ data file is missing
268~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
269
270Lines starting with a *#* character and blank lines are ignored to allow comments and spacing.
271See <a href="#testmanual_2_3_4">Interpretation of test results</a> chapter for details.
272
273If a line matches several rules, the first one applies.
274Rules defined in the grid are checked first, then rules in group,
275then rules in the test root directory. This allows defining some rules on the grid level
276with status IGNORE to ignore messages that would otherwise be treated as errors due to the group level rules.
277Example:
278
279~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
e5bd0d98 280 FAILED /\\bFaulty\\b/ bad shape
72b7576f 281 IGNORE /^Error [23]d = [\d.-]+/ debug output of blend command
282 IGNORE /^Tcl Exception: tolerance ang : [\d.-]+/ blend failure
283~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
284
285@subsection testmanual_2_3 Test Grids
286
287@subsubsection testmanual_2_3_1 Grid Names
288
289Group folder can have several sub-directories (Grid 1… Grid N) defining test grids.
290Each test grid directory contains a set of related test cases.
291The name of directory should correspond to its contents.
292
293Example:
294
295caf
296 basic
297 bugs
298 presentation
299
300Where **caf** is the name of test group and *basic*, *bugs*, *presentation*, etc are the names of grids.
301
302@subsubsection testmanual_2_3_2 Grid’s *begin* File
303
304The file *begin* is a TCL script. It is executed before every test in current grid.
305Usually it sets variables specific for the current grid.
306Example:
307
308~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
309 set command bopfuse ;# command tested in this grid
310~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
311
312@subsubsection testmanual_2_3_3 Grid’s *end* File
313
314The file *end* is a TCL script. It is executed after every test in current grid.
315Usually it executes specific sequence of commands common for all tests in the grid.
316Example:
317
318~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
319 vdump $logdir/${casename}.gif ;# makes a snap-shot of AIS viewer
320~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
321
322@subsubsection testmanual_2_3_4 Grid’s *cases.list* File
323
324The grid directory can contain an optional file cases.list
325defining alternative location of the test cases.
326This file should contain singe line defining the relative path to collection of test cases.
327
328Example:
329
330../data/simple
331
332This option is used for creation of several grids of tests with the same data files
333and operations but performed with differing parameters.
334The common scripts are usually located place in common
335subdirectory of the test group (data/simple as in example).
336If cases.list file exists then grid directory should not contain any test cases.
337The specific parameters and pre- and post-processing commands
338for the tests execution in this grid should be defined in the begin and end files.
339
340@subsection testmanual_2_4 Test Cases
341
342The test case is TCL script which performs some operations using DRAW commands
343and produces meaningful messages that can be used to check the result for being valid.
344Example:
345
346~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
347 pcylinder c1 10 20 ;# create first cylinder
348 pcylinder c2 5 20 ;# create second cylinder
349 ttranslate c2 5 0 10 ;# translate second cylinder to x,y,z
350 bsection result c1 c2 ;# create a section of two cylinders
351 checksection result ;# will output error message if result is bad
352~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
353
354The test case can have any name (except reserved names begin, end, data, cases.list, parse.rules).
355For systematic grids it is usually a capital English letter followed by a number.
356
357Example:
358
359@verbatim
360 A1
361 A2
362 B1
363 B2
364@endverbatim
365
366Such naming facilitates compact representation of results
367of tests execution in tabular format within HTML reports.
368
369@subsection testmanual_2_5 Directory *data*
370
371The test group may contain subdirectory data.
372Usually it contains data files used in tests (BREP, IGES, STEP, etc.)
373and / or test scripts shared by different test grids
374(in subdirectories, see <a href="#testmanual_2_3_4">Grid’s *cases.list* file</a> chapter).
375
376@section testmanual_3 Creation And Modification Of Tests
377
378This section describes how to add new tests and update existing ones.
379
380@subsection testmanual_3_1 Choosing Group, Grid, and Test Case Name
381
382The new tests are usually added in context of processing some bugs.
383Such tests in general should be added to group bugs, in the grid
384corresponding to the affected OCCT functionality.
385New grids can be added as necessary to contain tests on functionality not yet covered by existing test grids.
386The test case name in the bugs group should be prefixed by ID
387of the corresponding issue in Mantis (without leading zeroes).
388It is recommended to add a suffix providing a hint on the situation being tested.
389If more than one test is added for a bug, they should be distinguished by suffixes;
390either meaningful or just ordinal numbers.
391
392Example:
393
394~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
395 12345_coaxial
396 12345_orthogonal_1
397 12345_orthogonal_2
398~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
399
400In the case if new test corresponds to functionality for which
401specific group of tests exists (e.g. group mesh for BRepMesh issues),
402this test can be added (or moved later by OCC team) to this group.
403
404@subsection testmanual_3_2 Adding Data Files Required for a Test
405
406It is advisable that tests scripts should be made self-contained whenever possible,
407so as to be usable in environments where data files are not available.
408For that simple geometric objects and shapes can be created using DRAW commands in the test script itself.
409If test requires some data file, it should be put to subdirectory data of the test grid.
410Note that when test is integrated to master branch,
411OCC team can move data file to data files repository,
412so as to keep OCCT sources repository clean from big data files.
413When preparing a test script, try to minimize size of involved data model.
414For instance, if problem detected on a big shape can be reproduced on a single face
415extracted from that shape, use only this face in the test.
416
417@subsection testmanual_3_3 Implementation of the Script
418
419Test should run commands necessary to perform the operations being tested,
420in a clean DRAW session. This includes loading necessary functionality by *pload* command,
421if this is not done by *begin* script. The messages produced by commands in standard output
422should include identifiable messages on the discovered problems if any.
423Usually the script represents a set of commands that a person would run interactively
424to perform the operation and see its results, with additional comments to explain what happens.
425Example:
426
427~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
428 # Simple test of fusing box and sphere
429 box b 10 10 10
430 sphere s 5
431 bfuse result b s
432 checkshape result
433~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
434
435Make sure that file parse.rules in the grid or group directory contains
436regular expression to catch possible messages indicating failure of the test.
437For instance, for catching errors reported by *checkshape* command
e5bd0d98 438relevant grids define a rule to recognize its report by the word *Faulty*: FAILED /\\bFaulty\\b/ bad shape
72b7576f 439For the messages generated in the script the most natural way is to use the word *Error* in the message.
440Example:
441
442~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
443 set expected_length 11
444 if { [expr $actual_length - $expected_length] > 0.001 } {
445 puts *Error: The length of the edge should be $expected_length*
446 }
447~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
448
449At the end, the test script should output *TEST COMPLETED* string
450to mark successful completion of the script.
451This is often done by the end script in the grid.
452When test script requires data file, use Tcl procedure *locate_data_file*
453to get path to the data file, rather than explicit path.
454This will allow easy move of the data file from OCCT repository
455to the data files repository without a need to update test script.
456Example:
457
458~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
459 stepread [locate_data_file CAROSKI_COUPELLE.step] a *
460~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
461
462When test needs to produce some snapshots or other artifacts,
463use Tcl variable logdir as location where such files should be put.
464Command *testgrid* sets this variable to the subdirectory of the results folder
465corresponding to the grid. Command *test* sets it to $CASROOT/tmp unless it is already defined.
466Use Tcl variable casename to prefix all the files produced by the test.
467This variable is set to the name of the test case.
468Example:
469
470~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
471 xwd $logdir/${casename}.gif
472 vdisplay result; vfit
473 vdump $logdir/${casename}-axo.gif
474 vfront; vfit
475 vdump $logdir/${casename}-front.gif
476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
477
478could produce:
479
480~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
481 A1.gif
482 A1-axo.gif
483 A1-front.gif
484~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
485
486@subsection testmanual_3_4 Interpretation of Test Results
487
488The result of the test is evaluated by checking its output against patterns
489defined in the files parse.rules of the grid and group.
490The OCCT test system recognizes five statuses of the test execution:
491
492 * SKIPPED: reported if line matching SKIPPED pattern is found (prior to any FAILED pattern). This indicates that the test cannot be run in the current environment; most typical case is absence of the required data file.
493 * FAILED: reported if some line matching pattern with status FAILED is found (unless it is masked by preceding IGNORE pattern or a TODO statement, see below), or if message TEST COMPLETED is not found at the end. This indicates that test produces bad or unexpected result, and usually highlights a regression.
494 * BAD: reported if test script output contains one or several TODO statements and corresponding number of matching lines in the log. This indicates a known problem (see 3.5). The lines matching TODO statements are not checked against other patterns and thus will not cause a FAILED status.
495 * IMPROVEMENT: reported if test script output contains TODO statement for which no corresponding line is found. This is possible indication of improvement (known problem disappeared).
496 * OK: If none of the above statuses have been assigned. This means test passed without problems.
497
498Other statuses can be specified in the parse.rules files, these will be classified as FAILED.
499Before integration of the change to OCCT repository, all tests should return either OK or BAD status.
500The new test created for unsolved problem should return BAD.
501The new test created for a fixed problem should return FAILED without the fix, and OK with the fix.
502
503@subsection testmanual_3_5 Marking BAD Cases
504
505If the test produces invalid result at a certain moment then the corresponding bug
e5bd0d98 506should be created in the OCCT issue tracker http://tracker.dev.opencascade.org,
507and the problem should be marked as TODO in the test script.
72b7576f 508The following statement should be added to such test script:
509
510~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
511 puts *TODO BugNumber ListOfPlatforms: RegularExpression*
512~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
513
514where:
515
516 * BugNumber is an ID of the bug in the tracker. For example: #12345
517 * ListOfPlatform is a list of platforms at which the bug is reproduced (e.g. Mandriva2008, Windows or All).
518
519*Note: the platform name is custom for the OCCT test system;*
520*it can be consulted as value of environment variable os_type defined in DRAW.*
521
522Example:
523
524~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
525 Draw[2]> puts $env(os_type)
526 windows
527~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
528
529 * RegularExpression is a regular expression which should be matched against the line indicating the problem in the script output.
530Example:
531
532~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
533 puts *TODO #22622 Mandriva2008: Abort .* an exception was raised*
534~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
535
536Parser checks the output of the test and if an output line matches
537the RegularExpression then it will be assigned a BAD status instead of FAILED.
538For each output line matching to an error expression a separate TODO line
539must be added to mark the test as BAD.
540If not all the TODO statements are found in the test log,
541the test will be considered as possible improvement.
542To mark the test as BAD for an incomplete case
543(when final TEST COMPLETE message is missing)
544the expression *TEST INCOMPLETE* should be used instead of regular expression.
545
546Example:
547
548~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
549 puts *TODO OCC22817 All: exception.+There are no suitable edges*
550 puts *TODO OCC22817 All: \\*\\* Exception \\*\\**
551 puts *TODO OCC22817 All: TEST INCOMPLETE*
552~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
553
554@section testmanual_4 Extended Use
555
556@subsection testmanual_4_1 Running Tests on Older Versions of OCCT
557
558Sometimes it might be necessary to run tests on previous versions of OCCT (up to to 6.5.3)
559that do not include this test system. This can be done by adding DRAW configuration file DrawAppliInit
560in the directory which is current by the moment of DRAW startup,
561to load test commands and define necessary environment. Example
562(assume that d:/occt contains up-to-date version of OCCT sources
563with tests, and test data archive is unpacked to d:/test-data):
564
565~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
566 set env(CASROOT) d:/occt
567 set env(CSF_TestScriptsPath) $env(CASROOT)/tests
568 source $env(CASROOT)/src/DrawResources/TestCommands.tcl
569 set env(CSF_TestDataPath) d:/test-data
570 return
571~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
572
573Note that on older versions of OCCT the tests are run in compatibility mode
574and not all output of the test command can be captured;
575this can lead to absence of some error messages (can be reported as improvement).
576
577@subsection testmanual_4_2 Adding Custom Tests
578
579You can extend the test system by adding your own tests.
580For that it is necessary to add paths to the directory where these tests are located,
581and additional data directory(ies), to the environment variables CSF_TestScriptsPath and CSF_TestDataPath.
582The recommended way for doing this is using DRAW configuration file DrawAppliInit
583located in the directory which is current by the moment of DRAW startup.
584
585Use Tcl command *_path_separator* to insert platform-dependent separator to the path list.
586Example:
587~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
588 set env(CSF_TestScriptsPath) \
589 $env(TestScriptsPath)[_path_separator]d:/MyOCCTProject/tests
590 set env(CSF_TestDataPath) \
591 d:/occt/test-data[_path_separator]d:/MyOCCTProject/tests
592 return ;# this is to avoid an echo of the last command above in cout
593~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~