| 8 | | == General rules == |
| | 9 | == General conventions == |
| | 10 | Here is some general information about the testing infrastructure of Sophie2: |
| | 11 | * We are using JUnit as a testing framework. |
| | 12 | * The base classes that are used for testing are in the [source:/trunk/sophie2-platform/modules/org.sophie2.core/src/main/java/org/sophie2/core/testing org.sophie2.core.testing] package. |
| | 13 | * Tests should either extend one of UnitTestBase, IntegrationTestBase and SystemTestBase or be marked with @UnitTest, @IntegrationTest or @SystemTest annotation when they need to inherit another class. |
| | 14 | * All resources needed for tests should be placed in the /src/test/resources folder of the module where the test is in. |
| | 15 | * Unit tests are required for all classes, integration tests should be written where is needed, system tests are not required at the current stage of development. |
| 10 | | * The test2/net/asteasolutions/veda/testing/ contains the base infrastructure for testing |
| 11 | | * test2/net/asteasolutions/veda/ contains some suites - !AllTests, !AllUnitTests, !AllSystemTests and !AllIntegrationTests. You can imagine what they are doing by their names. |
| 12 | | * Tests should either extend one of (!UnitTestBase, !IntegrationTestBase, !SystemTestBase) or be marked with (@!UnitTest, @!IntegrationTest, @!SystemTest) when they do not need to inherit the infrastructure. |
| 13 | | * Tests should not initialize instance fields outside of the setUp methods. This is dirty. |
| 14 | | * Tests should really test only what they are intended to test. |
| 15 | | * If you override setUp(), the first line of your setUp should be super.setUp()! |
| 16 | | * If you overrid tearDown(), the last line of your tearDown should be super.tearDown()! |
| 17 | | * Put all resources needed for testing (sample book, images, etc) into !TestBase.TEST_RES_PATH. Do not hardcode this path! |
| 18 | | * All temporary files should be created in !TestBase.TEST_TMP_PATH. You should make sure that you erase them upon successful run. |
| 19 | | * Most test classes should contain several (or many) test methods. It is stupid to have only one test method for unit test. |
| 20 | | * All test methods should be short, and check ONLY what they are intended to check. |
| 21 | | * there are helper classes for the test cases in test2 directory - halos/HalosConstants, menus/demo, menus/mock, logic/LogicHelper, model/book/DefaultBookResource, model/persist/MiscTest, prolib/DummyListener, view/MockBookView, src/research/dialogs/TestingDialogManager |
| | 17 | Here follow some conventions about writing tests (these apply to all kinds of auto tests): |
| | 18 | * A test method should test a specific thing only. |
| | 19 | * A test class should have several test methods. |
| | 20 | * A test class/method should test only what it is intended to. |
| | 21 | * Test methods should be independent from one another. |
| | 22 | * Tests should not initialize instance fields outside of the setUp methods. This is dirty. |
| | 23 | * If you override setUp(), the first line of your setUp should be super.setUp(). |
| | 24 | * If you overrid tearDown(), the last line of your tearDown should be super.tearDown(). |
| | 25 | * Don't test trivial things |
| | 26 | * Don't use random test data. |
| | 27 | * Test special (boundary) cases. |
| 25 | | The easiest way to make good unit test, is to use Eclipse generate its skeleton (that is, one test for each method of the tested class). See http://www.basilv.com/psd/blog/2006/how-to-write-good-unit-tests and http://en.wikipedia.org/wiki/Unit_testing for directions. |
| 26 | | Then change your class to extends !UnitTestBase and start filling the tests. If a method is not trivial (you need to check several things) add extra test methods like: |
| | 41 | == Unit Testing == |
| | 42 | Units are the smallest testable part of an application. In our case these are methods and classes. Unit tests ensure that every part of the application is working as a unit. We require unit tests to be written for all classes. That is because they: |
| | 43 | * Make refactoring easier - if you break something, you will notice immediately. |
| | 44 | * Simplify integration - when you know that separate units are working, you can easily integrate them. |
| | 45 | * Provide documentation - unit tests actually provide examples of using the features. |
| | 46 | * Provide design benefits - unit tests actually define the required functionality of the tested unit. |
| 28 | | * testCurrentBookView |
| 29 | | * testCurrentBookView_remembers |
| 30 | | * testCurrentBookView_validation |
| | 48 | You can quickly start writing a unit test by making Eclipse generate its skeleton (i.e. add one test method for each method of the tested class. Follow these steps to do that: |
| | 49 | * Go to the /src/test/java folder of the module the tested class is in. |
| | 50 | * Create a package with the same name as the package of the tested class (if it does not already exist). |
| | 51 | * Select '''New''' -> '''JUnit Test Case''' from the File or context menu. |
| | 52 | * Fill in the name of the test class and the class under test. |
| | 53 | * The name of the test class should start or end with Test. It is usually the name of the tested class, followed by test. |
| | 54 | * You have to start typing in the class under test selection dialog in order to get some results |
| | 55 | * Since Javadoc is required, select '''Generate comments''' and click next. |
| | 56 | * Select the methods you want to test and click '''Finish''' |
| | 57 | * Simple getter methods do not need to be tested. |
| | 58 | * The new class is generated. Don't forget to write more informative Javadoc and actually fill in the test methods. |
| | 59 | * Change your class to extend UnitTestBase and feel free to add more test methods. |
| 84 | | '''Units to be tested''' |
| 85 | | 1. !FileMenu - in progress |
| 86 | | 2. !MainMenu |
| 87 | | 5. Chaining |
| 88 | | 6. !DragAndDrop |
| 89 | | 7. Search |
| | 80 | === Maven === |
| | 81 | When building, Maven automatically runs all tests (except when the -Dmaven.test.skip=true is passed as an argument). The results from the tests can be found in the \trunk\sophie2-platform\modules\''<module-name>''\target\surefire-reports folder. If you want to run the tests without building, you have two ways of doing that: |
| | 82 | * In Eclipse - right-click on the pom.xml file of the module that you want to run all tests for (sophie2-platform if you want to test all modules) and select '''Run as...''' -> '''Maven test'''. |
| | 83 | * In console - from the root folder of the module you want to test (or sophie2-platform if you want to test all modules), run the ''mvn test'' command. |
| | 84 | |
| | 85 | === Hudson === |
| | 86 | When building, Hudson also automatically runs tests. You can see the test results using the web interface at http://sophie2.org:8080 When you select the Sophie2.0 project from the home page, you will be taken to a summary page with a link to the latest test result. From there you can select tests and see their output. You can also select a build and see its tests - this is useful to see when a test has been broken. |
| | 87 | |
| | 88 | == GUI testing == |
| | 89 | |
| | 90 | == Reviewing == |
| | 91 | This section contains rules for reviewing auto-tests as well as for doing and reviewing the testing phase of tasks. |
| | 92 | |
| | 93 | === Rules === |
| | 94 | Implementation reviewers should make sure that the created auto tests comply with the [wiki:PLATFORM_STANDARDS_CODE code] and [wiki:JAVADOC_CONVENTIONS Javadoc] conventions and the rules specified in this document. Otherwise an implementation should not pass. An implementation with unit tests that fail due to a bug in other than the tested functionality may pass but that should be commented in the ticket. |
| | 95 | |
| | 96 | In the testing phase all auto tests should be run. If there is a failiure it should be reviewed: |
| | 97 | * If it is due to bad implementation of the task (i.e. functionality cover by this task is causing errors), a super review should be requested. |
| | 98 | * If it is due to an error in the test itself, it should be corrected. |
| | 99 | * If it is due to a bug in another part of the application, that bug should be reported. |
| | 100 | * If it is introduced by code commited after that, there is no universal rule. Seek the best solution in each specific case. |
| | 101 | |
| | 102 | In all three cases the results from the tests should be described in the Testing section of the task's wiki page. Here is a more thourough description of what this section should contain: |
| | 103 | * A link to the user documentation describing this task. |
| | 104 | * A link to the release documentation if the result of this task will be part of the next release. |
| | 105 | * Links to use cases in [http://sophie2.org/testlink Testlink] where applicable. |
| | 106 | * see [wiki:PLATFORM_STANDARDS_MANUAL_TESTS] for more information on Testlink and manual testing scenarios. |
| | 107 | * related test cases should be considered and listed as well. |
| | 108 | * Links to all auto tests related to this tasks. |
| | 109 | * (recommended) A link to the Hudson test report regarding these tests. |
| | 110 | * Explanations of the results of the tests. |
| | 111 | * when there are failures an explanation of what the errors are due to. |
| | 112 | * A brief explanation of the bugs reported with links to their trac tickets. |
| | 113 | * links to related bugs should be provided as well. |
| | 114 | |
| | 115 | === Scoring === |
| | 116 | Reviewers should either follow the standards in this document or comment them in the ''Comments'' section of this page. If you state a task does not comply with the standards, point to the requirements that are not met. Scores are in the range 1-5. Here are the rules for scoring the testing phase: |
| | 117 | * Score 1 (fail): The testing phase is not structured according to the standards (or is to very little extent). |
| | 118 | * Score 2 (fail): The testing phase is structured according to the standards in the most part but has some things that are missing or bugs that are not linked or explained or test cases are not suitable, etc. - in general - the testing does not cover all aspects of the functionality. |
| | 119 | * Score 3 (pass): The testing phase is structured according to the standards, covers the functionality but lacks some descriptions and more things can be added. |
| | 120 | * Score 4 (pass): The testing phase is structured according to the standards and provides detailed information according to the requirements mentioned above. |
| | 121 | * Score 5 (pass): The testing phase is structured according to the standards and there's nothing more to be added - it's perfect in such a way that a person who is not quite familiar with the project can clearly see that the feature(s) is/are implemented really well. |
| | 122 | |
| | 123 | All reviews should be motivated. A detailed comment about why the testing phase fails is required. For a score of 3 a list of things that could be better should be provided. Comments are encouraged for higher scores as well. Non-integer scores are STRONGLY disencouraged. If you give the testing a score of 3.5, then you probably have not reviewed it thoroughly enough and cannot clearly state whether it is good or not. Once the testing phase has been reviewed, it cannot be altered. If you think it is wrong, you should request a super review. Currently all super reviews should be discussed with Milo. Make sure you are able to provide clear arguments of what the testing lacks before you request a super review. |
