= Using Opensocial test framework... = = Introduction = This document describes the inner workings of the Compliance gadget described at ComplianceTests in technical detail. It gives an overview of the class structure and files that implement the testing framework used by the gadget. The test framework is implemented entirely in xml and javascript. If you'd like to modify a test, or add additional tests or test suites, check the code out from the svn server, and modify the xml and javascript files. There's a list of test suites and their description, as well as individual tests at ComplianceTests. If you have made changes that you think might be valuable to others, please submit them back to svn - we'll all benefit! = Getting the Source Code = You can check out the source code from SVN here: {{{ svn co http://opensocial-resources.googlecode.com/svn/tests/trunk }}} = Test hierarchy = There are three levels in the hierarchy of tests: test, test suite, and test suite directory. Following are descriptions of the three levels of hierarchy, and required attributes for each level. * Test - this is a basic building block of a test suite. Tests have the following attributes: * id: A short, symbolic id for the test * name: The human-readable name of the test * run: The main function of this test * description: A longer, human-readable description of the test * timeout: The number of milliseconds that the run function will have to finish before it has a timeout error. by default is 60000 milliseconds * queue: The testing queue this test belong to, by default queue is 6 * tags: List of tags for this test * bugs: List of bugs associated with this test * manual: A flag indicating the test is manual and should not be executed * priority: Specifies a priority for the test * data: Additional user data should be stored here * Test suite - a logical collection of tests. A test suite has the following attributes: * id: A short, symbolic id for the test * name: The human-readable name of the suite * suiteSetup: A method that will run only once before the test suite is run * suiteTearDown: A method that will run only once after the entire suite has run * tests: List of tests that belong to this suite * Test suite directory: Collection of suites to be executed in single gadget * id: A short, symbolic id for the suite directory * name: The human-readable name of the suite directory * suites: List of suites that belong to this directory == Typical test suite == Here's a sample test suite implementation: {{{ function testSuiteName() { this.name = 'Test Suite Name'; this.id = 'TestSuite ID'; this.tests = [ { name: 'test name', id: 'test id', priority: Test.PRIORITY.P0, // {P0 to P4} description: 'description of test', run: function(context, callback, result) { .... .... result.addSubResult(this.name, Assert.assertEquals, Helper.getObjectOptions(this.name), this.name); .... callback(result); } }, // ... more tests here ] }; }}} Each test case has a run method which will be called by the validator that executes the test cases. It takes three parameters. They are propagated here by the testrunner in the common directory at [http://opensocial-resources.googlecode.com/svn/tests/trunk/common/ http://opensocial-resources.googlecode.com/svn/tests/trunk/common/] # Context: Context in which the gadget is being run. i.e. OWNER and VIEWER information of gadget. # Callback: Callback function called upon completion of test. If you don't call the callback, your result will not be registered with the framework, so make sure to call this upon completion. # Result: Object that stores the outcome of the test. It provides a few methods to save information. Each result object is closely tied with the individual test. == How to add a test suite in the framework == Assuming that you have written a new suite with a couple of tests, you'll need link them with existing framework. This is as easy as editing two text files: # Add references to your suite .js file to three locations in the gadget .xml file For the main gadget, for example, you'd add the .js file to your local version of the file at: http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/compliance/reference/reference.xml - add tags to the CDATA sections for canvas, profile, and home view as with the existing suites. # Include your suite in controller.js Edit your local copy of: http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/compliance/reference/controller.js to add an instantiation of your test suite inside the Master.suites array. == Framework == Here's a broad overview of the components of the test framework. === TestRunner === This is the entry point of the tests. Configure the test runner with the test suite directory or test suite (either is acceptable). TestRunner will first setup the tests, then start execution with the help of the validator. *setupTests:* This method will see if we are running tests in the right context, by verifying viewer and owner information is provided. It also retrieves the baseline results for these tests from a previous baseline run, if any. Finally, it asks logger to prepare the report format and show it to the user. We show the skeleton of the report at this point, and will fill in the details as each test reports it's result. This makes sure that end user see progress on the fly rather than waiting for all tests to finish. It's particularly helpful if we have over a hundred tests to run. After setup is complte, it's time to run the tests! *executeTests:* This method will call validator with the location of baseline results, how to run manual tests, what to do on completion of each test, and when all tests are complete. It will then ask the validator to execute the tests. === Validator === This is where the call to the test you wrote will be made. Calls are asynchronous in nature - we call each test in sequence and ask the test to report on completion by giving a callback. We don't wait for the test to complete synchronously. runTests: This method will prepare queues of tests to run merging all suites. This is done get parallel execution. {{{ BEFORE suites to run | |-----------------------------------------------------------------\ | | | suite-1 suite-2 ... suite-n | | | | | | q1---q2-----qn q1---q2-----qn q1---q2-----qn | | | | | | | | | | | t6 | | t12 | | t18 | t4, t5 | t10, t11 | t16, t17 t1, t2, t3 t7, t8, t9 t13, t14, t15 AFTER suites to run | | q1---t1, t2, t3 , t7, t8, t9 | q2---t4, t5, t10, t11, t16, 17 | | qn---t6, t12, t18 }}} Once segregated, validator calls runQueues. *runQueue:* This method takes the current queue sequence and picks one test at a time from it. It creates a result object for each test and initializes it with appropriate status, 'running'. It calls the setUp method for the test, if any. After setup, it checks if the test is manual, and if so skip it. It also runs any pre-condition of the test before calling the run method of the test. It passes in a newly created result object, callback and context information to run object. The callback passed in is the captureResult method of the validator. *captureResult:* All tests must call captureResult upon completion. It changes the status of the result to 'finished'. Next, it calls the test-finish function in testrunner to display the result of the test. If we have finished all suites, it calls the all-finished function in testrunner. Else, if just the suite is completed, it calls runQueues to pick another suite. Finally, if no suites are completed, it calls runQueue to proceed. ===HtmlLogger:=== This is a reporting module, which displays test results in html format. HtmlLogger: * Displays html grid in the beginning of test execution * Updates test result, suite summary table, and final summary table for each test upon completion of test execution * Updates test suite headers upon completion of a test suite * Provides JSON output for the results and displays baseline results in JSON format *Assert:* This method of HtmlLogger provides a different assert methods as in junit and jsunit *Helper:* This provides helper methods used by the framework. Most important are bind and function runner.