#summary Compliance tests for OpenSocial and Gadgets = Quick Start Guide = The compliance tests can be used to quickly determine if an !OpenSocial container is compliant with the spec. To run the compliance tests simply install the appropriate gadget ([http://opensocial-resources.googlecode.com/svn/tests/trunk/suites/0.8/compliance/reference.xml JavaScript] or [http://opensocial-resources.googlecode.com/svn/tests/trunk/suites/0.8/restful/restful.xml REST]) and load the canvas view. To publish the results, click the "Generate XML Report" link and save the output to your local machine. Then visit the [http://opensocial-compliance.appspot.com/ Compliance matrix] and upload the results. You should also add a link to your compliance test results to your container page on the [http://wiki.opensocial.org/index.php?title=Containers OpenSocial OpenSocial wiki] ([http://wiki.opensocial.org/index.php?title=Contributing_Container_Info howto]). Following is more detail on what is tested, how to interpret the results, and most importantly, how you can help make the tests better! = Introduction = The !OpenSocial Compliance tests are gadgets that you can install in any !OpenSocial container. These gadgets run a series of tests to exercise the container's support for the JavaScript and REST APIs. The gadgets will output whether the container passes or fails compliance for each of over 100 tests. There are three levels of tests, p0, p1, and p2. The p0 tests must pass in order for a container to be spec compliant. The p1 and p2 tests should pass but are more about developer usability issues and are not required for spec compliance. Lower numbered issues are more urgent, for example, a p1 test failure indicates a higher priority issue than p2 test failure. Sets of related tests are grouped into test suites. A list of all current test suites and tests is at the end of this document. These tests currently cover the !OpenSocial and gadgets specs, and are a work in progress. If you wish to contribute to the tests by refactoring, fixing problems with the tests themselves, or adding your own test cases, all patches are very welcome! The tests currently live in the svn tests directory as [http://opensocial-resources.googlecode.com/svn/tests/trunk http://opensocial-resources.googlecode.com/svn/tests/trunk] The structure of the gadget and test framework, as well as how to get the source code and add tests, is described more fully in OpenSocialTestFramework. Please go through the *[http://code.google.com/p/opensocial-resources/wiki/ContributingToTestFramework !JavaScript style guide]* to make sure your patches adhere to it. = Running the test gadgets against your container = You can quickly try the latest submitted version of the tests by installing the app with the public SVN hosted URL for the gadget XML file: ||0.8 !JavaScript||http://opensocial-resources.googlecode.com/svn/tests/trunk/suites/0.8/compliance/reference.xml|| ||0.8 REST||http://opensocial-resources.googlecode.com/svn/tests/trunk/suites/0.8/restful/restful.xml|| ||0.7 !JavaScript||http://opensocial-resources.googlecode.com/svn/tests/trunk/suites/0.7/compliance/reference/reference.xml|| Simply add the gadget to your container with this URL. You should disable caching when working with this gadget as it may be changing rapidly. For example, in the orkut sandbox, append &bpc=1 to the page URL when viewing this app. = Getting an overview of test results = The gadgets will take some time to render, as they run over a hundred tests covering many aspects of the !OpenSocial spec. Please be patient when first loading the page. After the gadget first renders, you'll see a message something like the following - the number of passed, failed, warn, and unverified tests will update dynamically as more tests are completed: {{{ Executing... Compliance Tests Directory Passed Failed Warnings Unverified Total 33 1 2 9 45 }}} === What do these states mean? === * *Pass* indicates the test performed exactly as designed. * *Fail* indicates the test failed in verification. * *Warning* indicates the test behaved as designed, but didn't get the exact message or value for an undefined area. e.g. Unexpected error message on failure * *Unverified* indicates the test didn't perform any check. e.g. The test exited because the feature is not supported, or the test is manual. * Total is the number of tests that should have run. When the test is complete, you'll see a summary box at top, followed by detail sections for each test suite. In the summary box, you will see output like the following: {{{ Finished Compliance Tests Directory Passed Failed Warnings Unverified Total 136 13 4 13 166 Generate XML report Priority Description Failures P0 You are not doing something stated in the spec. You are not spec compliant 4 Activity Test Suite :: ACT100 Activity Test Suite :: ACT102 Activity Test Suite :: ACT300 Opensocial spec 0.7 Definition :: DEFINE P1 Important bugs not exactly falling in the P0s. But have negative impact 3 Activity Test Suite :: ACT200 Activity Test Suite :: ACT203 Activity Test Suite :: ACT202 P2 Functional nice to have breaks. i.e Exceptions for undefined behavior 6 People/Person Requests Test Suite :: PPL801 People/Person Requests Test Suite :: PPL506 MakeRequest Test Suite :: MKRT013 gadgets.io.* TestSuite :: GIO102 gadgets.io.* TestSuite :: GIO103 gadgets.io.* TestSuite :: GIO104 }}} You'll see failures broken down into the three categories of test priority, p0, p1, and p2, with a count and list of failed test names. We'll explain how to get more detail on a particular failed test below. If you see no failures, your container passed all tests. Congratulations! = Diving in on single test failures = We'll use an example to illustrate how to debug a single test failure. In the output above, we saw a P0 test failure: "Activity Test Suite :: ACT100" This text indicates a failure of a test in the Activity Test Suite. Below the summary box, there are many boxes, one per test suite. One of them is labeled "Activity Test Suite". If there were p0 failures in this suite, the text will be red to indicate failures. It looks something like this: {{{ (Expand) Activity Test Suite Passed Failed Warnings Unverified Total 11 6 0 0 17 }}} To find the failure above, click the Expand button. This will expand the Activity Test Suite box to display all tests in the suite and their status. In our example, we're looking for test ACT100. Scroll down in the list of expanded tests, and you'll find one test box labeled [ACT100]. Inside this box, click the Expand button to get more detail on the particular test. The expanded box will look something like this: {{{ [ACT100] [P0 ]:: opensocial.requestCreateActivity using title: FAILED [1109 ms] Stable Test (Expand)(Run again) Description> Create activity using title, verify if fetching is supported. [ACT100.0] dataResponse.hadError(): FAILED: (got 'true'), expected 'undefined' }}} In this example, the test is creating an activity using a title, then verifying that a fetch of activities returns an activity with that title. The test is failing because calling hadError() on the response returns True unexpectedly. = Running specific tests in your container = Running the entire set of tests suites each time you make a change to your container and deploy a fix can be time consuming. After running the entire set of test suites once, you can select an individual test suite or test to run again to speed this cycle up. To run one suite of tests, first run all tests once. After the run completes, scroll down to the suite you'd like to run. Click the Expand All button to view details about this suite. Click the Re run suite button to run this suite of tests again, without running the other suites. If there is just one test in the suite, you can run that test as described below. To run just one test, first run all tests once. After the run completes, expand the test suite containing the test you'd like to run, then scroll down to the test and click the (Run again) button to rerun just this one test. To run one suite of tests as individual gadget you can select .xml file for that suite and load it separately. i.e. To run people tests you can pick .../suites/0.7/people/people.xml = Generating an XML report of test results = The gadget can generate a report of test results. This can be uploaded to the Compliance Matrix at http://opensocial-compliance.appspot.com, and used for tracking feature coverage. To generate this report, first install the app, then allow all tests to run to completion. Once the run is complete, click on the (Generate XML report) link at the top of the gadget UI. This will pop up a new window with an XML representation of the test suites, tests, and the result of running the tests in this container. The text from this popup window can be saved to a text file and easily translated into other formats. (Currently you must cut-and-paste the XML text, as it is dynamically generated) = Saving a baseline of results for later comparison = The gadget is capable of storing a baseline result from running the tests in a given container, then compare those results with a later run. To save a baseline, first install the app, then allow all tests to run to completion. Once the run is complete, scroll to the bottom of the gadget. You'll find a text area labeled "Results:". This should contain a JSON data representation of the results of the text. Click on the (Save these results as baseline) button. This will use the OpenSocial data store mechanism to save the output of this test to the container, for comparison with later runs. Next time the gadget is viewed, the results should appear in the Baseline Results box for comparison with the current run. = Current Test Suites = Here is the list of test suites that are currently covered by the 0.7 test gadget: * !OpenSocial spec 0.7 Definition This suite contains a single test to verify the existence of expected datastructures, constants, and APIs in the gadgets, and opensocial APIs. It does not call methods to verify they work, or validate the values of constants. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/compliance/reference compliance/reference]. * Activity Test Suite This suite tests opensocial APIs to create and request activities. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/activities activities]. * AppData Test Suite This suite tests single and batched update and fetch of opensocial PersonAppData. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/appdata appdata]. * Environment Test Suite This suite tests opensocial.Environment calls. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/environment environment]. * gadgets.io Test Suite This suite tests gadgets.io.encodeValues and gadgets.io.getProxyUrl with various inputs. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/gadgets/core gadgets/core]. * gadget.Prefs Test Suite This suite tests setting and retrieving individual gadget prefs, and container-level prefs. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/gadgets/core gadgets/core]. * gadgets.util Test Suite This suite tests escape and unescape of strings, getting features, parameters, url params, etc. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/gadgets/core gadgets/core]. * MakeRequest Test Suite This suite tests making request to fetch content of various mimetypes, both GET and POST, unsigned, and signed. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/makerequest makerequest]. * Miscellaneous This suite tests creating messages, requesting user permission, and collections. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/misc misc]. * People/Person Requests Test Suite This suite tests opensocial Person fields and methods. See tests for this suite at [http://code.google.com/p/opensocial-resources/source/browse/tests/trunk/suites/0.7/#0.7/people people]. = Current Test Cases = http://code.google.com/p/opensocial-resources/wiki/ComplianceTestsDirectory