Regression and unit tests
From LimeSurvey Manual
Since LimeSurvey 2.65.6 you have the possibility to add unit tests to LimeSurvey.
Introduction
In an ideal world, a regression test is added every time a new bug is discovered. That way you make sure you never have to fix the same bug twice.
Installation
LInux
To enable unit testing, you have to issue the command
$ touch enabletests
in the LimeSurvey root folder.
Make sure you have Composer installed.
The test system uses phpunit and Selenium, which you need to install first with the following commands:
$ composer require --dev phpunit/phpunit
$ composer require --dev phpunit/phpunit-selenium
Go to the Selenium webpage and download the standalone server. Make sure you have the latest version of Java installed so you can run it. Then start the server with command:
$ java -jar selenium-server-standalone-<version>.jar -enablePassThrough false
Another way to run Selenium is with xvfb
. This way, you get rid of the annoying Firefox popup at each test run.
$ xvfb-run java -jar selenium-server-standalone-<version>.jar -enablePassThrough false
Windows
TODO
Usage
After all steps for the installation are completed, you should be able to write
$ SUBDOMAIN=your_limesurvey_subdomain ./vendor/bin/phpunit
in the LimeSurvey root folder to run the tests.
If LimeSurvey is installed on your web root, just skip the SUBDOMAIN
part.
This is an example of the output:
$ phpunit
PHPUnit 5.6.2 by Sebastian Bergmann and contributors.
.... 4 / 4 (100%)
Time: 713 ms, Memory: 14.00MB
OK (4 tests, 24 assertions)
You can get some more information if you use the switch --testdox
:
$ phpunit --testdox
PHPUnit 5.6.2 by Sebastian Bergmann and contributors.
ls\tests\DateTimeForwardBack
[x] Q 1
ls\tests\DateTime
[x] Wrong input
[x] Correct date format
[x] Q 1
What can be tested?
Right now, there are tests for the expression manager, qanda and updatedb helper. You cannot test Javascript yet, but it can be added.
What should be tested?
Everything that can be reported as a bug should be tested, but especially critical is of course the relation between user input and database, that is, that the correct answer is saved.
Adding a test
First of all, make sure to head over to the phpunit web site and read the introduction.
When done with that, I will assume you want to add a new test for a question type. Tests for questions are stored in folder tests/questions
.
You should create a minimal survey to reproduce your bug. Export it as an lss file and put it in tests/data/surveys
. Then create a class for your test. Name the file the same as the class, like "DateTimeForwardBackTest.php". The class name must end with "Test", so phpunit can find it.
Inside the class you might want to add code to import and delete the survey. I usually do this class-wide, which means that the survey gets imported when the class is loaded and then deleted when all tests in the class is done (whether failed or successfully).
The static methods setupBeforeClass
and teardownAfterClass
is used to setup and teardown the test fixture, respectively. You can read more about fixtures in the phpunit manual.
Let's list what we know so far:
namespace ls\tests;
use PHPUnit\Framework\TestCase;
/**
* @since 2017-06-16
* @author Olle Haerstedt
*/
class DateTimeDefaultAnswerExpressionTest extends \TestCase
{
public static function setupBeforeClass()
{
// Init stuff
}
public static function teardownAfterClass()
{
// Tear down what you setup before.
}
}
TODO