Regression and unit tests: Difference between revisions
From LimeSurvey Manual
(→Linux) |
|||
Line 29: | Line 29: | ||
== Installation == | == Installation == | ||
=== Docker Setup === | |||
Install [https://www.docker.com/ Docker Desktop]. | |||
TODO | |||
=== Linux === | === Linux === | ||
Line 46: | Line 51: | ||
=== Windows === | === Windows === | ||
TODO | TODO | ||
Revision as of 13:04, 7 October 2022
limesurvey
database as well as the config.php
file. Always do a backup of important data before running the tests.
Since LimeSurvey 2.65.6 you have the possibility to add unit tests to LimeSurvey.
Since LImeSurvey 3.0.0, we use Selenium, headless Firefox (geckodriver) and Facebook web driver to test with the browser from PHPUnit.
Introduction
An extensive test suite will give the programmer a better sense of security, and allow him/her to make bigger re-factorizations without risking the soundness of the codebase. In an ideal world, a regression test is added every time a new bug is discovered. That way you make sure you never have to fix the same bug twice.
You can run the tests locally if you want, or you can setup a docker container that mimics the Travis CI environment. Alternatively, you can visit Travis CI after you've pushed to github to make sure your commits didn't break anything. If you think Travis is falsely reporting a bug, you should contact the person who constructed the test.
Requirements
- A working LimeSurvey development environment
- FireFox
- Selenium - cross-platform Java archive .jar (tool for authoring functional tests across most modern web browsers)
https://selenium-release.storage.googleapis.com/3.7/selenium-server-standalone-3.7.1.jar
- Gecko Driver - platform specific selenium browser driver for FireFox
https://github.com/mozilla/geckodriver/releases
Notes
PhpUnit in container
The current PhpUnit config expects Selenium to be running on the same machine as a PhpUnit process. If running the PhpUnit process from within a Docker container or a VM this might be problematic. It is possible to configure PhpUnit to connect to a selenium process on another machine. It may however be simpler to just run PhpUnit at the operating system level.
Installation
Docker Setup
Install Docker Desktop.
TODO
Linux
(All commands below are executed in the LimeSurvey web root folder.)
The browser tests uses Firefox, so you have to install it before running.
- A good idea is to clear the database and make a fresh install using the command-line (backup any data you want to save first):
DROP DATABASE limesurvey; CREATE DATABASE limesurvey
Make sure your config.php has the correct database credentials.$ DBENGINE=INNODB php application/commands/console.php install admin password TravisLS no@email.com verbose
- To enable unit testing, you have to issue the command
$ touch enabletests
- To install Facebook web driver and PHPUnit, run
$ composer install
Install node (needed for EM tests).Not used right now.- Download Selenium and the geckodriver:
$ wget "https://selenium-release.storage.googleapis.com/3.7/selenium-server-standalone-3.7.1.jar"
$ wget "https://github.com/mozilla/geckodriver/releases/download/v0.23.0/geckodriver-v0.23.0-linux64.tar.gz"
$ tar xvzf geckodriver-v0.23.0-linux64.tar.gz
- Make sure you have Java installed, and start Selenium in a separate terminal:If you want to use the headless browser, prepend with:
$ java -jar selenium-server-standalone-3.7.1.jar -enablePassThrough false
MOZ_HEADLESS=1
- Run tests with where "<local domain>" is replaced with your localhost domain, e.g. "localhost" or "localhost/limesurvey".
$ DOMAIN=<local domain> ./vendor/bin/phpunit
Windows
TODO
Mac
TODO
Usage
After all steps for the installation are completed, you should be able to write
$ PASSWORD=adminpassword DOMAIN=your_limesurvey_domain phpunit
in the LimeSurvey root folder to run the tests.
If LimeSurvey is installed on your web root, use "DOMAIN=localhost".
This is an example of the output:
$ phpunit
PHPUnit 5.6.2 by Sebastian Bergmann and contributors.
.... 4 / 4 (100%)
Time: 713 ms, Memory: 14.00MB
OK (4 tests, 24 assertions)
You can get some more information if you use the switch --testdox
:
$ phpunit --testdox
PHPUnit 5.6.2 by Sebastian Bergmann and contributors.
ls\tests\DateTimeForwardBack
[x] Q 1
ls\tests\DateTime
[x] Wrong input
[x] Correct date format
[x] Q 1
During testing, you might want to use --group <test group>
and/or --stop-on-failure
.
What can be tested?
Right now, there are tests for the expression manager, qanda and updatedb helper. The Selenium driver makes it also possible to test the full browser experience, including JavaScript.
What should be tested?
Everything that can be reported as a bug should be tested, but especially critical is of course the relation between user input and database, that is, that the correct answer is saved.
Adding a test
First of all, make sure to head over to the phpunit web site and read the introduction.
When done with that, I will assume you want to add a new test for a question type. Tests for questions are stored in folder tests/questions
.
You should create a minimal survey to reproduce your bug. Export it as an lss file and put it in tests/data/surveys
. Then create a class for your test. Name the file the same as the class, like "DateTimeForwardBackTest.php". The class name must end with "Test", so phpunit can find it.
Inside the class you might want to add code to import and delete the survey. I usually do this class-wide, which means that the survey gets imported when the class is loaded and then deleted when all tests in the class is done (whether failed or successfully).
The static methods setupBeforeClass
and teardownAfterClass
is used to setup and teardown the test fixture, respectively. You can read more about fixtures in the phpunit manual.
Let's list what we know so far:
namespace ls\tests;
use PHPUnit\Framework\TestCase;
/**
* @since 2017-06-16
* @author Olle Haerstedt
*/
class DateTimeDefaultAnswerExpressionTest extends \TestCase
{
public static function setupBeforeClass()
{
// Init stuff
}
public static function teardownAfterClass()
{
// Tear down what you setup before.
}
}
To test with Selenium and the Facebook web driver, check out the wiki page on github.
TODO
Github Actions
All tests are run automatically after each commit using Github Actions.