ExpressionScript for developers

This wiki page is meant for the LimeSurvey development team and others wishing to contribute to LimeSurvey. It provides details about how to work with, test, and extend Expression Manager (EM).

Note, this document is currently incomplete.

Getting Started

The best way to get started with EM is to:

• Install LimeSurvey 1.92
• Load and play with the Expression Manager demos (located in /docs/demosurveys of the distribution)
• Navigate through all of the test cases
  • Select a survey, click Tools, then Expression Manager
• Read the documentation
  • Main user documentation for Expression Manager
  • Expression Manager How Tos

EM Source Code Organization and Purpose

ExpressionManager (EM)

Location

• Version 1.92: /classes/expressions/ExpressionManager.php
• Version 2.0: /application/helpers/expressions/em_core_helper.php

Purpose

Implements a recursive descent parser in PHP and JavaScript which let you create variables and securely expose a specified set of pre-defined functions.

Recursive Descent Parser (all variables and functions starting with RDP_)

• This is the core of EM, a compiler which builds and evaluates the parse tree for the expression
• Unless you are an expert in compiler theory, you should not touch the RDP_ code.

API Functions to the RDP

• RegisterFunctions - an interface to add external functions to EM
• sProcessStringContainingExpressions() - splits string on expressions, evaluates each, then joins together the parts
• ProcessBooleanExpression() - evaluates whether an equation (not surrounded by curly braces) is true
• Get*() - returns information about the most recently processed expressions

LimeExpressionManager (LEM)

• Location
  • Version 1.92: /classes/expressions/LimeExpressionManager.php
  • Version 2.0: /application/helpers/admin/expressions/em_manager_helper.php
• Purpose - this centralizes the integration of LimeSurvey with EM

Navigation Functions

These process the current responses, update the database, find the next relevant set of questions, and create the metadata needed to render those questions

• NavigateForwards()
• NavigateBackwards()
• JumpTo()
• GetLastMoveResult() -

Initialization Functions

• StartSurvey() - initializes the survey, setting all variable mappings
• setVariableAndTokenMappingsForExpressionManager() - post-processes CreateFieldMap() to create metadata needed for all variables and token values
• _CreateSubQLevelRelevanceAndValidationEquations() - processes advanced question attributes such as array_filter, min_answers, and min_value
• ProcessAllNeededRelevance() - computes the relevance results (and JavaScript) for all applicable questions
• _ProcessGroupRelevance() - computes the relevance of a group based upon its own relevance criteria and the status of the questions it contains.

Functions to Generate Tailored Content

• StartProcessingPage() - called at beginning of each page
• StartProcessingGroup() - called for each group on a page
• FinishProcessingGroup() - collects all tailoring for this group so can render appropriate JavaScript
• FinishProcessingPage() - completes the EM-related processing, and serializes LEM into $_SESSION['LEMsingleton']
• GetRelevanceAndTailoringJavaScript() - collects all relevance, validation, and tailoring logic and generates ExprMgr_process_relevance_and_tailoring() JavaScript function

Response Management

• ProcessCurrentResponses() - validates and processes the $_POSTed responses
• _UpdateValuesInDatabase() - saves values to database, including NULLing irrelevant values

Caching

LEM tries to avoid calling the Initialization functions each page, storing state in $_SESSION['LEMsingleton']

• SetDirtyFlag() - when called, tells LEM to force a rebuild of the Initialization functions (e.g. of admin has added/removed/updated any questions, groups, conditions, or defaults)
• SetSurveyId()- calls SetDirtyFlag if the user wants to work with a different survey from the one that is cached
• SetEMLanguage() - calls SetDirtyFlag() if the user starts to use a language different from the cached one

Validation Functions

• _ValidateSurvey() - validates the entire survey by calling _ValidateGroup()
• _ValidatGroup() - validates a group and returns its status by calling _ValidateQuestion() on all of its questions
• _ValidateQuestion() - validates a question, determining whether it is relevant, hidden, or fails any validation and/or mandatory criteria

EM-related JavaScript

• Location
  • Version 1.92: /classes/expressions/em_javascript.js
  • Version 2.0: /scripts/admin/expressions/em_javascript.js
• Purpose - contains all of the custom javascript needed for EM

JavaScript equivalents of PHP functions

About 20 functions from phpjs.org

Core Functions

• LEManyNA() - checks whether any of the variables are irrelevant
• LEMval() - retrieves the value for any variable, or its metadata (via the dot notation syntax)
• LEMsetTabIndexes() - ensures that the tab sequence will act as expected even if input elements change visibility.

EM-related Test Cases

• Location
  • Version 1.92: /classes/expressions/test/*.php
  • Version 2.0: /application/views/admin/expressions/*
• Purpose
  • Available Functions - runs EM::ShowAllowableFunctions to display functions and syntax from EM->RDP_ValidFunctions
  • String Splitter - runs EM::UnitTestStringSplitter() to show how it parses strings with curly braces
  • Tokenizer - runs EM::UnitTestTokenizer() to show how EM detects and categorizes tokens (e.g. variables, string, functions, operators)
  • Unit Tests of Isolated Expressions - runs EM::UnitTestEvaluator() for unit tests of each of Expression Manager's features (e.g. all operators and functions). Color coding shows whether any tests fail. Syntax highlighting shows cases where Expression Manager properly detects bad syntax.
  • Unit Tests of Expressions Within Strings - runs LEM::UnitTestProcessStringContainingExpressions() to show how Expression Manager can process strings containing one or more variable, token, or expression replacements surrounded by curly braces.
  • Unit Test Dynamic Relevance Processing - runs LEM::UnitTestRelevance() to show how questions and substitutions should dynamically change based upon values entered.
  • Preview Conversion of Conditions to Relevance - runs LEM::UnitTestConvertConditionsToRelevance() to show relevance equations for all conditions in the database, grouped by question id
  • Bulk Convert Conditions to Relevance - actually performs the conversion, saving the generated relevance equations (while retaining the original conditions)
  • Test Navigation - unit tests LEM::NavigateForwards() for the selected survey, generating the following information based upon the selected debugging options:
    • Detailed Timing - shows low-level timing information for each part of EM (e.g. to examine duration of database calls)
    • Validation Summary - shows one line per group and question, showing its relevance equation, and indicating whether it was irrelevant, hidden, mandatory and/or failed validation criteria. Also shows the generated SQL per navigation step to update the database
    • Validation Detail - shows extra details per question, including
      • Validation Tip, Equation, JavaScript equivalent of the Equation
      • Lists of sub-questions; which are relevant; and which are unanswered
      • List of array filters applied, by sub-question
    • Pretty Print Syntax - syntax highlights all of the equations so that you can see errors, and also click on variable names to jump to those questions and edit them.
  • Show Survey Logic File - generates the logic file which is available via the "QA" buttons in the admin console.

How EM Works

What is an Expression?

Anything surrounded by curly braces is an Expression, with two exceptions: If there is whitespace after the opening brace or before the closing brace, it is ignored.

• This is so that EM can ignore embedded JavaScript.
• So, if you have JavaScript that might be parsed by EM, make sure to add a space or newline after the opening brace.

1. Escaped curly braces are ignored (e.g. \{ and \})

Note that EM does support Expressions within strings. Moreover, Expressions can contain nested strings, but not nested expressions. So, the following red sections are valid Expressions and will cause substitions to occur within the containing strings.

• <img src="images/mine_{Q1}.png">
• <img src="images/mine_{if(Q1=="Y",'yes','no')}.png">
• <img src="images/mine_{if(Q1=="Y",'single quote with {nested braces}',"double quote with {nested braces}")}.png">

What does EM do with text containing expressions?

1. A regular expression divides the source line into STRING and EXPRESSION tokens
2. Each EXPRESSION is parsed by ExpressionManager, a recursive descent parser.
   1. If there are syntax errors, EM returns an HTML string that syntax-highlights the equation and puts red-lined boxes around syntax errors
   2. If there are no syntax errors, EM returns the result of evaluating the expression
3. EM re-joins the STRING and EM-evaluated EXPRESSION parts.
4. EM optionally appends the translation activity to structures used by GetRelevanceAndTailoringJavaScript()

How can we be sure that EM accurately parses the equations?

EM was originally written in 1999-2000 by Dr. Tom White (TMSWhite) for a different project (Dialogix) in Java, using JavaCC, an open source compiler compiler (parser generator). That Java-based project has been in production for over a decade, and has been fully vetted for unit and integration tests.

Since there is no production-grade parser generator for PHP and JavaScript (although Antlr is coming close), TMSWhite created a custom recursive descent parser for LimeSurvey. To ensure its accuracy, EM's logic is based upon the JavaCC source code for Dialogix. The JavaCC syntax mirrors the functionality needed for a recursive descent parser. JavaCC happens to build a state-based compiler, which is a little more efficient than a recusive descent parser. However, state-based compilers are impossible to read, understand, or expand without JavaCC-like source code, so it did not make sense to try to port the JavaCC output directly to PHP.

Futhermore, there are comprehensive unit and integration test suites for EM. These make it easy to validate the accuracy of the EM system. Each test suite includes dozens to hundreds of test cases, and it is trivial to add addition test cases.

How does the Recursive Descent Parser work?

EM must do the following:

1. Tokenize the expression - separating strings, words (variable names vs. functions), and punctuation; and categorizing the types of each.
2. Analyze the tokens to build a parse tree, checking for syntax errors along the way.
3. Return the result of evaluating the expression (using PHP) (or return syntax-highlighted HTML if there are syntax errors).
4. Create a safe JavaScript equivalent of the expression so that expressions can be dynamically re-computed client-side.
5. Determine which variables are used in each expression (so can make sure they are available client-side).

How does EM integrate into LimeSurvey?

The LimeExpressionManager (LEM) class manages the integration of EM into LimeSurvey. LEM must:

1. Initialize all of the variables needed by LimeSurvey (e.g. for TOKENS, INSERTANS, and templatereplace())
2. Know which Group and Question are being processed
3. Record the results and metadata about all of the text that LimeSurvey asks it to process
4. Output static HTML that reflects the results of that processing
5. Output JavaScript that lets those results be dynamically re-computed if values on the page change.

Extending EM

Adding Test Cases

Please do! The testing frameworks are solid. You just need to add more tests following the examples in the code.

Adding Functions

Functions are stored in LimeExpressionManager::amValidFunctions[]. Some existing examples are:

• 'abs' => array('abs', 'Math.abs', 'Absolute value', 'number abs(number)', 'http://www.php.net/manual/en/function.checkdate.php', 1),
• 'if' => array('exprmgr_if', 'LEMif', 'Excel-style if(test,result_if_true,result_if_false)', 'if(test,result_if_true,result_if_false)', , 3),
• 'max' => array('max', 'Math.max', 'Find highest value', 'number max(arg1, arg2, ... argN)', 'http://www.php.net/manual/en/function.max.php', -2),
• 'substr' => array('substr', 'substr', 'Return part of a string', 'string substr(string, start [, length])', 'http://www.php.net/manual/en/function.substr.php', 2,3),

The syntax for each function (func) is:

• 'func' => array(details)

The details which must be included in the array are:

1. PHP function name - this is the PHP function that will be called for that func.
2. JavaScript function name - this is the JavaScript function that will be called for that function
3. Meaning - this is a short description of what the function does
4. Syntax - this shows the valid syntax for the function
5. Reference - this is an optional URL showing more details about the syntax (e.g. a link to the PHP documentation)
6. Number of required arguments

• • Multiple values are allowed at the end of the array. For example, substr() above can take 2 or 3 arguments
  • Negative values mean that the function accepts a variable number of arguments
  • Negative values less than -1 mean that the function requires at least abs(N)-1 arguments (so -2 means it requires at least 1 argument)

If you add a function that does not exist in PHP, then add it to the body of em_core_helper.php

• FIXME: Eventually we should separate out such add-on functions into their own php file

If you add a function that does not exist in JavaScript, then add it to the body of em_javascript.js

Make sure to add Unit tests for new functions to UnitTestEvaluator

• Syntax is ExpectedResult~Expression, such as:
• 212~5 + max(1,(2+3),(4 + (5 + 6)),[[]],[[7 + 8) + 9),( (10 + 11), 12),(13 + (14 * 15) - 16) )

If your test case should return an error, use NULL as the expected value, like this:

• NULL~four * / seven
• NULL~(5 + 7) = 8