ExpressionScript Engine roadmap

This page is for tracking the development status and TODO lists for ExpressionManager

Status - Ready for 1.92RC1

As of revision 11661, Expression Manager is fully integrated into the 1.91+ code base.

There are no known bugs remaining to be addressed.

This will now be back-ported into main development branch so that we can release an official 1.92RC1.

This will also be forward-ported to the _CI branch so that it is available for the Yii conversion.

Features present in 1.91+ but DIFFERENT in 1.92  RC1

Mathemematical Comparisons against Empty (no response)

1.91+ and 1.92 give different answers for less-than / greater-than comparisons against empty values.

One of the LimeSurvey demo surveys uses a set of conditions that translates to this relevance equation:  {(age < 16) or (age == 20) or ... or (age == 80)}. In LimeSurvey 1.91+, (age < 16) is FALSE when there is no answer (the value is blank).  However, in LimeSurey 1.92, (age < 16) is TRUE when there is no answer, since both PHP and JavaScript treat blank as 0 in mathematical comparisons.  Thus, 1.91+ would hide that question when age was unanswered, but 1.92 would show it.

We went to great pains to prevent this, but since we needed to have the Expressions generate identical results in PHP and JavaScript, there was no way to make 1.92 treat "" < 16 as FALSE.  Fortunately, there is an easy work-around for this.  If you want (age < 16) to be FALSE, then use this expression instead: {(!is_empty(age) and age < 16)}.  You can use the new Survey Logic File view to quickly identify and fix any such comparisons in your survey.

Display of Conditions

In 1.91+, the list of conditionally related questions is shown in the following.  In 1.92, the new approach is shown after '=>' for each entry

• Question view => now shows syntax-highligted relevance equation
• Question re-order view => now shows syntax-highligted relevance equation
• Group re-order view => now shows syntax-highligted relevance equation (but only at group-level)
• Printable Survey => missing - plan to show relevance and validation equations
• Data Entry => missing - plan to show relevance and validation equations

Features present in 1.91+ but MISSING in 1.92  RC1

Display conditions and validation rules in printable survey and data entry views

Solution:

• Instead of showing conditions, use LimeExpressionManager::GetQuestionStatus(gid) validation and relevance tips to show the business rules.
• Ensure that LimeExpressionManager:ValidateSurvey() is called after submission of Data Entry screen

Some Question-level Validation Rules don't show when you first view a page

But they do show if you fail the validation requirements.

Solution:

• Requires refactoring of qanda.php to use LimeExpressionManager::GetQuestionStatus($qid) to use the validation and madatory tips

Features of original 1.92dev MISSING from 1.92 RC1

List (dropdown)

1. Random answer order
2. Get order from previous question
3. Category separator

List (radio)

1. Random answer order
2. Get order from previous question

List with comment

1. Random answer order
2. Get order from previous question

Array

1. Random answer order
2. Get order from previous question

Array (10 point choice)

1. Random answer order
2. Get order from previous question

Array (5 point choice)

1. Random answer order
2. Get order from previous question

Array (Increase/Same/Decrease

1. Random answer order
2. Get order from previous question

Array (Numbers)

1. Random answer order
2. Get order from previous question

Array (Texts)

1. Random answer order
2. Get order from previous question

Array (Yes/No/Uncertain)

1. Random answer order
2. Get order from previous question

Array (by column)

1. Random answer order
2. Get order from previous question

Array dual scale

1. Random answer order
2. Get order from previous question

Date/Time

1. Month display style
2. Minute step interval
3. Date/Time format

Multiple numerical input

1. Random answer order
2. Get order from previous question

Ranking

1. Random answer order
2. Get order from previous question

Multiple short text

1. Random answer order
2. Get order from previous question

Multiple choice

1. Random answer order
2. Get order from previous question
3. 'Other:' comment mandatory

Multiple choice with comments

1. Random answer order
2. Get order from previous question

BugFixes and Completed Wish List Items for 1.92 RC2 (as of revision 11958)

1. Upgrade EM to support group-level relevance - revision 11674
2. 1.92 does not highlight sub-questions unanswered or invalid sub-questions - it just flags the whole question-- Fixed in revision 11958

• • Used to generate field-specific validation JavaScript - e.g. color code only the sub-question responses that violate the validation rules

Wish List for 1.92+

1. Incorporate EM-related mandatory and validity checks into submit function (so can check on client-side instead of having to wait for server-side submission)
   1. EM already generates needed JavaScript to check validation
   2. Requires extension of LimeExpressionManager::GetRelevanceAndTailoringJavaScript() to utilize the validation JavaScript - e.g. to use JQuery to change the background color of input boxes to red if they fail validation
   3. Probably don't want to put that JavaScript in an onsubmit() function - since it could be helpful to let users submit the data they have on the page, even if some is invalid
2. Remove need for call to ExprMgr_process_relevance_and_tailoring prior to document.getReady()
   1. Only needed because qanda not always inserting proper question/subquestion-level visibility - those values are available from EM - just need to be inserted in qanda.
3. Refactor other parts of qanda.php
   1. Use GetQuestionStatus(qid) to get answer lists, help, etc - will eliminate the need to do SQL queries within qanda for some question types
4. Enhance EM'support for array_filter
   1. If the filter is completely dependent upon questions on prior pages, then irrelevant filtered values don't need to be generated in the HTML
   2. Currently, all array_filter items are generated, but appropriately hidden.  For large arrays, this can lead to a lot of never-to-be-used code
5. Eliminate $_SESSION['fieldarray']
   1. Refactor group.php to use GetQuestionStatus(qid) instead of $_SESSION['fieldarray'] to display questions and answers
   2. Refactor return_timer_script() in qanda.php to use GetQuestionStatus(qid) instead of $_SESSION['fieldarray']
   3. Remove UpdateFieldArray()
   4. Eliminate $_SESSION['fieldarray'] from buildsurveysession()
6. Upgrade EM ShowLogicFile function to show array filters applied to List question type
7. Upgrade EM to support sub-question-level relevance (e.g. use relevance logic for array_filter)
   1. Javascript support already exists
   2. Update admin interface to allow custom relevance to be added to sub-questions (in addition to auto-computed relevance from array_filter)
   3. Modify ProcessAllNeededRelevance() to also compute and  set $_SESSION['relevanceStatus'][SGQA] for each sub-question that has a relevance equation
8. Eliminate buildsurveysession()?
   1. Appears to be a subset of what EM creates.  Assess gap and remove it if possible.
9. Eliminate createFieldMap()?
   1. Appears to be a subset of what EM creates.  Assess gap and remove it if possible.

Wish List for beyond 1.92+

1. GUI (using CodeMirror)

• • Syntax highlighing lets users quickly debug equations

Historical Roadmap  Notes

Completed ToDo and Wish List Items (as of revision 11661)

1. Testing / Development
   1. If there are only Equations within a Group, and none are visible, ensure that the Equation values are saved to the database before moving to the next Group (thereby ensuring that user does not see a blank screen).

• • • Add LEM->NextRelevantSet(mode={question,group,survey},direction={forward,back}) - This will use relevance, hidden, and mandatory status to determine which are the next set of questions to show.
      • If going forward, and there are any hidden but relevant equations, this function will save those values to the DB
      • If going forward and there are any irrelevant questions, those values will be NULLed in the DB

1. 1. In deployed surveys, Mandatory is not being trumped by relevance - so insists that irrelevant questions be answered before continuing.

• • • Will be fixed by LEM->NextRelevantSet()

1. 1. Remove all back-end processing of Conditions and Mandatory.  These will be processed by EM.

• • • Conditions are already convertable to Relevance
      • Call LEM->UpgradeConditionsToRelevance() and LEM->UpgradeRelevanceAttributeToQuestion() as a database upgrade
      • Make sure that changes to conditions via GUI are saved as Relevance (call UpgradeConditionsToRelevance(surveyID=sid,qid=qid) once conditions are saved)
    • Mandatory will be dealt with by LEM->ValidateCurrentSet()

1. 1. Convert most validation to using EM

• • • Create LEM->ValidateCurrentSet() to compose validation from attributes as server- and client-side
    • See below for details of how attributes convert to relevance and validation
      • Some are relevance, but require sub-question level control of visibility - must modify LEM->ProcessRelevance to generate show/hidden controls for sub-questions (e.g. use id='tbdispXXXX' rather than id='displayXX')
      • Validation ones can be composed into a single EM equation which can be tested on client and server-side.

1. Integration into other survey styles
   1. question.php.  Works, but shows irrelevant questions

• • • Use LEM->NextRelevantSet() to ensure get next relevant question

1. 1. dataentry.php - whole page could be dynamic, as is already true in survey_format.php
   2. Note, EM capabilities can  not be fully supported in dataentry.php since it does not have the same div fields controlling question visibility. Instead, we could have have people use Survey_format.php.  If people want the dataentry.php look-and-feel, we could use  Survey_format.php as a template (since it works), and change the question-level views to look more  like the dataentry.php style - that way the dynamic question visibility and tailoring would continue to work.
2. Question Focus / Tab Order
   1. Make sure that TAB and SHIFT-TAB work as user expects, even if relevance causes form elements to appear and disappear
3. Performance Tuning
   1. Ensure that EM-based replacements are properly cached
4. EM-Enable other Question Attributes
   1. array_filter - which is done via JS

• • • http://docs.limesurvey.org/Question+type+-+Array#Array_filter
    • http://bugs.limesurvey.org/view.php?id=5435

1. 1. array_filter_exclude
   2. code_filter
   3. Validation (min/max)
2. Confirm EM does relacements for the following
   1. dualscale_headerA, dualscale_headerB
   2. prefix
   3. suffix
3. Cross-Site Scripting Protection
   1. Ensure that if person enters a [script] block into a free-text field, then inserts that value into a subsequent question, that the [script] does not run
4. Integration with Assessments
   1. Use the Assessment Value field for questions if there is one.  Otherwise use the codeValue
5. Caching
   1. Currently, EM maximizes caching.  This might cause problems as questions are added/removed - needs to be tested and force a refresh from source if needed.
6. BugFixes
   1. Default values being set in Question-by-Question mode, but not in Group or All-in-one modes
   2. JumpTo - must process validation of all intervening questions/groups
   3. Submit from Index - must process validation of all intervening questions/groups
   4. Submit from Index in Question-by-Question mode is only advancing one question rather than submitting
   5. Resume Later
      1. Shows navigation error after save instead of informational message
      2. Not re-loading data when click on emailed link
   6. Partially completed session data not reloaded
   7. Activate Survey - should validate that there are no EM-related errors
      1. New Syntax-Highlighted Logic File lets uses detect any syntax errors at Question, Group, or Survey levels before activating survey
   8. Prevent use of browser back-button to resubmit a prior page.
   9. Irrelevant mandatories not always being ignored (e.g. in AutoMEQ) -- was a problem in how PHP handles 0:  {'NA' > 0} == true!
   10. Switch survey languages

Have EM control processing of many Question Attribute

array_filter: 1ABCEF:;MPL

If want to filter question Q2 on Q1, where each have subquestions SQ1-SQ3, this is equivalent to relevance equations of:

Question	Relevance
Q2_SQ1	Q1_SQ1!=~146<span style="color:146~
Q2_SQ2	Q1_SQ2!=~146146~
Q2_SQ3	Q1_SQ3!=~146 Move relevance to Question table. Would allow more complex relevance (like combinations of conditions) Would require a way to edit sub-questions - like existing, but adding column for relevance Would need view of current sub-questions so can syntax-highlight relevance to show any inaccuracies Invisibly create appropriate relevance equations Would not allow user over-rides - would be entirely behind-the-scenes Challenges/Approach: Relevance is currently done at the question level, not subquestion level.  Would need to: Have are relevance parser know about subbquestions, and show/hide tbdispSGQA instead of displayN Check whether all SQ are hidden - if so hide entire question Treat hidden SQ as  irrelevant (so that cascade relevance?) or just hidden? array_filter_exclude:  1ABCEF:;MPL Similar to array_filter: Question Relevance Q2_SQ1 Q1_SQ1==~146146~ Q2_SQ2 Q1_SQ2==~146<span style="color:146~ Q2_SQ3 Q1_SQ3==~146146~ code_filter">  WZ TODO - how is this supposed to work?  If the goal is to allow more complex relevance for sub-questions, most robust way would be to add relevance to sub-questions equals_num_value:  K Validation:= sum(sq1,...,sqN) == value (which could be an expression). Creates a "Total" cell that shows the sum of the elements, and color-codes the background as green or red depending upon whether it passes validation Currently creates complex JavaScript which could be replaced. For example, add the following text to the question which would be dynamically tailored: {if(value>max,implode(' ','Answer is invalid. The total of all entries should not add up to more than', max,' ')} {if(value<min,implode(' ','Answer is invalid. The total of all entries should add up to at least', min,' ')} The same functions could be used to generate and alert(), or change the title/tooltip for the total box. exclude_all_others:  M Can have multiple SQ codes (that must be attached to that question).  If any of them are checked, then uncheck and visually disable the other inputs. Should they also be made irrelevant so that questions dependent upon them are properly processed? Since they are unchecked, that may be enough. If made irrelevant, they would also be hidden, which may not be the desired behavior exclude_all_others_auto:  M If all except the exclude_all_others is checked, then uncheck/disable them all and check the exclude_all_others entry TODO:  How should this work if there are multiple values for exclude_all_others? max_answers:  MPR Validation:= count(sq1,...,sqN) <= value (which would be an expression). TODO - what should happen when it passes/fails this validation? May not need EM for Ranking.  TODO - check how M and P work. max_num_value:  K Validation:  sum(sq1,...,sqN) < value (which could be an expression) max_num_value_n:  N Validation:  var < value (which could be an expression) max_num_value_sgqa: K Validation:  sum(sq1,...,sqN) < value (which could be an expression) min_answers:  MPR Validation:= count(sq1,...,sqN) >= value (which would be an expression). TODO - what should happen when it passes/fails this validation? min_num_value:  K Validation:  sum(sq1,...,sqN) >= value (which could be an expression) min_num_value_n:  N Validation:  sum(sq1,...,sqN) >= value (which could be an expression) min_num_value_sgqa:  K Validation:  sum(sq1,...,sqN) >= value (which could be an expression) multiflexible_max:  : Validation:  sum(sq1,...,sqN) <= value (which could be an expression) TODO - is this correct, or does it sum by row/column/scale?? multiflexible_min:  : Validation:  sum(sq1,...,sqN) <= value (which could be an expression) TODO - is this correct, or does it sum by row/column/scale?? num_value_equals_sgqa:  K Validation:  sum(sq1,...,sqN) == value (which could be an expression) show_totals:  : (rows, columns, both) Have rows/columns use {sum{v1,...,vN)} so EM will auto-generate replacements Or, just keep current, which uses multi_set() function. assessment_value:  MP TODO - how should this work? Access to additional question and answer attributes Create JavaScript/PHP Arrays so that functions can do lookup of Question and Answer metadata Question sequence number code type text number of enumerated answer choices Answer code text sequence number (so know it is 4th in the enumerated list) Create JavaScipt Mapping Arrays to streamline on-page lookup Variable aliases to canonical variable name question code and INSERTANS:SGQA both map to the javaSGQA or answerSGQA JavaScript variable name Canonical variable name maps to an array of question attributes questionNum (from SGQA) sequenceNum code type text number of enumerated answer choices canonical AnswerId canonical SubQuestionId questionNum to array relevanceStatus displayStatus canonical AnswerId to ordered list of array of code text sequencenceNum Use dot notation to provide read-only access to attributes.  For example, say we have a question q1, we could have variables such as: q1 - returns the current value for q1 as stored in the database q1.num - the question number q1.type - either the single character that represents the type, or a short string that describes the type q1.text - the text of the question as parsed by EM (with substitions) q1.rawtext - the raw, unparsed text of the question q1.answerId - the answer identifier for the question q1.answerCount - the number of enumerated answer options Performance Tuning ExpressionManager can be called dozens to hundreds of times per page.  Instead, it should be possible to concatenate together the roughly tailored parts of the survey page and call Expression Manager once per page (or at least fewer times). To accomplish this, two things are needed: Ensure that all replacement values are correct when  called.  Some values like {QUESTION} change multiple times per page.  Others are constant across all pages. Find way to concatenate together all page components priort to calling ExpressionManager. For values that change multiple times per page (like {QUESTION}), I'd like to replace calls to templatereplace(), replaceTokens(), insertAnsReplace(), dTexts, and replaceFields() (several of which call ExpressionManager) as follows: Templates: (except for Group and Question) - return as is so they get processed at the end Group Template:  Replace {GROUP} with something like {GROUP_N} which ExpressionManager would then substitute at the end Question Template:  Replace {QUESTION}  and {ANSWER} with something like {QUESTION_N} and {ANSWER_N} which would also get replaced after the whole page was composed. When are each of the following attributes changed? Here are the possibilities: global - means that the value is global for the LimeSurvey instance (across all surveys and sessions survey - means each survey might have a different value session - means the value should be static across a given session page - means that the value on each page might differ group - means might differ for each group questions - means might differ for each question answer - means  might differ for each answer option no processing needed - means EM should only do a plain replace on these, no processing of the contents Attribute When Changed AID answer ANSWER answer ANSWERSCLEARED survey ASSESSMENTS page ASSESSMENT_CURRENT_TOTAL page ASSESSMENT_HEADING survey CHECKJAVASCRIPT survey - no processing needed CLEARALL survey CLOSEWINDOW survey COMPLETED session DATESTAMP page - no processing needed EXPIRY survey - no processing needed GID group GROUPDESCRIPTION group GROUPNAME group LANG page LANGUAGECHANGER page LOADERROR survey LOADFORM survey LOADHEADING survey LOADMESSAGE survey NAVIGATOR page NOSURVEYID survey - no processing needed NUMBEROFQUESTIONS survey PERCENTCOMPLETE page PRIVACY survey PRIVACYMESSAGE survey QID question QUESTION question QUESTIONHELP question QUESTIONHELPPLAINTEXT question QUESTION_CLASS question QUESTION_CODE question QUESTION_ESSENTIALS question QUESTION_FILE_VALID_MESSAGE question QUESTION_HELP question QUESTION_INPUT_ERROR_CLASS question QUESTION_MANDATORY question QUESTION_MAN_CLASS question QUESTION_MAN_MESSAGE question QUESTION_NUMBER question QUESTION_TEXT question QUESTION_VALID_MESSAGE question REGISTERERROR survey REGISTERFORM survey REGISTERMESSAGE1 survey REGISTERMESSAGE2 survey RESTART survey RETURNTOSURVEY survey SAVE survey SAVEALERT survey SAVEDID survey SAVEERROR survey SAVEFORM survey SAVEHEADING survey SAVEMESSAGE survey SGQ question SID survey SITENAME global SUBMITBUTTON survey SUBMITCOMPLETE survey SUBMITREVIEW survey SURVEYCONTACT survey SURVEYDESCRIPTION survey SURVEYFORMAT survey SURVEYLANGAGE survey SURVEYLIST survey SURVEYLISTHEADING survey SURVEYNAME survey TEMPLATECSS survey TEMPLATEURL survey THEREAREXQUESTIONS suvey TOKEN session - no processing needed URL survey WELCOME survey Where within LimeSurvey is each Keyword Used? A green 1 means that the keyword is used in that file at least once. Pre-combine all of the page parts prior to calling  Expression Manager? CodeIgniter has the option to have calls to views return a string - like $result = $CI->load->view('xxxx',$data,TRUE). However, each of the views uses a mix of echo statements and composition of $strings that are then returned from the function. Do we need to pick on output strategy (return $string vs. echo)? Will the echo statements be properly captured? Do echo vs. $string creation have different performance characteristics?  Some old PHP manual posts suggest that echo is less efficient.