ExpressionScript Engine roadmap

This page is for tracking the development status and TODO lists for ExpressionManager

Status - Ready for 1.92RC3

As of revision 12142, Expression Manager is fully integrated into the 1.91+ code base.

There are no known bugs from 1.92RC2 remaining to be addressed.

Features present in 1.91+ but DIFFERENT in 1.92  RCx

Mathemematical Comparisons against Empty (no response)

1.91+ and 1.92 give different answers for less-than / greater-than comparisons against empty values.

One of the LimeSurvey demo surveys uses a set of conditions that translates to this relevance equation:  {(age < 16) or (age == 20) or ... or (age == 80)}. In LimeSurvey 1.91+, (age < 16) is FALSE when there is no answer (the value is blank).  However, in LimeSurey 1.92, (age < 16) is TRUE when there is no answer, since both PHP and JavaScript treat blank as 0 in mathematical comparisons.  Thus, 1.91+ would hide that question when age was unanswered, but 1.92 would show it.

We went to great pains to prevent this, but since we needed to have the Expressions generate identical results in PHP and JavaScript, there was no way to make 1.92 treat "" < 16 as FALSE.  Fortunately, there is an easy work-around for this.  If you want (age < 16) to be FALSE, then use this expression instead: {(!is_empty(age) and age < 16)}.  You can use the new Survey Logic File view to quickly identify and fix any such comparisons in your survey.

Display of Conditions

In 1.91+, the list of conditionally related questions is shown in the following.  In 1.92, the new approach is shown after '=>' for each entry

• Question view => now shows syntax-highligted relevance equation
• Question re-order view => now shows syntax-highligted relevance equation
• Group re-order view => now shows syntax-highligted relevance equation (but only at group-level)
• Printable Survey => now shows relevance and validation equations
• Data Entry => now shows relevance and validation equations

Features present in 1.91+ but MISSING in 1.92 RC

There are no reported missing features.

BugFixes and Completed Wish List Items for 1.92 RC3 (as of revision 12151)

1. Remove need for call to ExprMgr_process_relevance_and_tailoring prior to document.getReady() -  Fixed in revision 12117
2. Upgrade EM to support sub-question-level relevance (e.g. use relevance logic for array_filter) - Fixed in revision 12117
3. New Feature question_attribute 'em_validation_q': "Boolean equation to validate the whole question" - Revision: 12142
4. New Feature question_attribute 'em_validation_q_tip': "Tip to show user describing the Question Validation Equation" - Revision: 12142
5. New Feature question_attribute 'em_validation_sq': "Boolean equation to validate each sub-question." - Revision: 12142
6. New Feature question_attribute 'em_validation_sq_tip': "Tip to show user describing the Sub-Question Validation Equation." - Revision: 12142
7. New Feature EM reserved word "this" which refers the the current sgqa code - Revision: 12142
8. Fixed issue#05450: Question code can be empty when a space is entered - Revision: 12136
9. Fixed issue #05673: Wrong sum value with min/max_num_value in multiple numeric when being array_filtered - Revision 12117
10. Fixed issue #05674: Question preview does not include any javascript - Revision: 12106
11. Fixed issue #05671: validation tips only appear when there are errors, and are not color-coded - Revision: 12105
12. Updated feature array_filter - now also available for multiple_numeric and multiple_short_text - Revision: 12099
13. Updated feature exclude_all_others - now also available for multiple_choice_with_comments - Revision: 12099
14. Updated feature min_num_value_n - now also available for multiple_numeric - Revision: 12099
15. Updated feature max_num_value_n - now also available for multiple_numeric - Revision: 12099
16. Updated feature min_answers - now also available for all array types, multiple_numeric and multiple_short_text - Revision: 12099
17. Updated feature max_answers - now also available for all array types, multiple_numeric and multiple_short_text - Revision: 12099
18. Fixed Issue #05670: Blank answers in non-mandatory questions prevent forward movement if there are validation rules - Revision: 12098
19. Fixed Issue Array Filters and tailoring sometimes not working in Question-by-Question mode - Revision: 12096
20. Fixed Issue Question Preview errors - Help value is wrong; notice of undefined indexes aid and sqid
21. Fixed Issue #05669: Fatal Error: Undefined function set_answer_time() - Revision: 12094
22. Fixed Issue Show Logic File button does not show assessment values when called from admin menus - Revision: 12082
23. Fixed Issue #05661: Survey without welcome page cannot show first question group/page correctly - Revision: 12033
24. Fixed Issue #05660: Warning: usort() [function.usort]: Invalid comparison function - Revision: 12017
25. Fixed Issue Validation messages too verbose - Revision 12151
26. Fixed issue Print Answers showing some irrelevant questions - Revision 12162

BugFixes and Completed Wish List Items for 1.92 RC2 (as of revision 11976)

1. Upgrade EM to support group-level relevance - Fixed in revision 11674
2. 1.92 does not highlight sub-questions unanswered or invalid sub-questions - it just flags the whole question - Fixed in revision 11958

• • Used to generate field-specific validation JavaScript - e.g. color code only the sub-question responses that violate the validation rules

1. Some Question-level Validation Rules don't show when you first view a page - Fixed in revision 11965
2. Incorporate EM-related mandatory and validity checks into submit function (so can check on client-side instead of having to wait for server-side submission) - Fixed in revision 11965
3. Display conditions and validation rules in printable survey and data entry views - Fixed in revision 11976

Wish List for 1.92+

1. Add support for "this"
   1. Should let you access this, this.NAOK, this.shown, etc. and show as valid within relevance equations
   2. Should provide access to all array values too - so create several:  this, this_nc (no comments), this_oc  (only comments)
   3. Should be expanded by EM during JavaScript generation to be comma separated values
   4. So, if you want to sum all values for an array, you'd say sum(this.NAOK)
   5. Macro expansion should occur in group.php as  cycles through each question
   6. Value should be forcibly cleared at start of each group/page otherwise use of this in a group description would have unpredictable results
2. Add aliases for common array processing values
   1. qcode_vars - acts like this, so can do sum(qcode_vars.NAOK), count(qcode_vars.NAOK), implode(', ', qcode_vars.shown), stddev(qcode_vars.NAOK), etc.
   2. qcode_vars_nc (no comments), qcode_vars_oc (only comments)
3. Add aliases needed to do row and column sums within arrays yet still properly handle array_filter
   1. qcode_rowname_vars
   2. qcode_colname_vars
4. Get exclude_all_others_auto working
5. Get variable assignment working (=)
   1. jsVarName.value = jsInputName.value = <result of evaluating expression>
6. Add email() function to EM
7. Refactor other parts of qanda.php
   1. Use GetQuestionStatus(qid) to get answer lists, help, etc - will eliminate the need to do SQL queries within qanda for some question types
8. Enhance EM'support for array_filter
   1. If the filter is completely dependent upon questions on prior pages, then irrelevant filtered values don't need to be generated in the HTML
   2. Currently, all array_filter items are generated, but appropriately hidden.  For large arrays, this can lead to a lot of never-to-be-used code
9. Eliminate $_SESSION['fieldarray']
   1. Refactor group.php to use GetQuestionStatus(qid) instead of $_SESSION['fieldarray'] to display questions and answers
   2. Refactor return_timer_script() in qanda.php to use GetQuestionStatus(qid) instead of $_SESSION['fieldarray']
   3. Remove UpdateFieldArray()
   4. Eliminate $_SESSION['fieldarray'] from buildsurveysession()
10. Upgrade EM ShowLogicFile function to show array filters applied to List question type
11. Upgrade EM to support sub-question-level relevance (e.g. use relevance logic for array_filter)
    1. Update admin interface to allow custom relevance to be added to sub-questions (in addition to auto-computed relevance from array_filter)
12. Eliminate buildsurveysession()?
    1. Appears to be a subset of what EM creates.  Assess gap and remove it if possible.
13. Eliminate createFieldMap()?
    1. Appears to be a subset of what EM creates.  Assess gap and remove it if possible.

Wish List for beyond 1.92+

1. GUI (using CodeMirror)

• • Syntax highlighing lets users quickly debug equations

Historical Roadmap  Notes

Features of original 1.92dev MISSING from 1.92 RC

1. List (dropdown)
   1. Random answer order
   2. Get order from previous question
   3. Category separator
2. List (radio)
   1. Random answer order
   2. Get order from previous question
3. List with comment
   1. Random answer order
   2. Get order from previous question
4. Array
   1. Random answer order
   2. Get order from previous question
5. Array (10 point choice)
   1. Random answer order
   2. Get order from previous question
6. Array (5 point choice)
   1. Random answer order
   2. Get order from previous question
7. Array (Increase/Same/Decrease
   1. Random answer order
   2. Get order from previous question
8. Array (Numbers)
   1. Random answer order
   2. Get order from previous question
9. Array (Texts)
   1. Random answer order
   2. Get order from previous question
10. Array (Yes/No/Uncertain)
    1. Random answer order
    2. Get order from previous question
11. Array (by column)
    1. Random answer order
    2. Get order from previous question
12. Array dual scale
    1. Random answer order
    2. Get order from previous question
13. Date/Time
    1. Month display style
    2. Minute step interval
    3. Date/Time format
14. Multiple numerical input
    1. Random answer order
    2. Get order from previous question
15. Ranking
    1. Random answer order
    2. Get order from previous question
16. Multiple short text
    1. Random answer order
    2. Get order from previous question
17. Multiple choice
    1. Random answer order
    2. Get order from previous question
    3. 'Other:' comment mandatory
18. Multiple choice with comments
    1. Random answer order
    2. Get order from previous question

Completed ToDo and Wish List Items (as of revision 11661)

1. Testing / Development
   1. If there are only Equations within a Group, and none are visible, ensure that the Equation values are saved to the database before moving to the next Group (thereby ensuring that user does not see a blank screen).

• • • Add LEM->NextRelevantSet(mode={question,group,survey},direction={forward,back}) - This will use relevance, hidden, and mandatory status to determine which are the next set of questions to show.
      • If going forward, and there are any hidden but relevant equations, this function will save those values to the DB
      • If going forward and there are any irrelevant questions, those values will be NULLed in the DB

1. 1. In deployed surveys, Mandatory is not being trumped by relevance - so insists that irrelevant questions be answered before continuing.

• • • Will be fixed by LEM->NextRelevantSet()

1. 1. Remove all back-end processing of Conditions and Mandatory.  These will be processed by EM.

• • • Conditions are already convertable to Relevance
      • Call LEM->UpgradeConditionsToRelevance() and LEM->UpgradeRelevanceAttributeToQuestion() as a database upgrade
      • Make sure that changes to conditions via GUI are saved as Relevance (call UpgradeConditionsToRelevance(surveyID=sid,qid=qid) once conditions are saved)
    • Mandatory will be dealt with by LEM->ValidateCurrentSet()

1. 1. Convert most validation to using EM

• • • Create LEM->ValidateCurrentSet() to compose validation from attributes as server- and client-side
    • See below for details of how attributes convert to relevance and validation
      • Some are relevance, but require sub-question level control of visibility - must modify LEM->ProcessRelevance to generate show/hidden controls for sub-questions (e.g. use id='tbdispXXXX' rather than id='displayXX')
      • Validation ones can be composed into a single EM equation which can be tested on client and server-side.

1. Integration into other survey styles
   1. question.php.  Works, but shows irrelevant questions

• • • Use LEM->NextRelevantSet() to ensure get next relevant question

1. 1. dataentry.php - whole page could be dynamic, as is already true in survey_format.php
   2. Note, EM capabilities can  not be fully supported in dataentry.php since it does not have the same div fields controlling question visibility. Instead, we could have have people use Survey_format.php.  If people want the dataentry.php look-and-feel, we could use  Survey_format.php as a template (since it works), and change the question-level views to look more  like the dataentry.php style - that way the dynamic question visibility and tailoring would continue to work.
2. Question Focus / Tab Order
   1. Make sure that TAB and SHIFT-TAB work as user expects, even if relevance causes form elements to appear and disappear
3. Performance Tuning
   1. Ensure that EM-based replacements are properly cached
4. EM-Enable other Question Attributes
   1. array_filter - which is done via JS

• • • http://docs.limesurvey.org/Question+type+-+Array#Array_filter
    • http://bugs.limesurvey.org/view.php?id=5435

1. 1. array_filter_exclude
   2. code_filter
   3. Validation (min/max)
2. Confirm EM does relacements for the following
   1. dualscale_headerA, dualscale_headerB
   2. prefix
   3. suffix
3. Cross-Site Scripting Protection
   1. Ensure that if person enters a [script] block into a free-text field, then inserts that value into a subsequent question, that the [script] does not run
4. Integration with Assessments
   1. Use the Assessment Value field for questions if there is one.  Otherwise use the codeValue
5. Caching
   1. Currently, EM maximizes caching.  This might cause problems as questions are added/removed - needs to be tested and force a refresh from source if needed.
6. BugFixes
   1. Default values being set in Question-by-Question mode, but not in Group or All-in-one modes
   2. JumpTo - must process validation of all intervening questions/groups
   3. Submit from Index - must process validation of all intervening questions/groups
   4. Submit from Index in Question-by-Question mode is only advancing one question rather than submitting
   5. Resume Later
      1. Shows navigation error after save instead of informational message
      2. Not re-loading data when click on emailed link
   6. Partially completed session data not reloaded
   7. Activate Survey - should validate that there are no EM-related errors
      1. New Syntax-Highlighted Logic File lets uses detect any syntax errors at Question, Group, or Survey levels before activating survey
   8. Prevent use of browser back-button to resubmit a prior page.
   9. Irrelevant mandatories not always being ignored (e.g. in AutoMEQ) -- was a problem in how PHP handles 0:  {'NA' > 0} == true!
   10. Switch survey languages

Have EM control processing of many Question Attribute

array_filter: 1ABCEF:;MPL

If want to filter question Q2 on Q1, where each have subquestions SQ1-SQ3, this is equivalent to relevance equations of:

Question	Relevance
Q2_SQ1	Q1_SQ1!=~146<span style="color:146~
Q2_SQ2	Q1_SQ2!=~146146~
Q2_SQ3	Q1_SQ3!=~146 Move relevance to Question table. Would allow more complex relevance (like combinations of conditions) Would require a way to edit sub-questions - like existing, but adding column for relevance Would need view of current sub-questions so can syntax-highlight relevance to show any inaccuracies Invisibly create appropriate relevance equations Would not allow user over-rides - would be entirely behind-the-scenes Challenges/Approach: Relevance is currently done at the question level, not subquestion level.  Would need to: Have are relevance parser know about subbquestions, and show/hide tbdispSGQA instead of displayN Check whether all SQ are hidden - if so hide entire question Treat hidden SQ as  irrelevant (so that cascade relevance?) or just hidden? array_filter_exclude:  1ABCEF:;MPL Similar to array_filter: Question Relevance Q2_SQ1 Q1_SQ1==~146146~ Q2_SQ2 Q1_SQ2==~146<span style="color:146~ Q2_SQ3 Q1_SQ3==~146146~ code_filter">  WZ TODO - how is this supposed to work?  If the goal is to allow more complex relevance for sub-questions, most robust way would be to add relevance to sub-questions equals_num_value:  K Validation:= sum(sq1,...,sqN) == value (which could be an expression). Creates a "Total" cell that shows the sum of the elements, and color-codes the background as green or red depending upon whether it passes validation Currently creates complex JavaScript which could be replaced. For example, add the following text to the question which would be dynamically tailored: {if(value>max,implode(' ','Answer is invalid. The total of all entries should not add up to more than', max,' ')} {if(value<min,implode(' ','Answer is invalid. The total of all entries should add up to at least', min,' ')} The same functions could be used to generate and alert(), or change the title/tooltip for the total box. exclude_all_others:  M Can have multiple SQ codes (that must be attached to that question).  If any of them are checked, then uncheck and visually disable the other inputs. Should they also be made irrelevant so that questions dependent upon them are properly processed? Since they are unchecked, that may be enough. If made irrelevant, they would also be hidden, which may not be the desired behavior exclude_all_others_auto:  M If all except the exclude_all_others is checked, then uncheck/disable them all and check the exclude_all_others entry TODO:  How should this work if there are multiple values for exclude_all_others? max_answers:  MPR Validation:= count(sq1,...,sqN) <= value (which would be an expression). TODO - what should happen when it passes/fails this validation? May not need EM for Ranking.  TODO - check how M and P work. max_num_value:  K Validation:  sum(sq1,...,sqN) < value (which could be an expression) max_num_value_n:  N Validation:  var < value (which could be an expression) max_num_value_sgqa: K Validation:  sum(sq1,...,sqN) < value (which could be an expression) min_answers:  MPR Validation:= count(sq1,...,sqN) >= value (which would be an expression). TODO - what should happen when it passes/fails this validation? min_num_value:  K Validation:  sum(sq1,...,sqN) >= value (which could be an expression) min_num_value_n:  N Validation:  sum(sq1,...,sqN) >= value (which could be an expression) min_num_value_sgqa:  K Validation:  sum(sq1,...,sqN) >= value (which could be an expression) multiflexible_max:  : Validation:  sum(sq1,...,sqN) <= value (which could be an expression) TODO - is this correct, or does it sum by row/column/scale?? multiflexible_min:  : Validation:  sum(sq1,...,sqN) <= value (which could be an expression) TODO - is this correct, or does it sum by row/column/scale?? num_value_equals_sgqa:  K Validation:  sum(sq1,...,sqN) == value (which could be an expression) show_totals:  : (rows, columns, both) Have rows/columns use {sum{v1,...,vN)} so EM will auto-generate replacements Or, just keep current, which uses multi_set() function. assessment_value:  MP TODO - how should this work? Access to additional question and answer attributes Create JavaScript/PHP Arrays so that functions can do lookup of Question and Answer metadata Question sequence number code type text number of enumerated answer choices Answer code text sequence number (so know it is 4th in the enumerated list) Create JavaScipt Mapping Arrays to streamline on-page lookup Variable aliases to canonical variable name question code and INSERTANS:SGQA both map to the javaSGQA or answerSGQA JavaScript variable name Canonical variable name maps to an array of question attributes questionNum (from SGQA) sequenceNum code type text number of enumerated answer choices canonical AnswerId canonical SubQuestionId questionNum to array relevanceStatus displayStatus canonical AnswerId to ordered list of array of code text sequencenceNum Use dot notation to provide read-only access to attributes.  For example, say we have a question q1, we could have variables such as: q1 - returns the current value for q1 as stored in the database q1.num - the question number q1.type - either the single character that represents the type, or a short string that describes the type q1.text - the text of the question as parsed by EM (with substitions) q1.rawtext - the raw, unparsed text of the question q1.answerId - the answer identifier for the question q1.answerCount - the number of enumerated answer options Performance Tuning ExpressionManager can be called dozens to hundreds of times per page.  Instead, it should be possible to concatenate together the roughly tailored parts of the survey page and call Expression Manager once per page (or at least fewer times). To accomplish this, two things are needed: Ensure that all replacement values are correct when  called.  Some values like {QUESTION} change multiple times per page.  Others are constant across all pages. Find way to concatenate together all page components priort to calling ExpressionManager. For values that change multiple times per page (like {QUESTION}), I'd like to replace calls to templatereplace(), replaceTokens(), insertAnsReplace(), dTexts, and replaceFields() (several of which call ExpressionManager) as follows: Templates: (except for Group and Question) - return as is so they get processed at the end Group Template:  Replace {GROUP} with something like {GROUP_N} which ExpressionManager would then substitute at the end Question Template:  Replace {QUESTION}  and {ANSWER} with something like {QUESTION_N} and {ANSWER_N} which would also get replaced after the whole page was composed. When are each of the following attributes changed? Here are the possibilities: global - means that the value is global for the LimeSurvey instance (across all surveys and sessions survey - means each survey might have a different value session - means the value should be static across a given session page - means that the value on each page might differ group - means might differ for each group questions - means might differ for each question answer - means  might differ for each answer option no processing needed - means EM should only do a plain replace on these, no processing of the contents Attribute When Changed AID answer ANSWER answer ANSWERSCLEARED survey ASSESSMENTS page ASSESSMENT_CURRENT_TOTAL page ASSESSMENT_HEADING survey CHECKJAVASCRIPT survey - no processing needed CLEARALL survey CLOSEWINDOW survey COMPLETED session DATESTAMP page - no processing needed EXPIRY survey - no processing needed GID group GROUPDESCRIPTION group GROUPNAME group LANG page LANGUAGECHANGER page LOADERROR survey LOADFORM survey LOADHEADING survey LOADMESSAGE survey NAVIGATOR page NOSURVEYID survey - no processing needed NUMBEROFQUESTIONS survey PERCENTCOMPLETE page PRIVACY survey PRIVACYMESSAGE survey QID question QUESTION question QUESTIONHELP question QUESTIONHELPPLAINTEXT question QUESTION_CLASS question QUESTION_CODE question QUESTION_ESSENTIALS question QUESTION_FILE_VALID_MESSAGE question QUESTION_HELP question QUESTION_INPUT_ERROR_CLASS question QUESTION_MANDATORY question QUESTION_MAN_CLASS question QUESTION_MAN_MESSAGE question QUESTION_NUMBER question QUESTION_TEXT question QUESTION_VALID_MESSAGE question REGISTERERROR survey REGISTERFORM survey REGISTERMESSAGE1 survey REGISTERMESSAGE2 survey RESTART survey RETURNTOSURVEY survey SAVE survey SAVEALERT survey SAVEDID survey SAVEERROR survey SAVEFORM survey SAVEHEADING survey SAVEMESSAGE survey SGQ question SID survey SITENAME global SUBMITBUTTON survey SUBMITCOMPLETE survey SUBMITREVIEW survey SURVEYCONTACT survey SURVEYDESCRIPTION survey SURVEYFORMAT survey SURVEYLANGAGE survey SURVEYLIST survey SURVEYLISTHEADING survey SURVEYNAME survey TEMPLATECSS survey TEMPLATEURL survey THEREAREXQUESTIONS suvey TOKEN session - no processing needed URL survey WELCOME survey Where within LimeSurvey is each Keyword Used? A green 1 means that the keyword is used in that file at least once. Pre-combine all of the page parts prior to calling  Expression Manager? CodeIgniter has the option to have calls to views return a string - like $result = $CI->load->view('xxxx',$data,TRUE). However, each of the views uses a mix of echo statements and composition of $strings that are then returned from the function. Do we need to pick on output strategy (return $string vs. echo)? Will the echo statements be properly captured? Do echo vs. $string creation have different performance characteristics?  Some old PHP manual posts suggest that echo is less efficient.