ExpressionScript Engine roadmap
From LimeSurvey Manual
This page is for tracking the development status and TODO lists for ExpressionManager
Status - Ready for 1.92RC1
As of revision 11661, Expression Manager is fully integrated into the 1.91+ code base.
There are no known bugs remaining to be addressed.
This will now be back-ported into main development branch so that we can release an official 1.92RC1.
This will also be forward-ported to the _CI branch so that it is available for the Yii conversion.
Features present in 1.91+ but DIFFERENT in 1.92 RC1
Mathemematical Comparisons against Empty (no response)
1.91+ and 1.92 give different answers for less-than / greater-than comparisons against empty values.
One of the LimeSurvey demo surveys uses a set of conditions that translates to this relevance equation: {(age < 16) or (age == 20) or ... or (age == 80)}. In LimeSurvey 1.91+, (age < 16) is FALSE when there is no answer (the value is blank). However, in LimeSurey 1.92, (age < 16) is TRUE when there is no answer, since both PHP and JavaScript treat blank as 0 in mathematical comparisons. Thus, 1.91+ would hide that question when age was unanswered, but 1.92 would show it.
We went to great pains to prevent this, but since we needed to have the Expressions generate identical results in PHP and JavaScript, there was no way to make 1.92 treat "" < 16 as FALSE. Fortunately, there is an easy work-around for this. If you want (age < 16) to be FALSE, then use this expression instead: {(!is_empty(age) and age < 16)}. You can use the new Survey Logic File view to quickly identify and fix any such comparisons in your survey.
Display of Conditions
In 1.91+, the list of conditionally related questions is shown in the following. In 1.92, the new approach is shown after '=>' for each entry
- Question view => now shows syntax-highligted relevance equation
- Question re-order view => now shows syntax-highligted relevance equation
- Group re-order view => now shows syntax-highligted relevance equation (but only at group-level)
- Printable Survey => missing - plan to show relevance and validation equations
- Data Entry => missing - plan to show relevance and validation equations
Features present in 1.91+ but MISSING in 1.92 RC1
Display conditions and validation rules in printable survey and data entry views
Solution:
- Instead of showing conditions, use LimeExpressionManager::GetQuestionStatus(gid) validation and relevance tips to show the business rules.
- Ensure that LimeExpressionManager:ValidateSurvey() is called after submission of Data Entry screen
Some Question-level Validation Rules don't show when you first view a page
But they do show if you fail the validation requirements.
Solution:
- Requires refactoring of qanda.php to use LimeExpressionManager::GetQuestionStatus($qid) to use the validation and madatory tips
Features of original 1.92dev MISSING from 1.92 RC1
List (dropdown)
- Random answer order
- Get order from previous question
- Category separator
List (radio)
- Random answer order
- Get order from previous question
List with comment
- Random answer order
- Get order from previous question
Array
- Random answer order
- Get order from previous question
Array (10 point choice)
- Random answer order
- Get order from previous question
Array (5 point choice)
- Random answer order
- Get order from previous question
Array (Increase/Same/Decrease
- Random answer order
- Get order from previous question
Array (Numbers)
- Random answer order
- Get order from previous question
Array (Texts)
- Random answer order
- Get order from previous question
Array (Yes/No/Uncertain)
- Random answer order
- Get order from previous question
Array (by column)
- Random answer order
- Get order from previous question
Array dual scale
- Random answer order
- Get order from previous question
Date/Time
- Month display style
- Minute step interval
- Date/Time format
Multiple numerical input
- Random answer order
- Get order from previous question
Ranking
- Random answer order
- Get order from previous question
Multiple short text
- Random answer order
- Get order from previous question
Multiple choice
- Random answer order
- Get order from previous question
- 'Other:' comment mandatory
Multiple choice with comments
- Random answer order
- Get order from previous question
BugFixes and Completed Wish List Items for 1.92 RC2 (as of revision 11958)
- Upgrade EM to support group-level relevance - revision 11674
- 1.92 does not highlight sub-questions unanswered or invalid sub-questions - it just flags the whole question-- Fixed in revision 11958
- Used to generate field-specific validation JavaScript - e.g. color code only the sub-question responses that violate the validation rules
Wish List for 1.92+
- Incorporate EM-related mandatory and validity checks into submit function (so can check on client-side instead of having to wait for server-side submission)
- EM already generates needed JavaScript to check validation
- Requires extension of LimeExpressionManager::GetRelevanceAndTailoringJavaScript() to utilize the validation JavaScript - e.g. to use JQuery to change the background color of input boxes to red if they fail validation
- Probably don't want to put that JavaScript in an onsubmit() function - since it could be helpful to let users submit the data they have on the page, even if some is invalid
- Remove need for call to ExprMgr_process_relevance_and_tailoring prior to document.getReady()
- Only needed because qanda not always inserting proper question/subquestion-level visibility - those values are available from EM - just need to be inserted in qanda.
- Refactor other parts of qanda.php
- Use GetQuestionStatus(qid) to get answer lists, help, etc - will eliminate the need to do SQL queries within qanda for some question types
- Enhance EM'support for array_filter
- If the filter is completely dependent upon questions on prior pages, then irrelevant filtered values don't need to be generated in the HTML
- Currently, all array_filter items are generated, but appropriately hidden. For large arrays, this can lead to a lot of never-to-be-used code
- Eliminate $_SESSION['fieldarray']
- Refactor group.php to use GetQuestionStatus(qid) instead of $_SESSION['fieldarray'] to display questions and answers
- Refactor return_timer_script() in qanda.php to use GetQuestionStatus(qid) instead of $_SESSION['fieldarray']
- Remove UpdateFieldArray()
- Eliminate $_SESSION['fieldarray'] from buildsurveysession()
- Upgrade EM ShowLogicFile function to show array filters applied to List question type
- Upgrade EM to support sub-question-level relevance (e.g. use relevance logic for array_filter)
- Javascript support already exists
- Update admin interface to allow custom relevance to be added to sub-questions (in addition to auto-computed relevance from array_filter)
- Modify ProcessAllNeededRelevance() to also compute and set $_SESSION['relevanceStatus'][SGQA] for each sub-question that has a relevance equation
- Eliminate buildsurveysession()?
- Appears to be a subset of what EM creates. Assess gap and remove it if possible.
- Eliminate createFieldMap()?
- Appears to be a subset of what EM creates. Assess gap and remove it if possible.
Wish List for beyond 1.92+
- GUI (using CodeMirror)
- Syntax highlighing lets users quickly debug equations
Historical Roadmap Notes
Completed ToDo and Wish List Items (as of revision 11661)
- Testing / Development
- If there are only Equations within a Group, and none are visible, ensure that the Equation values are saved to the database before moving to the next Group (thereby ensuring that user does not see a blank screen).
- Add LEM->NextRelevantSet(mode={question,group,survey},direction={forward,back}) - This will use relevance, hidden, and mandatory status to determine which are the next set of questions to show.
- If going forward, and there are any hidden but relevant equations, this function will save those values to the DB
- If going forward and there are any irrelevant questions, those values will be NULLed in the DB
- Add LEM->NextRelevantSet(mode={question,group,survey},direction={forward,back}) - This will use relevance, hidden, and mandatory status to determine which are the next set of questions to show.
- In deployed surveys, Mandatory is not being trumped by relevance - so insists that irrelevant questions be answered before continuing.
- Will be fixed by LEM->NextRelevantSet()
- Remove all back-end processing of Conditions and Mandatory. These will be processed by EM.
- Conditions are already convertable to Relevance
- Call LEM->UpgradeConditionsToRelevance() and LEM->UpgradeRelevanceAttributeToQuestion() as a database upgrade
- Make sure that changes to conditions via GUI are saved as Relevance (call UpgradeConditionsToRelevance(surveyID=sid,qid=qid) once conditions are saved)
- Mandatory will be dealt with by LEM->ValidateCurrentSet()
- Conditions are already convertable to Relevance
- Convert most validation to using EM
- Create LEM->ValidateCurrentSet() to compose validation from attributes as server- and client-side
- See below for details of how attributes convert to relevance and validation
- Some are relevance, but require sub-question level control of visibility - must modify LEM->ProcessRelevance to generate show/hidden controls for sub-questions (e.g. use id='tbdispXXXX' rather than id='displayXX')
- Validation ones can be composed into a single EM equation which can be tested on client and server-side.
- Integration into other survey styles
- question.php. Works, but shows irrelevant questions
- Use LEM->NextRelevantSet() to ensure get next relevant question
- dataentry.php - whole page could be dynamic, as is already true in survey_format.php
- Note, EM capabilities can not be fully supported in dataentry.php since it does not have the same div fields controlling question visibility. Instead, we could have have people use Survey_format.php. If people want the dataentry.php look-and-feel, we could use Survey_format.php as a template (since it works), and change the question-level views to look more like the dataentry.php style - that way the dynamic question visibility and tailoring would continue to work.
- Question Focus / Tab Order
- Make sure that TAB and SHIFT-TAB work as user expects, even if relevance causes form elements to appear and disappear
- Performance Tuning
- Ensure that EM-based replacements are properly cached
- EM-Enable other Question Attributes
- array_filter - which is done via JS
- array_filter_exclude
- code_filter
- Validation (min/max)
- Confirm EM does relacements for the following
- dualscale_headerA, dualscale_headerB
- prefix
- suffix
- Cross-Site Scripting Protection
- Ensure that if person enters a [script] block into a free-text field, then inserts that value into a subsequent question, that the [script] does not run
- Integration with Assessments
- Use the Assessment Value field for questions if there is one. Otherwise use the codeValue
- Caching
- Currently, EM maximizes caching. This might cause problems as questions are added/removed - needs to be tested and force a refresh from source if needed.
- BugFixes
- Default values being set in Question-by-Question mode, but not in Group or All-in-one modes
- JumpTo - must process validation of all intervening questions/groups
- Submit from Index - must process validation of all intervening questions/groups
- Submit from Index in Question-by-Question mode is only advancing one question rather than submitting
- Resume Later
- Shows navigation error after save instead of informational message
- Not re-loading data when click on emailed link
- Partially completed session data not reloaded
- Activate Survey - should validate that there are no EM-related errors
- New Syntax-Highlighted Logic File lets uses detect any syntax errors at Question, Group, or Survey levels before activating survey
- Prevent use of browser back-button to resubmit a prior page.
- Irrelevant mandatories not always being ignored (e.g. in AutoMEQ) -- was a problem in how PHP handles 0: {'NA' > 0} == true!
- Switch survey languages
Have EM control processing of many Question Attribute
array_filter: 1ABCEF:;MPL
If want to filter question Q2 on Q1, where each have subquestions SQ1-SQ3, this is equivalent to relevance equations of:
Question | Relevance | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Q2_SQ1 | Q1_SQ1!=~146<span style="color:146~ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Q2_SQ2 | Q1_SQ2!=~146146~ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Q2_SQ3 | Q1_SQ3!=~146
Challenges/Approach: Relevance is currently done at the question level, not subquestion level. Would need to:
array_filter_exclude: 1ABCEF:;MPLSimilar to array_filter:
code_filter"> WZTODO - how is this supposed to work? If the goal is to allow more complex relevance for sub-questions, most robust way would be to add relevance to sub-questions equals_num_value: KValidation:= sum(sq1,...,sqN) == value (which could be an expression). Creates a "Total" cell that shows the sum of the elements, and color-codes the background as green or red depending upon whether it passes validation Currently creates complex JavaScript which could be replaced. For example, add the following text to the question which would be dynamically tailored:
The same functions could be used to generate and alert(), or change the title/tooltip for the total box. exclude_all_others: MCan have multiple SQ codes (that must be attached to that question). If any of them are checked, then uncheck and visually disable the other inputs.
exclude_all_others_auto: MIf all except the exclude_all_others is checked, then uncheck/disable them all and check the exclude_all_others entry
max_answers: MPRValidation:= count(sq1,...,sqN) <= value (which would be an expression). TODO - what should happen when it passes/fails this validation? May not need EM for Ranking. TODO - check how M and P work. max_num_value: KValidation: sum(sq1,...,sqN) < value (which could be an expression) max_num_value_n: NValidation: var < value (which could be an expression) max_num_value_sgqa: KValidation: sum(sq1,...,sqN) < value (which could be an expression) min_answers: MPRValidation:= count(sq1,...,sqN) >= value (which would be an expression). TODO - what should happen when it passes/fails this validation? min_num_value: KValidation: sum(sq1,...,sqN) >= value (which could be an expression) min_num_value_n: NValidation: sum(sq1,...,sqN) >= value (which could be an expression) min_num_value_sgqa: KValidation: sum(sq1,...,sqN) >= value (which could be an expression) multiflexible_max: :Validation: sum(sq1,...,sqN) <= value (which could be an expression) TODO - is this correct, or does it sum by row/column/scale?? multiflexible_min: :Validation: sum(sq1,...,sqN) <= value (which could be an expression) TODO - is this correct, or does it sum by row/column/scale?? num_value_equals_sgqa: KValidation: sum(sq1,...,sqN) == value (which could be an expression) show_totals: : (rows, columns, both)Have rows/columns use {sum{v1,...,vN)} so EM will auto-generate replacements Or, just keep current, which uses multi_set() function. assessment_value: MPTODO - how should this work? Access to additional question and answer attributes
Performance TuningExpressionManager can be called dozens to hundreds of times per page. Instead, it should be possible to concatenate together the roughly tailored parts of the survey page and call Expression Manager once per page (or at least fewer times). To accomplish this, two things are needed:
For values that change multiple times per page (like {QUESTION}), I'd like to replace calls to templatereplace(), replaceTokens(), insertAnsReplace(), dTexts, and replaceFields() (several of which call ExpressionManager) as follows:
When are each of the following attributes changed?Here are the possibilities:
Where within LimeSurvey is each Keyword Used?A green 1 means that the keyword is used in that file at least once. Pre-combine all of the page parts prior to calling Expression Manager?CodeIgniter has the option to have calls to views return a string - like $result = $CI->load->view('xxxx',$data,TRUE). However, each of the views uses a mix of echo statements and composition of $strings that are then returned from the function.
|