Deliverable D8.4 Demonstrators assessment

Transcript

FP7ICTSTREPProject
DeliverableD8.4
Demonstratorsassessment
http://www.learnpad.eu
Templatev.0.3
ProjectNumber
:
FP7-619583
ProjectTitle
:
LearnPAd
ModelBasedSocialLearningforPublicAdministrations
DeliverableNumber
:
D8.4
TitleofDeliverable
:
Demonstratorsassessment
NatureofDeliverable
:
Report
Disseminationlevel
:
Public
Licence
:
CreativeCommonsAttribution3.0License
Version
:
4.0
ContractualDeliveryDate
:
31October2016
ActualDeliveryDate
:
31October2016
ContributingWP
:
WP8
Editor(s)
:
AndreaSergiacomi(MAR)
Author(s)
:
CinziaAmici(MAR),AlessiaBalducci(MAR),SerenaCarota
(MAR),LauraGiorgio(MAR),BarbaraRe(UniCam),Andrea
Sergiacomi(MAR)
Reviewer(s)
:
AntoniaBertolino(CNR),GuglielmoDeAngelis(CNR),Andrea
Polini(Unicam),BarbaraRe(Unicam)
Abstract
Work Package 8, titled Demonstrators, aimed at providing support to the technical and research workpackages for better focusing on their activities, and at assessing the applicability, acceptance and
effectiveness of the proposed solutions. An early validation was carried out considering the first running
instance of the Learn PAd platform available (since 06 November 2015). A group of users, sampled from
both project demonstrators (EPBR and SUAP), tested the Learn PAd platform and provided us with
feedbackandcommentstoimproveit.Fromamethodologicalperspective,intheearlyvalidationweruna
questionnaire and collected technical and detailed opinions from a focus group with influential people,
skilledincomputerscienceorineducation.
Inthisdeliverablewepresenttheaims,settingandresultsofthefinalvalidationrunonthelatestrunning
instanceoftheLearnPAdplatformavailable,whichincludesalltheLearnPAddefinitivecomponents.From
a methodological perspective, in the final validation we submitted a questionnaire ex-ante to assess
learners’competencesandexperiences,andaquestionnaireex-posttoassesstheirpossibleimprovements
inknowledge,skillsandcompetence,acquiredattheconclusionofatrainingsessionlastingtwoworking
weeks (since 14 September 2016). The ex-post questionnaire also aimed at surveying the degree of user
satisfactionandengagementandataskingtheiropinionregardingusability,efficacyandcompletenessof
theLearnPAdPlatform.Thefinalvalidationinvolved61users(31learnerswereinvolvedinusingtheLearn
Padplatformand30learners,referredtoasthematchingcontrolgroup,workedwithinanotherregionalelearningplatformcalledMARLENE).
Keywordlist
BusinessProcess,EuropeanProjectBudgetReport,ModellingEnvironment,LearningPlatform,Public
Administration,SportelloUnicoAttivitàProduttive,OneStopShop,Validation,Accessibility,User
Experience,Gamification,TitoloUnico,SegnalazioneCertificatadiInizioAttività,UrbanVariant,Conference
ofServices,EuropeanQualificationsFramework,LearningMaterial,Simulation,Recommender,Key
PerformanceIndicators,Learningapproach
ii
iii
Document History
Version
Changes
Author(s)
0.1
TableofContent(TOC)
1.0
FirstDraft:Contenttobe
improved
1.1
SecondDraft:Contenttobe
mergedandfixed
AndreaSergiacomi(MAR),Alessia
Balducci(MAR),LauraGiorgio
(MAR),AntoniaBertolino(CNR)
2.0
Internalrelease:Contenttobe
finalized
2.1
Secondinternalrelease:
Conclusionstobefixed
Balducci(MAR)
2.2
Thirdinternalrelease:Allcontent
addedandcheckedforthe
internalreview
Balducci(MAR)
3.0
Submission
Document Review
Release
Date
Ver.
Reviewer(s)
Comments
ToC
October3,2016
0.1
AntoniaBertolino,
BarbaraRe
Suggestiontoadd
thelearning
approach
description
Draft
October14,2016
1.x
AntoniaBertolino,
GuglielmoDe
Angelis,Andrea
Polini
addedpartner
integrationsand
contributesfrom
othertechnical
deliverables
Internalrelease
October24,2016
2.x
AntoniaBertolino
Candidatefinal
October27,2016
3.0
iv
v
Glossary, acronyms & abbreviations
Item
Description
Accessibility
Incomputerscienceisthecharacteristicofacomputingsystemtobeeasilyusable
byanyuser,regardlessofhissensory,motororpsychicindividualcapacities,ofany
temporaryorpermanentdisability,ofevenculturalaspectsoraspectsrelatingto
familiaritywiththeuseofICTinterfaces.
Bloom's
taxonomy
isasetofthreehierarchicalmodels(coveringthelearningobjectivesincognitive,
affectiveandsensorydomains)usedtoclassifyeducationallearningobjectivesinto
levelsofcomplexityandspecificity.
BP
BusinessProcess
CoS
ConferenceofServices
EPBR
EuropeanProjectBudgetReport
EQF
EuropeanQualificationsFramework
Gamification
Theuseofelementsborrowedfromthegamesandotherfuntechniques,adapted
toexternalcontexts(eg.scores,levels,rewards,badges,giftsandotherloyalty
methodsorpracticestoarouseinterestorcontributetotheusersatisfaction)
Item
Description
KPI
KeyPerformanceIndicatorssettomeasurestrategic,organizationalandlearning
goals
Learning
Approach
TheoverallarchitectureadoptedbytheLearnPAdproject,mappingthe
Methodology(inspiredbytargets,contextandexistinglearningparadigmsforPAs),
thechoicesmade(referredtothecentralityoftheworkprocesses,the
transmissionofknowledgeandusefulcompetences,theinteractivedidactic
methodologiesandthemixofe-learningmodels:selflearning,assistedlearning,
collaborativelearning,organizationallearning),thesubjectstobeinvolved
(mandants,modelers,learners,experts)andtheevaluationsystem(basedonaKPI
Dashboard)
Learning
Material
Allentitiesnotrepresentingmodelsbutrelevantforlearningasbooks,tutorials,
learningaudioandvideofilebutalsobrowsingandsimulation
MARLENE
MarcheLearningNetworke-learningplatform
OSS
OneStopShopforProductiveActivities(synonymofSUAP)
PA
PublicAdministration
vi
Item
Description
Recommender
Learningmodethatprovidesemanticallycontextualizedlearningcontent
(accordingtotheuserprofileorthefunctionalitieshe/sheisplaying)throughaside
baroftheLearnPAdplatform
SCIA
‘SegnalazioneCertificatadiInizioAttività’meansinEnglishcertifiedreportfor
startingabusinessactivity;hereusedtonameanotherkindofSUAPbusiness
process
Simulation
Learningmodethatsimulatestheexecutingof(partof)aprocessinatraining
environmentwithtrainingdata
SUAP
SportelloUnicoAttivitàProduttive
TitoloUnico
‘Titolounico’meansinEnglishstandardrequestforstartingabusinessactivity;
hereusedtonamethebusinessprocessconsidered
UrbanVariant
IsanotherkindofSUAPbusinessprocess,accordingtowhichtheentrepreneur's
requestmakesnecessarytoinvolvethecitycouncilinformallychangingthe
urbanisticdestinationofazonefromitsoriginalplan
UX
UserExperience.ThesubjectivefeelingsthatanuserexperiencedusinganICT
system,withreferencetotheempiricandaffectiveaspects,tothevalueand
meaningattributionhe/sherelatedtotheavailabilityandtheinteractionwiththe
solution,aswellastotheoverallperceptionsandpersonalthoughtshe/shehad
aboutitseffectiveness,theeaseofuseoftheinterface,thesystem'sefficiency,etc.
vii
TableofContent(ToC)
ListofTables......................................................................................................................................................x
ListofFigures...................................................................................................................................................xi
1Introduction...................................................................................................................................................1
1.1Purpose...................................................................................................................................................1
1.2Relateddeliverables................................................................................................................................1
1.3Structureofthedeliverable....................................................................................................................2
2Learningapproach..........................................................................................................................................4
2.1Explanationofthemethodology.............................................................................................................4
2.2Thechoices..............................................................................................................................................6
2.3Subjectsinvolved.....................................................................................................................................7
2.4Evaluationsystem...................................................................................................................................8
3LearnPadPlatform:finalvalidation...............................................................................................................9
3.1Resultsfromearlyvalidation...................................................................................................................9
3.2Validationintopractice.........................................................................................................................10
3.2.1Referencedplatforms.....................................................................................................................11
3.2.2Validationscenario.........................................................................................................................18
3.2.3Usedmethodology.........................................................................................................................21
3.2.4Questionnaires...............................................................................................................................22
3.3Resultsfromquestionnaires..................................................................................................................27
3.3.1Profilingandcontextdata..............................................................................................................27
3.3.2Competenceself-assessmentdata.................................................................................................31
3.3.3Learningassessmentdata..............................................................................................................34
3.3.4Usabilitydata..................................................................................................................................37
3.4Synthesisofthecollectedfeedback......................................................................................................39
3.4.1Positiveoverallopinionsandplatformsstrengths.........................................................................39
3.4.2Negativeoverallopinionsandmissingfunctionalities...................................................................40
4Resultsanalysisandconclusions..................................................................................................................42
Appendix.........................................................................................................................................................49
ExanteQuestionnaire(inItalian)................................................................................................................49
ExpostQuestionnaire(inItalian)................................................................................................................53
Collectedfeedback(inItalian).....................................................................................................................61
Collectedfeedback(inenglish)...................................................................................................................67
viii
ix
ListofTables
Table1CentralchoicesmappedinthemodelsanddidacticsolutionsadoptedwithintheLearnPAd
approach...........................................................................................................................................................7
Table2Usabilityresultsfromtheearlyvalidation..........................................................................................10
Table3DescriptorsdefininglevelsintheEuropeanQualificationsFramework(EQF)...................................25
Table4Aggregateddatarelatedtothetwosamplesinvolvedinthetesting................................................28
Table5PerceivedimprovementsinEQFskills................................................................................................33
Table6Profitresultsforeachclusterofquestions.........................................................................................35
Table7Dataandfiguresrelatedtotheusabilityexpostanswers.................................................................38
Table8Overallstatisticsforprofitanswersresults(themeanofthetotalscoreacquiredandthemedian
score)...............................................................................................................................................................43
Table9Pearsoncorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors(PA
experience,SUAPexperience,anddedicatedtime).......................................................................................43
Table10Kendalltaucorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors
(age,SUAPexperience,anddedicatedtime)..................................................................................................44
Table11Pearsoncorrelationbetweenthenumberofindividualcorrectanswersandthesumofdeclared
levelsinthefiveEQFcompetences.................................................................................................................46
Table12Pearsoncorrelationbetweenthenumberofindividualcorrectanswersandtheusersoverallrating
oftheplatforms..............................................................................................................................................47
x
ListofFigures
Figure1TheLearnPAdplatformhomepage..................................................................................................11
Figure2Thebrowsingmode:awikipagerepresentingatask.......................................................................12
Figure3Thevalidatedoutputofacontributionbythequalityassessmentsystem......................................14
Figure4Afterthesimulationisactivated,therecommenderbarproposescontextualizedsimilarcases.....15
Figure5TheLearnPAdKPIDashboard...........................................................................................................16
Figure6TheMARLENEinterfacetoaccessaSCORMcourse..........................................................................16
Figure7ApageofcontentfromtheMARLENEcourseaboutSUAPCoS........................................................17
Figure8TheSUAP“TitoloUnico”BusinessProcess.......................................................................................19
Figure9The“ConferenceofServicesmanagement”sub-processwithinthe“TitoloUnico”BP...................20
Figure10a/bTheLearnPAdandMARLENEsamplesbygender....................................................................28
Figure11a/bTheLearnPAdandMARLENEsamplesbyage..........................................................................29
Figure12a/bTheLearnPAdandMARLENEsamplesbyqualificationineducation.......................................29
Figure13a/bTheLearnPAdandMARLENEsamplesbypublicrole...............................................................30
Figure14a/bTheLearnPAdandMARLENEsamplesbyyearsofworkat(oronbehalfof)aPA...................30
Figure15a/bTheLearnPAdandMARLENEsamplesbyyearsofexperienceinOSS/SUAP...........................31
Figure16Theplatformsoverallusagetime(inhours)...................................................................................31
Figure17ThedeclaredimprovementinEQFskillswithinthetwosamples...................................................32
Figure18ThepercentageofdeclaredimprovementineachrelevantEQFskillwithinthetwosamples......33
Figure19Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestionswithin
thetwosamples..............................................................................................................................................34
Figure20Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestionswithin
thesamplesandperclustersofquestions......................................................................................................36
Figure21Acomparingoverviewonacquiredopinionsaboutusability,userexperienceandplatform
effectivenessbytheusersofLearnPAdandMARLENE..................................................................................39
Figure22a/bThePOSITIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE.......................................39
Figure23a/bTheNEGATIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE.....................................40
xi
1Introduction
1.1Purpose
InthecontextoftheLearnPadproject,demonstrators(i)helpthetechnicalandresearchwork-packagesto
betterfocustheiractivities;and(ii)theypermittoassesstheapplicability,acceptanceandeffectivenessof
theproposedsolutionwithinrealworkingcontexts.AccordingtoWP8objectiveswehadtovalidateproject
results considering the platform effectiveness in supporting civil servants training and collaborative
activities, by directly involving the end users. During the third year of execution of the project, the
objectives of the demonstrators evolved according to the state of development of the project, taking in
deep consideration the effort of the whole consortium and of the partners in each single WP and
deliverable that represented, for the platform, a new release, an empowerment or a content adapting
request.
IntheY2reportthereviewerssetsomespecificrequirementsforsuchvalidationrelatedtothesizeandthe
qualityoftheexperiment,whichhadtoprovidestatisticallymeaningfulconclusions.Theyalsoaskedtotest
theeffectivenessoftheconceptuallearningframeworkinordertoassesssomerealtangiblelearninggoals
achievedbythecivilservantsusingtheLearnPAdplatformintheworkplace.
WefocusedonthecaseofOSS(SUAP)tobeabletoinvolvethelargestnumberofavailableusersarounda
community of practice clearly identified. To pursue the required results, the final validation has been
planned to last a minimum of two weeks, while the experimenters, involved in the SUAP management
processes,hadcontinuedworkingasusualintheirdailydutiesandroutines.
The validation, better described in Section 3.2, was carried out dividing the 72 recruited people in two
groups, equally populated and as homogeneous as possible: the users of arunning instance of the Learn
PAd platform (available at the address http://Learn PAd.regione.marche.it:8080/xwiki/) and the control
group, using MARLENE, a more traditional e-learning Moodle platform, already available in the Marche
Region(http://marlenescuola.regione.marche.it/moodle/).
Apreviousquestionnairewasprovidedtoalltheexperimenters,todefineexantetheirownmasterdata
andthestatusoftheirexperience,EQFcompetencesandexpertiseinthematterofthecourse.
Subsequently,afterthetwoweekstrial,amoredetailedexpostquestionnaire-alwaysthesameforboth
groups-hadbeensubmittedtoretrieveusefuldatainordertounderstandifandhowthoseplatformshad
helped them to learn, and the judgment they gave to this experience, in terms of usability and
effectiveness.Attheendoftheexperimentationwereceivedback61compiledquestionnaires.
Thesedata,bothmeasurableandbasedoncommentstobeinterpretedandrepresentedinverbalform,
were anonymized, aggregated and processed to provide a clearer picture of the evaluation results, as
explainedlater.
1.2Relateddeliverables
Thefollowingdocumentshavebeentakenasreferencepoints:
• D1.2.RequirementsAssessmentReport.Thisdeliverablewasconsideredasitreportsonassessmentof
realizationstatusofthoserequirementsthathavebeencollected,analysed,andreportedinD1.1.
1
•
•
•
•
•
•
D2.4 Core Platform Implementation – Second Version. The deliverable consists of a second and final
versionoftheLearnPAdcoreplatform,andisofsoftwarenature.
D5.4 - KPI Ontology and Learners Assessment Mechanism. This deliverable describes the design of a
comprehensive solution to enable PA employees meeting their business goals on a strategic and
operational level, deriving operational goals, learning goals and competencies needed to achieve a
correctexecutionofrelevantbusinessprocesses.ItoutlinesthetypicalgoalsandKPIsrequiredtoplan
andtracklearninginarealworkplaceenvironmenttakingasexamplethe“TitoloUnico”SUAPprocess.
D8.1.DemonstratorsBPandKnowledgemodels.Thedocumentidentifiesthedemonstratorsrelatedto
the Learn PAd project within the two following scenarios: European Project Budget Reporting and
“SportelloUnicoAttivitàProduttive”.
D8.1. (Addendum) User Perspective and Project Evaluation Strategies. This additional deliverable was
triggered by the exigency of a deeper understanding of user needs is required. It is referred as it
providedearlyinsightsandguidelinesonhowtheevaluationoftheimplementationshouldbecarried
out.
D8.2.PlatformPrototypeEarlyValidationResults.Itreportstheresultsofthepreviousvalidationwhich
havebeencomparedwiththeresultsshowedinthepresentdeliverable.
D8.3. Demonstrators Populated Learning Platform. It consists of the final version of the Learn PAd
contentforthetwodemonstratorsdevelopedintheproject:EuropeanProjectBudgetReport(EPBR),
andSportelloUnicoAttivitàProduttive(SUAP).Thedeliverableisofsoftwareandreportnatureandit
providestheofficialreleasedversionofcontent,alsoincludinginformationonhowsuchcontentscan
be accessed. In particular, we refer to both the Learn PAd Modeling Environment and the Learn PAd
LearningPlatform.
Moreover,thevalidationusedasabackgroundthefollowingdeliverableswhosenatureisReport,Software
orboth:
• D3.3-FinalLearnPAdMetamodelsandImplementationofModelTransformationsforManaging
BusinessProcessesModelsinPublicAdministrations
• D4.3-QualityAssessmentMechanismsImplementation
• D5.1-ModelsforSettingtheWiki
• D5.5-SemanticsfortheWiki–FinalIteration
• D6.3-LearnPAdSimulationEnvironment:Finalrelease
• D7.2IntegrationToolsandSystems
• D9.5Technology-orientedLearnPadwhitepaper
• D9.7LearnPAdExploitationPlan–FinalIteration
1.3Structureofthedeliverable
Besidesthisbriefintroduction,thisdeliverableisstructuredinthreeparts,respectivelyfocusingon:
A. Learningapproach(Chapter2)
In this chapter we highlight the learning framework defined by the consortium, describing it in
termsofadoptedmethodology(inspiredbytargets,contextandexistinglearningmodelsforPAs),
choices made, involved subjects and evaluation system. As required in the Y2 review report, the
conceptualframeworkforlearningintheworkplacehasbeenusedasareferenceforbothtothe
evaluationapproachandthelearninggoalsmappedinthedeployedKPIDashboard.
2
B. Finalvalidationoftheplatform(Chapter3)
ThissectiondescribesthevalidationofthefinalversionoftheLearnPAdplatform.Inparticular,at
firstweprovidetheresultsfromthepreviousearlyvalidation,asapossibleyardsticktointerpret
the actual findings. Then we explain how we conducted the validation describing: the
functionalities of the platforms that have been tested, the chosen scenario, the validation
methodology,andthedevelopedquestionnaires.Finally,theresultsobtainedfromtheanswersto
thequestionnairesarerepresented,andasummaryofthecollectedfeedbackisreported.
C. Resultsandconclusions(Chapter4)
Thissectionprovidesevidenceandreportssomeconclusions,summarizingandfinalizingthework
doneforthefinalvalidation.
3
2Learningapproach
This Chapter will illustrate the learning approach utilised for theLearn PAd project and its impact on the
didacticandtechnologicalchoices.
2.1Explanationofthemethodology
Thearchitectureofthelearningmodelproposedderivesfromacross-analysisofthreesystemvariables:
a. Targetcharacteristics
b. Characteristicsofthecontextinfluencingthemodelsandthelearningobjectives
c. ExistinglearningmodelsforPA
a. Targetcharacteristics
The Learn PAd project is aimed at the civil servant, therefore an adult in a working environment, who is
generallyusedtodoinghighlyprescriptive(typicallynormedbystrictrules)professionalactivities.
The working context is still characterised by particularly fragmented working processes among the
employees, a condition that does not facilitate the understanding of the logic at the basis of the
professionalactivityofeachindividual.
Beinganadult,thecivilservantisparticularlyorientedat:
•
•
•
Learning from experience: this highlights the need to focus on experiential techniques when
constructing the learning setting. Such techniques make use of the learner’s experience. Among
them,themostefficientandcommonare:simulations,problemsolving,andcasestudies.
Selective learning: the adult only learns what he/she needs to know and to do in order to
efficientlyfacerealsituationsinlifeandatwork.
Beingengagedbysupportinglearningprocesseswhentheseareshowntoassisthimintasksand
problemsfacedindailylife.Infacttheadultgainsnewknowledge,understanding,abilities,values
and attitudes more effectively when these are presented and applied in the context of real
situations.
From a technological skills perspective, the civil servant is generally accustomed to the technologies that
supporthis/herprofessionalactivities,aswellaslearningsystems.Furthermoretherearewidespreadand
ampletechnologicalfacilitiesintheworkplace.
b.
Characteristicsofthecontextinfluencingthemodelsandthelearningobjectives
Europeanandnationaldirectivesregardingsimplification,anti-corruptionandefficiencyinthePApushfor
reorganisation of the PA based on lean work processes, in order to ensure procedure deadlines and
transparencyofthedecisionmakingprocesses.
Thischangeinpacehasanenormousimpactonthelearningsystemsthatmustsupportsaidchangeand
thereforemustreorganiseknowledgeintermsofcompetenceandworkprocesses.
4
c.
ExistinglearningmodelsforPA
Afterananalysisofthetargetandtheenvironmentinwhichitoperates,thechoiceofmethodologieswas
basedonthelearningexperiencesclosesttothePAcontextandthemostcommone-learningmodels.
Regarding the learning experience, the didactic model of the Training School of the Region of Marche,
whichasaninternalstructureexclusivelytrainscivilservants,wasadopted.
Regardinge-learningmodels,referencewasmadetothedirectiveoftheMINISTRYFORINNOVATIONAND
TECHNOLOGYandtheMINISTRYFORPAconcerninge-learningmodelsforPA
Thisdirectiveidentifies:
•
•
•
The Self learning model, in which the technology predominantly provides access to learning
materials offering repositories of material structured and organised around self learning. Static
learningcoursesandtoolsforself-assessmentareproposed.Inthiscasetheuserinteractsalmost
exclusivelywiththecontentmanagementsystem.
The assisted learning model, in which assistants, tutors and technological staff play a role,
interactingwithlearners,andhelpingthemusethematerials.Inthismodeltheaccesstomaterials
remains important but less than in the previous model. In fact human communication mediated
throughthecomputerispresentfacilitatingandenrichingthelearningprocesswhichisnolonger
purelybasedonstudyingmaterials.
The collaborative learning model, in which interpersonal relations are ever more important, not
only with the facilitators but also between learners. This model comprises systems based on
collaboration and community practice. Activities involving cooperation between the participants
areofgreatimportance,aswellastechnologyaimedatfavouringinteraction.
RegardingthemodelutilisedbythetrainingschooloftheRegionofMarcheforPA,thetraditionallearning
model based on the transfer of knowledge has been superseded in favour of a model centred on the
developmentofcompetencesthroughtheuseofactivedidacticmethodologies.
The basis of this methodology derived from the most common knowledge theories is the following: ‘the
adult in a working context learns (gains new competences and develops new attitudes) if the following
conditionsaremet:
•
•
•
Trainingcontract:thisisdeterminedwhenthetrainingofferedisrecognisedasadvantageousand
usefulforprofessionalimprovement,bothforthelearnerandfortheorganisation.Inorderforthis
conditiontoexisttheproposalmustbemadeonthebasisofapreliminaryanalysisoftheworkthat
allowsthedesignertoidentifytheareasofresponsibilityforeachparticipantandthereafterdesign
acoursethatiscoherentwiththeirworkingprocesses.
Activeparticipation:thelearnerneedstoplayanactiveroleinpersonallyverifyingtheefficiencyof
the information provided, even if simulated, within his/her professional role. To guarantee this
condition the school adopts an interactive didactic methodology oriented to solving real
professionaltasks.
MandantInvolvement:fromtheanalysisstage,thetrainingmustbestrictlyrelatedtotheoperative
goals of the organisational structure and consequently the responsibility for the learning/change
does not depend solely on the school and the individual learner, but also on the entire
organisationalsystemandinparticularthemanagement.
5
•
Assessmentoftheresults:afurtherconditionforeffectivenessisrelatedtotheassessmentofthe
learning outcomes. To this end, the school adopts a system monitoring all the variables that
influence the overall result: assessment of the training quality, teacher assessment, competence
assessmentandassessmentoftheoutputquality.
2.2Thechoices
On the basis of the aforementioned criteria, principled choices on the matter were made, which
determined the orientation of the Learn PAd approach. These choices focus on five main principal
features:
1. Thecentralityoftheworkprocessesasafactororientingtheorganisationoftheinformationand
thecontent.
2. The transmission of knowledge and competences seen as useful for the learner in that they are
directlyconnectedtoconcreteprofessionalexperience.
3. Functional development sustaining interactive didactic methodologies oriented to solving real
professionaltasks,whichfosteractivelearnerparticipation.
4. The adoption of a mix of e-learning models from which the most functional and target oriented
featureswereselected.
5. Theadoptionofacollaborativelearningperspectivethatcansupportchangesintheorganizational
processes.
The table below presents the solutions selected at a training system level based on the choices in the
matter.
Modelsandstandardsatthe
basisofLearnPAd
Didactic/technologicalsolutions
adopted
Centralityofthework
processes
• Businessprocessmodel
• Organisationalmodelrelated
torecommender
• Documentmodel
• KPImodel
• Competencemodel
• Documentmodel
• KPImodel
Interactivegraphicrepresentationof
theworkprocesses
Transmissionof
knowledgeand
competencesseenas
useful
Interactivedidactic
methodologiesoriented
tosolvingreal
professionaltasks
• Organisationalmodel
• CaseManagementModel
Interactivegraphicrepresentationof
theworkprocesses
Simulationbasedonrealcase
studies
Availabilityofformstosupport
explanationoftheprofessional
activity
Simulation
Gamification
Recommender
Commentingandproviding
contributions
6
Mixofe-learningmodels UseofWIKItechnology
Selflearningmodel
• Browsingmode
• Integrativedidacticmaterial
forselflearningandindividual
in-depthstudy
Assistedlearningmodel
• Expertconsultation
• Recommender
• Simulation
Collaborativelearningmodel
• Feedbackvs.modeller
• Commentvs.platform
• Collaborativesimulation
Feedbacks;
Commentingandproviding
contributions.
Adoption
of
a UseofWIKItechnology
collaborative
learning perspective that can
support changes in the
organizationalprocesses.
Table1Centralchoicesmappedinthemodelsanddidacticsolutionsadopted
withintheLearnPAdapproach
2.3Subjectsinvolved
LearnPAdlearningmodelinvolvesinitsprocessesasetofactorsthatsupportthedifferentphaseofthe
processmodellingandimplementation.Theyare:
Themandant
Themandantisalwaysaninternalstaffmemberwithintheorganisationpromotingthetrainingprocess,
whocorrespondsto(someof)thedescriptionsbelow:
•
•
•
•
•
He/shecontributestothedescriptionandmappingofthebusinessprocessesandorganisational
model.
He/shedefinesthestrategic,organisationalandoperativegoalsincollaborationwiththemodeller.
He/sheisthedecisionmakerwithrespecttolearningpolicieswithintheunitororganisation.
He/shereceivestheresultsofthelearningprocessesandinthisrolehe/sheisalsoresponsiblefor
thelearningimpact.
He/sheisthefinaldecisionmakerwithrespecttoproposalsforinnovationpresentedbythe
modelleronthebasisoffeedbackfromthelearners.
Themodeller
Themodellermanagesthedesign,maintenance,andmodificationsofmodelsandcontentsoftheplatform,
alsotakingintoaccountlearnerfeedback.Inthisenvironmenthe/sheisresponsiblefortheassessmentof
feedbackqualityandofitsimpactonthemodels.
Itisuptothemodellertovalorisepotentialaspectsforinnovationcomingfromthelearnerfeedback.He/
sheisthereforetheprincipalguarantorfortheprocessofmodelinnovation.
7
Thelearner
Thelearnerisalwaysaninternalstaffmemberwithintheorganisationpromotingthetrainingprocess.
He/sheisresponsibleforindividuallearning.
The said learner is also co-producer of knowledge through the production of feedback, interaction with
experts,productionoflearningobjects,andinteractionwithotherlearners.
Theexpert
Maybeinternal/externaltotheorganisation.
•
•
•
He/sheisresponsibleforthethematicin-depthlearning.
He/sherespondstolearnerrequestsregardingspecificstothecourse.
He/she assesses the importance and the quality of feedback relative to potential innovation or
changes within the business process. He/she relates to the modeller and the mandant regarding
decisionimpactingonthemodel.
2.4Evaluationsystem
The evaluation approach adopted, in coherence with the methodological choices, has two primary
objectives:
a. Evaluate the learning process by measuring competences before and after the training and
evaluatetheimprovementinworkerperformance.
ThelearningassessmentismadebyutilisingthedefinitionsofKPIsetswhichgroupdatarelativeto
thequalityofworkerperformance,correlatedtostrategic,organisationalandoperativegoals.
Inthismannerthelogicbehindtheworkprocesses,afundamentalelementinthelearningprocess,
isretained.
Such validation should be done over a long period, and extended to the organization (not only
individual)level.
b. Evaluatetheefficiencyoftheplatformusingindicatorsthatmonitorthelevelsandqualityofthe
learner/platforminteraction.Atthesametimetheefficiencyofthedidacticmethodologyadopted
canbeevaluated.
The KPI dashboard collects and provides information relative to the intensity of platform usage
(greater or lesser application of the tools present on the platform such as simulation, the use of
materials and the recommender etc.) and therefore contributes to a better understanding, also
indirectly,oftheeffectivenessofsomemethodologiesinrespecttoothers,andinrelationtothe
typeofuser.
TheevaluationsystembasedonKPImodelisdescribedindetailinDeliverable5.4.
8
3LearnPadPlatform:finalvalidation
According to Deliverable 8.1 addendum and considering the latest suggestions recommended by the
reviewers,theLearnPAdconsortiumcarefullyconsidereddifferentstrategiesfortheassessmentofproject
resultsandforgettingindicationsthatmayhelptheanalysisofimpactforpotentialadopters.Duringthe
early validation we aimed to collect as much feedback as possible to properly address the prototyping
activityofthefinalversionoftheLearnPAdplatform.Inthefinalvalidationwecollectedamoredetailed
amountofdataandcommentstobeevaluated.
This chapter describes the final validation we carried out on the Learn PAd platform at the end of the
project.TheassessmentoftheLearnPAdplatformhasbeendoneontheversiondeployedonatest-bed
settled in the cloud datacenter of Marche Region in Ancona, and available at the address http://Learn
PAd.regione.marche.it:8080/xwiki/ (release 14 September 2016). In parallel a control group tried and
testedanotherregionale-learningplatform“MARLENE(http://marlenescuola.regione.marche.it/moodle/)
to be able to compare the achievements, in terms of measurable learning goals, and the opinions of the
learnersaboutthetwodifferentlearningenvironments.
Despite the different ways of use, the content enjoyed by the two groups were essentially similar and
relatedtotheSUAP“TitoloUnico”workprocessesintheorganizationofaconferenceofServices.
Wedecidedtoperformanassessmentaimingatexaminingtheoverallstatusoftheprojectaccordingto
thedifferentlearningparadigms,asdescribedinthepreviouschapter,andthevariousaspectsrelatedto
theactivitiesthatanemployeehastocarryoutwithinaBPenactment.
Allthedidacticandtechnologicalsolutions,releasedandadoptedcoherentlywiththeLearnPAdlearning
approach, have been evaluated considering both the users learning profit results (also analyzing the
answerstheygivegroupedbyclustersofenablingfunctionalities),andtheiropinionsontheusabilityofthe
proposedfunctionalities.
WefocusedonaspecificsetofmodelsrelatedtotheSUAPscenario,whichhasbeenimplementedinterms
ofcontentsandlearningmaterials,accordingtothespecificfunctionalitiesofeachplatform,aswebetter
describe later. In particular, the platform final validation involved a quantitative assessment and the
collectionandanalysisofdataandfeedbackviaquestionnaires.
Thechapterisorganizedasfollows.Insection3.1weresumetheresultsobtainedfromtheearlyvalidation
at M18. In Section 3.2 we describe how and for what scenario we conducted the validation. We also
introducethemainfeaturesofthetwoplatformsandpresentthequestionsaskedinthequestionnaires,
togetherwiththeratiooftheirprovision.InSection3.3wepresentinfactsandfiguresthecharacteristic
datarepresentingthesamplesandtheresultsobtainedfromthecollectedanswers.TheninSection3.4we
report a summary derived from all the verbose feedback we collected from the users (which will be
reportedinappendix).
3.1Resultsfromearlyvalidation
Themainresultsobtainedfromtheearlyvalidationaresummarizedinthetablebelow,whichrepresents
theaverageopinionsofthememberswhoparticipatedinthe2015tests,answeringonspecificquestions
related to the perceived characteristics of the platform (usability, effectiveness, accessibility, innovation,
etc.). The majority of these questions was proposed again in the final validation, in order to set a
comparative meter with reference to the possible evolutions found on the current Learn PAd platform
9
functionalities.Sothesevalues,evenifcollectedbyonly7testersofanearlierversionoftheplatform(asof
6 november 2015) that had been assisted by platform experts, will be also considered and compared in
section3.3.
Criterions
Averageratingduringtheearlyevaluation
phase[from1=lowto5=high]
Overalldegreeofsatisfactionintheuseofthise-learningsolution
Degreeofeffectivenessofthevariousfunctionalitiesavailablewithin
theplatform
Degreeofusability,easeofuseandaccessibility
Degreeofcompletenessoftheavailablefeatures
Degreeofintegrationandhomogeneityofallthemodulesavailable
withinthelearningenvironment
Degreeofinnovationoftheproposedsolution
Degreeofusefulnessandappropriatenessoftheplatformwith
respecttothetrainingneedsoflearners
Detectedneedofagreaterpresenceofawizard-drivencontent
fruition
Detectedneedofagreaterpresenceofgamificationfeatures
Detectedneedofagreaterpresenceofanappropriateuseof
multimedia
Detectedneedofagreaterpresenceofinteractivitybetweenlearner
andcontent
Detectedneedofagreaterpresenceofmultilingualismor
explanatorytechnicalvocabularies
Detectedneedofagreaterpresenceoftutoringandexternalteaching
support
3,14
3,32
2,42
3,42
3,42
4,33
4,00
3,71
4,20
4,71
4,50
4,28
4,00
Table2Usabilityresultsfromtheearlyvalidation
3.2Validationintopractice
InthissectionwedescribeindetailhowwepracticallyrunthevalidationtestsoftheLearnPAdplatform,in
combination with a matching control group using MARLENE e-learning platform. We report the used
methodology and we describe the validation scenario supported by the final instance of the Learn PAd
platform(aswellasbythemoodletrainingmoduleproposedwithinMARLENE).Finally,weintroducethe
exanteandexpostquestionnaireswedevelopedfortheevaluationpurposes(intheappendixwereport
theItalianverbatimversionofthesequestionnaires).
Suchvalidationinvolvedatotalnumberof72end-usersoftheSUAPdemonstrator,whohadnoknowledge
of the learning platforms before the validation, and that have been invited to join by Marche Region,
havingconfirmedtheirreadinesstoparticipateinthetests.36wereaskedtousetheLearnPAdplatform;
36toowereaskedtotesttheMARLENEplatform.
Eventually, we collected 61 answered questionnaires (31 from the Learn PAd users and 30 from the
MARLENEusers).ThereceiveddataandfeedbackarereportedinSections3.3and3.4ofthisdeliverable.
AsfortheLearnPAdplatformvalidationresults,wecananticipate(seeinthefollowingofthisdocument-
Chapter 4 - a more detailed analysis and explanation) that the actual release is on average perceived as
quiteeffective,evenif,probably,itneedsahigherusagetimeinordertorecordmoresignificantlearning
outcomes and even if some further development and content customization are needed. We expect to
10
fulfillthoseevolutionarymaintenancerequests,abletotransformaprototypeintosomethingmoresimilar
toamarketstablerelease,duringthenextcomingexploitationphase.
3.2.1Referencedplatforms
We now turn to describe briefly each of the platforms used in the experiment and their main
functionalities.
3.2.1.1LearnPAd
Figure1TheLearnPAdplatformhomepage
Briefly, in the Learn PAd web/wiki platform, after the log-in (using the assigned credentials, as from the
users list represented in https://github.com/Learn PAd/Learn PAd/blob/master/lp-collaborative-workspace/lp-cwcomponent/lp-cw-component-ui/src/main/resources/LPUsers/UsersListY3Demo.xml#L42) and the selection of the
specific case/course “ModelSet – SUAP Titolo Unico”, the end user can find the list of the managed
BusinessProcesses(orsignificantpartsofthem),withasetoflinks,iconsandbuttons,mostlyto:
• browse hierarchically the single tasks or choice gateways, roles in the organizational units,
documentsandotherobjectsinvolvedanddescribedbythemodels
• viewanavigablemapoftheprocessorsub-processactivities
• contributeandgenerallytointeractwiththecontentandthefunctionalitiesoftheplatform.
• simulatearealcase
11
Asanindicative,althoughapproximate,unitofmeasuretheusershadattheirdisposalforthevalidationa
huge amount of material (almost 90 wiki pages), including BP and sub-process tasks and other modeling
objectstobevisited,collaborativeandsimulationactivitiestoplayandrecommendedcontenttoexplore.
Figure2Thebrowsingmode:awikipagerepresentingatask
More in detail, the main implemented functionalities of the Learn PAd platform under validation have
been:
• Thehomepage(providingintroductiveinformationandexplanations,anonlinehow-tohandbook,
someguidedtoursonfeatures,etc.)–Fig.1;
• TheBrowsingMode(whichallowstonavigateamongthetextualcontentsandtheimagesrelated
to the interconnected models – BP, documents, organizational structures, … - , and in particular
amongtheprocesstasks,representedwithinawikipage,thataretheknowledgesegmentsofan
overallproceduralflow)–Fig.2;
• TheCollaborativefeatures(includingthepossibilityto:(i)providecommentsineachpage,(ii)give
feedback to the modelers reporting logical errors or suggesting variations in the flow [this,
indirectlyandpotentially,transformanindividuallearningprocessintoanorganizationallearning
process], (iii) contribute in writing extended wiki documents – that can also be analyzed and
validatedbythequalityassessmentmechanismintermsofqualityforcorrectness,simplicity,non
ambiguity,contentclarity,presentationclarityandcompleteness)–Fig.3;
12
•
•
TheSimulator(inwhichtheusercanpracticeanexercisebasedonarealusecasehe/shechooses-
inthiscontextaSUAPinstance-followingsomeexecutivesteps,providingresultsandanswering
questions. As soon as the user correctly performs all the required jobs, the simulator returns an
overallscore.Thesimulationcanbeperformedinindividualmode,withrobots,orinvolvingother
physicalend-userstoplaytheroleofdifferentactorsinvolvedasthirdpartiesintheprocess)–Fig.
4;
The Recommendation bar (which provides additional semantically contextualized contents –
meaning experts to be contacted, more learning materials from the document model, real use
casessimilar–thatvariesaccordingtospecificparametersoftheuserprofile,typesofwikipage
theuserslandin,andthespecificcontentthereinselectedbytheusers)–Fig.4.
Thesefunctionalitiesarebetteroutlinedinthedeliverablesthatwetakeasreferenceorbackground,and,
ofcourse,havebeenimplementedintheMarcheRegiontest-bedandwillbedemonstratedduringtheY3
review.
Anotherfundamentalfunctionality,whichwasnotplannedtobeincludedinthefinalvalidationbutworth
mention,istheKPIdashboard(“pannellodicontrollo”–Fig.5).Itrepresentsalltherelevantaspectsofa
learning scorecard, in terms of learning and strategic goals, their connection to organizational goals, KPIs
forassessingtheirachievement.Itcollectsalsoinputvaluesthatresultfromtheactionsthatusersmakeon
thecollaborativeworkspaceandthesimulationenvironmentoftheplatform(eg.n°ofvaluablefeedback,
totaltimeofuse,globalscorepersimulationanduser,globalactionsperusers,etc.).
Someoftheseinputvaluesneedtoreachacertainthreshold,otherwisetherecommendersystemprovides
someadvicesbothtothelearnersandtotheirorganizationalunits,inordertoimprovebadperformance
scores.AllthesefunctionalitiesarebetterdescribedinDeliverable5.4.
Forthelearningassessmentandthefinalevaluationoftheplatformeffectiveness,wepreferredtousea
platform-independent tool, structuring specific questionnaires, also in order to compare more
homogeneousresultsgotbytheLearnPAdtestandthecontrolgroup’sone.TheuseoftheKPIDashboard
for evaluation purposes makes a greater sense in the context of a real training in the workplace preorganized set, where the aspects of motivation of the users and PA commitment exist, and the
organizationalgoalshadbeendefinedandtunedaccuratelywiththePAmanagement.
13
Figure3Thevalidatedoutputofacontributionbythequalityassessmentsystem
14
Figure4Afterthesimulationisactivated,therecommenderbarproposescontextualizedsimilarcases
15
Figure5TheLearnPAdKPIDashboard
3.2.1.2MARLENE
Figure6TheMARLENEinterfacetoaccessaSCORMcourse
The MARLENE (MARche LEarning NEtwork) platform is the regional web e-learning system, adopted,
followingasignedagreementwiththeTuscanyRegion,forthereuseoftheentiresystemcalledTRIOand
its available learning objects, with the goal of maximizing the public investments dedicated to the
developmentofanyonlinetrainingactivityforcivilservantsofthelocalPAs.
16
Thesystemisoperationalandofferstoitsusersawiderangeoflearningresources,basedonAICC/SCORM
andMoodlestandards.
Among the available courses, the module “SF0015-MD-2015-W” (La conferenza di servizi: quadro
normativo di riferimento in ambito nazionale e regionale) provides a lesson about the regulatory
framework, at national and regional level, on the conference of services. This module has been taken as
referencefortheexperimentationofthecontrolgroup,asitprovidesmanycontentsrelatedtothesame
scenarioimplementedwithintheLearnPAdSUAPdemonstrator,evenifatamoretheoreticallevel.Asan
indicative, although approximate, unit of measure the users had at their disposal for the validation 28
pagesofcontent.
Withregardstothefunctionalities,theplatformprovidesamanualandsomeonlinehelp.Whenthecourse
begins,awebpop-upwindowsappearstoguidethelearnersequentiallythroughthecontents,offeredin
the form of slides, accompanied by timed and animated texts, images and audio support (Fig. 7). Some
utilitybuttonsallowtheusertohandlethesubtitle,topausetheaudiospeech,tomovebetweenthenext
orpreviousslides,toaccesstheglossaryandthereferences,toconsultagraphicalhelponline,toprintthe
contentsofthecurrentpage.
Figure7ApageofcontentfromtheMARLENEcourseaboutSUAPCoS
17
The slides alternate with stages of verification (by multiple choice questions) exclusively for learning
reinforcement, and, at the end of the fruition, a final assessment questionnaire is proposed to test the
overall knowledge gained. The course is successfully passed answering correctly at least 70% of the final
proposedquestions.
A MARLENE back-end is also available to assess the state of completion of the course by each learner
engaged,returninginformation(takenintoaccounttovalidatesomeofthequestionnairesanswersandto
assess the final conclusions) on the dates of registration and completion, score points, total time of use,
numberofaccesses,statusand%ofcompletion.
Ifneeded,wewillbeabletoshowpracticallytheoperationofthissystemduringtheY3review,asitcanbe
reached through the Internet at the address http://marlenescuola.regione.marche.it/moodle/ by logging
withtheusercredentialsownedbytheofficersofMarcheRegion.
3.2.2Validationscenario
TheLearnPAdprojectdemonstratorsfocusontwodifferentworkplacecontexts.
The first demonstrator, referred to the European Project Budget Reporting (EPBR) scenario, engages a
PublicAdministration(PA),partnerinafundedproject,whichhavetomanagesomeinternaladministrative
documentationandfinancialproceduresinordertoreportdataandinformationtotheEUCommunity;in
thiscasethemodelstrytomapaBusinessProcess(BP)whichdoesnotcrosstheborderofaPA.
Theseconddemonstrator,referredtothe“SportelloUnicoAttivitàProduttive”(SUAP)scenario,represents
a more complex inter-organizational scenario, involving many Pas engaged in providing a unique and
sharedformalanswer(orauthorization)toentrepreneurswhowanttostartabusinessactivityortobuild
productiveorcommercialpremises.
Both of them were developed on the basis of several interviews with civil servants working in the Public
Administrationandinvolvedinthecases.
For both scenarios, the models we produced are those resulting from the WP3 (Organizational models,
Business Process models, Documents and Knowledge models Competence models, Business Motivation
models,KPIs,etc.).Inparticular,weconsideredtheorganizationalstructureofeachoftheinvolvedPAs.
In some cases (mainly due both to reasons of synthesis, readability, complexity decrease and space
availability) we focused only on part of those structures, considering the offices and the employees that
mainlyimpactontheanalyzedscenario,exercisingtheirrole.Ahugeeffortwasdonewithrespecttothe
competencesofemployees,sincetheyarerelevantintermsoflearning.Thesewereelaboratedaccording
tothewellknowEuropeanstandardcompetenciesframework(EQF).
It was originally planned to carry out a final validation with users on both SUAP and EPBR scenario.
However,weinsteadconcentratedalleffortsforfinalvalidationonlyonSUAP.OnereasonwasthattheY2
review report requested to collect a sufficient number of users in the final validation (30 + 30) to get
statistically significant results and the number of EPBR users would be quite far from such target.
Moreover,weconsideredriskytodividetheconsortiumeffortsandattentionontwodemonstratorsinthe
relativelylimitedtimeavailable,andpreferredtofocusallattentioninonestudy.
For these reasons we focused on the case of SUAP, and in particular on the Conference of Services subprocessforthe“Titolounico”process,tobeabletoinvolvethelargestnumberofavailableusersarounda
communityofpracticeclearlyidentified.
18
Inthefollowing,weintroduceandexplainbrieflytheconsideredSUAPscenario,chosenforthevalidation.
ThescenarioisalreadydetailedandexplainedinDeliberable8.1:wejustwouldliketoprovideanoverview,
within a self-contained document, to better highlight some of the choices we made in the setting of the
validationstudy.
The SUAP office can be considered as an intermediary between entrepreneurs and local government
bodies.Thismeansthat,ifanentrepreneurwantstostartanewbusinessactivityormakeaconstruction
interventiononafactory,he/sheonlyneedstocontacttheSUAPofficers.Thelatterareinchargeof:
• collectingandformallyverifyingtheapplicationandanysupportingdocumentation(eg.featuresof
theapplicant,possessionoravailabilityofthepropertytitles,etc.);
• acquiringanyintermediateanswersfromalltheotherauthoritiesoroffices,involvedonthebasis
oftheirindividualresponsibilities(eg.firesafety,buildingpermissions,protectionoflandscape,
protectionofarchaeologicalsitesandculturalheritage,environmentalpermits,foodhygiene,
sewagedischarges,connectiontothemunicipalutilities,etc.);
• issuinganofficialfinaldecisionthatisenforceableagainstthirdparties.
A SUAP office can be organized as an office of the Municipality, or as a district office - managed by a
consortium of PAs, by an aggregated administrative entity or by a locally superordinate public body (eg.
mountain unions) - that provides services on behalf of several Municipalities. This distinction was
considered in terms of Organizational Model: we assigned the user credentials distinguishing whether a
user belonged - or simulated to belong - to a "district desk" or a "municipal desk"; this parameter has
enabled,forexample,tovary,dependingontheuser,thehintoftheappropriatereferencedexpertsonthe
recommenderbar.So,alltheusersreallyworkingforamunicipalSUAP,orforathirdpartyadministration,
weregivena“municipaldesk”account;alltheusersreallyworkingforadistrictualSUAP,orforanotherPA,
indirectly involved in the SUAP processes for coordination, training and support, were given a “district
desk”account.
Figure8TheSUAP“TitoloUnico”BusinessProcess
The “Titolo Unico” application, also called standard request - Fig. 8 -, is one of the processes run by the
SUAP offices (typically the most ordinary one). The legislation introduced other peculiar types of
procedures, such as the “SCIA” or the “Urban Variant”. Here, for the validation purposes, we have not
19
includedthose,eventhoughalongtheprojecttheyhavebeenthesubjectofintenseanalysisandmodeling
activities.
Figure9The“ConferenceofServicesmanagement”sub-processwithinthe“TitoloUnico”BP
One of the sub-processes specified under the “Titolo Unico” is the management of the Conference of
Services(CoS).
TheCoSisameeting,activatedwhenrequiredbylaw,inordertodrawacollectivedecisionasquicklyas
possible; typically it is required when is necessary to obtain an agreement, a clarification, an act of
concertationorassentbyoneormorePAsinvolved.Thiscanhappenoncomplexdossiers,orwhensome
PAs show a delayed response time if compared to the standard timing of completion of the overall
procedure, or in any case where there is no consensus among all the public actors involved (and even,
sometimes,thesameprivateapplicant).Inparticular,thesimultaneousCoS,mappedinthesub-processin
Fig. 9, is a meeting where all the involved participants (i.e SUAP offices, other municipal offices, other
government units and possibly the applicant) discuss on the case, from all points of view, together and
concurrently, and decide whether the request could be eligible or not. According to the procedural flow,
theSUAPoffice:(i)announcestheCoSandconvenestheadministrationsinvolved-sendingalltheenclosed
documentation, (ii) manages the agenda and the works of the meeting, (iii) collects the opinions
summarizing them in a report, and then (iv) takes the final decision (which can be positive, negative,
conditionedtothefulfillmentoffurtherrequirementsbytheentrepreneur,etc.).
When using the Learn PAd platform, the learners were asked to play the role of a SUAP representative
contact person. In some phases of the simulation, they had also the option to play the role of a third
administrationreferee,calledtoexpressitsopinioninaCoS.
20
A new regulation, changing consistently many circumstances and aspects of procedural flow of the SUAP
CoS,hasbeenrecentlyintroducedbytheDecree“DLgsn.12730/06/2016”.Unfortunatelythishappened
atalatestageofthedeploymentofthecontentsontheLearnPAdplatform;similarlywedidnothavethe
time to adapt the CoS moodle course on the MARLENE platform. For validation purposes we decided to
leave, in both platforms, the outdated content, ie responsive to the previous legislation dictates.
Nevertheless we reacted offering the users some additional learning material, as better described in the
nextsection.
3.2.3Usedmethodology
At the beginning of August we invited via email more than 90 public managers, officers, employers, civil
servants or even freelance professionals, involved in the provisioning of the OSS/SUAP services towards
entrepreneurs,tojointheLearnPAdfinalvalidationactivities.TheywerepeopleworkingforSUAPoffices
or for local government bodies called to give an opinion on OSS practices or even other public
administrationsindirectlyinvolvedintheOSSprocessesforcoordination,training,technicalandregulatory
support,etc.
We explained them about the Learn PAd project - and its aim to develop a prototype platform for the
trainingofPublicEmployeesbasedonworkprocesses-andweaskedfortheiravailabilityandwillingness
to participate in this trial, carried out by the Marche Region ICT Department, in collaboration with the
Regional School of Public Administration, as well as for their help in finding other people to engage. We
informedthemaboutestimatedtiming,modalitiesandrequiredeffortforthevalidationexperiment.We
told them that we would have assessed by questionnaires, aggregating their answers, the degree of
teachingeffectivenessofthenewWikimodel-basedLearnPAdplatform,accessibleviaawebbrowser24
hourson24duringthetrialtimewindow,alsoincomparisonwithmoretraditionale-learninginstruments.
We also specified that the contents found in the platform, related to the SUAP, were to be considered
purelyasameanstotestitsvalidity.Forthisreasonwewouldnothaverequiredanypreviousorextensive
preparationonthesubject,meaningthatevenaninexperiencednovice,aslongasinsider,couldattendthe
testtoseeiftheplatformhadprovedausefullearningtool.
Wereceivedback72candidaturesfrompeopleavailabletojointheexperimentation.
InSeptemberwecommunicatedtheofficialstartofthetrial,from15/09till26/09,sendingtheusersafirst
exantequestionnairetoassess,briefly,their:
• personaldetails,relevantcharacteristicsandprofileinformation,neededtoclassifyandaggregate
thesamples
• aself-assessmentofthelevelofexpertise–from1to8–theybelievedtohavewithrespectto5
mainEQFqualificationsrecommendedtoworkintheSUAParea
Beforethetestsbegan,wegatheredthecompletedquestionnaires.
We divided the candidates in two groups (the main one of Learn PAd users and the control one of
MARLENEusers),ashomogeneousaspossible,consideringtheirexperienceandexpertiseinOSS/SUAP,on
thebasisofourknowledgeandunderstanding.Asaresult,eachofthetwogroupsresultedcomposedof:
almosta50%ofnoviceusers(withlessthan1yearofworkpractice),a25%ofintermediateprofessionals
anda25%ofexpertusers(withmorethan5yearsofworkpractice).
21
Then we provided the users with all the technical detailsto start (web address: respectively http://Learn
PAd.regione.marche.it:8080/xwiki and http://marlenescuola.regione.marche.it/moodle/, credentials,
operativeinstructionsandmanuals,telephoneandemailcontacttorequestsupport).Wealsoaskedthem
to keep track of the real daily time spent in the learning activities, as this would also be the subject of
specificfollow-upquestions.
For the Learn PAd sample, we created and assigned to users individual accounts to distinguish them by
profiles(novice,intermediateprofessional,expert)andtypeofSUAP(districtualormunicipal)accordingto
the ex ante information provided. Those profiles were needed to customize the user experience with
referencetothecontentproposedbytherecommendationsystem.Thisdistinctionhasnotbeennecessary
fortheusersofthecontrolgroup,whoaccessedbyauthenticatingthemselveswiththelocal“FedCohesion
/SPID”identificationsystem.
AssaidbeforetheDecree“DLgsn.12730/06/2016”introducedsignificantchangesinmanycircumstances
and aspects of procedural flow of the SUAP conference of services. However, for integration and
rectification,inthetrialstartingemailwealsoattachedthreedocuments:(i)thenewlegislationitself,(ii)a
brief guide to the changes introduced, (iii) an extensive interpretative guidance of the new regulatory
framework.Thesesupplementarydocumentsweredraftedwiththreelevelsofdetail,respectively:(i)for
experienced staff, (ii) for personnel with intermediate knowledge (iii) for staff with basic skills. Both the
MARLENE and Learn PAd users were advised to consult the information material consistently with their
knowledge; moreover, in the Learn PAd case, the documents were also referenced as learning materials
that the recommender would have returned on the basis of the appropriate user profile (if expert,
intermediateornovice).
ParticularlyforLearnPAd,sinceitisaprototype,somesectionsoftheplatformcouldhaveresulted,toa
first approach, not very easy to to use and not immediately understandable. For this reason, during the
tests, two information meetings (on 16 and 21 September at the Marche Region) had been provided to
illustrate, with practical demonstrations, the platform main features. They were attended by almost 10
users.
Attheendofthetrialperiodwesenttheusersanexposttestingquestionnairetoassesstheeffectiveness
ofthetrainingplatformandofthepeculiarteachingmodule.
We informed them that the information and data retrieved would have been handledby the partners of
the Project Learn PAd so as to ensure, under the Italian “Privacy code (Dlgs 196/2003)”, its security and
confidentiality. So we therefore asked them to sign a waiver of consent to the processing of the
informationanddataprovided,whendeliveringthecompletedquestionnaire.
Finallywecollectedtheexpostquestionnaires(31fromtheLearnPAdsampleand30fromtheMARLENE
sample). The tests results have been processed anonymously and on an aggregative basis, and analyzed
comparingthetwosamples.Theresultsarereportedinsection3.3and3.4.Theconclusionsarereportedin
Chapter4.
3.2.4Questionnaires
Asalreadyexplainedwedevelopedandsubmittedtothelearnerstwokindofquestionnaires.
22
3.2.4.1Theexantequestionnaire
Theexantequestionnaire(fullyreportedintheappendix)askedtheuserssomepersonaldetails,inorder
(i) to be able to classify and aggregate the samples by relevant characteristics, (ii) to give each user an
appropriate profile with respect to the parameters of experience (expert, intermediate, novice) and
organization of their SUAP offices (districtual or municipal), and (iii) to verify a properly homogeneous
distribution of the users between the Learn PAd group and the MARLENE one. We collected useful
information regarding Gender, Age, Geographical distribution within Marche region, Qualification, Public
role, Years of work at (or on behalf of) a PA, Years of experience in OSS/SUAP, Type of public body they
serve.
Moreover,theexantequestionnaire,submittedbeforethestartingoftheplatformtestingactivities,aimed
also to get from each user a self-assessment of the level of expertise with respect to 5 main European
QualificationsFrameworkskillsweanalyzeasrecommendedtoworkintheSUAParea.
TherecommendedEQFskillswere:
• Front-officeactivities,information,communicationandmanagementoftheexternalrelationswith
thepublic[neededforthetasksrelatedtothepreparationofworkandtoprovideinformationto
theSUAPdeskexternalusers]
• Assessing the administrative and procedural regularity of a request, through checking its
completenessandformalcorrectness[neededforthetasksrelatedtotheexecutionofaBPandto
checktheadmissibilityofaSUAPapplication]
• Managementandcoordinationofspecificadministrativeprocedures[neededforthetasksrelated
to the execution of a BP and i.e. to send documentation to other PAs, to ask for integrations, to
activatetheCoS,tosolicitthethirdpartiesincaseofdelayornon-compliance,etc.]
• Verifying the congruency and pertinence of data and documentation submitted in an instance,
consideringthemeritswithrespecttotheactivitythattheuserintendstostartortheintervention
tobeundertaken[neededforthetasksrelatedtodeepcheckingandmonitoringandnotonlyto
accomplish a final check of the correctness of the whole acquired documentation, but also to be
abletoissueanexpertopinionontheadmissibilityofaSUAPauthorization]
• Drawingupformaldocuments(Decrees,reports,letters,etc.)duringtheexecutionorattheendof
an administrative procedure [needed for the tasks related to the final audit and release of SUAP
formalactsandtobeabletowriteandorganizeadministrativetextsinapropermanner]
Weaskedlearnerstoattributethemselvesalevel,from1(low)to8(high),asspecifiedinthetablebelow,
derivedfromtheEuropeanQualificationsFramework(EQF).
23
Skills
[cognitive(involvingtheuseof
EQFlevel
Knowledge
[theoreticaland/orfactual]
logical,intuitiveandcreative
thinking),andpractical
(involvingmanualdexterityand
Competence
[responsibilityandautonomy]
theuseofmethods,materials,
toolsandinstruments)]
Level1
Basicgeneralknowledge
Basicskillsrequiredtocarryout
simpletasks
Level2
Basicfactualknowledgeofafield
ofworkorstudy
Basiccognitiveandpractical
Workorstudyunder
skillsrequiredtouserelevant
supervisionwithsome
informationinordertocarryout autonomy
tasksandtosolveroutine
problemsusingsimplerulesand
tools
Level3
Knowledgeoffacts,principles,
Arangeofcognitiveand
processesandgeneralconcepts,in practicalskillsrequiredto
afieldofworkorstudy
accomplishtasksandsolve
problemsbyselectingand
applyingbasicmethods,tools,
materialsandinformation
Takeresponsibilityfor
completionoftasksinworkor
study;adaptownbehaviourto
circumstancesinsolving
problems
Level4
Factualandtheoreticalknowledge
inbroadcontextswithinafieldof
workorstudy
Arangeofcognitiveand
practicalskillsrequiredto
generatesolutionstospecific
problemsinafieldofworkor
study
Exerciseself-management
withintheguidelinesofworkor
studycontextsthatareusually
predictable,butaresubjectto
change;supervisetheroutine
workofothers,takingsome
responsibilityfortheevaluation
andimprovementofworkor
studyactivities
Level5
Comprehensive,specialised,
factualandtheoreticalknowledge
withinafieldofworkorstudyand
anawarenessoftheboundariesof
thatknowledge
Acomprehensiverangeof
cognitiveandpracticalskills
requiredtodevelopcreative
solutionstoabstractproblems
Exercisemanagementand
supervisionincontextsofwork
orstudyactivitieswherethere
isunpredictablechange;review
anddevelopperformanceof
selfandothers
Workorstudyunderdirect
supervisioninastructured
context
24
Level6
Advancedknowledgeofafieldof
workorstudy,involvingacritical
understandingoftheoriesand
principles
Advancedskills,demonstrating
masteryandinnovation,
requiredtosolvecomplexand
unpredictableproblemsina
specialisedfieldofworkorstudy
Managecomplextechnicalor
professionalactivitiesor
projects,takingresponsibility
fordecision-makingin
unpredictableworkorstudy
contexts;takeresponsibilityfor
managingprofessional
developmentofindividualsand
groups
Level7
Highlyspecialisedknowledge,
someofwhichisattheforefront
ofknowledgeinafieldofworkor
study,asthebasisfororiginal
thinkingand/orresearch
Specialisedproblem-solvingskills
requiredinresearchand/or
innovationinordertodevelop
newknowledgeandprocedures
andtointegrateknowledgefrom
differentfields
Manageandtransformworkor
studycontextsthatare
complex,unpredictableand
requirenewstrategic
approaches;takeresponsibility
forcontributingtoprofessional
knowledgeandpracticeand/or
forreviewingthestrategic
performanceofteams
Criticalawarenessofknowledge
issuesinafieldandatthe
interfacebetweendifferentfields
Level8
Knowledgeatthemostadvanced Themostadvancedand
frontierofafieldofworkorstudy specialisedskillsandtechniques,
andattheinterfacebetweenfields includingsynthesisand
evaluation,requiredtosolve
criticalproblemsinresearch
and/orinnovationandtoextend
andredefineexistingknowledge
orprofessionalpractice
Demonstratesubstantial
authority,innovation,
autonomy,scholarlyand
professionalintegrityand
sustainedcommitmenttothe
developmentofnewideasor
processesattheforefrontof
workorstudycontexts
includingresearch
Table3DescriptorsdefininglevelsintheEuropeanQualificationsFramework(EQF)
3.2.4.2Theexpostquestionnaires
Ontheotherside,theexpostquestionnaire(fullyreportedintheappendix)askedeachlearnertorepeat
thesameEQFself-assessmentattheendofthetrainingactivities,inordertoverifywhethertheuseofthe
platformhadincreased-andtowhatextent-hisknowledgeandhisskills.
We also asked the user how much time he/she devoted to the use of the platform, in terms of overall
period(indays)andintermsofactualtime(inhours).
At the same time, we asked the users to answer 15 multiple choice questions related to specific SUAP
content. Each question answered correctly was worth 1 point and only one of the possible answers was
correct. Multiple choices were generally 5 for each question, except for the questions: 1 , 2, 9, 10 (the
possiblechoiceswere3)andforthequestions:5,14(thepossiblechoiceswere6).
The15questionscouldbegroupedinto4homogeneousclusters,bytypeofskills/knowledgeassessed:
• group A (theoretical questions: n° 4, 5, 6 and 7) aimed at verifying a successful learning of content
deliveredthroughdidascalicfunctions.
25
•
•
•
group B (questions on the new CoS regulatory reform introduced by the Decree “DLgs n. 127
30/06/2016”: n° 3, 9 and 10) aimed at verifying a successful learning of content delivered by
downloadingsupplementarymaterial.Inotherwords,wewantedtotestiftheusershadbeenableto
read the documentation attached to an email, or downloaded as an additional learning material, to
analyzeitandtolearninamoretraditionalmode(notfullybrokeredbytheplatform).
group C (questions related to BPs: n° 1, 2, 8, 11 and 12) aimed at verifying a successful learning of
content delivered through Business Process Modeling and the logic of the procedural flows. It was
requiredthecapabilityoftracingamentalpathandmovingbetweenthedifferenttaskswhichmakeup
theCoSprocess,incompliancewiththeassociatedrulesandtiming.
groupD(questionsaboutpracticalcases:n°13,14and15)aimedatverifyingasuccessfullearningof
content delivered through simulations or hands-on practice on real and concrete use cases. It was
requiredthecapabilityofapplyingthetheorytoaparticularconcretecasetakenfromtherealworld,
synthesizingandevaluatingthemainelements,andstrivingtosolveit.
Finally, we asked the users to give their opinion, mostly expressing a numerical value to the elements
submitted to their judgment [on a Likert rating scale from 1 = low to 5 = high], about usability, user
experience,platformeffectivenessandothertopics.Indetail,theaspectsbroughttotheirattentionwere:
1. Overalldegreeofsatisfactionintheuseofthise-learningsolution
2. Degreeofeffectivenessofthevariousfunctionalitiesavailablewithintheplatform
3. Degreeofusability,easeofuseandaccessibility
Weinformedtheuserthat"Accessibility",inourcontext,meansthecharacteristicofacomputer
systemtobeeasilyusablebyanyuser,regardlessofhissensory,motororpsychicindividual
capacities,ofanytemporaryorpermanentdisability,ofevenculturalaspectsoraspectsrelatingto
familiaritywiththeuseofICTinterfaces.
4. Perceiveduserexperiencewithregardsto:
a. graphicsandcolors
b. theunderstandingofsymbolsandicons
c. themeaningoftextualcontents
d. searchandsuggestionofcontenttheywanttofind
e. orientationandguidanceinbrowsingthepagesorthecontentstructure
f. thestructuring,placementandpositioningofthecontent
Weinformedtheuserthat“UXoruserExperience”,inourcontext,meansthesubjectivefeelings
thathe/sheexperiencedusingtheplatformwithreferencetotheempiricandaffectiveaspects,to
thevalueandmeaningattributionhe/sherelatedtotheavailabilityandtheinteractionwiththe
solution,aswellastotheoverallperceptionsandpersonalthoughtshe/shehadaboutitslearning
effectiveness,theeaseofuseoftheinterface,thesystem'sefficiency,etc..
5. Degreeofcompletenessoftheavailablefeatures
6. Degreeofintegrationandhomogeneityofallthemodulesavailablewithinthelearning
environment
7. Degreeofinnovationandnoveltyoftheproposedsolution
8. Degreeofusefulnessandappropriatenessoftheplatformwithrespecttothetrainingneedsof
learners
9. Detectedneedofagreaterpresenceof:
a. awizard-drivencontentfruition
b. gamificationfeatures
26
c.
d.
e.
f.
Weinformedtheuserthat"Gamification",inourcontext,meanstheuseofelements
borrowedfromthegamesandotherfuntechniques,adaptedtoexternalcontexts(eg.
scores,levels,rewards,badges,giftsandotherloyaltymethodsorpracticestoarouse
interestorcontributetotheusersatisfaction)
anappropriateuseofmultimedia
interactivitybetweenlearnerandcontent
interactivityandknowledgeexchangebetweenplatformusers
multilingualismorexplanatorytechnicalvocabularies,glossaries,dictionaryofacronyms,
bibliographicalreferences,etc.
tutoringandexternalteachingsupport
g.
Theobtainedresults,bothfromcontentandusabilityquestions,oftheLearnPAdgrouphavebeen
comparedwiththecontrolgroup.
Moreover we compared the Learn PAd usability results to the feedback received in the early evaluation
phase(exceptfortheelements:4-fromatof-and9e,thatwerenotpreviouslycollected).
Thequestionsn°1,2and5requiredalsoafree-textcomment(insynthesis,respectively:1-motivateyour
opinion;2-whichfeaturesaretobeconsideredmostuseful;5-whatfeaturesaremissingorinadequate,
and why, to consider the platform a valuable learning tool). Those feedback have been analyzed and
summarizedinsection3.4,alsoseparatingpositiveandnegativenotes.
We present all the data and figures received from the ex ante and ex post questionnaires in the next
section.
3.3Resultsfromquestionnaires
In this section the results obtained from the answers to the ex ante and ex post questionnaires are
representedintables(andalsographically,forthekeyaspectstohighlight).
3.3.1Profilingandcontextdata
We begin with representing the demographical data and the context information of the Learn PAd and
MARLENEsamplegroups:
Sample:
LearnPAd
MARLENE
N°users
Invitedtotesting:36
Invitedtotesting:36
Questionnairescollected:31
Questionnairescollected:30
Gender
Males:17
Males:18
Females:14
Females:12
Age
Mean:43,61
Mean:44,40
Compositionbyagegroups:2under30
Compositionbyagegroups:2under30
years,9from30to40,14from40to50,6
years,6from30to40,16from40to50,6
over50
over50
Geographical 18intheprovinceofAncona,4inthe
15intheprovinceofAncona,2inthe
distribution
provinceofAscoliPicenoandFermo,4in
provinceofAscoliPicenoandFermo,7in
withinMarche theprovinceofMacerata,5intheprovince theprovinceofMacerata,6intheprovince
region
ofPesaroeUrbino
ofPesaroeUrbino
Qualification
Universitygraduates:23
Universitygraduates:23
Schoolgraduates:8
Schoolgraduates:7
27
Publicrole
0executivemanagers,12officers,12
1executivemanager,17officers,8
employees,7freelancersservingaPA
employees,4freelancersservingaPA
Yearsofwork Mean:14,81
Mean:13,27
at(oron
CompositionbyPAexperiencedclusters:7 CompositionbyPAexperiencedclusters:6
behalfof)aPA upto5years,4from6to10,5from11to
upto5years,4from6to10,10from11to
15,8from16to20,7morethan20
15,5from16to20,5morethan20
Yearsof
Mean:2,35
Mean:3,87
experiencein CompositionbySUAPexperiencedclusters: CompositionbySUAPexperiencedclusters:
OSS/SUAP
20novices(upto1years),7intermediate
15novices(upto1years),7intermediate
professionals(from2to5),4experts(more professionals(from2to5),8experts(more
than5)
than5)
Typeofpublic 1workinadistrictualSUAP,3ina
7workinadistrictualSUAP,2ina
bodyserved
municipalSUAP,15inathirdparty
municipalSUAP,18inathirdparty
administration,12inanotherPA(indirectly administration,3inanotherPA(indirectly
involvedintheSUAPprocessesfor
involvedintheSUAPprocessesfor
coordination,training,support)
coordination,training,support)
Table4Aggregateddatarelatedtothetwosamplesinvolvedinthetesting
Figure10a/bTheLearnPAdandMARLENEsamplesbygender
28
Figure11a/bTheLearnPAdandMARLENEsamplesbyage
Figure12a/bTheLearnPAdandMARLENEsamplesbyqualificationineducation
29
Figure13a/bTheLearnPAdandMARLENEsamplesbypublicrole
Figure14a/bTheLearnPAdandMARLENEsamplesbyyearsofworkat(oronbehalfof)aPA
30
Figure15a/bTheLearnPAdandMARLENEsamplesbyyearsofexperienceinOSS/SUAP
As for the overall time spent on the platforms, Learn PAd usage time totalized 169 hours in 90 days (on
averageperuser:5:45hoursin3days);MARLENEusagetimetotalized96hoursin64days(onaverageper
user:3:20hoursin2days).
Figure16Theplatformsoverallusagetime(inhours)
3.3.2Competenceself-assessmentdata
Within the Learn PAd sample, 14 users out of 31 (45%) said they had improved one or more EQF skills;
regardingMARLENE,13usersoutof30(43%)saidthis.
31
Figure17ThedeclaredimprovementinEQFskillswithinthetwosamples
Thetotalperceivedimprovementsineachcompetence-calculatedasapercentageofthesumoftheex
postincreasesonthesumoftheexantevaluesdeclared-amountedtothefollowingvalues:
32
EQFSkill
LearnPAd
%ofusers
standard
perceived
deviation
improvement amongthe
onaverage
sample
9%
0,57
Front-officeactivities,information,
communicationandmanagementof
theexternalrelationswiththepublic
Assessingtheadministrativeand
9%
proceduralregularityofarequest,
throughcheckingitscompleteness
andformalcorrectness
Managementandcoordinationof
15%
specificadministrativeprocedures
Verifyingthecongruencyand
12%
pertinenceofdataand
documentationsubmittedinan
instance,consideringthemeritswith
respecttotheactivitythattheuser
intendstostartortheinterventionto
beundertaken
Drawingupformaldocuments
6%
(Decrees,reports,letters,etc.)during
theexecutionorattheendofan
administrativeprocedure
Table5PerceivedimprovementsinEQFskills
MARLENE
%ofusers
standard
perceived
deviation
improvement amongthe
onaverage
sample
2%
0,25
0,63
5%
0,37
0,70
6%
0,50
0,79
4%
0,34
0,47
6%
0,48
Figure18ThepercentageofdeclaredimprovementineachrelevantEQFskillwithinthetwosamples
33
3.3.3Learningassessmentdata
Summarizingthelearningoutcomesforthecorrectanswerstothemultiplechoicequestions:
In the Learn PAd sample, 15 users out of 31 (48% within the sample / 25% of the whole users) have
correctly answered more than 70% of questions (that is the minimum threshold set by the PA Training
Regional School to assess the successful passing of a course). The sample obtained a 62% of the total
achievablepoints.
IntheMarlenesample,16usersoutof30(53%withinthesample/26%ofthewholeusers)havecorrectly
answeredmorethan70%ofquestions.Thesampleobtaineda61%ofthetotalachievablepoints.
Figure19Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestions
withinthetwosamples
We can also go deeply in the analysis separating the 15 questions into the 4 homogeneous clusters
previouslyexplained:
34
Cluster
type
N°of
involv
ed
questi
ons
N°Users
with>70%
correct
answers
(outof31)
LearnPAd
Users
percentage
amongthe
sample
%oftotal
points
achieved
compared
tothe
maximum
achievable
52%
A
4
9
29%
theoretic
al
questions
B
3
18
58%
76%
questions
onanew
regulator
yreform
C
5
12
39%
61%
questions
relatedto
BPs
D
3
9
29%
59%
questions
about
practical
cases
Results
15
15
48%
62%
onthe
wholeset
of
questions
Table6Profitresultsforeachclusterofquestions
N°Users
with>70%
correct
answers
(outof30)
Marlene
Users
percentage
amongthe
sample
12
40%
%oftotal
points
achieved
compared
tothe
maximum
achievable
48%
20
67%
80%
11
37%
62%
10
33%
56%
16
53%
61%
35
Figure20Thenumberofuserswhoanswercorrectlytomorethan70%oftheproposedquestionswithinthe
samplesandperclustersofquestions
36
3.3.4Usabilitydata
AsfortheUsabilitycriterionswecalculatedtheseresults:
Criteria
Overalldegreeof
satisfactionintheuseof
thise-learningsolution
Degreeofeffectiveness
ofthevarious
functionalitiesavailable
withintheplatform
Degreeofusability,ease
ofuseandaccessibility
perceivedpositiveuser
experiencewithregards
tographicsandcolors
totheunderstandingof
symbolsandicons
tothemeaningoftextual
contents
tosearchandsuggestion
ofcontenttheywantto
find
toorientationand
guidanceinbrowsingthe
pagesorthecontent
structure
tothestructuring,
placementand
positioningofthe
content
Degreeofcompleteness
oftheavailablefeatures
LearnPAd
earlier
platformv.
6/11/2015
Average
rating
[from1=
lowto5=
high]
LearnPAdfinalplatformv.
14/09/2016
MARLENEplatform
Average
rating
[from1
=lowto
5=high]
N°users
who
expressed
anopinion
above
average
[over3]
%users
who
expressed
anopinion
above
average
[over3]
Average
rating
[from1
=lowto
5=high]
N°users
who
expressed
anopinion
above
average
[over3]
%users
who
expressed
anopinion
above
average
[over3]
3,14
2,97
8
26%
3,07
10
33%
3,32
2,94
10
32%
3,13
11
37%
2,42
2,90
12
39%
3,33
13
43%
N.A.
3,03
8
26%
3,50
18
60%
N.A.
2,84
7
23%
3,43
15
50%
N.A.
3,10
10
32%
3,60
17
57%
N.A.
2,68
5
16%
3,07
11
37%
N.A.
2,65
8
26%
3,57
18
60%
N.A.
2,97
11
35%
3,40
16
53%
3,42
3,06
10
32%
3,17
9
30%
37
Degreeofintegration
andhomogeneityofall
themodulesavailable
withinthelearning
environment
Degreeofinnovationof
theproposedsolution
Degreeofusefulnessand
appropriatenessofthe
platformwithrespectto
thetrainingneedsof
learners
Detectedneedofa
greaterpresenceofa
wizard-drivencontent
fruition
Detectedneedofa
greaterpresenceof
gamificationfeatures
Detectedneedofa
greaterpresenceofan
appropriateuseof
multimedia
Detectedneedofa
greaterpresenceof
interactivitybetween
learnerandcontent
Detectedneedofa
greaterpresenceof
interactivityand
knowledgeexchange
betweenplatformusers
Detectedneedofa
greaterpresenceof
multilingualismor
explanatorytechnical
vocabularies
Detectedneedofa
greaterpresenceof
tutoringandexternal
teachingsupport
3,42
3,16
9
29%
3,07
11
37%
4,33
3,61
19
61%
3,07
12
40%
4,00
3,06
11
35%
2,80
5
17%
3,71
3,19
9
29%
3,27
9
30%
4,20
2,90
9
29%
3,33
8
27%
4,71
3,45
6
19%
3,73
2
7%
4,50
3,35
6
19%
3,67
2
7%
N.A.
3,42
4
13%
3,77
5
17%
4,28
3,35
8
26%
2,73
13
43%
4,00
3,23
5
16%
3,27
8
27%
Table7Dataandfiguresrelatedtotheusabilityexpostanswers
38
Figure21Acomparingoverviewonacquiredopinionsaboutusability,userexperienceandplatform
effectivenessbytheusersofLearnPAdandMARLENE
3.4Synthesisofthecollectedfeedback
Inthissectionasyntheticsummaryofthecollectedverbosefeedbackisdescribed.Thefulllists(inItalian
andinEnglish)arereportedintheappendix.
3.4.1Positiveoverallopinionsandplatformsstrengths
Figure22a/bThePOSITIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE
39
RegardingtheLearnPAdplatform,wereceived20among31positivefeedbackrelatedtotheBPbrowsing
functionality, said to be a simple, logical, intuitive, fast and flexible way to navigate, by content or by
graphic schemes, through the workflow; the processes themselves and their related content were
describedasaccurate,wellstructured,clearlyexplainedandeasytounderstand.
10usersalsoappreciatedthesimulationmodeanditsexercisesbasedonrealcases.
4usershavepraisedthesocialinteractionfeaturesbasedoncomments,thechatandtheotherformsof
contribution;4more,inparticular,evidencedthefunctionalitiesfordrawingupandaddingcontributively
structuredtextsthroughthewikienvironment.
2learnersnoticedtheadvantagesofthedynamicrecommendationofnewcontentbasedonsemantics.
Finally2usershavespokenwellofthehomogeneousandresponsivevisualidentityand1userspokeofa
newerandinterestingwaytolearn.
About MARLENE 13 users among 30 liked the intermediate comprehension tests via multiple choice
questions.
6userssucceededinupdatingtheirknowledgeandinquicklymemorizingSUAPnotions,thankstothewell
describedandsyntheticcontentandtosomein-depthssections.
5userstalkedaboutaneasytousetool,withaclearinterfaceandasimplesequentialflowthatlinksallthe
pages.
3 users judged valid the fact that the content were presented through timed animation, highlights and
audio.
Finally2usersrecommendede-learningandonlinewebcontentasawaytosavetimeandmoney.
3.4.2Negativeoverallopinionsandmissingfunctionalities
Figure23a/bTheNEGATIVEFEEDBACKWord-CloudsforLearnPAdandMARLENE
40
Speaking about Learn PAd, 19 users among 31 have complained about an initial feeling of disconcert,
uncertainty, displacement, unclarity. They reported that the platform was difficult to explore since there
wasnotapredefinedorientingpathtofolloworexplainingtheprogression.Thereweretoomanyunknown
availablelinksandfeatures,vehicolatedthroughnonexhaustiveintroductionsandthroughatrickyandnot
so intuitive interface. One also remarked that just learning to use the platform itself is hardly time
consuming.
10learnerswrotethatsimulationdidnotseemtoworkwellorwasnotwellexplainedoreventhatitwas
shortlyeffectiveinstimulatingrealproblemsolvingattitudes
4usersreportedthatthegraphicalschemesrepresentingtheflowchartsweredifficulttouseandread
3usersnoticedthepresenceofdisturbingtypos(eg.e&apos)andenglishwords
3 users suggested to strengthen and make more attractive the social collaboration and interaction
capabilities
2 learners believe that the Learn PAd platform is not suitable for beginners as the course and even the
profitquestionnairerequiredasaprerequisiteapreviouslyacquiredknowledge.
2peopleaskedtointroducetheuseofvideo,multimediaandanimationsinordertofacilitatelearning.
2 users commented that the recommending bar showed poor content and it could have been more
effective.
AnisolatedopinionreportedthatcontentaboutCoSwasoutdated.
Concerning Marlene, 11 users among 30 suggested to introduce, as more advanced e-learning systems
provide,interactiveandsocialfunctionalities,togaintheopportunityofhavingchat,submittingquestions
on regulatory concerns and complex situation to handle, receiving in-depth opinions and legislation
interpretationfromexpertsorothercolleaguesattendingthecourseorexperts.
7 learners described the course as very basic and theoretical, suitable for beginners, lacking insights on
everydaypracticalwork,realapplicationsandcomplexsituationtohandle.
4userscomplainedaboutaninsufficientaccessibilitybecauseofthepop-upblockerandthebrowserorI/O
devicesconfigurationchangestobeputinplace.
Other4usersreportedthelackofacontentsearchsystemtogodirectlytothecontentofinterest.
ResidualsingleopinionsnoticedoutdatedcontentforCoS,thehighamountoftimeneededtocompleteall
thetrainingmodule,theabsenceofgamificationaspectsandofvideosasresourcesforlearning.
41
4Resultsanalysisandconclusions
Insummary,thevalidationdatacollectedareofdifferenttype.Precisely,foreachrespondent,theyinclude:
X.
Aself-assessmentevaluationonowncompetenceconcerningthe5expectedEQFcapabilities,
collectedexanteandexpost;
Y.
15answerstomultiple-choiceprofitquestionsrelatedtolearningcontents(withintheSUAP
contextasavalidationscenario);
Z.
Qualitativeanswers,onaLikertscale,on9questions(andfor3ofthembyanaccompanying
commentary),assessingusabilityandlikingoftheplatform.
Assaid,alltheabovedatahavebeenidenticallycollectedfrombothusersofLearnPAdandofMarlene,as
abaseline.
Thequestionsweaddressedintheexperimentforvalidationcanbestatedasfollows:
VQ1: HowdoeslearningthroughtheuseofLearnPAdapproachcomparesagainstlearningthrough
theuseofamoretraditionale-Learningplatform(suchasMarlene),intermsoflearning
effectiveness?
VQ2:HowdousersassesstheusabilityandthefunctionalityoftheLearnPAdplatform?
Theconsiderationsrelatedtodatacollectedbynon-statisticallysignificantsamplesshouldbetakenwith
thenecessaryprecautions.Howeverwetriedtosummarizesomegeneralconclusions.
ToaddressVQ1weexamineboththequalitativeself-assessmentbyusers(answerstogroupXquestions)
andtheanswerstoquestionstestingusersacquiredknowledgeonthesubjectoflearningexperiment
(groupYquestions).Thefirstgroupofanswerssupportsasubjectivemeasureoflearningeffectiveness
whereasthesecondgroupsupportsamoreobjective-eventhoughlimited-measure.
Themeasurementoflearningeffectivenesshasbeenassessedalsoinrelationtothelearnersprofile(such
astheyearsofworkingexperienceinaPA).
ToaddressVQ2weexaminetheanswerstothegroupZquestions(boththeresultingratingsandthefreetextcomments,aggregatedforrelevancyandsimilarity).
RESULTSOFGROUPY(profitanswers)
We start the analysis considering answers to group Y. Overall we can see (Table 8) that the provided
answersarebetterfortheMarlenegroupthathaveahighermedianvalueandlowerstandarddeviation,
althoughthedifferencesaresmall.Wethenperformedastatisticaltesttoassessthenullhypothesisthat
thedifferencebetweenthenumberofcorrectanswersforthetwogroupsfollowsasymmetricdistribution
aroundzero,i.e.,thenullhypothesisisthatthemedianvaluesarestatisticallyequivalent.Becauseourdata
couldnotbeassumedtobenormallydistributed,weadoptedanon-parametricstatisticalhypothesistest,
theWilcoxonsigned-ranktest.Witharesultingp-valueof0.5604,thenullhypothesiscouldnotberejected,
i.e.,themedianvaluesobservedarestatisticallyequivalent.
42
Mean
Median
St.Dev.
Marlene
9,93
10
3,38
LearnPAd
9,23
9
3,5
Table8Overallstatisticsforprofitanswersresults(themeanofthetotalscoreacquiredandthemedian
score)
Werepeatedthetestforthe4clustersofquestionsdescribedinSection3.2.4.2(Atheoreticalquestions,B
questionsonanewregulatoryreform,CquestionsrelatedtoBPs,Dquestionsaboutpracticalcases)and
theresultsarethesame:themediansarestatisticallyequivalent.
Itmeansthatthelearningassessmentareequivalentbetweenthetwosamples,andwecansaythatitisa
good result for the Learn PAd platform, as it is a prototype with respect to the more tested and utilized
Marleneplatform.
We further examined these results looking for possible correlation with some of the factors that we
collected. We hypothesized that demographic characteristics of the learners, such as age or matured
experienceinthetopicofthelearningsession(SUAP),mightimpactthelearningresults.Wealsotestedthe
hypothesis that the time spent on the learning platform (either Marlene or Learn PAd) is not correlated
withsuchresults.
Weperformed,forasetofcomparisonvariablesconsideredsignificant,boththePearsoncorrelationtest
andtheKendalltaucorrelationtest.KendalltauissimilartothemorecommonlyusedPearsoncoefficient
butitdoesnotrequirethevariablestobenormallydistributed.TheresultsarereportedinTable9and10.
Marlene
LearnPAd
YearsofPA
experience
0,360
0,577
Yearsof
SUAP
experience
0,467
0,563
Hours
0,189
dedicatedto
testing
0,096
Table9Pearsoncorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors(PA
experience,SUAPexperience,anddedicatedtime)
With the Pearson index we noticed a no significant correlation of the individual learning results with
respect to the platform time usage and a moderate correlation, more evident for the Learn PAd sample,
with respect to the years of working experience within a PA and regarding SUAP. This is plausible and it
couldindicatethat,astheLearnPAdapproachiscenteredonworkprocesses,learningiseasierandmore
immediateforthosewhoarethelongestmorefamiliarwiththeseprocesses.
43
Marlene
LearnPAd
Ageof
learners
0.2827474
0.403308
Yearsof
SUAP
experience
0.5207782
0.469423
Hours
0.1636847
dedicatedto
testing
0.15134
Table10Kendalltaucorrelationbetweenthenumberofcorrectanswersandpossiblyinfluencingfactors
(age,SUAPexperience,anddedicatedtime)
ForinterpretingthedatadisplayedinTable10(whereallentriesarestatisticallysignificantata95%level),
we use the Guildford scale1 [1], in which correlations with absolute value less than 0.4 are described as
“low”,0.4to0.7as“moderate”,0.7to0.9as“high”,andover0.9as“veryhigh”.
Weobservedsomeinterestingandconvincingresults,asfollows.
The age of respondents has a low correlation with the number of correct answers both for Marlene and
LearnPAd.
Thetimespentontheplatformdoesnothaveastatisticalcorrelationwiththeamountofcorrectanswers.
Thisisinlinewiththecurrentstate-of-the-art.
Even if time had no relevance on the progression of learning, we found that Learn PAd, to be tested,
requiredagreatereffortinhoursthanMarlene(abouttwiceofwhatusersspentonMarlene).LearnPAd
sample users provided a higher level of commitment during the tests, maybe also because of the huge
amount of content that the platform contains (about three times the amount that is contained in the
Marlenecourse).Weacknowledgethattwoweeksoftrialhadbeentoofew,especiallyfortheLearnPAd
platform,whoseusageneedstobeextendedovertimebeforeitcanfulfillallitspotentialstrengths–such
asgeneratingarealcommunityofpractice-andbeforeitisgrantedthepossibilitytowalkthrough,study
andunderstandallthetasksofBP.However,fororganizationalreasons,thefinalvalidationcouldnothave
beenstructuredotherwise.
TheexperienceonSUAPhasamoderatecorrelationwiththenumberofcorrectanswersbothforMarlene
andLearnPAd,andrepresentsthefactorthatisbettercorrelatedwiththenumberofcorrectanswers.
Basedonsuchoutcome,andnoticingthatwithintheLearnPAdsamplealargerpartofsubjectsdeclaredto
be novices in terms of expertise about SUAP, we made a further analysis by computing the correlation
between SUAP and the number of correct answers after removing those users who declared less than 1
years of SUAP experience. The correlation coefficient improved for both Marlene (from 0.5207782 to
0.6958166)andLearnPAd(from0.469423to0.5957597).Suchresultsseemtosuggestthat,independently
1
J.P.Guilford.FundamentalStatisticsinPsychologyandEducation.McGraw-Hill,1942
44
fromtheplatformused,learningeffectivenessisbetterwhensomepreviousknowledgeexists,whichmight
beexplainedbytheextremecomplexityofSUAPregulations.
Asforthefinalprofitresultsoflearning,only48%ofLearnPAdusershasexceededthethresholdof70%of
correct answers (which defines the minimum acceptable level). Similarly, the percentage of MARLENE
sufficient results stood at 53% of users. Given that the Learn PAd group included a higher proportion of
learnerswithupto1yearsofSUAPexperience(20outof31)thanMarlene(15outof30),inpracticewe
canalsoaffirmthatourassessmentisslightlybiasedinfavorofMarlene.
Althoughthefinallearningoutcome,initself,isnotsatisfactory,wenoticedthatitisacross-platformeffect
(independentoftheplatformused).WhatitcanbesaidinfavorofLearnPAd-unlikeMARLENE-itisthat
Learn PAd is a more complex and multi-faceted platform, which surely needs more assimilation time to
deploybenefits(especiallythoseofthecollaborativelearning).
According to the Marche Region PA Training School, the learning results of online methods are generally
lowerthanthoseofthetrainingbasedoncoursesinpresence(andalsothedrop-outrateismuchhigher).
ThiscanbeduetoagenerallackofhumanizationandpersonalrelationshipsineLearningpractices,butas
forthefinalvalidationalsoalackofmotivation-fromthelearnersside-andtheabsenceofamandantor
anagreedtrainingplan-fromtheorganizationsinvolved-couldhaveinfluenced.
Finally, we also performed a Kendall tau correlation analysis between the experience with SUAP and the
individual’s self-assessment (ex post EQF skills values declared). We observed a negative correlation for
both platforms with correlation coefficients of -0.336554 and -0.05058633 for Marlene and Learn PAd,
respectively.Thisresultisconsistentwiththeexpectation:individualswhoarelessexperiencedwithSUAP
will probably notice a greater improvement after using the platform, than those who are already
experienced.
RESULTSOFGROUPX(EQFself-assessment)
WenowdiscusstheresultsfromgroupXanswers.Asimilarnumberofsubjects(45%ofLearnPAdusers
and43%ofMarleneusers)assessedthattheyhaveacquiredsomecompetence,althoughthisnumberis
nothigh.
Table5(andFigure18)inSection3.3.2showthat,attheendofthetest,theLearnPAdusers,morethan
the MARLENE ones, said they had achieved improvements in their competences. However the figures of
the standard deviation shows that, within the MARLENE sample, there has been a more equitable
distributionoftheimprovementsbetweenusers(andso,thatthehighestimprovementattributedtothe
useofLearnPAdmaybeduetothepresenceofsomeoutliers).Checkingdata,wefoundasingleLearnPAd
userself-assessingimprovementinhislevelsof2,3,3,4and2pointsforthefiveEQFSkills(thatcouldbe
exaggerated,orperhapsitcouldmeanthantheuserunderestimatedhisskillsintheexantequestionnaire).
ThepercentageofimprovementsvariesacrossthefivedifferentEQFcompetences,andwethinkitmaybe
usefultolookattheresultssingularlyastheyaddresseddifferenttypeofcompetences.
An interesting thing to notice is that the higher incremental percentage for the Learn PAd sample is the
15% on the EQF Skill “Management and coordination of specific administrative procedures”, which,
notably, is the most used skill when working, for instance, with the coordination of the tasks within the
“TitoloUnico”BP.And,bytheway,theBPbrowsingmodehasbeenconsideredthebestfunctionalityof
45
theLearnPAdPlatform,asassessedbythepositiveverbalfeedbackprovidedbythevastmajorityofthe
users(64%).
Also the better improvements in “Assessing the administrative and procedural regularity of a request,
throughcheckingitscompletenessandformalcorrectness”(9%amongtheLearnPAdsampleinsteadofthe
5%withinMarlene)andin“Verifyingthecongruencyandpertinenceofdataanddocumentationsubmitted
in an instance, considering the merits with respect to the activity that the user intends to start or the
intervention to be undertaken” (12% among the Learn PAd sample instead of the 4% within Marlene),
could be related to the Learn PAd case based activity, performed in simulation mode, where users were
askedtoreadasubmittedapplicationformtocheckitanddeterminehowtoproceed.
Marlene
LearnPAd
exantedeclared
levelinEQF
competences
0,463
0,454
expostdeclared
levelinEQF
competences
0,565
0,413
Table11Pearsoncorrelationbetweenthenumberofindividualcorrectanswersandthesumofdeclared
levelsinthefiveEQFcompetences
With the Pearson index we also noticed a weak correlation, but present in both cases, of the individual
learningresultswithrespecttotheself-assessedexanteEQFlevels.Forwhatconcernsthedeclaredexpost
EQF competence, it seems that MARLENE self-assessments has a slightly higher level of coherence with
respecttothelearningresults.
RESULTSOFGROUPZ(usabilityrelatedanswers)
As a general comment, we observed that the average of ratings stands to mid-range for almost all the
criteria,andforboththeplatforms.
There is a poor correlation between the profit total scores obtained by users and their ratings of overall
satisfactionontheuseoftheplatform.Andatthesametimethesejudgmentsareplacedonintermediate
valuesthatmakesuchanindicatorapoorproxy.
ForLearnPAdwerecordedvaluesbelowaverage,thoughslightly,on:
overallsatisfaction;effectivenessofthevariousfunctionalities;usabilityandaccessibility;userexperience
(UX) for symbols and icons; UX for search and suggestion of content; UX related to orientation and
guidanceinbrowsingthepagesorthecontentstructure;UXforstructuring,placementandpositioningof
thecontent.Webelievethatthenon-positivescore,mostlyabouteffectivenessoffunctionalities,usability
and accessibility, search and suggestion of content, guidance for browsing the content structure (also
46
corroborated by the verbal comments of users) will have undoubtedly to inspire the next actions for the
evolutionoftheplatform.
Marlene
LearnPAd
Overalldegreeof
satisfactionintheuseof
thise-learningsolution
0,357
0,242
Table12Pearsoncorrelationbetweenthenumberofindividualcorrectanswersandtheusersoverallrating
oftheplatforms
About the perceived degree of usability, ease of use and accessibility, the judgment of the users is
improvedcomparedtothepreviousassessmentintheearlyevaluation.
With the exception of the need for greater presence of multilingualism or explanatory technical
vocabularies, Learn PAd presented better results than the previous assessment and than MARLENE
concerningalltheother10indicatorsmonitoringdeficientorabsentfeatures,meaningthat:
•
the consortium addressed many of the comments that emerged in the early evaluation, solving
requestsandissues
•
on average Learn PAd is considered, with a very slight margin of advantage, a complete
environment,concerningtheneededfunctionalitiestobeeffectiveintermsoflearning.Clearlythis
does not imply that the quality of the implemented features is good enough for users or may
requirefurtherimprovements.
LearnPAdpresentedbetterresultsthanMARLENEonlyon:degreeofintegrationandhomogeneityofall
the modules available within the learning environment; degree of innovation of the proposed solution;
degree of usefulness and appropriateness of the platform with respect to the training needs of learners
(and this last is a good result because it considers the content of the platform and how they are
transmitted).
Unless for completeness, innovation and usefulness for training needs, MARLENE scored for the other
criterionsthehighestnumberofusersthatexpresspositivejudgments.
Thisresultisnotsurprising,becauseMarleneisaplatformusedinproductionforseveralyears,whileLearn
PAd,atthemoment,isjustaresearchprototypethat(althoughproducingevidencesoflearning)isheavily
basedonabusinessprocessmodel,inwhichthelocalcivilservantshavelittleornoexperienceatall.
Analyzingthefree-textfeedbackprovidedbytheusers,wenoticedthatmanyofthemconsideredMarlene
amoresuitabletoolforbeginnersthanLearnPAd,mainlybecauseitusesamoresimplifiedandsequential
process of acquiring information and knowledge. Instead Learn PAd would be more suitable for
experienced people, also because it makes a wide use of features for socialization, sharing, contribution
andcollaboration.
47
It is not easy to deduce clear recommendations from the verbal feedback related to the Learn PAd
platform. For example, simplifying the analysis, 10 users among 31 argued that simulation is a very good
idea,but,ontheotherend,thesameamountofuserssaiditdidnotworkwellandshouldberevised;20
outof31usersbelievedvaluablethebrowsingmode,but,atthesametime,other20userstoldtheydid
notfeelwell-orientedduringthenavigation.
Trying to draw some conclusions we believe that, unsurprinsingly, there is still work to do around the
platform.Theactualrelease(andthelearningapproachitrepresents)isonaverageperceivedaseffective,
evenif,probably,itneedsahigherusagetimeinordertorecordmoresignificantlearningoutcomesand
evenifsomefurtherdevelopment(oradjustment)offeaturesandcontentcustomizationareneeded.We
expect to fulfill those evolutionary maintenance requests, able to transform a prototype into something
moresimilartoamarketstablerelease,duringthenextcomingexploitationphase.
We provide a quite high effort to update content in the platform: from process modeling within a single
referencemeta-model,tillthecollectionofeducationalmaterials,theeditingofexplanatorytextsandthe
verification of the state of a constantly changing legislation, passing through the personalization of the
simulation exercises, of the recommender suggestions and of the KPI Dashboard goals to be fed. At the
same time it was also challenging to maintain aligned the technologies and the functionalities of the
platform around a single and integrated application infrastructure. It is therefore logical to assume that
those who intend to pursue the result of making useof Learn PAd as a teaching tool, are equipped with
resourcestomaintainup-to-datetheplatform,eventechnologically,andtocustomizeitscontent(andthe
managedbusinessprocesses)accordingtosimilarneedsandrequestscomingfrommorethanonePAasa
profitable market segment, and consequently they should be subjects able to reap economic or
organizationalgainsfromtheseactivities.
48
Appendix
ExanteQuestionnaire(inItalian)
Datianagraficiediprofiloutente
NomeeCognome…………………………………………..…………………………………………..………………………………………..
Sesso(M,F)……...
Età……………………..
Sedelavorativa(Comune)……………………………………………………………………………………………………………………….
Titolodistudio………………………………………………………………………………………………………………………………………..
Ruolopubblico(Dirigente,Funzionario,Impiegato,Liberoprofessionistaoaltrafiguraalserviziodiuna
PA)……………………………………………………………………………………………………………………………………..
• Annidilavoropresso(opercontodi)PA………………...............
• AnnidilavorosvoltiinambitiattinentialSUAP……………………
• Entediappartenenza/svolgimentoservizioetipologia(Enteterzo,SUAPcomprensoriale,SUAP
municipale,Altro)…………………………………………………………………………………….………………………………………......
Lechiediamodicompilare,primadell’avviodelleattivitàformative,un’autovalutazionerispettoallivello
europeo(EQF)dicompetenzeposseduteinambitoSUAP(SportelloUnicoAttivitàProduttive)ocomunque
neiprocessidilavorodiunaPubblicaAmministrazione.
PerognispecificacompetenzadescrittanellaTab.1(colonnacompetenza),dovràindicareillivellodi
competenzacheritienedipossedere(1=bassa/nulla–8=alta).Idiversilivellisonomegliodescrittinella
Tab.2
Tab.1
Competenza
Livelloda1a8(rispetto
allatabellaEQFsottostante)
a. Attivitàdisportello,diinformazione,dicomunicazioneedigestionedelle
relazioniconilpubblico
b. Verificadellaregolaritàamministrativaeprocedimentalediun’istanza,
attraversocontrollidicompletezzaecorrettezzaformale
c. Gestioneecoordinamentodispecificheprocedureamministrative
d. Verificadimeritodiun’istanzainordineallacongruenzaeallapertinenza
deidatiedelladocumentazionepresentatarispettoall’attivitàchel’utente
intendeavviareoall’interventochesiintenderealizzare
e. Stesuradidocumentiformali(atti,verbali,lettere,etc.)nell’esecuzioneo
allaconclusionedeivaripassaggiprocedurali
Tab.2
•
•
•
•
•
•
Livello
EQF
Conoscenza
Abilità
Competenza
49
1
Generaledibase.
Basilarinecessariepersvolgere
Lavorareostudiaresottola
compitisemplici,anchein
direttasupervisione,inun
assenzadiesperienzapregressa.
contestostrutturato.
Cognitiveepratichedibase
necessarieperutilizzarele
2
Praticadibaseinunambito
informazionirilevantialfinedi
lavorativoodistudio.
svolgerecompitierisolvere
problemidiroutineutilizzando
Lavorareostudiaresottola
supervisioneconunacerta
autonomia.
regoleestrumentisemplici.
Cognitiveepratichenecessariea
Conoscenzadifatti,principi,
3
processieconcettigenerali,inun
ambitolavorativoodistudio.
svolgerecompitierisolvere
problemiscegliendoe
Assumersilaresponsabilitàper
ilcompletamentodelleattività
nellavoroenellostudio.
applicandometodidibase,
Adeguareilproprio
strumenti,materialied
comportamentoalle
informazioni.
circostanzenelrisolvere
problemi.
Autogestionenell'ambitodelle
lineeguidaincontestidilavoro
odistudiochesono
solitamenteprevedibili,ma
4
Cognitiveepratichenecessariea soggettiacambiamenti.
risolvereproblemispecificiinun
inunambitolavorativoodistudio.
Supervisionareillavorodi
campodilavoroodistudio.
routinedialtri,assumendosi
Praticaeteoricainampicontesti,
unacertaresponsabilitàperla
valutazioneeilmiglioramento
diattivitàlavorativeodistudio.
Gestireesorvegliareattivitàin
Praticaeteorica,completae
specializzatainunambito
5
lavorativoodistudioe
consapevolezzadeiconfiniditale
conoscenza.
6
Unagammacompletadiabilità
cognitiveepratichenecessarie
persvilupparesoluzionicreative
aproblemiastratti.
contestidilavoroodistudio
espostiacambiamenti
imprevedibili.
Controllareesvilupparele
prestazioniproprieedialtri.
Avanzatainunambitolavorativoo Avanzate,chedimostrino
Gestireattivitàtecnicheo
distudio,chepresupponeuna
professionalicomplesseo
padronanzaeinnovazione
50
comprensionecriticaditeoriee
necessariearisolvereproblemi
progetti,assumendosila
principi.
complessiedimprevedibiliinun
responsabilitàperilprocesso
ambitospecializzatodilavoroo
decisionaleincontestidilavoro
distudio.
odistudioimprevedibili.
Assumersilaresponsabilitàdi
gestirelosviluppo
professionaledipersonee
gruppi.
Gestireetrasformarecontesti
Altamentespecializzata,chepuò
7
Problemsolvingspecializzato
conoscenzainunambito
necessarionellaricercae/o
lavorativoodistudio,comebase
nell'innovazione,alfinedi
delpensieroe/odiricerca
svilupparenuoveconoscenzee
Assumersilaresponsabilitàdi
originale.Consapevolezzacritica
procedureeperintegrare
contribuireallaconoscenzae
delleproblematichelegatealla
conoscenzeprovenientida
allapraticaprofessionalee/odi
conoscenzainuncampoe
ambitidiversi.
verificareleprestazioni
all'interfacciatracampidiversi.
specializzate,tracuilasintesie
Livelloconoscitivopiùavanzatoin
unambitolavorativoodistudioe
all'interfacciatracampi.
imprevedibilicherichiedono
nuoviapproccistrategici.
strategichedeigruppi.
Tecnichepiùavanzatee
8
complessidilavoroodistudio
costituirel'avanguardiadella
lavalutazione,necessarieper
risolvereproblemicomplessi
dellaricercae/odell'innovazione
eperestendereeridefinirele
conoscenzeesistentiolapratica
professionale.
Dimostrareeffettivaautorità,
innovazione,autonomiae
integritàtipicadellostudiosoe
delprofessionistaeunimpegno
continuonellosviluppodi
nuoveideeoprocessi
all'avanguardiaincontestidi
lavoroodistudio,tracuila
ricerca.
Ai sensi del Dlgs196/2003 - "Codice Privacy”, desideriamo informarla che le informazioni ed i dati personali forniti in tale contesto
(sperimentazioneevalidazionedellapiattaformaLearnPAd)sarannoutilizzatiesclusivamenteperfinalitàstatisticheescientificheetrattatiin
conformità al documento programmatico per la sicurezza informatica in vigore, nonché nel rispetto della normativa vigente, in modo da
garantirnelasicurezzaelariservatezza.IrisultatideitestediquestionaricompilatisarannoanalizzatiedutilizzatidaipartnersdelProgetto
LearnPAd(RegioneMarche:PFSistemiInformativieScuoladiformazioneregionale,CNRConsiglioNazionaledelleRicerche:ISTI,Universitàdi
Camerino,Universitàdell’Aquila,UniversitàFHNWSwitzerland,BOCAssetManagementGmbHAustria,NoMagicEuropeLituania,Linagora
GSOFrancia,XWIKISasFrancia),confrontaticonquelliraccoltidaglialtripartecipantialtestecomunquerielaboratiinformaaggregataed
anonimaaifinidiunapresentazioneversoisoggettiterziabilitati(ades.funzionarievalutatoriUE).Essisarannoutilizzatipercomprenderei
livellidiefficaciaraggiunticonilnuovomodelloformativosperimentatonelprogettoeuropeoLearnPAd.
Tali dati sono conservati da Regione Marche ed accessibili solo da parte di personale autorizzato fino alla conclusione del progetto; la
cancellazionedeidatiavvienesurichiestaviamail,faxoletteradegliinteressati,senzaindugio.Inognicasoidatisarannocancellatialtermine
dell'iniziativaeaseguitodelloroutilizzoneiterminiindicati.
I soggetti cui si riferiscono i dati personali hanno il diritto, ai sensi dell'art. 7 del D.Lgs. n.196/2003, in qualunque momento, di ottenere la
confermadell'esistenzaomenodeimedesimidatiediconoscerneilcontenutoel'origine,verificarnel'esattezza,chiedernel'integrazioneo
l'aggiornamento, oppure la rettifica, la cancellazione, la trasformazione in forma anonima o il blocco dei dati trattati in violazione di legge,
nonchédiopporsiinognicaso,permotivilegittimi,allorotrattamento.
51
InbasealdocumentoprogrammaticosullasicurezzadeidatipersonalidellaRegione,iltitolaredeltrattamentodeidatipersonalièlaGiunta
regionale, il responsabile è il dirigente titolare della PF SISTEMI INFORMATIVI E TELEMATICI e gli incaricati sono i responsabili regionali dei
procedimentiedeiprocessicorrelatialprogettoLearnPAd.
Percomunicazioni:
RegioneMarche
P.FSISTEMIINFORMATIVIETELEMATICI
DirigenteDott.ssaSERENELLACAROTA
ViaTiziano4460125Ancona(AN)
Fax0718063071
[email protected]
Ilsottoscrittoacconsentepertanto,almomentodiriconsegnareilquestionariocompilato,aconcedere
unaliberatoriaperiltrattamentodelleinformazioniedeidatiraccolti,neiterminisopraindividuati.
Infede____________________________________________________
52
ExpostQuestionnaire(inItalian)
SPERIMENTAZIONEPIATTAFORMA…………………………..–QuestionariodivalidazioneexPOST
NomeeCognome…………………………………………..…………………………………………..………………………………………..
NomeutenteLearnPAdassegnato…..…………………………………………………………………………………………………..
Quantotempohadedicatoallafruizionedellapiattaforma,interminidiarcotemporalecomplessivoin
giorni………………………………………edinterminiditempoeffettivodiutilizzoinore……………….…………………
Lechiediamodicompilare,alterminedelleattivitàformative,un’autovalutazionerispettoallivelloeuropeo
(EQF)dicompetenzecheritienediaveracquisitoneiprocessidilavorodiunaPubblicaAmministrazione
e/oinrelazioneaitemidelSUAP(SportelloUnicoAttivitàProduttive);ciòascopodiverifica,relativamente
alfattosel’utilizzodellapiattaforma,asuoavviso,abbiaincrementato–einchemisura–lesueabilitàela
suaconoscenza.
PerognispecificacompetenzadescrittanellaTab.1(colonnacompetenza),dovràindicareillivellodi
competenzacheoraritienedipossedere(1=bassa/nulla–8=alta).Idiversilivellisonomegliodescrittinella
Tab.2riportatanelquestionarioexante
Tab.1
Competenza
Livelloda1a8(rispetto
•
•
•
allatabellaEQFgiàproposta
nelquestionarioexante)
f.
Attivitàdisportello,diinformazione,dicomunicazioneedigestionedelle
relazioniconilpubblico
g. Verificadellaregolaritàamministrativaeprocedimentalediun’istanza,
attraversocontrollidicompletezzaecorrettezzaformale
h. Gestioneecoordinamentodispecificheprocedureamministrative
i. Verificadimeritodiun’istanzainordineallacongruenzaeallapertinenza
deidatiedelladocumentazionepresentatarispettoall’attivitàchel’utente
intendeavviareoall’interventochesiintenderealizzare
j. Stesuradidocumentiformali(atti,verbali,lettere,etc.)nell’esecuzioneo
allaconclusionedeivaripassaggiprocedurali
DomandearispostamultiplasuicontenutirelativiaSUAP,TitoloUnico,ConferenzadiServizi
(soltantounadellepossibilirisposteècorretta–EVIDENZIARELARISPOSTASELEZIONATAOSCRIVERELA
LETTERAALLAFINEDELLADOMANDA)
1. nelcasoincuialSUAPpervengaunaistanzaperlarealizzazionediunanuovastrutturasanitaria,di
cuiallaL.R.20/00,inqualicasipuòindirelaconferenzadiservizi?
a. Potràindirelaconferenzadiservizisolosesonodecorsi30giornidallarichiestadeipareri
alleamministrazioniinteressate(AziendaSanitariaUnicaRegionaleASUR;ServizioSanità
dellaRegioneMarche,ecc.)senzachequestesisianoinalcunmodopronunciate
b. Dovràinognicasoindirelaconferenzadiservizi,inquantoènecessario
c. Nonpotràmaiindirelaconferenzadiservizi,madovràrichiedereipareriallesingole
amministrazioniinteressate(ASUR;ServizioSanitàdellaRegioneMarche,ecc.)
53
2. incasodiassenzadell’ASUR,regolarmenteconvocata,allaconferenzadiservizi,comedeveessere
intesoilrelativoparere?
a. Espressofavorevolmente
b. Nondeveessereintesoinalcunmodo,mailprocedimentoamministrativovieneinterrotto
finoall’acquisizionedelpareredapartedell’ASUR,chepotràessereacquisitoanchealdi
fuoridellaconferenzadiservizi
c. Espressonegativamente
3. Checosaèlaconferenzadeiservizisincrona?
a. MeetingtrairesponsabiliSUAPcoinvoltinelmedesimoprocedimentoamministrativo
b. IncontrotrailegalirappresentantidiciascunComune
c. Tuttelerisposte(trannelae)
d. StrumentocheriuniscenellostessoluogolePP.AA.coinvoltenelprocedimentoamm.
e. Nessunadellerisposte
4. Qualèloscopodellaconferenzadeiservizi?
a. Riunireiresponsabilidelleamministrazionicoinvoltenelmedesimoprocedimento
b. FarincontrareilegalirappresentantidiciascunComune
c. Tuttelerisposte(trannelad)
d. Nessunadellerisposte
e. Ridurreitempieilcontemperamentodegliinteressipubblicicoinvolti
5. Qualisonolenormechesonointervenuteprevedendol’utilizzodistrumentiinformaticiperilavori
dellaconferenzadiservizi?
a. Tuttelerisposte
b. laLegge18giugno2009n.69
c. ilD.L.5/12,convertitonellaL.35/12
d. laLegge11febbraio2005n.15
e. laLegge7agosto2015n.124
f. ilDLgsn.127del30/06/2016
6. Qualisonoivantaggidellaconferenzadeiservizitelematica?
a. Prevalentementelasemplificazionedeirapportitraisoggettipartecipantialleconferenze
servizi
b. Prevalentementel’azzeramentodelledifficoltàcheconsentonodipartecipareintempi
diversi
c. Prevalentementelatrasparenzaelatracciabilitàdellaproceduranonchélariduzionedei
tempidirisposta
d. Prevalentementeladematerializzazionedelladocumentazioneeladigitalizzazionedel
procedimento
e. Tuttelerisposte
7. InqualeprocedimentodicuialD.P.R.160/2010trovaapplicazionelostrumentodellaconferenzadi
servizi?
a. Tuttelerisposte(trannelad)
b. Nelprocedimentoordinario
c. Nelprocedimentodivalutazionedellaconformitàdeiprogettipreliminari
d. Nessunadellerisposte
e. Nelprocedimentoinvarianteallostrumentourbanistico
8. Aisensieperglieffettidell’art.7delD.P.R.160/2010,entroquantigiornidalricevimento
dell’istanzailresponsabiledelSUAPdeveverificarelacompletezzaformaledell’istanza?
54
9.
10.
11.
12.
13.
a. 30giorni
b. 20giorni
c. 25giorni
d. 35giorni
e. 15giorni
IlDLgs30giugno2016n.127introduceunanuovadisciplinae5tipologiediConferenzediServizi
attivabili;essesono:
a. Decisoria,Automatica,Consensuale,inVariante,Istruttoria
b. Decisoria,Semplificata,Simultanea,Preliminare,Istruttoria
c. Decisoria,Automatica,Parziale,Preliminare,Procedimentale
ChiconvocalaConferenzadiServiziDecisoria
a. L’enteterzoincasodiemissionediparerenonfavorevolealrilasciodelTitolounico
b. Iltitolarerichiedente(osuointermediariodelegato)quandol’attivitàdelprivatoè
subordinataadununicoattodiassensodaadottareaconclusionediunprocedimentodi
competenzadiunaamministrazionepubblica
c. L’amministrazioneprocedente,nellapersonadelresponsabiledelprocedimento,quando
perlaconclusionedelprocedimentodevonoessereacquisitialmenoduepareri,intese,
concerti,nullaostaoaltriattidiassenso,dapartedidiverseamministrazioni,inclusii
gestoridibenioservizipubblici.
Inunaistanzadititolounico,qualora,aseguitodell’esameformaledapartedelSUAP,la
documentazioneallegatarisultiesserenonconformeallanormativa:
a. IlSUAPrichiedeulteriorichiarimentieintegrazionialtitolaredellapratica
b. Glientiterzicoinvoltiemettonoparerenegativo,ciascunoperl’endo-procedimentodi
competenza(ovveroperognisub-attività-BusinessProcesstask-incuivengonocoinvolti
echiamatiademetterepareredimerito)
c. IlSUAPdeveconvocareunaconferenzadiservizi
d. IlSUAPrigettal’istanzainquantoinammissibileoirricevibile
e. IlTribunaleAmministrativoRegionale(TAR)deveprendereinesamed’ufficiol’istanzaela
documentazioneallegata
Se,aseguitodiunaconferenzadiserviziodelcontrollodimeritodiun’istanza,unenteterzo
esprime,perquantodicompetenza,unparerefavorevolecondizionatoall’avviodiun’attività
produttiva,ciòsignificache:
a. L’enteterzohanecessitàdiulteriorichiarimentiprimadipotersiesprimere
definitivamente,pertantoemetteunpareresospensivodeiterminidelprocedimento
b. L’attivitàproduttivadaavviaresiintendeesercitabile,previaverificadellecondizionidi
idoneitàprofessionaledapartedelrichiedente
c. L’enteterzohaespressounpareresostanzialmentefavorevole,previoadempimentoda
partedelrichiedentediunaopiùprescrizionivincolantiindividuate
d. L’utenterichiedenteègiudicatoincondizionediesercitarel’attivitàproduttiva,masoloin
riferimentoallespecifichecompetenzeapronunciarsichel’enteterzohaincapoper
norma
e. L’enteterzohaespressounparerepositivo,segnalandoall’utenteunaseriediinterventi
aggiuntiviopzionalinonobbligatoridaporreinessere
UnutentehacorrettamentepresentatoistanzadiTitoloUnicoalSUAPdelComunediSenigallia,in
ordineallaristrutturazionediunostabilimentobalnearesuunaspiaggiaricadenteinarea
55
paesaggisticatutelataanormadellaparteIIIdelD.Lgs.42/2004;IlSUAPconvocaunaconferenzadi
servizi:qualidiquestientiterziverrannosicuramenteinvitati?
a. MultiserviziSpAinquantoentegestoredellaretefognaria(perl’esamedelrefertoanalitico
sulloscaricodiacquereflue)eARPAM(perlavalutazionediimpattoacusticoaisensidella
L.447/95edellaL.R.14novembre2001,n.28)
b. IlDipartimentodiigienealimentareenutrizionedell’AziendaSanitariaUnicaRegionale-
ASUR(perilparereigienicosanitariopreventivoinordineall’esecuzionedeilavoriedili)ela
PFVALUTAZIONIEDAUTORIZZAZIONIAMBIENTALIdellaRegioneMarche(perlarichiestadi
autorizzazioneintegrataambientaledicuialDLgs59/05,inordineaimpianticoncapacità
produttivapariosuperioreaivaloridisogliaindicatinell’AllegatoI)
c. L’areaEcologiadellaProvincia(perilparereinordineall’adozionedell’AUA-autorizzazione
unicaambientale–perleemissioniinatmosfera)edilDIPARTIMENTODIPREVENZIONE
INCENDIdeiVigilidelFuoco(perlavalutazionedelprogettoaisensidell’art.3delDPR
151/2011)
d. LaReteFerroviaItalianaSpA-exFF.SS.(perilparereinmeritoaiconfiniedistanzedalla
lineaferroviariaaisensiDPR753/1980)edilDIPARTIMENTODIPREVENZIONEESICUREZZA
NEILUOGHIDILAVOROdell’ASUR(perl’acquisizionedell’autorizzazionedicuiall’art.5del
R.D.147/27inmateriadiutilizzodigastossici)
e. Gliufficicomunaliambiente(perilpareredicompatibilitàpaesaggistica)ededilizia(per
l’ottenimentodelpermessodicostruire)
14. ConclusasiunaconferenzadiserviziincuisiastataconvocatalaSoprintendenzaperibeniartistici,
architettonici,ambientaliestorici,affinchésiesprimaaisensidell'art.146DLgs42/2004esmie
art.14-tercomma3bisL.241/1990,qualoralaSoprintendenzanonabbiaemessoparereformale
néabbiapartecipatoailavori:
a. IlSUAP,aisensidellanormativasulsilenzioassenso,intendeacquisitounparerefavorevole
b. LaSoprintendenzaha30giorniditempodall’emissionedelverbaleperintegralocon
eventualiesuccessiveosservazioni,richiestedichiarimento,prescrizioni
c. IlSUAPsollecitalaSoprintendenzaadesprimeredefinitivamenteilparereentroulteriori15
giornidall’emissionedelverbale
d. LaSoprintendenzaha15giorniditempodall’emissionedelverbaleperconvocareuna
nuovaconferenzadiservizi
e. L’utenterichiedenteacquisiscedidirittol’autorizzazionearealizzarel’interventooavviare
l’attivitàd’impresa
f. IlSUAPècostrettoacomunicareall’utenteildiniegoall’autorizzazione
15. Conclusasiunaconferenzadiservizirelativaadun’istanzadiTitolounicoincuihannopresoparte,
oltrealSUAP,tuttiglientiterzinecessari,qualoralasoprintendenzaperibeniartistici,
architettonici,ambientaliestoriciabbiaemessoparerefavorevoleaisensidell'art.146DLgs
42/2004esmieart.14-tercomma3bisL.241/1990,l’ufficiopatrimoniodelcomuneabbia
accordatoilNULLAOSTAall’interventoinqualitàdiEnteproprietariodell'area,l’ufficioambiente
delcomuneabbiaevidenziatolacompatibilitàdell’interventoalivellopaesaggisticoaisensidell'art.
146DLgs42/2004el’ufficiocomunaleediliziaabbiaevidenziatolanonammissibilitàdell'intervento
aseguitodellacontestualerichiestadelpermessodicostruire:
a. AlterminediprocedimentoilSUAPemetteràtitolounicoaprocedereall’intervento
b. Ilconsigliocomunalesaràchiamatoapronunciarsisulprocedimentodivarianteurbanistica
c. AlterminediprocedimentoilSUAPcomunicheràalrichiedenteildiniego(oilpreavvisoal
diniego)perl'effettuazionedell'intervento
56
d. L’ufficiocomunaleediliziasaràinvitatoariesaminareilpropriopronunciamentolegatoal
permessodicostruireentroilterminedi15giorni
e. Alterminediprocedimentol’utentepotràpresentarealSUAPlaSegnalazioneCertificatadi
InizioAttività
Domandesull’usabilitàdellapiattaforma
(EVIDENZIARELARISPOSTASELEZIONATAOAGGIUNGEREUNAXNELLARELATIVACASELLA)
1. Ingenerale,quantoèsoddisfattodellafruizioneinmodalitàe-learningacuihaavutoaccesso?
(Scaladiriferimento1=insoddisfatto;5=totalmentesoddisfatto)
1
2
3
4
5
Puòspiegarciinsintesiperché……………………………………………………………………………………………………………………….
……………………………………………………………………………………………………………………………………………………………………….
2. Comegiudicanelcomplessol’efficaciadellediversefunzionalitàresedisponibilidallapiattaforma?
(Scaladiriferimento:1=pernienteefficaci;5=moltoefficaci)
1
2
3
4
5
Inparticolarequalifunzionalitàgiudicapiùutili:ades.esercizidisimulazionebasatisucasireali,
raccomandazionedinamicadicontenuti,navigazionesequenzialedelflussodilavoro,interazionesocialcon
commentiealtreformedicontribuzione,elementidiapprofondimentoevidenziatiattraversograficao
multimedia,momentidiverificaintermedicondomandearispostamultipla,attribuzionedipunteggifinali,
etc.......................................................................................................................................................................
……………………………………………………………………………………………………………………………………………………………………….
……………………………………………………………………………………………………………………………………………………………………….
3. Comegiudica“facilitàd’uso”ed“accessibilità”deiserviziresidisponibili?Per“accessibilità”siintendela
caratteristicadiunsistemainformaticodiesserefacilmentefruibiledaqualsiasitipologiadiutente,
indipendentementedallecapacitàsensoriali,motorieopsichichedell’individuo,daeventualidisabilità
temporaneeostabili,oanchedaaspetticulturaliorelativialladimestichezzaconl’usodelleinterfaccee
delletecnologieICT.
(Scaladiriferimento1=difficile;5=facile)
1
2
3
4
5
4. Interminidi“usabilità”ed“esperienzautente”comevalutaiseguentiaspetti?Per“esperienzautente”
(UserExperience-UX)siintendelasensazionesoggettivacheaveteprovatoutilizzandolapiattaforma
conriferimentoagliaspettiesperienziali,affettivi,diattribuzionedisensoevalorecollegatialla
57
disponibilitàdelprodottoeall’interazioneconessononchéallepercezionicomplessiveeaipensieri
personalicircal’efficacianell’apprendimento,lasemplicitàdiutilizzodell’interfaccia,l’efficienzadel
sistema.
(Scaladiriferimento:1=esperienzadeltuttonegativa;5=esperienzadeltuttopositiva)
Graficaecolori
1
2
3
4
5
Comprensionedeisimboli
1
2
3
4
5
Significatodeitesti
1
2
3
4
5
Ricercaesuggerimentodeicontenutidesiderati
1
2
3
4
5
Orientamentonellanavigazione
1
2
3
4
5
Strutturaeposizionamentodeicontenuti
1
2
3
4
5
Altro(specificareeaggiungererighesenecessario)
1
2
3
4
5
1
2
3
4
5
…………………………………………………………………………………………………………………………………………..
…………………………………………………………………………………………………………………………………………..
5. Comegiudicala“completezza”dellefunzionalitàresedisponibili?
(Scaladiriferimento1=incomplete;5=complete)
1
2
3
4
5
Inparticolarequalifunzionalitàmancherebberooasuogiudizioreputacarenti(eperché)perpoter
considerarelapiattaformaunvalidostrumentodiapprendimento………………………………………………………………
……………………………………………………………………………………………………………………………………………………………………….
……………………………………………………………………………………………………………………………………………………………………….
6. Quantovalutaefficacel’integrazionedegliambientididatticipresenti?Inaltreparoleleèsembratodi
operareinuncontestodiapprendimentosufficientementeomogeneo,rispettoatuttiimoduli,i
contenutielefunzionalitàesplorati?
(Scaladiriferimento1=pernienteintegrati;5=moltointegrati)
1
2
3
4
5
7. Comevalutailgradodiinnovativitàdellasoluzioneproposta?Leèsembratodiapprendereinuna
modalitànuovarispettoadesperienzedidattichegiàavute?
(Scaladiriferimento1=pernienteinnovativo;5=moltoinnovativo)
58
1
2
3
4
5
8. Comevalutal’utilitàdellasoluzioneproposta?Lareputaadeguatarispettoallesueesigenzeformative?
(Scaladiriferimento1=pernienteutile;5=moltoutile)
1
2
3
4
5
9. Aifinidiunamiglioreefficaciadellapiattaformainterminidiapprendimentoritieninecessariauna
maggiorepresenzadeiseguentiaspetti?
(Scaladiriferimento:1=pernientenecessario;5=deltuttonecessario)
Fruizioneguidatadeicontenuti
1
2
3
4
5
“Gamification”ovverol'utilizzodielementimutuatidaigiochiedelletecnicheludiche
riadattateacontestiesterni(ades.punteggi,livelli,ricompense,distintivi,doniealtri
metodidifidelizzazioneoperdestareinteresseocontribuireallasoddisfazionedegli
utenti)
1
2
3
4
5
Multimedialità(usoappropriatodiaudio,video,immagini,…)
1
2
3
4
5
Interattivitàtradiscenteecontenuti
1
2
3
4
5
Interattivitàescambiodiconoscenzatrautentidellapiattaforma
1
2
3
4
5
Multilinguismoocomunqueancheusodididascalieingradodispiegareilsignificatodi
acronimi,vocabolitecnicidaaddettiailavori,etc.
1
2
3
4
5
Tutoring(animazioneperidiscenti,supportoalladidattica)
1
2
3
4
5
Altriaspettinecessari(specificareeaggiungererighesedelcaso)
1
2
3
4
5
1
2
3
4
5
…………………………………………………………………………………………………………………………………………..
…………………………………………………………………………………………………………………………………………..
Ai sensi del Dlgs196/2003 - "Codice Privacy”, desideriamo informarla che le informazioni ed i dati personali forniti in tale contesto
(sperimentazioneevalidazionedellapiattaformaLearnPAd)sarannoutilizzatiesclusivamenteperfinalitàstatisticheescientificheetrattatiin
conformità al documento programmatico per la sicurezza informatica in vigore, nonché nel rispetto della normativa vigente, in modo da
garantirnelasicurezzaelariservatezza.IrisultatideitestediquestionaricompilatisarannoanalizzatiedutilizzatidaipartnersdelProgetto
LearnPAd(RegioneMarche:PFSistemiInformativieScuoladiformazioneregionale,CNRConsiglioNazionaledelleRicerche:ISTI,Universitàdi
Camerino,Universitàdell’Aquila,UniversitàFHNWSwitzerland,BOCAssetManagementGmbHAustria,NoMagicEuropeLituania,Linagora
GSOFrancia,XWIKISasFrancia),confrontaticonquelliraccoltidaglialtripartecipantialtestecomunquerielaboratiinformaaggregataed
anonimaaifinidiunapresentazioneversoisoggettiterziabilitati(ades.funzionarievalutatoriUE).Essisarannoutilizzatipercomprenderei
livellidiefficaciaraggiunticonilnuovomodelloformativosperimentatonelprogettoeuropeoLearnPAd.
Tali dati sono conservati da Regione Marche ed accessibili solo da parte di personale autorizzato fino alla conclusione del progetto; la
cancellazionedeidatiavvienesurichiestaviamail,faxoletteradegliinteressati,senzaindugio.Inognicasoidatisarannocancellatialtermine
dell'iniziativaeaseguitodelloroutilizzoneiterminiindicati.
I soggetti cui si riferiscono i dati personali hanno il diritto, ai sensi dell'art. 7 del D.Lgs. n.196/2003, in qualunque momento, di ottenere la
confermadell'esistenzaomenodeimedesimidatiediconoscerneilcontenutoel'origine,verificarnel'esattezza,chiedernel'integrazioneo
59
l'aggiornamento, oppure la rettifica, la cancellazione, la trasformazione in forma anonima o il blocco dei dati trattati in violazione di legge,
nonchédiopporsiinognicaso,permotivilegittimi,allorotrattamento.
InbasealdocumentoprogrammaticosullasicurezzadeidatipersonalidellaRegione,iltitolaredeltrattamentodeidatipersonalièlaGiunta
regionale, il responsabile è il dirigente titolare della PF SISTEMI INFORMATIVI E TELEMATICI e gli incaricati sono i responsabili regionali dei
procedimentiedeiprocessicorrelatialprogettoLearnPAd.
Percomunicazioni:
RegioneMarche
P.FSISTEMIINFORMATIVIETELEMATICI
DirigenteDott.ssaSERENELLACAROTA
ViaTiziano4460125Ancona(AN)
Fax0718063071
[email protected]
Ilsottoscrittoacconsentepertanto,almomentodiriconsegnareilquestionariocompilato,aconcedere
unaliberatoriaperiltrattamentodelleinformazioniedeidatiraccolti,neiterminisopraindividuati.
Infede____________________________________________________
60
Collectedfeedback(inItalian)
LearnPAd
Negativeoverallopinionsandmissingfunctionalities
• L’impattoinizialeconlapiattaformasconcertaegeneraincertezze–nonessendociunpercorso
predefinitodaseguire,maunaseriedifunzionalitàattivabiliapiaceredicuiinizialmentenonsene
conosconoutilitàocomportamento,l’utentesitrovaspiazzato->siconsigliadipotenziaregliaspetti
introduttivi(es.testoesplicativoiniziale,videotutorial,etc.).
• Perquantohoavutomododivedere,alcunefunzionalitàsembranononfunzionarebene(es.barra
raccomandazioni:nonvedoalcunicontenuticheimieicolleghiinvecehannosperimentato)–altrenon
sonobenspiegate(es.simulationnonèchiara,néneicontenuti,nénellapresentazionedelleazionida
fare)
• Difficoltàanavigareall’internodellapiattaforma.Honotatochevisonoalcuneimprecisionisui
contenutirelativiallaconferenzadiservizi(nonsonocontemplatigliaggiornamentidicuialrecente
Dlgsdiriformadellamateria)
• Pocoefficacelasimulazionecherisultaesseresemplicementeunacompilazionedicampi(quasiun
test)senzaprevederefunzionichestimolanoilproblemsolving.Inoltreipunteggifinalinonsono
rapportatiadunascaladiriferimentopercuiperdonoognisignificatovalutativo
• Interfacciapocointuitiva.Difficilecapiredovecisitrova
• Conlavastitàdelleinformazioniedeilinkchecisitrovadavantirisultanonfacilecapiresesisiano
esploratetuttelefunzionalità
• Lapiattaformaèancorainunostatoembrionaleincuiicontenutiguidasonopocochiarielafruizione
delleinformazionirisultaosticatramitel’interfacciaproposta
• Ilcorsorichiedeungradopregressodiconoscenzaecompetenzaspecificasullamateriatrattata(non
adattoaneofiti)
• LAPARTEDESCRITTIVAE’COMPLETAEDIFACILECOMPRENSIONE(ADECCEZIONEDIALCUNIERRORIDI
ORTOGRAFIAES.e&apos);ICONTENUTISONOBENDESCRITTI,MAHOTROVATODIDIFFICILE
FRUIBILITA’ELETTURAGLISCHEMIGRAFICI
• Lafruizionedellapiattaformanonèmoltointuitiva,bisognaprendercidimestichezzaeconciliare
concettualmenteleduemodalitàdiorganizzazionedeicontenuti(quelladiflusso,cheèmoltochiarae
avvienetramitelaschematizzazione,equelladispiegazionedeiprocessi)
• Lapiattaformanonmièsembrataparticolarmenteintuitiva,la“mappa”navigabileèinteressante,ma
nonèimmediatoilcollegamentoconilinkeitestisottostanti
• Ibusinessprocessdovrebberoregolareilfunzionamento,manondiventareprotagonistidelcorso,
l’utentenondovrebbevederli,lafruizionedelcorsodeveessereaccompagnatadacalltoactionso
descrizionisemplicicheloguidinoattraversoicontenutieleesercitazioni/simulazioni.Finchénonme
l’hannospiegatononsonoriuscitoacapirciniente,mancanoledescrizionidellesezioni,mainognicaso
unsoftwareèusabilequandosispiegadasoloenonhabisognodiguide
• Pocointuitiva
• L’attivitàformativasullaconferenzadeiservizi,comeprimapagina,illustrailflussodocumentalechesi
attivainrelazioneadunaConferenzadeiServizi,tuttavianonappareconevidenzanellapagina
introduttival’illustrazionedellefinalitàedellastrutturadellapiattaforma(oalmenoiononsono
riuscitaatrovarlaconimmediatezza–poihovistodopolasezioneHOWTO).LasezioneHOWTO
dovrebbeessereresapiùevidentenellaprimapaginaocostituireproprioicontenutidellaprimapagina
peresplicitarelefinalitàdell’attivitàchesiandràasvolgere.E’essenzialechelefinalitàegliobiettivi,
nonchélastrutturaelefunzionalitàdellapiattaformasianoresechiaresindasubito.Noncredoqueste
61
•
•
•
•
•
•
•
•
•
•
•
informazionipossanoesserelasciateallaliberasceltadichiaccedeallapiattaforma.E’essenzialeche
sianoenunciateesplicitamenteprimadiavviarelanavigazioneperunaesigenzadichiarezzae
trasparenzaeperconsentireaifruitoridiorientarsimaggiormentenelpercorso,avendoanchechiarala
direzioneacuitendere.
Visualizzareilcontenutodei“tasti”checostituisconoildiagrammadiflussononèsemplice.Sonodi
dimensionitroppopiccolee,dunque,scarsamentefruibili(daunpuntodivistagrafico).D’altrocanto,è
ottimochesianocliccabili,permettendoachientrainpiattaformadiorientarsiinognifasedel
percorso,dopounprimomomentodiconfusione(dovutaalfattochenonsonochiarigliobiettiviele
finalitàdell’attività,amioparere).Lapiattaformanonrispondeacriteridiaccessibilitàpertutti(glistep
neldiagrammadiflussosonodescritticoncaratteritroppopiccoliehoavutodifficoltàacapirecosaci
siascrittoinognitasto).
Ingenerale,l’interfacciaapparepocointuitivaedinonimmediatacomprensioneperipotenzialifruitori
chesiavvicininodaprofaniall’utilizzodellapiattaforma.Adesempio,nellaschermatainiziale
dell’attivitàsipotrebbeindicaregliobiettiviformativiechiarirelefunzionalitàfondamentali,ovvero:
navigare,affrontaresimulazionidilavorodigruppoopersingoliutenti,contribuireattraversoproprie
considerazioniedannotazioni.
E’ottimal’ideachesiapossibilefaredellesimulazioni,anchesepoinonsonoriuscita(ononsiriesce)
adavviarle.E’comeselasimulazionefosseraccontata,manoncifosselapossibilitàdiattivarla.Nonho
bencapitocometalefunzionalitàsiaattivabile.Hoprovatoacliccaresuimenùatendinaperavviare
unasimulazione,maalsubmitnonèsuccessonulla.Nonsocosailsistemaabbiagenerato,seabbia
generatoqualcosaecomequestopossaattivareilcontrollo(casierisposteinserite–cheperònon
sonoriuscitaadinterpretare).Inaltreparolelasimulazione,amioparere,nonfunzionainmodo
adeguato(pocochiareleistruzioniperfarlapartire-mancanodocumentinecessarixcomprendereil
casoreale).
Imodellidilavoro(formatconvocazioni,formatpercomunicareorichiedereintegrazioni,formatper
redigereverbali,etc.)nonsonoutilizzabilienonsembrasianodisponibilinelcorso.
L’interfacciagraficaèassolutamentedamigliorareperchérappresentailgrandelimitedella
piattaforma,rendendolapocointuitiva,pocoaccessibileenonchiaramenteorganizzataperun’ideale
fruizionedellefunzionalitàcheprevede.
Perpoterrisponderealledomandedicontenutoneltestènecessariofareriferimentoadocumentiche
nonsonodisponibiliinpiattaforma
Hotrovatoilsistemaabbastanzaapprossimativoepocointuitivo
Ladimensionedeifontètroppopiccola
Lefunzionalitàdicollaborazionesocialandrebberoirrobustiteeresepiùaccattivanti
Lefunzionalitànecessariesonotuttepresenti,dovrebberoperòessereimplementateinmanierapiù
efficaceinterminidi:organizzazionelogica,interattività(illivellodiinterattivitàtrauserecontenutiè
troppobasso),progressionenelcompletamentodelpercorsodiapprendimento(nonsiintuiscequalèil
complessivoammontaredeicontenutidafruirepercuinonsipercepisceilquantoèstatofattoe
quantodeveessereancorafatto);esplicitazionedegliobiettividelpercorsoformativo
Lanavigabilitàpotrebbeesseremigliorata,cosìcomeleinformazionirelativeadovecisitrovainogni
pagina(breadcrumbsinsufficienti)
Nonèchiaralamodalitàdifunzionamentodellasimulation
Lefunzionalitàdisimulationerecommendationsonoancoraincompleteoinunostatoembrionaleche
nerendedifficilelafruizione.Avolteitestidell’interfacciasonoininglese
62
•
•
•
•
•
•
•
Itutorialsonobensviluppatiearticolati.Occorrededicarecomunquemoltopiùtempo.Lascrivente,
perilpocochehautilizzato,hatrovatovantaggio.Almenoduegiornatelavorativedovrebberoessere
dedicate(apprendereautilizzarelapiattaformaèdipersétimeconsuming)
LAPARTEDEICASIPRATICINONRISULTAFUNZIONANTE
Risultacarentelafunzionalitàdellesimulazioni,cheandrebberoresepiùinterattiveemenocomplesse
Mancano:lapossibilitàdifareannulla/escidallasimulazione;unostrumentodidesigndeiprocessidi
businessaccessibile;unpercorsoformativoatappe,nelsensochetuttiicontenutisembrano
rappresentatiinmodonondifferenziatoallostessolivello(mentrealcunirisultanoosticiedaltripiù
accessibili)epersonalmentenonhocoltoqualiargomentisono“propedeuticiper…”
Sarebbeopportunoinserire,sullasinistra,una“barra”deicontenuticheevidenzilastrutturadelcorso
elagerarchiadeicontenutistessi.L’utilizzodivideoedianimazioniagevolerebbel’apprendimento
Cisonodellepartiinlinguainglesecheandrebberotradotte;igrafici,moltoutili,dovrebberoesserepiù
leggibili
Aggiungerevideo:nellahomeperintrodurreilsistemaeilsuofunzionamento,nellevarielezioniper
spiegareicontenutichevisitroveranno.Aggiungerespiegazionitestualidicosafaognisingola
pagina/sezionechesivisita(adesempiolasimulazione)
Positiveoverallopinionsandstrenghts
• Nelcomplessoèunmodototalmentenuovoedinteressantediimparare
• Moltovalidoilmodellorelativoalprocessodilavoroeilfattochesianavigabile
• Superatoilmomentodismarrimentoinizialeecompresalalogicadicomeinteragireconlapiattaforma
lafruizioneèrisultatasempliceeintuitiva
• Modalitàdie-learningveloceinquantopermettedisaltareicontenutichenoninteressanoper
dedicarsiaquellidaapprofondire
• L’usodeibusinessprocessmodelinquestoambitodaungrandevaloreaggiunto,poichécredosiano
utiliperavereunavisionegraficadiunprocessocomplesso.Tuttavial’aggiuntadellainterattività
nell’immaginechedescriveilBP,perrenderlanavigabile,garantirebbeunamaggioreusabilitàdella
piattaforma
• Benstrutturatoedifacilefruizione
• L’illustrazionedelflussodocumentaleècompletaechiaraalivelloconcettuale.Itastidiognistepsono
cliccabilierimandanoadognifasedilavoro.Cliccandoognistep,compareunasinteticaechiara
descrizionedellafasedilavoroincuicisitrova.
E’ottimocheinognistepcisiaanchelapossibilitàdiredigeredocumentiattraversoilwiki,lavorando
informacollaborativa
Ingenerale,lefunzionalitàessenzialicisono.Inparticolareildiagrammadiflusso,cliccabileadogni
stepconlerispettiveopzioni,èunottimostrumento,chiaroecomprensibiledalpuntodivista
concettuale
• Lagraficasempliceelineareaiutaefavoriscel'apprendimentoattraversolanavigazionedelprocesso
• Utilel’eserciziodisimulazionebasatosucasireali(mavacuratomeglio);efficace(madaimplementare
meglio)ilmeccanismodelloSCORE
• Interessantelaraccomandazionedinamicadicontenuti(potenzialitàsemanticachetuttavianonho
avutomododivedereinesecuzione–mièstataraccontata)
• Utilissimalanavigazionesequenzialedelflussodilavoroancheattraversol’immaginedelprocesso,ma
daottimizzare(purtroppol’immaginenellafasedinavigazioneètroppopiccolasulloschermo,maho
apprezzatocheisingolielementisonocliccabili/ancorapiùutilesel’esplorazionepotesseavvenire
63
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
comenellasimulazione,doveadognistepsievidenziagraficamentel’attivitàdelprocessoincuisono.
Nellasimulazioneilriquadrocheevidenziaquestoscorrerenelflussoètropporistretto)
L’interazionesocialconcommentiealtreformedicontribuzioneèabbastanzautile
Buonaidealafunzionalitàcontribuisci,conlapossibilitàdiaggiungeretestoesplicativo
Lanavigazionesequenzialedelflussodilavoroèl’aspettopiùinnovativoedefficacedellapiattaforma
Giudiziopositivoperlaraccomandazionedinamicacontenuti
Lasimulationèunafunzionalitàutile,mavaresapiùintellegibile
Presenzadeglistessielementidiapprofondimentoinognipagina,evidenziatiattraversografica
omogenea
Lasimulazionesucasirealiaiutamoltissimonelprocessoditraining.Unicoaspettonegativoèchelo
spazioristrettodellafinestracontenenteilflussodeitaskviaviaattivinonpermettevadicapire
immediatamente(senzascrollorizzontaleeverticale)inqualefasedelprocessomitrovavo
Leimmaginidelprocessorendonochiaralavisioned'insieme.Chiaroiltestodescrittivodeisingoli
passaggi
Validigliesercizidisimulazionebasatisucasirealinonchélefunzionidiinterazionesocialcon
commentiedaltreformedicontribuzione.Ledescrizionideicontenutisonocompleteedifacile
comprensione
Traimodulipiùinteressantisegnalo:lanavigazionesequenzialedelflussodilavoro;l’interazione
collaborativadegliutenti;lesimulazioni,madarenderepiùstrutturateepiùaderentiacasireali
complessieverosimili
Hoapprezzatogliesercizidisimulazionebasatisucasireali,lanavigazionesequenzialedelflussodi
lavoroelavisionegraficadelprocesso
Ritengoutililemappenavigabili,l’interazioneneicommentielesimulazionidicasireali
Lasimulazioneèunabuonaeutileidea,mal’usoditerminicomeroboteumanoandrebberocambiati
ospiegatiinqualchemodo
OKperinterazionesocial,simulazione,navigazionesequenziale
Lanavigazionesequenzialedelflussodilavorobasatasuclicèfacilmentecomprensibilealivello
concettuale.Ledescrizionisonochiareesintetiche.Ottimalapossibilitàdico-produrredocumentiedi
contribuireinmodalitàcollaborativa(wiki)
Puntidiforza:l’eserciziodisimulazioneconcasireali;lanavigazionesequenzialedelflussodilavoro;
linearitàesemplicitàdellagrafica
LANAVIGAZIONESEQUENZIALEDELFLUSSODILAVOROMISEMBRALAMIGLIOREFUNZIONALITA’DEL
SISTEMA
Tralefunzionalitàpiùutili:navigazionesequenzialedeiflussidilavoro
•
MARLENE
• Difficoltànell’integrareledispenseconleinformazionifornitenelcorso
• Duranteilcorsoinmodalitàe-learningsonotroppelesezioniincuivienerichiestaunasemplicelettura
deltesto(senzal’ausiliodellavocenarrante)
• Mancanoapprofondimentisucasirealiedesercizipratici
• Fruibilitàdellapiattaformascarsa,perviadelbloccopopupedinumerosiblocchidurante
l’avanzamentodelleslide
• Corsomoltoteoricoconpocheevidenzepraticherispettoallavoroquotidiano
64
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
IcontenutinonsonoaggiornatirispettoalnuovoDLGScheriformalanormativasulleConferenzedi
Servizi
Icontenutipropostisonobasicisarebbemeglioaverecontenutipiùapprofonditi
Utileperdipendentipubblicialleprimearmi,maicontenutivannoaggiornatirispettoallanormativa
vigenteperleconferenzediservizi
Icontenutidialcuniargomentinonsonoaggiornati,(ALMENOPERITESTACUIHOAVUTOACCESSO),
sarebbeutileaverelapossibilitàdiinteragirepersottoporredomandesudubbinormativi
Probleminelladisattivazionedellepolicydisicurezzadelbrowserperlariproduzionepluginjava;Non
sufficientequalitàdell’insegnamentorispettoalmio(basso)livellodiconoscenza
Occorrededicaremoltotempoalcompletamentodelmoduloformativo.Servirebbeunamodalitàpiù
snella
Occorremaggiorespaziocollaborativoefunzionidedicateperdiscutereeapprofondirenormativa,
interpretazioni,prassitracolleghidelSUAP
Esistonosistemidie-learningpiùall’avanguardiacuiispirarsi
Mancanospecifichepaginededicateallarisoluzionedicasirealioconinterpretazionedellenormeper
situazioniparticolarmentedifficilidagestire
Strumentodiapprendimentononevolutointerminidiweb2.0(interazionetradiscenti,possibilitàdi
postaresuisocialnetworkiprogressi,interoperabilitàconyoutubeoaltricontenutiwebesterni,…)e
websemantico(ricercacontenuti,propostaautomaticadicontenutiinbaseainteressiecaratteristiche
dell’utente,…)
Nonpresenti:spazidiapprofondimentocheintegranolevariazioninormativeintercorserispettoalla
pregressadisciplina
Perpoterconsiderarelapiattaformaunvalidostrumentodiapprendimentomancaunsistemadi
ricercadeicontenutiall’internodellapiattaforma.Lostrumentohaunapprocciosolosequenziale
Mancanochatostrumentidiscambioinformazionitrapartecipantialcorso
Carenze:possibilitàdiscambiarsiopinioniconespertioaltriutentisucasistichereali;gamification,
comepiùavantidescritta
NONMISEMBRAADATTALAPOSSIBILITA’DIINTERAGIREPERPOTERPORREDOMANDEOPOTER
ESPRIMERELANECESSITA’DIUNAPPROFONDIMENTO
Mancanoaltricanaliutiliall’apprendimentoqualiilconfrontoconglialtridiscentioconundocente
espertoperapprofondiretemispecifici.Inoltreicontenutipresentati,sebbeneutili,nonentranonel
meritodelleconoscenzecheservononell’esperienzaquotidianadellavoro–ècomesesirimanessein
unpianoancorateorico(percuirestacomunqueunostrumentoutileperriceverenozioniadesempio
normative)
Nonc’èlapossibilitàdisaltarealcuneslideperdedicarsidirettamenteaicontenutidiinteresse
Nonhotrovatospazicollaborativi,didiscussioneediinterscambiotrautentieversountutoresperto
Assentelafruizionedivideo
•
•
•
• VALIDOPERLAPRESENZADIFUNZIONALITA’DIFFERENTICOMECONTENUTIDINAMICI,TEST
INTERMEDI,APPROFONDIMENTI
• Icontenutisonoesaustiviedisemplicefruizione
• Buonstrumentodiaggiornamento
• Conl’E-learningsievitadidoverpartecipareacorsiinpresenza(edunquedispostarsifisicamente)e
ciòpermetteunafruizionepiùamisuradegliimpegniprofessionaliinessere
65
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Abbastanzasemplicedausareeschematicopermemorizzareglielementiprincipali
Pratico,sinteticoeveloce
Interfacciachiara–flussosequenzialesempliceescorrevole
Utileperriceverenozioninormative,credosiaunvalidostrumento
UTILISSIMELEVERIFICHEINTERMEDIE
Apprezzabiliimomentidiverificaintermedi
Interessantelapresenzadilinkagliargomentitrattatinellevarieslideedeimomentidiverifica
intermedicondomandearispostamultipla
Puntidiforza:Verificaintermediaalterminedellesingoleunità;Guidaaudiosulleslide
Validoapproccioquellodelledomandediverificaintermedia
Ottimoilflussodiscorrimentosequenzialedelcorso
Benegliintervallidiverificaarispostamultipla
OKVERIFICHEINTERMEDIEEFINALI
Positivalafruibilitàonlinedeicontenuti
Maggiormenteutili:momentidiverificaintermedieriferimentinormativi
Piuttostoutiliimomentidiverificaintermedi
Validal’enfasisuicontenutiprincipalidatacongrassettoeriquadriinevidenza
Imomentidiverificaaiutanoaricapitolareeragionareeintervallandoilcorso,spezzanolamonotonia
dellalettura
Ledomandediverificaintermediapermettonodifarsiunrapidoriassuntomentaleeverificarei
principaliconcettiappresi
Imomentidiverificaintermedicondomandearispostamultiplamisonopiaciuti
66
Collectedfeedback(inenglish)
LearnPAd
• Theinitialimpactwiththeplatformdisconcertsandgeneratesuncertainty-sincethereisnota
predefinedpathtofollow,butavariousnumberoffeaturesthatcanbeactivatedatwill,whichutility
orbehaviorwedonotknowinitially.Sotheuserisdisplaced->werecommendtopotentiatethe
introductoryaspects(eg.theinitialexplanatorytext,videotutorials,etc.).
• AsfarasIgottosee,somefeaturesdonotseemtoworkwell(eg.recommendationsbar:icouldnot
seeallthecontentthatmycolleaguesontheotherhandhaveexperienced);otherfeaturesarenotso
wellexplained(eg.Simulationisnotclear,neitherincontentnorconcerningthepresentationofthe
actionstobedone)
• Itisdifficultnavigatingwithintheplatform
• Ihavenoticedthattherearesomeinaccuraciesincontentrelatedtoconferenceservices(thechanges
referredtotherecentreformDecreeonthematter,arenotcovered)
• Thesimulationisshortlyeffective:inthelongrunitissimplyacompilationoffields(almostatest)
withoutprovidingfunctionsthatstimulatearealproblemsolving.Inaddition,thefinalscoresarenot
computedonascaleofreferenceandsotheyloseanyevaluativemeaning
• unintuitiveinterface.Hardtoknowwhereyouare
• Withthevastnessoftheinformationandlinksthatyoufaceisnoteasytounderstandifyouhave
exploredallthefunctionality
• Theplatformisstillinanembryonicstateinwhichtheguidecontentsareunclearandtheuseof
informationbytheproposedinterfaceistricky
• Thecourserequiresapreviouslevelofknowledgeandspecificexpertiseonthesubjectmatter(not
suitableforbeginners)
• THEDESCRIPTIVEPARTISCOMPLETEANDEASYTOUNDERSTAND,EXCEPTFORSOMETYPOS(ES.
e&apos);THECONTENTSHEREINAREWELLDESCRIBED,BUTIFOUNDDIFFICULTTOUSEANDREAD
THEGRAPHICSCHEMES
• Theuseoftheplatformisnotveryintuitive,wemustgetfamiliarandconceptuallyreconcilethetwo
modesofcontentorganization(theflow,whichisveryclearandtakesplacethroughthe
schematization,andthedescriptive/explanatorysectionoftheprocesses)
• Theplatformdonotseemparticularlyintuitive;thenavigablemapisinteresting,butitisnot
immediatetocatchtheconnectionwiththelinksandtheunderlyingtexts
• Thebusinessprocessesshouldregulatetheoperation,butnotbecometheprotagonistsofthecourse:
theusershouldnotseethem;thefruitionofthecoursemustbeaccompaniedbycalltoactionsor
simpledescriptionsguidinghimthroughthecontentsandexercises/simulations.Aslongastheyhave
haveexplainedittome,iwasnotabletounderstandanything:therearemissingdescriptionsofthe
sections,butinanycaseasoftwareisusablewhenitisself-explanatoryanddoesnotneedguides
• Unintuitive
• Thefirstpageofthetrainingshowsthedocumentflowthatisactivatedinconnectionwitha
ConferenceofServices;however,itdoesnotappearclearly,intheintroductorypage,anillustrationof
theaimsandtheorganizationoftheplatform(oratleastIcouldnotfinditwithdirectness-onlylaterI
sawtheHOWTOsection).TheHOWTOsectionshouldbemademoreevidentwithinthehomepage,
orevenitshouldbecomeitselftheuniquecontentofthefirstpage,inordertoclarify,sincethe
beginning,thepurposeofthetasksyouwillperform.Itisessentialthatthelearningobjectives,aswell
67
•
•
•
•
•
•
•
•
•
•
•
•
asthecontentstructureandthefunctionsoftheplatform,aremadeclearfromtheoutset.Idonot
believethatthisinformationcanbelefttothefreechoiceofeachuseraccessingtheplatform.They
needtobeexplicitlysetoutbeforeyoustartnavigating,forclarityandtransparency,andalsotoallow
theuserstofocusmoreinwhattheywillfindduringthepath,havingasaprerequisiteanevident
directiontogo.
Viewingthedescriptionoftasksthatconstitutetheflowchartitisnotsimple.Theyaretoosmalland,
therefore,poorlyusable(fromagraphicalpointofview).Ontheotherhand,itisgreatthattheyare
clickable,allowingthosewhoentertheplatformtoorientthemselvesineverystepoftheway,
bypassingtheveryfirstmomentofconfusion(duetothefactthat,inmyopinion,theaimsand
purposesoftheactivitiestotryarenotsoclear).Theplatformdoesnotmeetthecriteriaof
accessibilityforeveryuser(thestepsintheflowchartaredescribedwithtoosmallfontsandIhad
troubleunderstandingwhatiswrittenoneachsquare).
Ingeneral,theinterfaceappearslessintuitiveandnotreadilyunderstandablebythepotentialusers,
approachingasprofanetheuseoftheplatform.Forexample,onthehomescreen,itshouldbe
indicatedwhatthetraininggoalsareandalsobesummarizedthefundamentalcapabilitiesofthe
platform,namely:browsing,tacklingteamworksimulationsor,forindividualusers,contributingby
postingtheirownconsiderationsandannotations.
It'sagoodideathatyoucandothesimulations,eventhoughIcannot(oramnotableto)startthem.
Thesimulationwasintroduced,buttherewasnochancetoactivateit.Idonotunderstandhowthis
featurecouldbeactivated.Itriedtoclickonthedropdownmenutostartasimulation,butafter
submittingnothinghappened.Ididnotknowwhatthesystemhavegenerated,ifihadreallycreated
somethingchosingarealcase,howthiscouldbecontrolledandwhatanswersineededtoinserted-
maybealsobecauseIfailedtounderstandsomeofthequestions.Inotherwords,andinmyown
opinion,thesimulationdoesnotworkproperly(mainlyduetounclearinstructionstogetitstarted-
andlackofthenecessarybackgrounddocumentstounderstandtherealcase).Theadministrative
modules(convocationsformat,communicationnotes,letterstorequireintegrations,draftsforreports,
etc.)doesnotseemtobeavailableandusableamongthesimulationstages.
TheGUIneedtobeimproved,becauseitrepresentsthemajorlimitationoftheplatform,makingit
non-intuitive,difficulttobeaccessedandnotclearlysetupforanidealuseoftheprovided
functionalities.
Inordertoanswertosomeofthequestionscontainedinthetest,ithasbeennecessarytoreferto
documentsthatwerenotavailablewithintheplatform
Ifoundtheprototypequiteroughandunintuitive
Thefontsizeistoosmall
Socialcollaborationcapabilitiesshouldbestrengthenedandmademoreattractive
Thefundamentalbasicfeaturesareallpresent,butshouldbeimplementedmoreeffectivelyinterms
of:overalllogicalorganization;interactivity(thelevelofinteractivitybetweenuserandcontentistoo
low);progressioninthecompletionofthelearningpath(icouldnotguesswhatthetotalamountof
contenttobeenjoyedwas,sowithoutthatperceptionicouldnotfindwhathasbeendoneandwhat
wasstillneededtobedone);explicationofthetrainingobjectives
Navigabilityshouldbeimproved,aswellasinformationaboutwhereyouareineachpage
(breadcrumbsseeminsufficienttome)
Itisnotcleartheoperationalmodeofthesimulation
Thesimulationandrecommendationfeaturesarestillincompleteorinanembryonicstate,which
makesitdifficulttouse.SometimestheinterfacetextsareinEnglish
68
•
•
•
•
•
•
•
Thetutorialsarewelldevelopedandarticulated.However,theplatformrequiresalotoftimetobe
devotedto.Forthelessiusedit,ihavefoundsomeadvantages.Atleasttwomoreworkingdaysshould
havebeenspenton(thatistosaythatjustlearningtousetheplatformitselfishardlytimeconsuming)
THEPRACTICALCASESARENOTWORKING
Thesimulationfunctionalityaredeficientandshouldbemademoreinteractiveandlesscomplex
Itlacks:thecapabilitytomakeundo/quitthesimulation;adesigntoolavailabletocorrectthe
businessprocessesonthego;atrainingcoursebystages,meaningthatallcontentappearslikeifitis
representedinanundifferentiatedmanner,allatthesamelevel(whileinrealitysomepartsaremore
difficultandsomeothermoreaccessible),andpersonallyIhavenotgraspedwhattopicsare
prerequisites"inpreparationfor..."
Itwouldbeappropriatetoinclude,ontheleft,a"bar"ofcontentsthatshowsthecoursestructureand
contenthierarchy.Theuseofvideoandanimationsfalsowouldfacilitatelearning
TherearesomepartsinEnglishthatshouldbetranslated;graphics,veryuseful,shouldbemore
readable
Videosneedtobeadded,i.e.inthehometointroducethesystemanditsoperation,inthevarious
lessonstoexplainthecontentthatyouwillbeabletofind.Alsoitwouldbeadvisabletoaddtextual
explanationsofwhateachsinglepage/sectionyouvisitstandsfor(duringsimulationitisreallyamust)
• Overall,itisatotallynewandaninterestingwaytolearn
• Themodeloftheworkprocessisveryaccuratedandthefactthatitisnavigableisavalidfeature
• Goingthroughaninitialmomentofconfusionandonceunderstoodthelogicofhowtointeractwith
theplatform,thefruitionwassimpleandintuitive
• Thise-learningmodeprovestobefaster,becauseitallowstheusertoskipnotofinterestcontent,to
embracedirectlythosejudgedtobeexplored
• Theuseofbusinessprocessmodelinthisareaisabigplus,becauseIthinktheyareusefultogetaeasy
understandablegraphicalviewofacomplexprocess.Howevertheadditionofmoreinteractivity
featureswithintheimagedescribingtheBP,egtomakeitfullynavigable,wouldguaranteeagreater
usabilityoftheplatform
• Wellstructuredandeasytousewebpages
• Theillustrationoftheflowofstepsoftheprocessiscompleteandclearataconceptuallevel.Theareas
ofeachstepareclickableandrefertoeachstageofwork.Clickingoneachsteparea,youwillseea
concisebutcleardescriptionoftheworkphaseinwhichyouarelocated
• Itisgreatthat,ineverytaskoftheBP,thereisalsothepossibilityofdrawingupdocumentsthrough
wikiandalsotovalidatethem,workingwithtextsinacollaborativeandassistedway
• Ingeneral,theessentialfeaturesareincluded.Inparticular,theflowchart,clickableateach
progressionandevidencingtherespectiveoptions,isagreattool,clearandunderstandablefroma
logicpointofview
• Thesimpleandstraightforwardgraphichelpsandencourageslearningthroughthenavigationofthe
process
• Userscangetprofitfromthesimulationbasedonrealcases(butitshouldbebetterconfiguredand
curated);themechanismoftheSCOREiseffective(butitneedstobetoimplementedmoreaccurately,
alsolettingtheuserstounderstandthelogicofthepointsawarding)
69
•
Aninterestingfeatureisthedynamicrecommendationofcontent(asemanticpotentialitythat
neverthelessihavenotnoticedwhilerunningthesystem,becauseitworksinbackground-Iwastold
aboutandthenirealizeditwatchingacollegueofminetestingtheplatform)
Theforwardandbackwardnavigationoftheworkflowthroughtheimageoftheprocessisveryuseful,
butstillneedtobeoptimized(indeed,unfortunately,thepictureshowedinthenavigationphaseistoo
smallonthescreen,evenifIappreciatedthattheindividualelementswereclickable;atthatpointigot
agreaterbenefitdownloadingthepicturetomydesktop).Forinstanceitwouldbemoreusefulifthe
explorationofthesingleactivitiesinthebrowsingmodecouldtakeplaceasinthesimulation,where
eachstepisgraphicallyhighlightedshowingthecurrentactivityoftheprocessinwhichwestand.
However,duringsimulation,thepanethatcontainsthedynamicflowpictureistoosmallandweneed
toscrollhorizontallyandvertically(anditbothers).
Thesocialinteractionwithcommentsandotherformsofcontributionisquiteuseful,alsotoaskhelpto
otherusers
Goodistheideaofthewindowtocontribute,withtheabilitytoaddbrandnewexplanatorytexts
Thesequentialbrowsableprocessworkflowisthemostinnovativeandeffectiveaspectoftheplatform
Apositiveratinggoestothedynamiccontentrecommendation
Thesimulationisausefulfeature,butitshouldbefixedandmademoreintelligible
Ilikedthepresenceofthesameelementsoneachpage,identifiedthroughstandardgraphicsandicons
Thesimulationofrealcaseshelpsalotduringthetrainingprocess.Theonlydownsideisthatthe
confinedspaceofthewindow,containingthetasksflowandtheactivetask,doesnotallowto
understandimmediatelyandwithoutscrollingatwhatstageoftheprocessIam
Theprocessimagesgiveaclearoverview.Alsocleararethetextualdescriptionsofeachindividualstep
Thesimulationexercisesbasedonrealcasesarevalid,aswellasthecollaboirativeandsocial
interactionfunctionsforpostingcomments(andcommentstocomments),givefeedbackandadda
contribution.Asforthecontributions,theheaderofeachsectionofthedocumentshouldbetranslated
inItalian.Thecontentdescriptionsarecompleteandeasytounderstand
Amongthemostinterestingmodulesipointout:1)thenavigableworkflow;2)thecollaborative
interactionoftheusers;3)thesimulationofarealcaseandthepossibilitiestochatwithotherlearners
whiledoingit,buttherealcaseitselfhastobemademorestructuredand-forlearningpurpose-more
adherenttolikelytohappencomplexcases
Ienjoyedthesimulationexercisesbasedonrealusecases,andthenavigationoftheConferenceof
Servicesworkflowplusthegraphicaloverviewoftheprocess
Iconsiderusefulthenavigablemaps,theoptiontoaddcommentsandthesimulationsoftheSUAP
applications
Thesimulationisagoodandusefulidea,buttheuseoftermssuchasrobotsandhumanhastobe
avoided...otherwisetheseoptions(andhowtousethem)havetobeexplainedinsomeway
OK:socialinteraction,simulation,sequentialnavigation
Thebrowsingmode,basedonclickstomovethroughtheTitoloUnicoprocess,iseasytouse,to
understandandeffectivefortrainingonthejob.Thedescriptionsofeveryactivityinvolvedareclear
andconcise.Excellenttooistheopportunitytoco-producedocumentsandtocontributeinthe
collaborativemode(wiki)
Strengths:theexercisessimulatingtherealcases;thesequentialnavigationoftheworkflow;the
linearityandthesimplicityofthegraphic,whichworksevenwithmobiledevice
THESEQUENTIALITYOFPROCESSBROWSINGSEEMSTOMETHEBESTFUNCTIONALITYOFTHESYSTEM
70
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Themostusefulfunctionalityisthenavigablebusinessprocess,bothbypageclickandbyinteracting
withtheimage
MARLENE
• Iexperienceddifficultiesinintegratingtheattacheddocuments(aboutthereformationofthe
conferenceofservices)withtheinformationprovidedwithintheMarlene'scourse
• Duringthecourseine-learning,toomanysectionsrequiredasimplereadingofthetext(withoutthe
helpofthenarratorvoice-functionthat,whenpresent,makesmelearnfaster).Atthebeginningihad
someproblemswiththeconfigurationofmyPCaudiodevices
• Itlacksinsightsonrealapplicationsandpracticalexercises.TheSUAPorganizationisdefinitivelymore
complexthanwhatthelawssay
• Poorusabilityoftheplatform,becauseofthepop-upblockerandvariousotherblocksduringthe
forwardmovementoftheslides
• Averytheoreticalcourse,withfewpracticalevidenceswithrespecttomydailywork
• ThecontentsareoutdatedcomparedtothenewLegislativeDecreereformingtheregulationof
ConferencesOfServices
• Theproposedcontentsarebasic.Itwouldbebettertohavemorein-depthcontent
• Usefulforcivilservantsbeginners,butthecontentsneedtobeupdatedwithrespecttothecurrent
regulationsforSUAPandforconferencesofservices
• TheSUAPcontents,atleastforsomeargumentstowhichihadaccessforthetest,areoutofdate.It
wouldbeusefultohavetheopportunitytointeracttosubmitquestionsonregulatoryconcernsandto
receiveopinionsfromothercolleaguesattendingthecourse
• Ihadtechnicalproblemsinthedeactivationofbrowsersecuritypolicyforthejavapluginplayback
• Idonotfindasufficientqualityoftheteachingaccordingtomy(low)levelofknowledge.Ifindithard
learning
• Ittakestoomuchtimetocompleteallthetrainingmodule.Itwouldservealeanermodetogothrough
thecontentandtogiveasfinishedeachslide/chapter
• Amorecollaborativespaceisneeded,andalsosomededicatedfunctionstodiscussandexploredeeply
thelegislation,togiveinterpretationsandtoexchangebestpracticesamongSUAPofficersand
managers
• Therearemanymoreadvancede-learningsystemstogetinspiredby
• nospecificpagesaddressedtothesolutionofrealcasesorinterpretationoftherulesforparticularly
difficultsituationstohandle
• Marleneisalearningtoolnotevolvedintermsofweb2.0(interactionbetweenlearners,abilitytopost
individualprogressonsocialnetworks,interoperabilitywithyoutubeorotherexternalwebcontent,...)
andthesemanticweb(contentsearching,automaticallyofferedcontentonthebasisoftheuser's
interestsandcharacteristics,...)
• Thereweren'tenoughstudiesthatintegratethechangesoccurredwithrespecttotheprevious
regulationsgoverningtheconferenceofservices
• Inordertoconsidertheplatformavaluablelearningtool,itlacksacontentsearchsystemwithinthe
platform.Theinstrumenthasonlyasequentialforwardandbackwardapproachtoreachcontent
• Achat-orevenasimplercommunicationtooltoexchangeinformationbetweentheparticipantsatthe
course-ismissing
71
•
•
•
•
•
•
Weaknesses:idonotfindanypossibilitytoexchangemypointsofviewwithexpertsorotheruserson
realcasestudies;alsogamificationaspects-asdescribedinthequestionnaire-arenotpresent
ITDOESNOTSEEMSUITABLETOME,ASALEARNINGINSTRUMENT,BECAUSEICAN'TFINDTHE
OPPORTUNITYTOINTERACTWITHSUAPEXPERTSINORDERTOASKQUESTIONSORTOBEABLETO
EXPRESSMYPECULIARNEEDSFORIN-DEPTH
Otherusefullearningchannels,suchascomparisonwithotherlearnersorwithanexperiencedteacher
tofocusonspecifictopics,needtobeaddedMoreover,thepresentedcontent,althoughuseful,donot
enterintothemeritsoftheknowledgeandskillsneededineverydaywork-itisasifthecourse
remainsinamoretheoreticallevel(soitisstillausefultooltoreceivesuchregulationsnotions,butnot
tolearnhowtoconductaconferenceofservices)
Istherethepossibilityofskippingsomeslidestogodirectlytothecontentofinterest?Icannotfindit
...
Ididnotfindcollaborativespacesfordiscussionandinformationexchangebetweenusersorfroman
experttutor
Absenceofanyconnectionorusageoflearningbyvideo
• THESYSTEMISVALIDFORTHEPRESENCEOFDIFFERENTFUNCTIONALITIESASDYNAMICCONTENTS,
INTERMEDIATECOMPREHENSIONTESTS,IN-DEPTHSSECTIONS
• Thecontentiscomprehensiveandthetooliseasytouse
• Agoodtooltoupdateone'sknowledge
• WiththeE-learningyouavoidhavingtotakecoursesinthepresence(andthereforeyouquitphysically
transferts).Thisenablesamoretailoredusageoftime,inrelationwiththeoutstandingprofessional
commitments
• Prettysimpletouse.ItProvidesaschematicapproachtostoreinmindquicklythemainelementstobe
learned
• Practical,synthetic,quicktounderstandandfasttouse
• Aclearinterface-Asimpleandsmoothsequentialflow
• Usefultoreceiveregulatorynotions,Ithinkitisagoodtool
• INTERMEDIATECHECKSAREUSEFUL
• Theintermediateverificationmomentsareappreciable
• Theplatformisinterestingforthepresenceoflinkstothetopicscoveredinthevariousslidesandfor
themultiple-choicequestions,bothforintermediateandfinalassessmentoflearning
• Amongthestrengths:intermediatechecking-bytheroreticalquestions-attheendofeachunit;audio
guideoneachslide
• Aviableapproach,thatofthemid-termexaminationquestions
• Excellentforthesequentialslipstreamofthecourse
• Wellfortheintervalswiththemultiplechoicequestions
• OKFORTHEINTERMEDIATEANDFINALCHECKS
• Positiveistheonlineavailabilityofthecontent
• Themostusefulfeaturesare:theintermediatemomentsofverificationandthepresenceofnormative
andstandardreferences
• Ratherusefultheintermediatemomentsofverification
• Howeffectiveistheemphasisonthekeycontent,conveyedthroughbold,colorsandhighlightboxes
72
•
•
•
Themomentsofverificationhelpthelearnerstosummarizeandreasonand,spacingoutthecourse,
theybreakupthemonotonyofreading
Theinterimreviewquestionshelpyoumakingaquickmentalrecapandcheckingmentallytheeffective
assimilationofthemainconceptstolearn
Ilikedthemomentsofintermediatecheckbasedonmultiplechoicequestions
73

Deliverable D8.4 Demonstrators assessment

Transcript

Documenti analoghi

scheda richiesta spazio 2017

Francese 1A - B Roveleto Libr - Istituto Comprensivo di Cadeo e

PROGRAMMA ANIMAZIONE E CULTURA ARCILOMBARDIA

sportello unico per le attivita` produttive attivita` di commercio su aree

sportello unico per le attivita` produttive attivita` di commercio su aree

asseverazione di altri tecnici incaricati

E` Natale ogni volta che permetti al Signore di

Scarica il modulo di partecipazione

mouse pad carta

SCARICA IL PROGRAMMA COMPLETO