| Notes |
1 The ADL Initiative – May 2019
The ADL Initiative
Total Learning Architecture:
2018 Reference Implementation
Specifications and Standards
Advanced Distributed Learning (ADL)
1901 N. Beauregard St., Suite 600
Alexandria, VA 22311
May 2019
This work was supported by the U.S. Advanced Distributed Learning (ADL) Initiative W900KK-17-D-0004. The
views and conclusions contained in this document are those of the authors and should not be interpreted as
representing the official policies, either expressed or implied, of the ADL Initiative or the U.S. Government. The
U.S. Government is authorized to reproduce and distribute reprints for Government purposes.
DISTRIBUTION STATEMENT A. Approved for public release.
TABLE OF CONTENTS
1. Introduction............................................................................................................................................... 1
1.1 Future Learning Ecosystem ............................................................................................................ 2
1.2 Specifications and Standards..........................................................................................................2
1.3 2018 TLA Reference Implementation ............................................................................................ 3
1.4 2018 TLA Test and Demonstration ................................................................................................7
1.5 2018 Data Analytics and Visualization ..........................................................................................8
2. TLA Specifications and Standards............................................................................................................ 9
2.1 Activity Registry........................................................................................................................... 10
2.1.1 Learning Resource Metadata Initiative (LRMI) ...................................................................... 10
2.1.2 IEEE 1484.12.1 Learning Object Model (LOM) ..................................................................... 11
2.1.3 Dublin Core Metadata Initiative (DCMI) Metadata................................................................ 12
2.1.4 Schema.org Vocabularies........................................................................................................ 12
2.2 Activity Streams ........................................................................................................................... 13
2.2.1 Experience API (xAPI)............................................................................................................. 14
2.2.2 IMS Global Caliper ................................................................................................................. 14
2.2.3 W3C Activity Streams 2.0 ........................................................................................................ 15
2.2.4 SISO Human Performance Markup Language (HPML).......................................................... 15
2.3 Competency Management ............................................................................................................ 16
2.3.1 Achievement Standards Network™ (ASNTM)......................................................................... 17
2.3.2 Competencies and Academic Standards Exchange™ (CASE™) ............................................ 17
2.3.3. O*Net...................................................................................................................................... 18
2.3.4 MedBiquitous Competency Framework................................................................................... 19
2.3.5 IEEE 1484.20.1 Reusable Competency Definitions (RCD)..................................................... 19
2.4 Credentials.................................................................................................................................... 19
2.4.1 Credential Transparency Description Language (CTDL)....................................................... 21
2.4.2 Credential Registry.................................................................................................................. 21
2.4.3 IMS Global Open Badges........................................................................................................ 21
2.4.4. W3C Verifiable Claims........................................................................................................... 21
2.5 Learner Profile .............................................................................................................................. 21
2.5.1. Comprehesive Learner Record (CLR).................................................................................... 22
2.5.2. Learner Information Package (LIP)....................................................................................... 22
2.5.3. IEEE Public and Private Informaiton for Learners (PAPI Learner) ..................................... 23
2.5.4. Airmen Learning Record (ALR).............................................................................................. 23
2.6 Recommendation Engine.............................................................................................................. 24
2.6.1. Fast Learning from Unlabeled Episodes for Next-Generation Tailoring (FLUENT)............ 24
2.6.2. Navigator for Integrated Learning Environments (NILE)...................................................... 24
2.6.3. Adaptive Instructional Sciences – IEEE (C/LT/AIS) P2247.1 ................................................ 25
2.7 Launch Service ............................................................................................................................. 25
2.7.1. xAPI Launch ........................................................................................................................... 25
2.7.2. cmi5......................................................................................................................................... 25
2.8 Assessment Service....................................................................................................................... 25
2.8.1. cmi5......................................................................................................................................... 25
2.9 Identity Management .................................................................................................................... 26
2.9.1. Privacy and Security for TLA (PS4TLA) ................................................................................ 28
2.10 Miscellaneous............................................................................................................................. 27
2.10.1. Learning Tools Interoperability (LTI).................................................................................. 27
2.10.2. Open Pathways..................................................................................................................... 27
2.10.3. Sharable Content Object Reference Model (SCORM)) ........................................................ 27
2.10.4. IEEE Actionable Data Book (ADB)...................................................................................... 27
2.10.5. ePub ...................................................................................................................................... 28
2.10.6. IEEE P1589 Augmented Reality Learning Experience Models (ARLEM) ........................... 28
3. 2018 TLA Reference Implementation .................................................................................................... 29
3.1 Logical Architecture ..................................................................................................................... 29
3.2 Physical Architecture.................................................................................................................... 30
3.3 System Components ..................................................................................................................... 33
3.3.1. Activity Registry...................................................................................................................... 33
3.3.2. xAPI Activity Streams ............................................................................................................. 34
3.3.3. Learner Record Store.............................................................................................................. 34
3.3.4. Competency Management System........................................................................................... 34
3.3.5. Learner Profile ....................................................................................................................... 35
3.3.6. Recommendation Engine ........................................................................................................ 35
3.3.7. Single Sign On ........................................................................................................................ 36
3.4 Concept of Execution.................................................................................................................... 36
3.5 Interface Design............................................................................................................................ 38
3.5.1 Interface Identification and Diagrams..................................................................................... 38
3.5.2 Advanced Message Queuing Protocol (AMQP) ...................................................................... 38
3.5.3 Representational State Transition (RESTful)........................................................................... 38
3.5.4 Experience Application Program Interface (xAPI) ................................................................. 38
3.6 Computational Resources............................................................................................................. 40
3.7 Lessons Learned and Future Research.......................................................................................... 41
4. 2018 TLA Test and Demonstration ........................................................................................................ 42
4.1 Technical Planning ....................................................................................................................... 43
4.2 TLA Content Demonstration ........................................................................................................ 44
4.2.1 CODIAC Course Description .................................................................................................. 45
4.2.2 CODIAC Competency Framework .......................................................................................... 47
4.2.3 CODIAC Content Discovery and Alignment ........................................................................... 48
4.2.4 Instrumenting CODIAC Content.............................................................................................. 49
4.2.5 Sequencing and Delivery ......................................................................................................... 52
4.3 Technical Integration .................................................................................................................... 54
4.4 TLA Test Event at Fort Bragg ...................................................................................................... 57
4.4.1 Logistics and Administration................................................................................................... 58
4.4.2 Hardware and Network Connectivity ...................................................................................... 58
4.4.3 Pre-Tests, Post-Tests, Surveys, and Focus Groups ................................................................. 59
4.4.4 Learner Management and User Experience ............................................................................ 60
4.4.5 Data Collection and Management........................................................................................... 61
4.4.6 Data Analytics and Visualization ............................................................................................ 62
4.4.7 Lessons Learned ...................................................................................................................... 66
5. Conclusion and Next Steps..................................................................................................................... 69
LIST OF TABLES
Table 1. 2018 TLA Components 5
Table 2. 2018 TLA Services 6
Table 3. Summary of TLA Specifications 9
Table 4. AWS Server Specifications 32
Table 5. TLA Reference Implementation Application Layer Protocol Port Usage 40
Table 6. Key Planning Events 44
Table 7. CODIAC ELOs and TLOs 46
Table 8. 2018 TLA Learning Record Providers 49
Table 9. Well Structured Tasks 52
Table 10. Ill-Defined Tasks 53
Table 11. Competency Classifications 70
LIST OF FIGURES
Figure 1. TLA Service Layers 1
Figure 2. 2018 TLA Reference Implementation: Logical View 4
Figure 3. 2018 TLA Test and Demonstration 7
Figure 4. Learning Object Metadata 11
Figure 5. Alignment between a Schema.org CreativeWork and a Node in an Educational Framework 13
Figure 6. HPML High-Level Conceptual Architecture 16
Figure 7. O*NET Content Model 18
Figure 8. CDTL Class Structure 20
Figure 9. Logical TLA Services Architecture 31
Figure 10. 2018 TLA Reference Implementation Physical Architecture 33
Figure 11. Competency and Skills System 35
Figure 12. TLA 2018 Reference Implementation Concept of Execution (COE) 37
Figure 13. TLA Reference Implementation Components and Interfaces 39
Figure 14. CODIAC 45
Figure 15. CODIAC Competency Framework 47
Figure 16. Competency Framework 48
Figure 17. xAPI Statement Mappings 50
Figure 18. TLA Metadata Spreadsheet 51
Figure 19. Well Structured Tasks 53
Figure 20. Ill-Defined Tasks 54
Figure 21. Home Page of 2018 TLA Test and Demonstration FLUENT Interface 54
Figure 22. FLUENT Interface Showing the CODIAC Skill Tree Structure and Goal Selection 55
Figure 23. SWEG(A) Classroom at Fort Bragg 56
Figure 24. Orientation Briefing 59
Figure 25. Activity Levels of All Learners 62
Figure 26. Statements Generated per Learner. 63
Figure 27. Statements Generated per Competency 63
Figure 28. Statements per Minute 64
Figure 29. Empirical Distribution Function 65
Figure 30. Competency States and Learner Paths 65
Figure 31. 2019 Technical Specifications and Standards 69
Figure 32. 2019 Data Streaming Architecture 71
Figure 33. 2019 Data Analytics and Visualization Environment 73
1 The ADL Initiative – May 2019
1. INTRODUCTION
The Total Learning Architecture (TLA) program sponsored by the ADL Initiative seeks to develop a set of
policy and standards defining the process for developing a learning ecology, where multiple services and
learning opportunities (of various modalities and points of delivery) can be managed in an integrated
environment. The TLA serves to:
1) shift from a course-centric focus to a learner-centric focus, where competence is managed more
efficiently (for savings in cost and schedule) and effectively for the learner’s professional
development path;
2) provide the opportunities for data interoperability between implementations to aid in enterprisewide decision support analysis and in reuse of competency and content between agencies, and;
3) capture all experiences, including on-the-job training (OJT), use of performance support and job
aids, simulation, distributed learning, and traditional classroom learning to provide a more rounded
portrait of personnel capability and aptitude to aid in the team building and detailing process.
The TLA employs a Service Oriented Architecture (SOA) approach to developing a flexible, modular
system to optimize the time, cost, and effectiveness of learning delivery and analysis. It is composed of
interchangeable software services and databases integrated to create an efficient learning environment for
the user. The layered architecture of the TLA results in a separation of concerns amongst the different TLA
components. Figure 1 shows the different layers within the TLA. These applications utilize common
services and shared data to deliver instructional activities and provide overall management of the
experience. TLA services are built around a set of defined set of specifications.
Figure 1. TLA Service Layers – The TLA’s service layers centralize external access to data and functions, hides the
complexity of implementation across physical components, and allows for versioning of the services across the TLA
ecosystem. This enables a suite of learning applications to use common services and shared data to glean insights
about learners, content, context, and competencies within the 2018 TLA Test and Demonstration.
2 The ADL Initiative – May FY19
1.1 FUTURE LEARNING ECOSYSTEM
Ecology is the branch of biology which studies the dynamic interactions between plants, animals, and
micro-organisms and their environment, working together as a functional unit. Ecologies are living systems
containing a diversity of factors that interact with each other and are self-organizing, adaptive, and fragile.
The term ecology has been applied to the concept of learning as a metaphor for the many different tools,
technologies, systems, and processes that interact with each other to promote knowledge. The term
ecosystem is often used to describe all sources of learning that are available to employees within their
organization; however, each organization has its own processes, contexts, relationships, and interactions
that provides opportunities and resources for learning, development, and achievement. Further, the current
landscape of institutional learning technologies consists of multiple data stovepipes even within the same
organization. Training and education systems do not share information well with human resourcing and
personnel systems; nor do they share data with operational systems that are the true indicators of
organizational performance.
The future learning ecosystem is defined by personalized and competency-based learning environments
that promote interoperability and accessibility of learning resources across organizational boundaries and
throughout an individual’s life. Today we understand that learning takes place both inside and outside the
classroom. We live and work within a digital web of learning resources and learning experiences; each
resource connects to others and feeds an overall structure in which learning takes place. At the heart of this
ecosystem is the learner and the associated policies and processes used to track their learning experiences
across the continuum of learning.
The key to managing this lifelong learning data is the interoperability afforded through the technical
standards, specifications, and practices that comprise the TLA research portfolio. These underpin the wide
range of learning resources an individual will encounter and enable a federated data strategy that facilitates
the sharing of this information with other systems within an organization and even across institutional
boundaries. Conceptual interoperability also includes shared vocabularies, system architectures,
frameworks, and reference models that help producers and consumers communicate effectively. The TLA
research portfolio is grounded in a chain of research that helps the DoD optimize training and education
resources, communicate information and knowledge anywhere and anytime, manage an overall data
strategy, and enable greater flexibility for how and when learning should occur. This document is divided
into the major sections outlined below.
1.2 SPECIFICATIONS AND STANDARDS
The future learning ecosystem powered by the TLA includes an interconnected,
evolving community of learners, tools, systems, activities, and content that must all
operate together. Making this shift requires the adoption of common technical
standards across institutions, systems, and instructional content.
Technical specifications and standards allow different learning resources to communicate with each other
using a mutually agreeable common language. These standards form the fundamental building blocks for
life-long learning by establishing consistent protocols that can be universally understood and adopted by
any component within the future learning ecosystem to enable data exchange about learners, activities, and
experiences. Beyond the interoperability standards required for integrating tools, learning experiences
(activities and content), and a myriad of different data types, TLA research is also looking at shared
vocabularies, linked data strategies, performance thresholds and an overall data strategy that ties them all
together.
3 The ADL Initiative – May FY19
The 2018 TLA research investigated relevant standards, specifications, and policies that relate to metadata
strategies, competency representation, activity tracking, Learner Profiles, and numerous communications
protocols, and interface specifications. A specification/standard provides definitions, rules, roles, and
responsibilities for data components within it. In many cases, a standard or specification will also define
how and where data is stored or registered. Numerous specifications and standards were reviewed to
investigate how they would influence the interoperable nature of the TLA. Candidate specifications were
then implemented and tested within the 2018 TLA Reference Implementation.
Within the TLA Reference Implementation, candidate standards and specifications are analyzed to
determine whether the spec/standard is partially or entirely appropriate. If changes are required to extend
the spec/standard, this work is documented to inform future standards/specification development bodies. It
is important to note that the TLA is looking at specifications and standards used across other industrial
sectors (e.g., finance, healthcare, manufacturing), across different communities (e.g., K-12 Education,
colleges and universities, corporate learning) and for different purposes (e.g., machine to machine
communication, evaluation/assessment). The 2018 research portfolio includes specifications and standards
that relate to the following:
• Interoperability standards: Characteristic definitions of a TLA component, activity providers,
and services including performance thresholds, interface specifications, data models/schemas, etc.
• Classification standards: Systematic groupings of data, components, systems, or services based
on different characteristics or properties.
• Standards for testing and evaluation: Characteristics and minimum thresholds measurements for
security testing, regression testing, conformance testing, etc.
• Standards for compliance and certification: Description of the functions and relationships
between administration, curation, quality assurance, lifecycle maintenance, and governance of
future learning ecosystem components, systems, and activities/content.
With an ever-increasing amount of options, the task of selecting an approved standard or specification can
be difficult. Each has their own advantages and drawbacks, and many have overlapping uses. The future
growth of personalization services, data analytics, and integration with other business systems presents
numerous challenges in achieving true interoperability between all the disparate data sources required for
learning across an enterprise. The ADL Initiative will continue to evaluate different specifications and
standards to determine how suitable they are for different aspects of the TLA. Section 2 of this document
will describe technical standards and specifications evaluated for use in the 2018 TLA Reference
Implementation.
1.3 2018 TLA REFERENCE IMPLEMENTATION
The goal of the 2018 TLA Reference Implementation is to evaluate candidate standards and specifications
for potential inclusion within the TLA. In creating the 2018 TLA Reference Implementation, the ADL
Initiative started with a foundational understanding of existing Information Technology (IT) components
commonly utilized in Distributed Learning (DL) today. The reference implementation builds upon these
components and in some instances, utilizes the same specifications and standards used by these components.
While these existing specifications and standards are considered a starting point for the development of the
TLA, the ADL Initiative is investigating additional specifications and standards to determine their
applicability and suitability for the future learning ecosystem.
4 The ADL Initiative – May FY19
The 2018 TLA Reference Implementation maintained a few high-level considerations and design
constraints. First, it was to be developed without proprietary data rights. The commercial products used
within the system are open source. All vendor product communication was performed using industry
standards. The system and user-action logs were maintained using the experience API. Systems
communicated using a RESTful implementation. The system was deployed using Amazon Web Services
(AWS) on virtualized server hardware, with purchased Platform and Backend as a Service (PaaS/BaaS).
Figure 2 shows the major components of the TLA architecture as they were implemented in support of a
TLA Test and Demonstration exercise at Fort Bragg in August 2018. This diagram establishes the context
for how specifications and standards are currently being applied. The TLA components used in this
implementation include a range of Technology Readiness Levels (TRLs) that impact the requirements and
even the availability of relevant standards or specifications for certain components of the architecture. A
primary goal of the TLA is to support plug and play interoperability of connected components, systems,
content, and data. Each implementation of the TLA will utilize any variation of components that are
required to meet the goals and objectives for that instance. Approved TLA components and associated
specifications/standards will change over time as systems, capabilities, tools, and technologies continue to
evolve.
Figure 2. 2018 TLA Reference Implementation: Logical View – Activity providers (green) are instrumented with
xAPI statements that are streamed to a Learner Record Store (LRS). The ADL Initiative services (blue) include the
LRS, Authentication, Identity Management, TLA component discovery, and launch services. A competency
management system (orange) collects xAPI assertions from a variety of assessments activities. These assertions are
transformed into estimated levels of competency for each learner in the system. A Recommendation Engine (yellow)
uses competency information and Learner Profiles to estimate mastery. The Learning Analytics Store collects
information about all users in the System. Learner Inference information is also collected about each individual
learner in the system. This information is used to augment a Collaborative Filtering algorithm with Content-based
filtering that matches content to user characteristics. Learner Inference data was also written back to the Learner
Profile that houses competency information. The recommender uses both algorithms to statistically rank each
recommendation that was presented to the learner. For the 2018 Reference Implementation, the Learner Profile
and the Activity Index were integrated into other TLA components. However, these will be separate components in
future implementations.
5 The ADL Initiative – May FY19
Section 3 provides a detailed description of the specific components and architecture used in the 2018 TLA
Reference Implementation. The data that flows between the different learning activities and the core TLA
systems and components are areas where specifications and standards were applied in 2018. TLA
components can generate terabytes of data in a typical course. Shared services between different
components exchange data using a common communication protocol. These protocols are built using
commonly available standards or specifications across separate web services. This allows the TLA to stay
independent of vendors, products, or technologies and maintains focus on the interoperability between
learning systems.
A current snapshot of TLA components and services are shown in Table 1 and Table 2. Entries within these
tables reflect current capabilities but will change as the TLA evolves. When instantiated in the 2018 TLA
Reference Implementation, some TLA components were aggregated with other TLA components to create
efficiencies or to facilitate the exploration of candidate specifications between components. Specifically,
the Activity Index was instantiated as part of the recommendation engine and the TLA Learner Profile was
instantiated as part of the Competency Management System. However, both components will be decoupled
from those systems and will be set up separately in future TLA iterations.
Table 1. 2018 TLA Components – The future learning ecosystem, powered by the TLA, contains several learning
systems that must share data and work together to initiate, tailor, and track different learning experiences. This table
lists major TLA components used in the 2018 TLA Reference Implementation.
TLA Component Short Description Related Standards
Activity Index /
Activity Registry
The Activity Index/Activity Registry manages the relationship between
activities, competencies, enabling and terminal learning objectives,
and other relevant information about each activity. It also contains
access information and permission to allow access to activities. For
2018, the Activity Index was part of the FLUENT architecture and in
the future instantiations will be set up as a separate TLA component.
LRMI, Dublin Core
Metadata, LOM
Activity Streams These streams are time-stamped data about learner experiences
tracked across learning activities and throughout the TLA. The
Experience API (xAPI), version 1.0.3, is used to track performance
inside TLA Learning Activities.
xAPI, Caliper,
HPML
Competency
Framework
The framework is a model that broadly describes performance
excellence within the TLA. Each framework includes numerous
competency objects that may be applied to multiple occupational
roles.
ASN™, CASE™,
O*Net,
Medbiquitous
Competency
Management
System
Competency Management System, CaSS, provides a common
language/framework to describe competencies, formalizes the
mechanism for collecting evidence of attainment, and manages the
lifecycle of learning.
RCD
Credential
Management
System
Credentials are learning artifacts gained through an organization
based on merit. CaSS is the backend to Credential Engine. For the 2018
implementation, CaSS was used for both competencies and
credentials.
CDTL, Open Badges
2.0, Verifiable
Claims
Data Dashboards,
Data Analytics,
and
Visualizations
These components process data to provide insight and reports.
Currently connected to LRS functionality. In future implementations,
they will pull data from different systems to mashup different data for
different users.
xAPI, TBD
Learner Profile Learner Profiles store learner data about competencies,
achievements, context, and other learner characteristics that may
influence different TLA components. For 2018, the Learner Profile was
part of the CaSS system. In future instantiations, the Learner Profile
will be set up as a separate TLA component.
AIS, CLR
6 The ADL Initiative – May FY19
TLA Component Short Description Related Standards
Learner Record
Provider (LRP)
TLA learning activities include videos, PDF documents, SCORM
courses, serious gaming applications, e-books, mobile learning
content, concept maps, and multiple different types of assessments.
xAPI
Learner Record
Store (LRS)
An LRS, Learning Locker, stores xAPI statements across a wide variety
of learning activities, experiences, devices, and platforms.
xAPI, Caliper,
HPML
Learning
Management
System (LMS)
An LMS delivers and manages instructional content, and typically
handles student registration, online course administration, and
tracking, and assessment of student work.
SCORM
Recommendation
Engine
FLUENT – Utilizes data from the LRS, Learner Profile, Activity Registry,
and metadata attached to learning content to adaptively sequence
the delivery of different learning activities to each individual.
AIS (Too early to
evaluate)
Single Sign-On For the purpose of the 2018 TLA Reference Implementation, the ADL
Initiative staff utilized Keycloak™ for identity and access management.
This capability allows a user to log-in once and have access to multiple
applications within the TLA ecosystem. This was also instrumental in
establishing universal unique identifiers for each learner to protect
each learner’s true identity.
OIDC, OAuth 2.0,
SAML 2.0
Table 2. 2018 TLA Services – This table lists the underlying services used to enable communications between the
different TLA components to communicate with each other.
TLA Services Short Description Related Standards /
Technologies
Assessment Within the 2018 TLA Reference Implementation, CaSS utilized a
component called an Evidence Mapper. This collected assessment
evidence and other data used to infer competency for a specific skill.
QTI, AICC, SCORM, IEEE
1484.11, xAPI
Authorization Authorization is the access granted by an authority to specific
activities/components
Keycloak™, CAC, web3,
OAuth 2.0
Authentication Authentication is the means by which identity is validated with the
purpose of access
Keycloak™, CAC, web3,
OIDC
Content
Discovery
This has not currently been implemented within the 2018 TLA
Reference Implementation. But it is being considered for future
iterations. This service will be closely tied to metadata generation.
OAI/PMH, SKOS
Identity
Management
Identity management is a coordination of services and protocols for
preserving and representing a learner’s uniqueness across the system.
Keycloak™, CAC, web3,
OIDC, UUID
Launch Launch refers to being responsible for delivery and session
management of a resource.
Cmi5, WebSockets
Messaging Within the 2018 TLA Reference Implementation, numerous messaging
services were implemented to push data from one component to
another.
http/s, Websockets,
AMQP
Metadata
generation
This capability was manually created for all content used in the 2018
TLA Reference Implementation. Research into automation tools,
techniques, and technologies is expected to inform future iterations.
Schema.org
Vocabularies, LRMI,
OASIS CMIS, LOM,
Dublin Core Metadata
Notifications Notifications is not currently implemented within the 2018 TLA
Reference Implementation but being considered for future iterations.
W3C Push Notifications
API, WhatWG
Notifications API
Search Search is not currently implemented within the 2018 TLA Reference
Implementation but being considered for future iterations.
SPARQL, PLQL,
ElasticSearch
TLA Service
Registry, aka
Discovery
For 2018, this service exposed a RESTful API for either manual or
dynamic IP address registration for all major components and
services.
XRDS, REST, WSDiscovery, RAMLET
7 The ADL Initiative – May FY19
1.4 2018 TLA TEST AND DEMONSTRATION
The TLA Test and Demonstration was held in August of 2018 to assess the 2018 TLA Reference
Implementation. The 2018 TLA Reference Implementation was developed by a cross-organizational team
to evaluate the various software services, technical components, and learning activities, all of which
exchanged data using a candidate set of technical standards and specifications. The 2018 TLA Test and
Demonstration provided hands-on experience with the 2018 TLA Reference Implementation by students,
instructors and administrators. Additional data was collected as content was tailored and instrumented to
work within the reference implementation.
A primary goal of this study was to better understand what the impediments to full adoption and/or diffusion
of the TLA might be with the current design. Another goal was to assess the practical functionality of the
system and the viability of the candidate specifications used in the 2018 TLA Reference Implementation.
Numerous communications protocols, interface specifications, and data models (schemas) are required to
facilitate the systems to systems interoperability required by the TLA. Many of these candidate
specifications required modifications or custom extensions to meet the requirements for delivering the
myriad of different learning activities. The 2018 TLA Test and Demonstration included traditional elearning content, a browser-based concept-map assessment application, an e-book, micro-learning activities,
a computer-based serious game, and physical (non-digital) instructor-led learning activities.
Figure 3. 2018 TLA Test and Demonstration - The TLA was tested in collaboration with the John F. Kennedy
Special Warfare Center and School (JFKSWCS), Fort Bragg, NC.
The 2018 TLA Test and Demonstration showed adaptation across these different learning activities, but
within the context of a single course. Future implementations of the TLA will provide the ability to adapt
across learning trajectories, careers, and experiences. The 2018 TLA Test and Demonstration used learning
content related to an existing course entitled Combat Observation and Decision-Making in Irregular and
Ambiguous Conflicts (CODIAC). As shown in Figure 3, it was tested by approximately 60 participants
from the Army Special Warfare Education Group (Airborne) (SWEG(A)) at the John F. Kennedy Special
Warfare Center and School (JFKSWCS). Testing consisted of users interacting with the CODIAC content
for 14 hours over five days, with data collected before, during, and after.
Automated data collection efforts at the event produced a significant quantity of data. The system collected
over 450,000 xAPI statements during the 4-day event, which were later anonymized and distributed for use
in a TLA hack-a-thon held at the 2018 iFest. Other than xAPI statements, the system captured each learner’s
final competency profile, their activity history, and statistically significant pre-test and post-test
differentials. Additionally, the surveys and focus groups collected learner opinions about usability,
relevance, and other impressions from various users of the reference implementation. Section 4 of this
document provides an overview of the 2018 TLA Test and Demonstration.
8 The ADL Initiative – May FY19
1.5 2018 DATA ANALYTICS AND VISUALIZATION
The goal of the hack-a-thon was to present opportunities for a broad community of experts to explore the
TLA in depth and discuss new or alternative methods to combine and process learning data to empower
analytics. The hack-a-thon investigated the potential of learning analytics, identified new ways to visualize
collected data, and exposed gaps in the practical use of the Experience API (xAPI) across different types
of media. The ADL Initiative provided a large set of TLA data from the 2018 TLA Test and Demonstration
for participants to aggregate, analyze, and visualize. High-level goals presented at the beginning of the
hack-a-thon included the following:
• Make multi-activity time-series data accessible to a non-data scientist.
• Correlate activities and outcomes for issuing awards, badges, and credentials (microcredentials).
• Create visualizations from the existing data set based on content, anomaly detection, or
other options.
• Identify publicly available data and third-party APIs that can be instrumented with xAPI
to drive analytics.
Hack-a-thon attendees came from a variety of organizations spanning industry, academia, and government.
Most participants identified as software developers or data scientists and experience levels were mixed.
Participants were organized into groups based on level of experience and focus area. Participants had the
opportunity to network with experienced developers, educators, and data scientists to learn best practices
for implementing, structuring, storing, and processing xAPI data and other data generated by TLA
components at scale. Section 5 of this document presents and discusses these results.
9 The ADL Initiative – May FY19
2. TLA SPECIFICATIONS AND STANDARDS
Research is continually performed to evaluate how standards and specifications might be utilized across the
future learning ecosystem, at various stages of a career trajectory, and across institutional boundaries. Table
3 below provides a mapping of relevant specifications and standards for each TLA component or service.
This table summarizes the candidate standards and specifications utilized or investigated for 2018. The
following sections provide an in-depth synopsis of how each specification or standard was investigated,
utilized, and/or modified to support the 2018 TLA Reference Implementation.
Table 3. Summary of TLA Specifications – This table breaks down each specification to show how and where it
was used. The specifications are grouped and listed according to the TLA component it is aligned with. There is an
expectation that TLA Components and their outputs adhere to the referenced specifications. Supporting vocabularies,
rules, and software ensure this process.
The following sections describe the candidate specifications and standards used for each TLA component
or service, providing a thorough evaluation of capabilities and recommended extensions to support TLA
requirements with additional insights describing how they complement or compete with other related
specifications.
TLA Component Standard/Specification Transport Data Store/Registry
Activity Registry LRMI Json Activity Index
IEEE 1484.12.1 LOM XML Activity Index
Dublin Core Metadata XML, RDF Activity Index
Schema.org Vocabularies Activity Index
Activity Stream IEEE xAPI http/https with json payloads Learner Record Store
IMS Global Caliper http/https with json-LD
payloads
Learner Record Store
W3C Activity Streams 2.0 Json, json-LD Needs Research
SISO HPML Human Performance Data
Model
Competency
Management
ASN™ CaSS
CASE™ CaSS
O*Net CaSS
Medbiquitous CaSS
RCD / RCDEO CaSS
Others CaSS
Credentials CTDL Credential Authoring
Language
Credential Registry
IMS Global Open Badge
W3C Verifiable Claims
Data Analytics and
Visualization
Environment
Multiple DAVE Algorithms, DAVE
Visualizations, Data Cards
Dashboards, TLA Portal
Learner Profile CaSS Proprietary
Airmen Learner Record
TBD Research
Learner Service TBD Research
Recommender AIS Needs Research N/A
Identify Management
/ Single Sign On
Keycloak™ OpenID Connect (profile of
OAuth2.0)
TLA Common Training
and Education Portal
Learner Record
Provider - eBooks
ePUB 3 (formerly IEEE
ADB), xAPI
10 The ADL Initiative – May FY19
2.1 ACTIVITY REGISTRY
The Activity Registry includes an Activity Index that stores metadata about each TLA learning activity.
The approach used in the 2018 Reference Implementation relied heavily on the Learning Resource
Metadata Initiative (LRMI) specification. While LRMI shows promise for tagging new content, there are
thousands of learning resources tagged using metadata schemas managed under different organizations.
Given the current state of technology, the TLA needs to be compatible with many of the more common
metadata formats.
All data formats currently being investigated only support human-readable descriptions and don’t consider
the type of metadata generated by modern machine learning algorithms. These systems generate metadata
based on a statistical analysis to describe concepts such as engagement, effectiveness, or any number of
other descriptors that would be useful to other TLA components. In the future, an Activity Registry will
include data and connected services that discover, catalog, and store information about TLA compatible
learning activities. A primary goal of the 2019 research is to understand how the Activity Registry drives a
Common Course Catalog, populated by different stakeholders. Other research slated for 2019 includes the
identification of approaches for integrating paradata into the Activity Registry. The following metadata
standards are being looked at to describe TLA content.
2.1.1 Learning Resource Metadata Initiative (LRMI)
The LRMI1
is a common metadata framework developed by Creative Commons (CC) and the Association
of Educational Publishers (AEP) for describing and tagging of educational resources in web-based
instruction and training. The LRMI metadata schema was adopted by Schema.org in April 2013. This
allows anyone who publishes or curates educational content to use LRMI markup to provide rich,
education-specific metadata about their resources with the confidence that this metadata will be recognized
by major search engines.
The LRMI specification is a collection of classes and properties for metadata markup and description of
educational resources. The specification provides clear guidance of the terms within and how to use them
both in coding and through descriptive language to be followed by those implementing it. The attributes
defined by the metadata are clear and the rules surrounding the specification drive interoperability. LRMI
1.1 specification is very stable and has seen significant adoption. Within the context of the 2018 TLA
Reference Implementation, LRMI was selected because it provided the most contextually relevant data
fields to empower the FLUENT recommender.
The LRMI specification includes an AlignmentObject as part of the specification. This object
describes an alignment between a learning resource and a node in an educational framework. This feature
was used extensively in the 2018 TLA Reference Implementation. The 2018 alignments included
educational audience, educational use, competency object, simple or complex, interactivity level, and others.
While limited, these use cases proved the ability of the AlignmentObject to serve its intended purpose.
The 2019 research effort needs to build out a minimum set of metadata requirements that should be
implemented for any future learning activities or content.
1
http://lrmi.dublincore.org/
11 The ADL Initiative – May FY19
2.1.2 IEEE 1484.12.1 Learning Object Model (LOM)
As shown below in Figure 4, Learning Object Model (LOM)
2 provides a broad framework for describing
learning objects to facilitate search, evaluation, acquisition, and use. LOM is used by the SCORM reference
model to provide descriptive information about learning objects, including the title, author, description,
keywords, educational objective, and other relevant information. The LOM data model specifies which
aspects of a learning object should be described and what vocabularies may be used for these descriptions;
it also defines how this data model can be amended by additions or constraints. The metadata is hierarchical,
starting with a root and expanding to leaf nodes. Alignment of an object can be connected to a discipline,
idea, prerequisite, educational objective, accessibility, restrictions, educational level, skill level, security
level, or competency.
Figure 4. Learning Object Metadata – LOM comprises a hierarchy of elements that includes 9 categories, each of
which contains sub-elements. The semantics of an element are determined by its context: they are affected by the
parent or container element in the hierarchy and by other elements in the same container.
https://commons.wikimedia.org/w/index.php?curid=16026109
The purpose of this Standard is to allow the creation of LOM instances in XML. This allows for
interoperability and the exchange of LOM XML instances between various systems. When implementing
the LOM as a data or service provider, it is not necessary to support all the elements in the data model, nor
need the LOM data model limit the information which may be provided. The creation of an application
profile allows a community of users to specify which elements and vocabularies they will use. Elements
from the LOM may be dropped and elements from other metadata schemas may be brought in; likewise,
the vocabularies in the LOM may be supplemented with values appropriate to that community. For example,
the Healthcare LOM3
, developed by the Medbiquitous consortium extends the LOM standard and provides
custom vocabularies for some metadata elements.
2
https://ieeexplore.ieee.org/document/1032843/
3
https://www.medbiq.org/working_groups/learning_objects/HealthcareLOMSpecification.pdf
12 The ADL Initiative – May FY19
The attributes defined by the metadata are clear and the rules surrounding the specification drive
interoperability. The LOM specification (1484.12.1-2002) is very stable and has seen significant adoption.
The ADL Initiative stakeholders have thousands of SCORM courses and other content that has been
encoded with LOM metadata so any TLA component that relies on metadata needs to be compatible with
LOM. As with most metadata formats, the consistency of quality metadata is a known issue.
2.1.3 Dublin Core Metadata Initiative (DCMI) Metadata
The Dublin Core Schema4
is a small set of vocabulary terms that describe digital resources (video, images,
web pages, etc.), as well as physical resources such as books or CDs, and objects like artworks. The Dublin
Core Metadata Element Set is a vocabulary of fifteen properties for use in resource description. It is part of
a larger set of vocabularies and technical specifications maintained by the DCMI. Dublin Core metadata
may be used for multiple purposes, from simple resource description to combining metadata vocabularies
of different metadata standards, to providing interoperability for metadata vocabularies in the linked data
cloud and Semantic Web implementations.
The full set of vocabularies also includes sets of resource classes, vocabulary encoding schemes, and syntax
encoding schemes. The terms in DCMI vocabularies are intended to be used in combination with terms
from other, compatible vocabularies in the context of application profiles and the DCMI Abstract Model.
As part of an extended set of DCMI Metadata Terms, Dublin Core became one of most popular vocabularies
for use with RDF, more recently in the context of the Linked Data movement. The DCMI assumed
stewardship of the LRMI specification in 2014. The Dublin Core schema is relevant because it is widely
used across the Internet. While the educational alignment aspects of DCMI are not as robust as LOM or
LRMI, the ability to tie into additional vocabularies provides an attractive mechanism to identify and
catalog content from different communities of interest, industries, or demographics.
2.1.4 Schema.org Vocabularies
Founded by Google, Microsoft, Yahoo and Yandex, Schema.org vocabularies5
are developed by an open
community with a mission to create, maintain, and promote schemas for structured data on the Internet.
Each schema.org item type has its own set of descriptive properties. The broadest item type is Thing, which
has four properties (name, description, URL, image). More specific types share properties with broader
types. For example, a Place, Person, or CreativeWork is a more specific type of Thing.
LRMI’s adoption into Schema.org vocabularies provides many benefits. In theory, nearly any schema.org
Thing could be a learning resource. Therefore, LRMI addresses those metadata properties that distinguish
content when it is deliberately used for learning. This was done by adding learning-resource properties to
key root types. Currently, CreativeWork properties include educationalUse and
educationalAlignment. A more specific type of CreativeWork includes a Course6 which may be
offered as distinct instances at which take place at different times or take place at different locations or be
offered through different media or modes of study. An educational course is a sequence of one or more
educational events and/or other types of CreativeWork which aims to build knowledge, competence or
ability of learners.
As shown in Figure 5, a learning activity is a schema.org→Thing→CreativeWork that inherits the
properties that every schema.org Thing has and can be used to support multiple forms of alignment that are
4
http://www.dublincore.org/specifications/
5
https://schema.org/
6
https://schema.org/Course
13 The ADL Initiative – May FY19
possible between a resource and an educational framework. The AlignmentObject can also be used
to distinguish between resources that teach and assess. This presents the ability to collect a substantial
corpus of paradata about how different types of content apply to different instructional domains and enables
new insights into which content is more effective.
Figure 5. Alignment between a Schema.org CreativeWork and a Node in an Educational Framework7 – The
2018 TLA Reference Implementation used LRMI’s AlignmentObject to reference a competency framework that
provided a structured description of required knowledge, skills, abilities, and their interrelated relationships.
2.2 ACTIVITY STREAMS
The future learning ecosystem will offer a wide range of learning content across the continuum of learning.
One learner may spend time reading technical articles or writing a blog post while another interacts with
video content and interactive exercises. Activity Streams are a list of events generated by individuals,
groups, applications, or learning activities that provide details about the ongoing experiences to other TLA
components.
The types and variety of activities that are used for learning can often be associated with a specific delivery
modality. Instructor-led classroom training will create one set of instructional activities, while serious
games and simulations have the potential of generating a completely different set of activities. This has
potential to have two similarly named activities with two different contexts for how those activities are
being applied and the experiences they encompass. A common vocabulary is necessary to ensure all
learning activities across different communities accurately describe the experience. By formalizing this
vocabulary, a set of attributes and rules about the data is established such as how they are stored, retrieved
and accessed by other components, systems or activities.
The different activity stream specifications investigated for inclusion in the TLA are similarly structured.
Each specification includes serialized data streams that consist of statements about activities. Such
statements typically involve a subject (the person doing the activity), a verb (what the person is doing), and
a direct object (what the activity is being done to or with). The subject of an activity is nearly always the
learner but could foreseeably be an instructor, cohort, or other. The direct object of an activity is presented
differently depending on its context. Verbs need to conform to a common vocabulary. Otherwise different
organizations will use different verbs to describe the same activity or the same verb to describe different
activities.
7
https://blogs.pjjk.net/phil/explaining-the-lrmi-alignment-object/
14 The ADL Initiative – May FY19
2.2.1 Experience API (xAPI)
The xAPI8
specification is in the process of becoming a standard through the Institute of Electrical and
Electronics Engineers – Learning Technology Standards Committee (IEEE-LTSC)
9
. The xAPI specifies a
structure to describe learning experiences and defines how these descriptions can be exchanged
electronically. The main components of xAPI are the data structure called Statements and the data
storage/retrieval capability called the Learning Record Store (LRS). The xAPI specification has stringent
requirements on the structure of this data and the capabilities of the LRS. Statements are data triples that
use an actor, a verb, and an object to describe any experience. Each statement also includes timestamps and
unique, resolvable identifiers. The transport is HTTP/HTTPS with JSON payloads.
Any enabled device can send xAPI statements including mobile phones, CPR dummies, serious games and
simulations, and any number of other learning systems. The “xAPI Profile Specification”
10 offers a common
way to express controlled vocabularies across these different mediums, provides instructions on the
formation of xAPI statements, and describes patterns of xAPI usage that provides additional context to a
domain, device, or system. The xAPI Profiles Specification also adds tools to support authoring,
management, discovery and/or adoption, including additional data elements and properties.
A Learning Record Store (LRS) is the implementation of the server-side requirements associated with the
xAPI specification. The LRS is a key component of the xAPI architecture. It is the application interface for
storing, accessing, and often visualizing the data about learning experiences, activities, and performance.
The LRS is essentially a web service that leverages REST to ensure all data from xAPI can be stored and
retrieved. It can share data and transcripts with other LRSs so learning experiences can follow you from
one organization’s LRS to another. An LRS could be optionally integrated with any application such as an
LMS, human resources (HR) system, or it could serve as centralized data store in an enterprise learning
ecosystem. Third party applications which send or retrieve learning activity data will interact with the LRS
as the data store for xAPI data. From this perspective, an LRS is also required to validate the format of the
statements as many of the requirements in the xAPI specification are targeted toward the LRS component.
2.2.2 IMS Global Caliper
The Caliper Analytics® Specification11 defines a set of concepts, relationships, and rules for describing
learning activities. Each activity domain is described in a metric profile that models a learning activity or
any supporting activities. Each profile provides a shared vocabulary that developers use to describe user
interactions in a consistent manner. Each profile is composed of one or more Event types (e.g.,
AssessmentEvent, NavigationEvent). Each Event type is associated with a set of actions
undertaken by learners, instructors, and others. A Caliper Event describes the purposeful action undertaken
by the actor within a given learning context. The data model is based upon the (Actor, Action,
Activity) triple. Various Entity types representing people, groups, and resources are provided to
describe the relationships established between participating entities and the contextual elements relevant to
the interaction (e.g., Assessment, Attempt, CourseSection, Person).
Storage or data retrieval capabilities are not defined as a part of the specification; however, the defined data
attributes about Events and Entities are clear and the rules surrounding the specification drive
8
https://github.com/adlnet/xAPI-Spec
9
https://www.tagxapi.org/
10 https://github.com/adlnet/xapi-profiles
11 https://www.imsglobal.org/sites/default/files/caliper/v1p1/caliper-spec-v1p1/caliper-spec-v1p1.html
15 The ADL Initiative – May FY19
interoperability. It is comparable to the xAPI specification but offers a more controlled vocabulary. It does
implement ideal behaviors that xAPI has identified for future profiles. The Caliper specification appears to
be gaining adoption across learning tool providers including Blackboard, Canvas, and others. Because other
IMS Global standards are prevalent in K-12 education, it is anticipated that adoption will continue to grow.
2.2.3 W3C Activity Streams 2.0
The W3C Activity Streams12 is an open format specification for activity stream protocols, which are used
to syndicate activities in social web applications and services, like those in Facebook or LinkedIn. An
Activity is a semantic description of an action. This specification describes a JSON-based serialization
syntax for the activity vocabulary that conforms to a subset of [JSON-LD] syntax constraints but does not
require JSON-LD processing. The Activity Streams 2.0 Vocabulary defines a set of abstract types and
properties that describe past, present, and future activities. For example, the activity vocabulary defines five
specific types of actors that include Application, Group, Organization, Person, and Service.
Every JSON object in an Activity Streams 2.0 document is either an object or a link. The object is the
primary base type for the Activity Streams 2.0 Vocabulary. In addition to having a global identifier
(expressed as an absolute Internationalized Resource Identifiers (IRI) using the id property) and an object
type (expressed using the type property), all instances of the object type share a common set of properties
normatively defined by the activity vocabulary. Object types are segmented into a set of eight core types,
but external vocabularies can be used to express additional detail not covered by the activity vocabulary.
This specification presents opportunities for future TLA implementations. The W3C specification is already
in use across many social platforms that are already part of the future learning ecosystem. Additional value
may also be found in the way this specification implements properties and their ability to push/pull
metadata about different objects referenced in the stream (e.g., information about places, content, other
people). The capacity to include content metadata as a payload presents interesting possibilities in the way
we catalog learning experiences and could have a huge impact on the way a competency management
system makes inferences about learner competencies after interacting with a TLA learning activity.
2.2.4 SISO Human Performance Markup Language (HPML)
HPML13 is an XML-Schema-based language intended to support trainers, instructors, leaders and others in
the evaluation of individuals and teams as they perform their job functions. HPML provides schemas for
organizing the information relevant to defining performance measurements, including computations,
measures, assessments, results, and instances/periods. Specifically, it is an XML based language designed
to express performance measurement concepts in a format that is both machine and human readable.
HPML enables the explicit combination and transformation of performance data into performance
measurements and assessments. These are decoupled from a specific training system but can be used to
specify the data elements to be collected from a system, the calculations to use to process the data, and
when to produce performance measurement results. As shown in Figure 6, HPML provides a flexible
framework for configuring and executing performance measurement and assessments. The schema is
separated into six distinct groups that make up the core components of HPML and can be added to or
expanded with additional links in the schemas.
12 https://www.w3.org/TR/activitystreams-core/#documents
13 https://discussions.sisostds.org/index.htm?A0=SAC-PDG-HPML
16 The ADL Initiative – May FY19
Figure 6. HPML High-Level Conceptual Architecture – HPML includes a Human Performance Data Model that
defines a set of rules that dictate how performance data is measures, computed, and assessed. It is agnostic about
the kinds of data used to compute the measurements and assessments.
HPML as a specification includes both the markup language used to author the activity stream and the
Human Performance Data Model used to measure and assess performance based on the incoming stream
of data. This is especially useful when considering approaches for collecting performance data out of live,
virtual, and constructive simulations, but also provides immediate value to the TLA in formalizing a method
for rolling up future learning ecosystem activities into meaningful assessments. More specifically, by
externalizing the measurement and assessment specifications, these can be used with any data source (e.g.,
other activity streams).
2.3 COMPETENCY MANAGEMENT
Competency management requires the generation of rich and traceable data about learning experiences,
how they relate to skill proficiency, and the knowledge, skills, abilities, and behaviors that individuals need
to do their job. Competencies describe specifically what people need to do to be effective in their roles, and
it clearly establishes how their roles relate to organizational goals and success. Each individual role has its
own set of competencies needed to perform the job effectively.
Competency-based learning (CBL) represents a transition from curricula focused on abstract knowledge
and pursuit of certificates to curricula built around authoritative performance indicators that guide learning
experiences based on challenges in the workplace that unlock human potential. Proficiency is a complex
and critical concept that requires relevant, trusted data that can be used as evidence about an individual’s
mastery against a specific set of competencies. A competency framework is a structure for defining the
knowledge, skills, abilities and attitudes (or other characteristics) required to do a job. Competency
frameworks are widely used in business for defining and assessing competencies within organizations in
both hard and soft skills that jointly define successful job performance. There are numerous competency
frameworks available and numerous specifications that drive them. Some of these competency
specifications are included later in this section.
17 The ADL Initiative – May FY19
2.3.1 Achievement Standards Network™ (ASNTM)
ASN™ provides access to machine-readable representations of learning objectives and curriculum
standards14. It provides an RDF-based framework based on the Dublin Core Metadata Initiative’s (DCMI)
syntax-independent abstract information model (DCAM). The DCAM is intended to support development
of Dublin Core Application Profiles (DCAP) of which the ASN DCAP is an example. The ASN™
framework is made up of standards documents, which represent the overall competency framework; and
Statements, that represent the individual achievements within the overall framework. A set of core
properties define the relationships between the two in terms of an Entity Relationship model. Structural
relationships replicate the relationships between different components of the Standards Document and
semantic relationships define the relationships of meaning between statements (e.g., assertion of
equivalence).
ASN™ is designed for interoperability and open access to learning objectives. It has seen wide adoption in
the K-12 community. The ASN Description Framework (ASN-DF) also provides the means to create ASN
profiles through inclusion of additional properties and classes from other namespaces and definition of
constraints on data patterns. ASN-DF provides a small vocabulary of classes and properties and a set of
mechanisms for tailoring an ASN profile based on the Dublin Core's conceptualization of application
profiles and description set templating as well as refinements to these conceptualizations developed by the
U.S. Library of Congress to support BIBFRAME profiling.
2.3.2 Competencies and Academic Standards Exchange™ (CASE™)
The IMS Global Competencies created the CASE™ specification15 to define how systems exchange and
manage information about learning standards and competencies in a consistent and digitally-referenceable
way. CASE™ connects standards and competencies to performance criteria and provides a way to transmit
rubrics between various platforms. Within CASE™, the underlying structure of both a competency and an
academic standard are represented using the same data model. The data model is composed of three core
constructs:
• The root definition document - the CFDocument is the top-level structure that holds the set of
statements that define the individual competencies/academic standards.
• The set of composition statements - the CFItem is the set of statements into which the top-level
competency/academic standards have been decomposed.
• The rubric – the CFRubric defines how mastery of the competency or standard can be achieved.
This requires the definition of specific criteria used for each of the scores that can be awarded
during an assessment.
The CASE™ specification defines internal relationships between Parent/Children statements like ,
Precedes, IsRelatedTo, Exemplar, and IsPartOf. All competency frameworks published in
CASE™ format can be linked together in a network of equivalent or aligned competencies. Having
universal identifiers for each competency makes it possible for any tools or applications to share
information between systems.
14 http://www.achievementstandards.org/
15 http://www.imsglobal.org/activity/case
18 The ADL Initiative – May FY19
2.3.3 O*Net
The Occupational Information Network (O*NET) 16 is sponsored by the U.S. Department of Labor,
Employment and Training Administration (USDOL/ETA) to facilitate and maintain a skilled workforce.
Central to the project is the O*NET database, which contains hundreds of standardized and occupationspecific descriptors on almost 1,000 occupations covering the entire U.S. economy. Every occupation
requires a different mix of knowledge, skills, and abilities, and is performed using a variety of activities
and tasks. As shown in Figure 7, the O*NET database identifies, defines, describes, and classifies
occupations through Experience Requirements (Training, Licensing), Worker Requirements (Basic and
Cross-Functional Skills), Occupation Requirements (Work Activities, Context), Worker Characteristics,
Occupation Specific Information, and Occupational Characteristics.
Figure 7. O*NET Content Model – This model provides the framework that identifies and organizes the
distinguishing characteristics of an occupation. The model defines the key features of an occupation as a standardized,
measurable set of variables called descriptors. This hierarchical model starts with six domains that expand to 277
descriptors.
The value of the O*NET Database to the TLA is in leveraging the existing resources these government
sponsored activities have already amassed. Their military transition search connects occupations in the
O*NET database to other classifications systems within the military17. This is accomplished through data
from the Military Occupational Classification (MOC) system at the Defense Manpower Data Center
(DMDC). This capability is also available via web services. Other resources include Careers in the Military
and links to the COOL projects for Army, Navy, Marine Corps, and Air Force.
16 https://www.onetonline.org/
17 https://www.onetcenter.org/crosswalks.html
19 The ADL Initiative – May FY19
2.3.4 MedBiquitous Competency Framework
The MedBiquitous Competency Framework18, ANSI /MEDBIQ CF.10.1-2012, is a technical standard for
representing competency frameworks in XML, accredited by the American National Standards Institute.
Organizations that publish competency frameworks can do so in this standard format, making it easier to
integrate competency frameworks into educational technologies like curriculum management systems. The
standard allows medical schools and other health professions schools to connect their curriculum, learning
resources, and assessment data back to a common set of competencies, ultimately enabling competencybased views of the curriculum and of learner performance. The data model establishes relationships between
competency objects that narrow or broaden the definition of overall competency.
The MedBiquitous Competency Object specification provides a consistent format and data structure for
defining a competency object, an abstract statement of learning or performance expectations, and
information related to the statement. Statements can be learning outcomes, competencies, learning
objectives, professional roles, topics, classifications/collections, etc. The competency object may include
additional data to expand on or support the statement. This specification is meant to be used with the
MedBiquitous Competency Framework specification.
2.3.5 IEEE 1484.20.1 Reusable Competency Definitions (RCD)
The current RCD standard19 defines a data model for describing, referencing, and sharing competency
definitions, primarily in the context of online and distributed learning. The standard provides a way to
represent formally the key characteristics of a competency, independently of its use in any specific context.
It enables interoperability among learning systems that deal with competency information by providing a
means for them to refer to common definitions with common meanings. This specification enables the
storage of a competency framework in the form of a Dynamic Acyclic Graph (DAG) where competency
objects define the knowledge, skills, conditions and other factors that role up into an ability to do a job. The
edges between each competency object inform the relationship between them and have the potential to
identify whether they’re dependent, complimentary, conflicting, or other. This provides an extensible
mathematical underpinning to a competency framework that accommodates the relationships defined in
any other foreseeable competency framework.
The Data Model for RCD Working Group (WG) 2020 was put together in September 2018 to revise the
2008 standard. The Competencies WG 20 intends to take the Credential Ecosystem Mapping Project’s
mapping of competencies metadata and update RCD to represent the most common elements that are found
in multiple standards addressing competencies and competency frameworks. The ADL Initiative will
monitor this WG for progress.
2.4 CREDENTIALS
A credential is a testament of qualification and competence issued to an individual by an authoritative third
party. Examples of credentials include academic diplomas, degrees, certifications, professional licenses, or
other proof of occupational qualification. Within the TLA, credentials are an exchange format by a trusted
party that formally encapsulates a set of competencies. By their nature, they are linked to other TLA
components such as competency frameworks, assessment rubrics, or talent management systems. This
requires the establishment of attributes and rules that allow TLA components to process and compare
18 https://www.medbiq.org/working_groups/competencies/index.html
19 https://ieeexplore.ieee.org/document/4445693
20 http://sites.ieee.org/sagroups-1484-20-1/
20 The ADL Initiative – May FY19
credentials in the same, interoperable way, particularly in Learner Profiles across instantiations of the TLA.
This also requires a common way to describe, store, and access credentials in order to make fair
comparisons of the achievement that was performed to acquire them.
2.4.1. Credential Transparency Description Language (CTDL)
CTDL21 is a vocabulary comprised of terms that are useful in making assertions about a credential and its
relationships to other entities. CTDL is modeled as a directed graph using RDF for describing data on the
web. Like an activity stream, the triple is the basic grammatical construct in making CTDL assertions about
Things and is comprised of three simple components: a subject, a predicate and an object. This structure
allows us to make simple statements that enable a rich description of credential-related resources including
credentialing organizations and specific subclasses of credentials such as Degrees, Certificates, and Badges.
The comprehensiveness and scope of this specification makes it ideal for defining TLA credentials. As
shown in Figure 8, two super classes called Agent and Credential define sets of subclasses. The
primary classes also include the ConditionProfile used to define sets of constraints on the credential
described by an AssessmentProfile, LearningOpportunityProfile, or
CompetencyFramework. These are used to express learning goals and outcomes.
2.4.2 Credential Registry
The Credential Registry22 is both a repository of information
regarding credentials and a set of services that make it easier
to use that information. The Credential Registry works with
CTDL to allow the storage of credentials that have been
expressed using that specification. It also registers
credentialing organizations that issue these credentials
including universities, colleges, schools, professional
associations, certification organizations, and more. Since it is
only a registry, it only stores the description of a credential in
the abstract sense and does not include any personal
information or personally obtained credentials. This
information will typically be stored in a Learner Profile. As a
registry, it does allow users to see what various credentials
represent in terms of competencies, transfer value, assessment
rigor, third party approval status and more.
Credential Engine developers are using Dublin Core Application Profiles to create systems that
communicate all aspects of credentials. The Technical Advisory Committee (TAC) promotes collaboration
across different standardization initiatives that are developing data models, vocabularies, and schemas for
credentials and competency frameworks. The Credential Registry uses CTDL to capture, link, update, and
share up-to-date information about credentials so it can be organized and centralized within the registry,
made searchable by customized applications and linked to from anywhere on the open Web.
21 http://credreg.net/ctdl/handbook
22 https://www.credentialengine.org/credentialregistry
Figure 8. CDTL Class Structure – Six
primary classes structure the data used to
express learning goals and outcomes.
21 The ADL Initiative – May FY19
2.4.3 IMS Global Open Badges
Open Badges23 are visual representations of verifiable achievements earned by recipients. Open Badges is
a technical specification and set of associated open source software designed to enable the creation of
verifiable credentials across a broad spectrum of learning experiences. The Open Badges standard describes
a method for packaging information about accomplishments, embedding it into portable image files as
digital badges, and establishing resources for its validation and verification. Open Badges contain detailed
metadata about achievements such as who earned a badge, who issued it, and what it means. The Open
Badges 2.0 specification expands on this capability to include versioning, endorsements, and full use of
JSON-LD.
Open Badges are expressed as linked data so that badge resources can be connected and reference by other
systems. This metadata can also be embedded within the graphic. The Open Badges vocabulary defines
several data classes used to express achievements. There are three core data classes (Assertion,
BadgeClass, and Profile) that define mandatory and optional properties as well as the restrictions on
the values those properties may take. Each badge object may have additional properties in the form of an
Open Badges Extension, a structure that follows a standard format so that other users can understand the
information added to badges. Assertions are representations of an awarded badge, used to share information
about a badge belonging to one earner.
2.4.4 W3C Verifiable Claims
The mission of the verifiable claims24 is to make expressing and exchanging credentials that have been
verified by a third party easier and more secure. The basic components of a set of verifiable claims include
a subject identifier, claims about the subject, claim set metadata, and a digital signature. Both the Entity
Profile Model and Entity Credential Model consist of a collection of name-value pairs which will be referred
to as properties in the data model. The link between the two is in the id property. The Entity Profile Model
defines a subject identifier in the id property, while the claims section of the Entity Credential Model uses
the id property to refer to that subject identifier. The first working draft of this specification was released
in August 2017. Additional research is required to better evaluate its applicability to the TLA. However,
the defined data model and structure of this specification implies that it could connect both semantically
and syntactically. The data associated with verifiable claims are largely susceptible to privacy violations
when shared with other TLA components so many of the verifiable claims use cases should be used to
inform how PII centered on credentials should be protected within the security boundaries of the TLA.
2.5 LEARNER PROFILE
There are numerous challenges in creating a lifelong learning profile. What elements of the Learner Profile
should the learner be in control of? How does learner information pass from one organization to another?
What authoritative systems have permissions to write to the Learner’s Profile? How do we represent learner
context?
Learner Profiles exist in many systems and are extremely diversified in nature. Within the context of the
future learning ecosystem, it is envisioned that a Learner Profile will house data about people across their
entire career. Adult learners are characterized by distinctive personal attributes, knowledge, skills and
competencies that they have gained in different contexts through a process of development and growth. A
23 https://www.imsglobal.org/activity/digital-credentials-and-badges
24 https://www.w3.org/TR/verifiable-claims-data-model/
22 The ADL Initiative – May FY19
Learner Profile may also include demographic data, student interests, learning preferences, descriptions of
preferred learning environments, inter/intrapersonal skills, existing competencies and those that need to be
developed, socio-emotional health, and a myriad of other data. Learner Profiles are dynamic and will
change over time. As student interests change and they become competent in new areas, the profile will
update to reflect the latest state of the learner.
While no formal specifications were used in the 2018 TLA Reference Implementation, numerous insights
were learned that will influence future research. The 2018 TLA Test and Demonstration exposed a
multitude of learner-related data was generated and consumed by different TLA components. This
illustrates the need for a Learner Profile Service that allows different components and activities to exploit
relevant information about each learner. To maximize the value these components can gain from the learner
data, a broad and consistent taxonomy is necessary to describe and characterize the different attributes about
each learner. Business rules are also needed to manage how this data is used across the Future Learning
Ecosystem.
Existing Learner Profile specifications (e.g., LIP, PAPI) are somewhat dated. Their bindings are described
in the XML language which is not appropriate for semantic interoperability. In some ways, these standards
hinder the process of sharing and reusing Learner Profiles because it is difficult for applications to exploit
profiles that are stored deep within various servers. Additionally, these specifications do not easily allow
extensions to the schema with additional information that is relevant to other TLA components.
2.5.1 Comprehensive Learner Record (CLR)
IMS Global led the development of the CLR25, formerly known as the Extended Transcript. It was originally
designed to support traditional academic programs as well as co-curricular and competency-based
education by capturing and communicating a learner's achievements in verifiable, digital form. The CLR
contains data about different learning experiences and achievements including course descriptions, syllabi,
rubrics, portfolios, performance evaluations, prior learning, internships, and other experiences or
assessments. As a digital transcript, the CLR closely follows the registrar guidance of the American
Association of Collegiate Registrars and Admissions Officers (AACRAO) to draw information from an
institution’s student information systems, learning management systems (LMS), or other internal
databases.26 The CLR is due to be completed in 2019 but will likely evolve to help foster adoption across
higher education institutions. Integration with existing institutional reporting systems and data structures
will be critical in enabling this effort to succeed. The ADL Initiative will continue to monitor this effort to
measure its applicability to the DoD and other Federal government stakeholders.
2.5.2 Learner Information Package (LIP)
The IMS LIP specification27 provides a standard means for recording information about learners. It is
designed to allow information about learners, including their progress to date and awards received, to be
transferred between different software applications. In this specification, the learner information is
separated into eleven main categories of information that are: Identification,
Qualifications/Certifications/Licenses (QCL), Accessibility, Activity, Goal, Competency, Interest,
Transcript, Affiliation, Security Key, and Relationship. The LIP specification describes the data structures,
XML binding, and best practices for the formatting, storage, and exchange of learner information.
25 https://www.imsglobal.org/activity/comprehensive-learner-record
26 https://library.educause.edu/~/media/files/library/2019/1/eli7164.pdf
27 http://www.imsglobal.org/profiles/index.html
23 The ADL Initiative – May FY19
The specification supports the exchange of learner information among learning managements systems,
human resource systems, student information systems, and other systems used in the learning process. LIP
is an older specification and its reliance on XML impacts the access speed for how these systems can load
learner information. The future learning ecosystem will require real time learner services that can send and
update records on the fly. An accessibility for LIP standard28 is extending the LIP Information Model to
better define accessibility preferences for people with disabilities. This work is intended to benefit all
learners in situations that require alternative modes of instruction, such as an extremely noisy environment
where captions are needed for a video or language barriers.
2.5.3 IEEE Public and Private Information for Learners (PAPI Learner)
The PAPI Learner specification29 is a multi-part standard that specifies the semantics and syntax for storing,
retrieving, searching, and exchanging learner information. It defines elements for recording descriptive
information about knowledge acquisition, skills, abilities, personal information, learner relationships,
preferences, performance, and similar types of information. An important feature is the logical division,
separate security, and separate administration of several types of learner information. This specification
partitions the learner record into six main information types that support extension:
• Learner Contact Information: describes information related to administration.
• Learner Relations Information: stores information about the learner’s relationships to other users
of learning systems, such as teachers and other learners.
• Learner Security Information: stores information about the learner’s security credentials, e.g.
passwords, private keys, public keys, biometrics.
• Learner Preference Information: describes preference information intended to improve human
computer interactions and the automatic adaptation and personalization of systems to specific needs
of the learner.
• Learner Performance Information: is about the learner’s history, current work, or future
objectives. PAPI Performance information is primarily created and used by learning technology
components to provide improved or optimized learning experiences.
• Learner Portfolio Information: is a representative collection of the learner’s works or references
to them that is intended for presentation and evidencing of his achievements and abilities.
This standard permits different views of the learner information and addresses issues of privacy and security.
The data models associated with this specification is not sufficiently complete to cover all the learner data
that needs to be exchanged between TLA components, particularly those related to
pedagogical/andragogical aspects such as defining an educational pathway.
2.5.4 Airmen Learning Record (ALR)
The vision of the Air Force Learning Services Ecosystem (AFLSE) is to support a continuum of learning
that deliberately prepares Airmen with the required competencies to meet the challenges of the 21st century.
The ALR is a comprehensive record of all learning Airmen achieve during their career including
educational transcripts, training records, performance reports, and ancillary training transcripts. The
Airmen’s learning record is a centralized location to record all learning, whether it occurs in a specialized
training or education program, on-the-job, or even off-duty. The Airmen’s learning record will enhance the
28 https://www.imsglobal.org/accessibility/acclipv1p0/imsacclip_infov1p0.html
29 http://metadata-standards.org/Document-library/Meeting-reports/SC32WG2/2002-05-Seoul/WG2-SEL042_SC36N0175_papi_learner_core_features.pdf
24 The ADL Initiative – May FY19
ability to analyze the readiness of the air force by capturing an Airmen's knowledge and skills gained
throughout the continuum of learning (training, education, and experiences), documenting progress and
achievements, and identifying gaps and opportunities for growth tied to mission accomplishment from both
an enterprise perspective as well as an individual level.
2.6 RECOMMENDATION ENGINE
A recommendation engine collects data, analyzes the data, filters the data, and makes a recommendation.
While recommender algorithms are not likely to be standardized, they require computationally accessible
data to reason over for which technical standards and specifications will play a role. Within the context of
the TLA, a recommendation engine uses data filtering algorithms that make use of metadata and paradata30
about learning resources, competency frameworks and the definitions of competence, Learner Profiles, and
other data to inform a probabilistic ranking that guides learner interactions. Beyond the existing
specifications and standards that a recommender must interface with, a recommender must be grounded in
the science of learning.
This is substantially different from the recommenders most people are familiar with from their experience
on Amazon or Netflix. These recommenders use collaborative filtering algorithms, content-based filtering
algorithms, or a hybrid of the two. These algorithms make recommendations based on other similar user
choices or on other similar content that someone might be interested in. Within the future learning
ecosystem, recommendations need to be informed by learning theory. This adds complexity because not all
people learn the same way which impedes the collaborative filtering approach. Additionally, the process of
learning is a constantly moving target, so content-based filtering is not an ideal situation either, although
both will certainly play a role.
2.6.1 Fast Learning from Unlabeled Episodes for Next-Generation Tailoring (FLUENT)
FLUENT31 is an open source recommendation engine used in the 2018 TLA Reference Implementation. It
utilized the LRMI metadata attached to various learning resources and information inferred about learners
to make its recommendations. FLUENT supports the ability to utilize different instructional frameworks
which provides the ability to deliver content recommendations based on different learning theories. This is
important as different instructional domains require different instructional strategies.
2.6.2 Navigator for Integrated Learning Environments (NILE)
NILE builds upon an existing commercial product that has been historically targeted at the K-12 educational
domain. They use a Google MapsTM approach to create a recommended learning path for learners to take.
NILE has a content discovery service that has aggregated millions of educational resources into their
platform. Recommended learner pathways change based on performance, instructor input, or student
choices. The ADL Initiative is funding the migration from propriety data structures and interface
specifications to a TLA-enabled ecosystem to better evaluate how TLA specifications and standards scale.
NILE includes numerous enabling technologies (e.g., content discovery service) that are also appealing to
the TLA ecosystem of tools and technologies.
30 Data about how content was being used that was collected.
31 https://github.com/soartech/tla/tree/master/learner-profile-interface/src/main/java/com/soartech/fluent
25 The ADL Initiative – May FY19
2.6.3 Adaptive Instructional Sciences – IEEE (C/LT/AIS) P247.1)
The purpose of the Adaptive Instructional Systems (AIS) Working Group32 is to investigate the market need
for standards across a group of technologies known as the AIS. The output of the working group will be
one or more PARs identifying and generating specifications and standards that improve the interoperability
across adaptive tools and technologies.
2.7 LAUNCH SERVICE
The ability to launch learning resources across devices, platforms, operating systems, and browsers is a
fundamental requirement of the future learning ecosystem. A TLA launch service will ultimately be
designed as a plugin-based protocol that can launch activities according to different launch standards that
are applicable to different systems, platforms, or learning modalities.
2.7.1 xAPI Launch
The xAPI Launch Specification33 is a method of launching xAPI-enabled content by online learning
modules, static HTML files, offline content, or immersive serious games and simulations. It does not require
the identity of the learner, the LRS endpoint, or the session information for how the events should be
grouped. Implementation requires a minimal HTTP request and handles the creation and transmittal of xAPI
data to the LRS on behalf of the activity. While this specification is mature, it has not been widely adopted.
2.7.2 cmi5
The cmi5 34 Profile replicates many of the features and capabilities associated with SCORM. The
specification provides definitions, a launch process, a course structure, and runtime communications
between a learning management system and learning content. The cmi5 allows the setup of several global
variables including actor, xAPI endpoint and security token, and the automated communication of some
xAPI statements. The launch portion of cmi5 allows developers to avoid “hard coding” the LRS information
into the LMS. The cmi5 could be expanded to support a secure, single sign-on experience by decoupling
the data models and behaviors from the implementation details. Implementation details could be defined as
part of the cmi5 launch profile which allows it to be used for web and native applications. There are known
security concerns over the JavaScript credentials and the code is not portable without modification.
2.8 ASSESSMENT SERVICE
Assessment within the future learning ecosystem will take many different forms. The nature of assessment
in a competency-based educational program is more focused on the learner’s ability to demonstrate their
understanding of key concepts by having them apply their learned skills in different contextual situations.
The 2018 TLA Reference Implementation used a variety of traditional assessment activities that included
multiple choice tests/quizzes, Situational Judgement exercises built in Unity3D, mobile applications and
live group activities. Future implementations require a common approach for communicating competency
assertions across TLA components and systems. Competency evidence will be aggregated from multiple
communities inside and outside the organization. Performance indicators, organizational metrics, and other
systems or databases in use across the enterprise will continually be analyzed and assessed to measure the
proficiency of individuals and teams within the organization.
32 http://sites.ieee.org/sagroups-2247-1/
33 https://github.com/adlnet/xapi-launch
34 https://github.com/AICC/CMI-5_Spec_Current/blob/quartz/cmi5_spec.md#content_launch
26 The ADL Initiative – May FY19
Many of the standards and specifications referenced in this document include rubrics for measuring
performance (HPML, CASETM, Open Badges). Each has its own data structure, evaluation metrics, and
format and each needs to be considered as part of the TLA assessment service. Controlled vocabularies are
also required to ensure assessments are all speaking the right language and crediting the right competencies.
2.8.1 Question & Test Interoperability (QTI®)
The IMS QTI® specification35 enables the exchange of item and test content and results data between
authoring tools, item banks, test construction tools, learning platforms, assessment delivery systems, and
scoring/analytics engines. The data model is described using UML to facilitate binding to a wide range of
data-modelling tools and programming languages. The IMS QTI specification has been designed to support
both interoperability and innovation through the provision of well-defined extension points. These
extension points can be used to wrap specialized or proprietary data in ways that allow it to be used with
other test items. QTI 2.2 is very stable and has been built on other stable, adopted versions of QTI. QTI 2.2
adoption is strong. TLA assessment activities could leverage the data models of QTI to gain interoperability.
2.9 IDENTITY MANAGEMENT
The future learning ecosystem requires data exchange across different organizational boundaries and
inherently different IT enclaves. The 2018 TLA Reference Implementation used Keycloak™, an open
source single sign-on capability to manage individual access and permissions to all TLA components and
activities used in the 2018 TLA Test and Demonstration. User names were anonymized through the creation
of a universal user ID that each component used when communicating data (e.g., activity tracking) about
each learner. Most tools and technologies are focused on protecting Personally Identifiable Information
(PII) inside the organizational firewall; however, there are numerous use cases within the TLA where
information about a learner needs to be communicated between different organizations.
This is currently achievable at a minimal level through services like ID.me36 and Login.gov37. ID.me
provides identify management, authentication and group affiliation verification for numerous government
and business customers. Login.gov allows users to log into multiple government agencies with a single user
account. The approaches used to manage these capabilities could be relevant to the TLA. Both platforms
follow the National Institute of Standards and Technology (NIST) SP 800-63 Digital Identity Guidelines.
38
2.9.1 Privacy and Security for TLA (PS4TLA)
The PS4TLA39 research seeks to make recommendations for implementing a privacy by design model
where privacy and security strategies are an inherent component of the future learning ecosystem. Areas of
study include user data characteristics, output characteristics, data location and ownership, data sharing,
and privacy support mechanisms. This research will result in a requirements specification for tools and
technologies that manage a learner’s PII and privacy preferences, while also providing individual learners
with the knowledge required to enact their own privacy control across TLA activities, systems, and services.
35 https://www.imsglobal.org/question/index.html#version2.2
36 https://www.id.me/
37 https://login.gov/
38 https://pages.nist.gov/800-63-3/
39 https://www.youtube.com/watch?v=opmiFzqfwXo
27 The ADL Initiative – May FY19
2.10 MISCELLANEOUS
2.10.1 Learning Tools Interoperability (LTI)
The IMS LTI Specification40 prescribes a way to connect learning applications and tools with platforms
like learning management systems, portals and learning object repositories, in a secure and standard manner.
LTI is comprised of a central core and optional extensions to add optional features and functions. The LTI
core establishes a secure connection and confirms the tool’s authenticity while the extensions add features
like the exchange of assignment and grade data between an assessment tool and LMS gradebook. While
LTI is tool-oriented, the underlying data model includes elements about the learner and could lend itself
well to a TLA Learner Profile.
2.10.2 Open Pathways
Open Pathways41 has the goal to allow publishing and consumption of learning pathways across multiple
services much like Open Badges. A learning pathway is an organized set of educational goals connected by
specific digital credentials and a community’s understanding of what people have accomplished in terms
of competencies, or other measures. This model is published using standardized data formats and a common
vocabulary across different stakeholder organizations. The Open Pathways effort is in the early stage of
development and has little community adoption. However, it has potential to define a common taxonomy
for TLA activities and could inform attributes and rules that TLA learning pathways should follow.
2.10.3 Sharable Content Object Reference Model (SCORM)
SCORM defines a specific way of constructing learning content so that it can be shared across learning
management systems. SCORM is a set of existing standards and specifications that control how content is
developed, packaged, described, and delivered across systems. A Sharable Content Object (SCO) is the
most granular piece of training in a SCORM world. The Content Aggregation Model (CAM) determines
how a piece of content should be delivered in a physical sense.
At the core of SCORM packaging is a document titled “imsmanifest.xml.” This file contains every piece
of information required by the LMS to import and launch content without human intervention. The SCORM
run-time specifies how the content communicates with the LMS while the content is playing. There are two
major components to this communication. First, the content has to find the LMS. Once the content has
found it, it can communicate through a series of get and set calls and an associated vocabulary. SCORM is
entrenched in the traditional distributed learning community and is still widely used for managing online
learning. It provides an important capability for the TLA in being able to deliver online content. The
SCORM/xAPI wrapper provides additional capabilities to enable SCORM courses to report to an LRS.
2.10.4 IEEE Actionable Data Book (ADB)
ADB is a transformative blend of experiential analytics and rich media, delivered through interactive eBook
technology. ADB was created as a reference model based solely on open standards. These include the
International Digital Publishing Forum’s ePub 3 specification, HTML5, Bluetooth, xAPI, and W3C
packaging standards for interactive content. It appears work on this specification is no longer continuing.
40 https://www.imsglobal.org/activity/learning-tools-interoperability
41 https://www.concentricsky.com/articles/detail/open-pathways-connect-badges-to-what-you-care-about
28 The ADL Initiative – May FY19
2.10.5 ePub
The ePub specification 42 defines a distribution and interchange format for digital publications and
documents. The EPUB format provides a means of representing, packaging and encoding structured and
semantically enhanced Web content — including HTML, CSS, SVG and other resources — for distribution
in a single-file container. EPUB has been widely adopted as the format for digital books (eBooks), and with
new versions continues to increase the format's capabilities to better support a wider range of publication
requirements, including complex layouts, rich media, interactivity, and global typography features. The
expectation is that the EPUB format will be utilized for a broad range of content, including books,
magazines and educational, professional, and scientific publications.
This specification is relevant to different types of content the TLA can support and was used as part of the
Personalized eBook for Learning (PeBL). EPUB in general has seen major adoption, but the adoption of
version 3.1 is inconclusive at this time. Rules are specific and will promote interoperability among tools
that leverage EPUB 3.1. This specification indirectly competes with the ADB specification. However, ADB
was designed with consideration of the EPUB specification.
2.10.6 IEEE P1589 Augmented Reality Learning Experience Models (ARLEM)
The purpose of the ARLEM specification43 is to become an overarching integrated conceptual model that
describes interactions between the physical world, the user, and digital information to establish the context
for Augmented Reality (AR) applications used for learning. It will define the required data models,
modeling languages and their bindings to chosen representation formats (e.g. XML, JSON). The
specification has not yet been released as a version 1.0 specification. Therefore, a thorough review has not
been possible. A review of draft documentation found that while ARLEM attempts to formulate a
standardized taxonomy around Augmented Reality (AR), it doesn’t provide enough vocabulary to support
Virtual Reality (VR) or Mixed Reality (MR) learning environments and therefore requires extrapolation.
As this specification matures, the ADL Initiative will continue to investigate to determine how these
learning activities can integrate with the TLA.
42 https://www.w3.org/Submission/2017/SUBM-epub31-20170125/
43 http://arlem.cct.brookes.ac.uk/
29 The ADL Initiative – May FY19
3. 2018 TLA REFERENCE IMPLEMENTATION
This section describes the logical, physical, and operational implementation of the FY18 TLA
demonstration. It introduces the interface definitions that will form the core of future TLA implementations,
as well as full descriptions of the commercial items used to provide the current test implementation. The
document is organized like the architectural elements defined in a system/subsystem design document
(SSDD), albeit the requirements traceability section is tailored out. A research summary and lessons
learned, from a technology standpoint, is added as Section 5. The research effort did not maintain detailed
requirements, although high level functional requirements for the TLA implementation were collected and
defined at a series of Technical Integration and Planning (TIP) meetings held throughout the year leading
up to the 2018 TLA Test and Demonstration held at Fort Bragg in August of 2018.
3.1 LOGICAL ARCHITECTURE
The TLA Logical Architecture decomposes complexity by separating the TLA into a set of related technical
services and principles that support the operation of the system. The service layer in Figure 1 acts as the
bridge between TLA components and shared data stores. Each service exposes the stored data to an
application so that information can be transformed into other meaningful data used by other TLA
components. Each service includes control logic and user interfaces for a set of functions as depicted in
Figure 9. The data contracts between data and service layers are shown based on the nature of the data
exchanged. The behavior and functionality of each service is defined and aligned with a TLA business
function. Input-output data flows are identified and aligned with the required TLA data stores. Data models
and protocols are defined around candidate TLA standards and specifications.
Specific implementation of data interfaces is described in Section 3.3. The service layers include:
• User Management: Handles protection of Personally Identifiable Information (PII), login
credentials and identification. In the 2018 implementation this was done primarily by Keycloak44
.
Keycloak™ is an open source Identify and Access Management solution. This service will typically
be inherited from the parent organization implementing a TLA instance.
• Resource Management: For the 2018 event, the physical location and endpoints of TLA
components were managed manually via a static index. Resource Management services are already
used by the ADL Initiative stakeholders to manage dorm rooms, live training logistics, simulator
schedules, and other capabilities. This data could be used to inform the TLA.
• Competency Management: Tracks overall competency state for selected goals and makes
assertions based on evidence provided via xAPI. The Competency and Skills System (CaSS)45
performed the competency service for the 2018 event. The CaSS project includes a Learner Profile
as an internal data store that archives mastery estimates and user metadata for each learner.
• Activity Management: Manages the thread of assigning content to satisfy an identified goal.
Activity management in 2018 included a recommender service that prioritized recommendations
based on inferences made from the learners’ performance. The FLUENT system performed the
activity management functions in the 2018 TLA Reference Implementation. FLUENT also acted
as the primary student interface for the TLA.
44 https://www.keycloak.org/index.html
45 www.cassproject.org
30 The ADL Initiative – May FY19
• Content Management: Manages the registration of content. In the 2018 Implementation, this was
a static file called the Activity Index, which included metadata for all content in the 2018 content
database. It was managed in an offline manual process, and physically co-located with the FLUENT
instance.
• Decision Management: Manages the presentation and analysis of aggregate data for making
learner, curricular, facility and other decisions. In the 2018 TLA Test and Demonstration, this was
an offline process using commercial tools built into the LRS, or with exports in Microsoft Excel.
Within the logical services architecture, learning content is technically not part of the TLA, but the content
was stored on TLA controlled servers for the purposes of the 2018 TLA Test and Demonstration, so it is
included in this description. Content viewers establish data contracts with the TLA ecosystem by acting as
“boundary objects.” The Content viewers and a Modified version of Moodle™ thus act as the Learning
Record Providers (LRP), generating xAPI statements of user learning experiences, and managing content
launch and user ID tokens. The LRPs generate xAPI statements that are streamed to the User LRS. These
statements are the raw evidence that the CaSS uses to make assertions of competence.
These assertions are transformed into estimated levels of competency (mastery estimates) for each learner
in the system. A recommendation engine (FLUENT) uses competency information and Learner Profiles
schedule the delivery of activities through a series of prioritized recommendations. The ADL Initiative
managed services include the LRS, authentication, identity management, content discovery (resource
management), and launch services.
3.2 PHYSICAL ARCHITECTURE
The 2018 TLA Reference Implementation was installed on a set of seven virtual machines hosted by AWS.
AWS provides the backend platform hosting, virtualization, and Domain Name Service (DNS) resolution
functions. Six of the servers were procured under contract to USALearning, maintained by the ADL
Initiative; the seventh was separately procured by the vendor on AWS under the prime CaSS contract. The
specifications for each server are listed in Table 4. All seven servers used Ubuntu 16 LINUX as the
operating system (OS). Web server configuration varied by vendor, with some teams utilizing traditional
deployment and others using containerization frameworks. The Server instances communicate between
themselves using HTTP over TCP/IP internally to the Amazon campus. The application ports and protocols
used to access each service are listed in Table 5 and described in Section 3.5.
31 The ADL Initiative – May 2019
<<Service layer>>
Resource Management
<<Service layer>>
Content Management
<<Service layer>>
Activity Management
<<Service layer>> Decision
Management
<<Service layer>>
Competency
Management
<<Service layer>>
Content Presentation Session management
Content Secure URI
Inferences
Goals Resource Request
Human Performance Data
KSAAO Inventory
<<Service layer>>
User Management
assertions
registry
Content options
Content metadata
Content paradata
Resource listings
Signature req
Adaptation Micro Macro?
<<Data Layer>>
Learner Profile
<<Data Layer>>
Content Catalog
<<Data Layer>>
Learning Record Store
Session data
Assessment data
User Tokens
User data
Content paradata
• Discover available network resources
•
• Manage Competency Framework
• KSAAO DAG into Badges
• Badges tree into individual career
path
•
• Schedule Activity/Identofy Goal
• Schedule/Priortitize Activity Content
• Manage Activity Session
• initialize
• start
• pause/restart
• report
• closeout
•
• find/register content
• advertise content
•
• Student Progress
• xAPI filters
• xAPI displays
•
• Content Launch
• Videos
• Docs
• LMS/SCORM
• Activity Reporting
• Adjudicate Endoresment of
competency
•
• Single Sign On
• PPI/PII Protection/Privacy
• Credential Certs/non repudiation
•
Figure 9. Logical TLA Services Architecture – The TLA is implemented as a combination of multiple services (and microservices) and their interactions. Hence
the communications between services and their coordination is vital for a successful realization of the architecture. Business functions within the TLA are often
comprised of multiple services. Therefore, services cannot always be mapped directly to a specific TLA system or component.
32 The ADL Initiative – May 2019
Table 4. AWS Server Specifications – The 2018 TLA Reference Implementation utilized a set of 7 virtual machines
hosted by AWS. CaSS, FLUENT, Moodle™, and two instances of the Learning Locker LRS were set up to run with a
content server and another server configured to run essential TLA services such as single sign-on authentication,
launch, and discovery.
Description EC2 Type Disk Volume Type Storage IOPS Base
Throughput
Snap
Storage
VM1-LRS X-Large General Purpose SSD (gp2) 200 150 128 150
VM2-LogLRS X-Large General Purpose SSD (gp2) 200 150 128 150
VM3-
ADLServer Large General Purpose SSD (gp2) 50 100 128 0
VM4-
SoarTech X-Large General Purpose SSD (gp2) 100 150 128 150
VM5-
Content Medium General Purpose SSD (gp2) 200 150 128 50
VM6-
Moodle X-Large General Purpose SSD (gp2) 100 150 128 100
VM7-CaSS X-Large General Purpose SSD (gp2) 100 150 128 100
While services and systems could be consolidated to run on fewer servers, the simplicity of this approach
reduced risk and delineated clear boundaries for what will run on each system in the cluster. Moodle™
requires significant computational resources for each connected learner. Each LRS and the FLUENT
recommendation engine also require significant resources for storing the different data elements about
systems, learners, and content. A second LRS was added to support the logging of system components
about different ongoing activities using xAPI (e.g., data about recommendations, competency assertions,
learner inference, etc.). The six elastic IP addresses and 1000GB of data transfer per month was also added
to AWS hosting requirements.
As shown below in Figure 10, the Student and Instructor Workstations communicated with the AWS
servers over the commercial internet using the Hypertext Transfer Protocol (HTTP), accessed through a
web browser. The Google Chrome™ 64-bit browser, version 69.0.3497.100, loaded on top of Windows10
OS was used as the principal interface. The initial login page and credential management were provided by
a Keycloak™ Server. Keycloak™ is a commercial, open source, single sign on application that also
generates 28 key user tokens to protect PII by avoiding the passage of names, SSN, or other private data.
The FLUENT user interface (UI), hosted on the SOAR Technologies server, provided the main control
panel page resources. Learning content pages used a set of browser-based content viewers to interact with
learning content. A launcher application installed on the workstations, as well as the Moodle™ Server,
managed launch of the individual content viewers of the selected type. A Keycloak™ client application
installed on the workstations centrally manages logins. All content viewers except PEBL could launch in
new browser sessions on the student and instructor PC workstations. The Personal eBook Learning (PEBL)
application launched from an iPad™ personal data tablet device, using the user login to the Keycloak™
app to match user tokens to identify the correct device to launch. This overall physical architecture is
depicted in Figure 3 in the first section of this document.
33 The ADL Initiative – May FY19
WiFi
Commercial
Internet
cass.tla.cassproject.org
Launch Client
moodletla.usalearning.net adltla.usalearning.net
logtla.usalearning.net
contenttla.usalearning.net
lrstla.usalearning.net
soartla.usalearning.net
Apache
Keycloak Client
• Domain Name Service
• Load Balancing
• Virtualization Services
• Network Services
UBUNTU
Windows/Browser
Keycloak Server
Student/Instructor
Workstations
Personal Data Device used to
Launch PEBL or PALMS
content
Associated by Keycloak User Token
Figure 10. 2018 TLA Reference Implementation Physical Architecture – The TLA’s physical architecture defines
the system elements that can perform the functions of the logical architecture. The 2018 TLA Reference
Implementation is built from systems, system elements, and all necessary physical interfaces between these elements.
3.3 SYSTEM COMPONENTS
3.3.1 Activity Registry
An Activity Registry is an approach to capturing, connecting and sharing data about learning resources
available to an organization. Key features include the ability to generate and manage metadata, taxonomies
and ontologies, the alignment of content with competencies, paradata, semantic search services, and
machine-actionable metadata. In the 2018 TLA Reference Implementation, the Activity Registry combined
cataloging information, usage, assertions, and analytical data into a single, sharable metadata repository.
The Activity Registry has a trusted relationship with connected learning resources and other TLA services
like launch and discovery. This growing collection of data about learning resources can be consumed by
any TLA component. The Activity Registry was deployed as part of the FLUENT recommender; however,
future versions will be implemented as a stand-alone capability.
A JSON metadata file was generated for each activity. Metadata was created following the LRMI
specification and made specific use of the extension for educational alignments. Beyond the general
metadata overhead (e.g., publisher, name, location), a very limited set of differentiators were implemented
for 2018. The FLUENT recommender required 5 different metadata types for making its recommendations.
Metadata was a semi-automated process using a spreadsheet to manually enter all metadata fields. A script
formatted all fields into LRMI formatted JSON file that was uploaded to the Activity Index.
34 The ADL Initiative – May FY19
3.3.2 xAPI Activity Streams
Each LRP is uniquely situated to produce learning records that connect a user to learning experiences
within an activity. The LRP is responsible for the formatting and population of a learning record that meets
xAPI requirements. These learning records can then be transmitted to the LRS. Within the 2018 TLA
Reference Implementation, the xAPI was used exclusively to track individual learner experiences across
different learning activities. System logs were also formatted into xAPI and sent to a separate LRS
3.3.3 Learner Record Store
The xAPI-enabled learning activities generate statements, or records of e-learning which include a basic
“triple” structure consisting of actor, verb, and object. These statements are transmitted over HTTP or
HTTPS to an LRS. The xAPI requires a central storage capability called a Learner Record Store (LRS). An
LRS is a data store that serves as a repository for learning records collected from connected systems where
learning activities are conducted. It is an essential component in the process flow for using the xAPI. The
2018 TLA Reference Implementation used the open source Learning Locker LRS46. Its main function is to
store and retrieve the data that's generated from xAPI statements such that all other TLA components may
access it without being dependent on direct communication with each other. With the statement structure
of the LRS records, native Learning Locker dashboards were generated using a number of different actor,
verb, and object combinations.
3.3.4 Competency Management System
A competency management system manages evidence of an individual skills, knowledge, abilities,
attributes, experiences, personality traits and motivators to predict their value towards effective
performance in a job. Competencies might include technical skills, business skills, leadership skills, people
skills, ethics, professionalism, or any number of topics related to a job. Within the context of the TLA, the
Activity Stream(s) stored in the LRS provide the evidence upon which competency assertions are made.
Key to establishing assertions of competency from an Activity Stream, is the reconciliation between the
Actor in a statement (stored in an LRS) and a persistent Learner Profile. As shown in Figure 11, the 2018
TLA Reference Implementation utilized the open source CaSS47 to manage this evidence and infer
competency against the competency framework.
For the 2018 TLA Reference Implementation, CaSS pulled xAPI statements from the LRS and checked the
verb to determine whether the statement claims any evidence of competency. For statements that infer
competency, the ObjectID from the xAPI statement was compared to the Activity Registry to determine
which competencies a statement is providing evidence for. The primary output from CaSS is a series of
mastery estimates for an individual over time. This data is managed within a service layer that provides a
common, semantic, and syntactically accessible network service for other systems to leverage. CaSS also
enables assertions about an individual's competencies to be collected, processed, and incorporated into a
machine-readable Learner Profile. Within the 2018 TLA Reference Implementation, AMQP was used to
push this information to other TLA components and other TLA components were able to write to the
Learner Profile by making assertions through CaSS.
46 https://www.ht2labs.com/learning-locker/
47 www.cassproject.org
35 The ADL Initiative – May FY19
Figure 11. Competency and Skills System – The 2018 TLA Reference Implementation used CaSS to create, manage,
and share information about competencies with other TLA components.
3.3.5 Learner Profile
In the 2018 TLA Reference Implementation, the Learner Profile was developed as part of the CaSS project.
It primarily contains information about competencies, competency assertions, and credentials. Current and
future ADL Initiative research is attempting to understand how different learner characteristics influence
the intelligent delivery of content to optimize how a person learns. For example, the 2018 TLA
test/demonstration populated the Learner Profile with additional learner inference data from FLUENT that
helped the system rank and prioritize recommendations tailored to each learner. Future implementations
will continue to explore how a Learner Profile might influence other TLA components.
3.3.6 Recommendation Engine
The open source FLUENT recommendation engine uses information about competencies, content, and
learners to recommend different learning activities for a user to follow at a given point in time. For the 2018
demonstration, the competencies were divided into knowledge domains which were linear in nature and illdefined tasks which required a higher level of understanding to apply the learned knowledge and skills.
FLUENT also made use of a small subset of learner information that primarily included mastery estimates
for each competency.
To make data-driven recommendations, FLUENT processes information about a learner including their
history and context for learning. Learner inference data was derived from the Learner Profile, the LRS, and
paradata from individual learners. FLUENT’s Collaborative Filtering algorithms look at a user’s past
behavior and aligns that with the behavior of other users to make a recommendation. Any information
derived about a learner was written back to the CaSS Learner Profile using xAPI assertions. FLUENT’s
content-based filtering algorithms utilize a series of discrete characteristics about an activity, as described
in its metadata, to recommend additional items with similar properties.
36 The ADL Initiative – May FY19
3.3.7 Single Sign On
Single sign-on (SSO) is a property of the TLA’s access control of multiple related, yet independent,
software systems. With this property, a user logs in with a single ID and password to gain access to all TLA
connected systems. While this is not a formal component of the TLA, the ADL Initiative expects that any
TLA component will need to operate within an ADL Initiative stakeholder SSO environment. Within the
2018 TLA Reference Implementation, Keycloak™ was used for this purpose. This capability has been
integrated to protect Personally Identifiable Information (PII) and to provide information security for
learners, content, and organizations.
3.4 CONCEPT OF EXECUTION
The basic user thread of execution in the TLA demonstration system starts with the Keycloak™ LOGIN
and proceeds otherwise through the FLUENT UI. Learners select goals, and then select from prioritized or
un-prioritized content work towards that goal, launch the content, and closeout their performance. The
experience closeouts, and any related assessments, are recorded as xAPI statements in the LRS. The goal –
activity – content thread continues in a loop until the goals are all satisfied.
Two micro-services were used to communicate data between the different systems. A JSON/REST API
was used to provide synchronous services between public facing components such as the recommender UI
and the LRS. A RabbitMQ-based implementation of AMQP was used to provide immediate data transfer
between server-side components, such as the mastery estimates coming from CaSS which needed to be
communicated to FLUENT. Within the 2018TLA Reference Implementation, all data exchange APIs were
required to support CRUD (Create, Read, Update, Delete) Operations with error messaging. System logs
were embedded into xAPI statements and sent to a dedicated logging LRS, providing additional insight into
the behaviors of each component.
The 2018 Test and Demonstration showed that a pub/sub architecture is necessary for achieving the desired
performance for the TLA. The quantity of messages and data that will be present when deploying at scale
is very large. While AMQP demonstrated the effect push can have on end users at a small scale, it may not
be adequate from a scalability standpoint. FLUENT reported latency issues with the use of linked data and
identified it as a performance bottleneck when attempting to achieve real-time data sharing performance.
Retrieving data involves following links many layers deep to retrieve sub-parts of the necessary data. If the
linked data structure is complex, the number of interface calls required to retrieve the whole data structure
can number in the hundreds or thousands. Other work-arounds need to be investigated.
The basic flow for the Run Time Mode is shown in the Unified Modeling Language (UML) sequence
diagram of Figure 12. There are administrative and diagnostic modes using the native User Interfaces of
Moodle™, CaSS, Keycloak™, and Learning Locker which are not shown in the diagram as part of the
overall flow of execution. These interfaces were not intended to be accessed by learners. Administrative
and diagnostic modes were used to:
• Modify the content discovery and metadata in the activity index.
• Unpin live-locked content by forcing a state change in Moodle™.
• Unpin live-locked competencies by forcing a state change in CaSS.
• Forcing global sign outs if Keycloak™ had an error.
Orange boxes represent system components in the data layer, blue boxes show components in the service
layer, and green boxes represent the boundary object of the ecosystem, as depicted in Figure 9.
37 The ADL Initiative – May 2019
CASS Fluent LRP User LRS Activity
Index Launcher
System
LRS Fluent UI Keycloak
loop
Loop unitl Logout
prioritize content
verify resources
Launch
initialize content
user perform
adjudicate
loop
until launch sesssion closed
update xAPI record
update xAPI record
update XAPI record to
closeout activity
request
content(filter) send
content ref (filter)
update performance Endorsement
Update Competency Network
update paradata
select Goal
update competency state
Goal Select
update inference
log recommendation
update assertions
Set launch
SSO Login
SSO Login
alt
All Recommendations
graphical skills tree
populate skill tree
collapse/expand skills
Focused Recommendations
set goal
display content & state
Display Content & state
select content
Select Content
request login
update xAPI
LOGIN
LOGOUT
All Activities
request content (all)
send content (all)
select content
loop
Until recommender pane options selected
request
content(filter)
send
content ref (filter)
Display Content & state
update paradata
update paradata
update xAPI record
update xAPI record
update xAPI record
Figure 12. TLA Reference Implementation Concept of Execution (COE) – This describes the overall learner experience from their first-time logging into the
2018 TLA Test and Demonstration. The learner signs into the TLA via Keycloak™ and is then directed to the FLUENT User Interface where they remain in an
infinite loop of receiving content recommendations until all competencies are completed.
38 The ADL Initiative – May FY19
3.5 INTERFACE DESIGN
The TLA is comprised of a collection of open specifications that define the specific message structures
supported by the application programming interfaces used to expose TLA data, services, and systems. The
next sections describe the interfaces used in the 2018 TLA Reference Implementation. These interfaces
represent the operationalization of service contracts defined in the logical architecture.
3.5.1 Interface Identification and Diagrams
Figure 13 shows the specific data interfaces between the low-level components of the System configuration
items in the TLA Reference Implementation (which are shown along the top as swim lanes in the diagram.
The FLUENT User Interface is the principle interface for the learner. It has the following functions:
• FLUENT launches via HTTP when the user logs into Keycloak™ successfully.
• FLUENT connects via REST calls to the recommender service within FLUENT to obtain the
current state of competencies and set user selected goals,
• FLUENT sends launch requests to the launch service via AMQP, connects to Keycloak™ via
AMQP, and launches the content viewers as second browser sessions.
• FLUENT connects to the CaSS using HTTP, shares assertions using AMQP and generates xAPI
statements to the system LRS by sending JSON over AMQP.
The TLA launch services were embedded into FLUENT which provided the following capabilities:
• The user can launch content for static content viewer or xAPI video viewer via HTTP connected to
launch client. This browser sends xAPI messages to the user LRS via JSON over AMQP.
• SCORM and assessment content launch in Moodle™, which loads in its own browser session with
the launch function natively communicating through HTTP and generating xAPI statements to the
user LRS in AMQP over HTTP.
• PEBL and PALMS content launches in their own browser session and generate xAPI to the user
LRS natively using AMQP over HTTP, they host the launch function natively.
3.5.2 Advanced Message Queuing Protocol (AMQP)
AMQP is an open standard application layer protocol for message-oriented middleware to enable a common
messaging service between TLA components. It provides an extensible markup language (XML) based
middleware application layer protocol for exchanging information via HTTP requests.
3.5.3 Representational State Transition (RESTful)
REST is an architectural paradigm that defines a set of constraints to be used for creating web services,
which maintain data and control logic on the edges of communication and uses a series of HTTP based
requests to a Universal Resource Locator (URL) or specified address for a network resource
3.5.4 Experience Application Program Interface (xAPI)
The xAPI is a specification currently going through the IEEE-LTSC standards development process
(https://www.tagxapi.org/). The main components of xAPI are the data structure called statements and the
data storage/retrieval capability called the Learning Record Store (LRS). The xAPI specification has
stringent requirements on the structure of this data and the capabilities of the LRS. Statements are data
triples that use an actor, a verb, and an object to describe any experience. Each statement also includes
timestamps and unique, resolvable identifiers. The transport is HTTP/HTTPS with JSON payloads.
39 The ADL Initiative – May FY19
Figure 13. TLA Reference Implementation Components and Interfaces – In its current form, the TLA follows a modular open systems architecture that
leverages technical standards to support a loosely coupled and highly cohesive system structure. This allows TLA components to be added, modified, replaced,
removed or supported by different vendors throughout their lifecycle. TLA components use carefully defined execution boundaries layered onto an integrated
framework of shared applications, services, and data. TLA standards are published documents that establish key interface specifications, communication protocols,
and data structures designed to facilitate interoperability between TLA components, enable compatibility with a wide range of learning content, and maximize the
efficiency for how people learn throughout their career from “hire to retire.” These standards form the fundamental building blocks for life-long learning by
establishing consistent protocols that can be universally understood and adopted by any TLA component to enable data exchange about learners, activities, and
experiences.
40 The ADL Initiative – May 2019
3.6 COMPUTATIONAL RESOURCES
The ADL Initiative performed a load test of the server configuration listed in Table 5 to determine the top
end on the performance of the current TLA Reference Implementation. This test was run over the course
of 90 minutes and simulated 1,000 instances of a single user signing in, viewing a course, opening and
closing a SCORM course, taking a quiz, viewing all users enrolled in a course, and then signing out. The
system failed at 10,000 users with Moodle™ unable to handle the load, failing on CPU overload. Memory
is not provided as it did not represent a significant resource limit in any case. The network uplink limited
component was the ADL Initiative server, likely Keycloak™ request for user ID.
Table 5. TLA Reference Implementation Application Layer Protocol Port Usage – Just as the IP address
identifies the computer on a network, the network port identifies the application or service running on the computer.
This allows TLA servers in the testing environment to run multiple applications and/or services.
HTTP
Service
IP Address / URL Port Endpoint
CaSS cass.tla.cassproject.org 80 http://cass.tla.cassproject.org/api
Moodle™ moodletla.usalearning.net 80 http://moodletla.usalearning.net
Static content
viewer
contenttla.usalearning.net 8000 http://contenttla.usalearning.net:8000
Video player contenttla.usalearning.net 8001 http://contenttla.usalearning.net:8001
User LRS lrstla.usalearning.net 8001 http://lrstla.usalearning.net:8001/data/xapi
Assessment contenttla.usalearning.net 8002 http://contenttla.usalearning.net:8002
Static content contenttla.usalearning.net 8003 http://contenttla.usalearning.net:8003
FLUENT UI soartla.usalearning.net 8004 http://soartla.usalearning.net:8004
PALMS contenttla.usalearning.net 8004 http://contenttla.usalearning.net:8004
Keycloak™ adltla.usalearning.net 8081 http://adltla.usalearning.net:8081/auth
Launch client adltla.usalearning.net 8081 http://adltla.usalearning.net:8081/applicationlauncher/client
Launch adltla.usalearning.net 8081 http://adltla.usalearning.net:8081/applicationlauncher/rest
Discovery adltla.usalearning.net 8085 http://adltla.usalearning.net:8085/endpoints
FLUENT UI
support
soartla.usalearning.net 8778 http://soartla.usalearning.net:8778
FLUENT soartla.usalearning.net 8979 http://soartla.usalearning.net:8979
Activity
Index
soartla.usalearning.net 8989 http://soartla.usalearning.net:8989/activityindex/activities
Learner
Inference
soartla.usalearning.net 8999 http://soartla.usalearning.net:8999
System LRS logtla.usalearning.net 80 http://logtla.usalearning.net/data/xapi
The point to point nature of all messaging within the 2018 TLA Reference Implementation was also a
limiting factor in our ability to aggregate real-time data from different systems into a TLA-driven
dashboard. The only analytics available within the 2018 implementation were those that were resident
within the Learning Locker LRS. This limited the analytics to xAPI statements which did drive interesting
statistics about content usage. However, the 2019 migration to a data streaming architecture enables a data
strategy that utilizes data from multiple sources to drive analytics and visualizations.
41 The ADL Initiative – May FY19
3.7 LESSONS LEARNED AND FUTURE RESEARCH
The use of the system LRS to record assertions and inferences generated almost 75% of the xAPI
statements. This is a performance bottleneck for the system as it scales. Since the assertions and inferences
are developed continuously, this represents system data with a relatively high Quality of Service (QoS)
requirement. This, along with several other concerns drives a desire to move to a streaming architecture for
FY19, to ensure scalability and QoS.
The FLUENT application basically served as a singleton for the entire TLA 2018 Reference
Implementation because it provides the primary user interface and manages the state space of the execution.
The primary interface for the TLA will likely migrate to a common TLA portal. Moreover, a true SOA
paradigm should be stateless between service boundaries. Moreover, the FLUENT application was single
threaded and caused some performance delays during test and execution. Moving the recommender service
to the system edge and improving its performance are goals for FY19 follow on work.
The CaSS calculations of convergence on competency is very sensitive to the weights assigned for the
contribution of each content type for making assertions. These weights were determined without scientific
rigor, and the FY19 architecture must move them to the edge to allow for human in the loop intervention.
Moreover, some students ended up reaching an asymptote below the arbitrary 80% threshold set for
competence and it was impossible to push the system farther forward.
The FLUENT recommender relied largely on the metadata assigned in twelve categories to select
recommendations. It did not address learning path recommendations, as goals were selected manually by
users. The FLUENT UI presentation of the CaSS maintained competency tree did not provide enough
information for users to optimize their own learning path. This automation “surprise” should be addressed
by the FY19 effort.
Keycloak™ was not implemented correctly, and while it was only chosen as one solution among many
possible for SSO, the interface between the TLA services and the backend services used to protect PII and
otherwise maintain information assurance should be addressed to facilitate future transition efforts.
The Static content viewer provides a very simple implementation for recording experiences with electronic
publications. PEBL content required a full copy of the PEBL content repository and specially rigged content
on each client. The ADL Initiative is reviewing the electronic publication technology strategy for FY19.
The Moodle™ version used was not the USALearning version, but required modifications for
interoperability with Keycloak™, xAPI, and launch client. The CODIAC SCORM course did not allow
fast forwarding to the appropriate point for the specific goal pursued and required the students to complete
the entire course. The LMS migration and integration plan must be reviewed for FY19.
42 The ADL Initiative – May FY19
4. 2018 TLA TEST AND DEMONSTRATION
The ADL Initiative worked with the Institute for Defense Analysis (IDA) and the Special Warfare
Education Group (Airborne) – SWEG(A), to plan logistics, scope, scale, and domain for the 2018 TLA Test
and Demonstration. IDA designed the experiment using SWEG(A) students and the ADL Initiative
integrated TLA systems and developed the course and its content. The ADL Initiative staff was augmented
by vendors under contract to work on various TLA-related systems.
During the initial meeting with SWEG (A) representatives, a request was made to move away from the
cyber and IT training used in the 2017 demonstration. They requested instruction that was more relevant to
their mission. In order of preference, they preferred small unit training, 18D medical training, or 18E
communications / radio training. The USJFCOM Combat Observation and Decision-Making in Irregular
and Ambiguous Conflicts (CODIAC) course was ultimately approved as the foundation for the 2018 TLA
Test and Demonstration. To assess the TLA, several research questions were posed. The 2018 TLA Test
and Demonstration had the following research objectives:
• Assess the TLA functional requirements, does the TLA 2018 Reference Implementation
demonstrate the intended functionality described below categorically?
• Assess the conformance to candidate specifications to allow TLA components and activities to
communicate with each other.
• Assess successfully indexing, discovering (service discovery within the architecture), and
launching a wide variety of learning activities across devices, operating systems, and modalities.
• Assess the ability of the TLA to track learner competence within a competency framework.
• Assess the ability to produce user learning activity recommendations based on the current state of
learner competence.
• Evaluate the metadata strategies used within the 2018 TLA Reference Implementation.
• Assess the potential of the TLA for supporting cross-activity dashboards for instructors, learners,
administrators, and others.
• Assess the TLA reference implementation for system-ilities (reliability, availability,
maintainability, scalability, interoperability, portability, or others).
• Assess usability and user experience when interacting with the 2018 TLA Reference
Implementation.
• Evaluate usability of the 2018 TLA Reference Implementation of the TLA by populations other
than end users.
• Evaluate the technical viability of implementation and the diffusion potential compared to previous
versions.
• Document system usage to determine whether the system functions as expected.
• Identify costs and potential benefits to different communities.
• Evaluate learning potential and the ability to enhance the learning experience.
43 The ADL Initiative – May FY19
A mixed-methods approach incorporating Delphi techniques, surveys, focus groups and a quasi-experiment
was used for assessing the TLA. For assessing technical decisions, documentation quality, and diffusion
potential, Delphi methods were used with panelists reacting to the API specifications, use-cases, and
implementation aids. To assess the APIs, a testing implementation of the TLA was deployed and the
functional and non-functional requirements tested with human subjects. To assess the functionality and
non-functional requirements of the 2018 TLA Test and Demonstration system, the unit of analysis for
examining behavior and results can no longer reside with human subjects but moves to the behavior of the
components and APIs comprising the reference implementation and the results as measured by the integrity
of the system’s behavior compared to requirements. These levels will be assessed through the usage of 2018
TLA Test and Demonstration system level data to determine descriptively the behavior of the testing
implementation, its potential, and experiences and levels of effort required to develop for and integrate with
the test and demonstration system using the TLA architecture.
4.1 TECHNICAL PLANNING
Technical planning and integration meetings were held throughout the year in preparation for the overall
2018 TLA Test and Demonstration event. The first technical interchange meeting was held over 2 days in
early March. The purpose was to bring TLA component developers, activity providers, researchers, and
end-users together to plan logistics, scope, scale, and domain for the TLA demonstration at Fort Bragg in
August 2018. The attendance of these events by representatives from the Special Warfare Education Group
(Airborne) – SWEG(A), at Fort Bragg provided the opportunity for dialogue about IT requirements, target
audience, instructional content, and other operating constraints. Other attendees included the ADL Initiative
SETA staff, government personnel, the Institute for Defense Analysis (IDA), and vendors that are working
on different elements of the TLA and the 2018 demonstration at Fort Bragg.
As shown in Table 6, numerous planning meetings were held. The first meeting focused on providing
background information to build a common understanding of all relevant TLA components, activities, and
the CODIAC Program of Instruction. Attendees were divided into three working groups based on their
expertise. Risks were identified within each working group. Mitigation strategies were discussed, progress
was updated, and a schedule of activities and dependencies was created and agreed upon by all participants
within each working group. The three working groups included:
1. CODIAC Content Demonstration
• Decompose the 200-hour CODIAC course into a smaller subset that can be used to support
the 2018 TLA Test and Demonstration.
• Create a related competency framework.
• Align/augment existing course content.
• Decompose and realign existing CODIAC source materials, identify new related content,
design new activities.
2. 2018 TLA Technical Integration
• Technical development and integration of TLA components and services.
• Protection of PII and IA conformance.
3. CODIAC Content Instrumentation (This working group was eventually absorbed by the Content
Demonstration working group)
• Define methods for instrumenting CODIAC learning activities for optimized reporting,
analytics, and discovery by the TLA recommendation engine (FLUENT).
44 The ADL Initiative – May FY19
To help integrate the products from each working group, regular meetings provided the project team with
an understanding of the technical underpinnings developed in each working group. A collaborative
SharePoint site was established to help the team maintain a clear understanding of critical dates, milestones,
objectives, and next steps for successfully executing the 2018 TLA Test and Demonstration in August 2018.
A pilot test was scheduled for late July where a substantially completed walk-through of the course was
performed. Numerous issues arose, and a second pilot test was more successful in early August. The ADL
Initiative and IDA worked with SWEG(A) to handle all logistics including IRB approvals, IT setup and
deployment, participant management and administrative tasks.
Table 6. Key Planning Events - Numerous in-person team meetings were held throughout the year. Working groups
met weekly.
Month Key Planning Event Accomplishments
February Initial Planning Meeting The ADL Initiative and SWEG(A) staff meet at Fort Bragg to discuss
all aspects of the 2018 TLA Test and Demonstration
March 6-7 Technical Integration and
Planning Meeting #1
In person planning meeting focused on engineering efforts for the
2018 TLA Reference Implementation, ISD efforts for the
curriculum/course development, and implementation guidance for
associated activities
May 7-8 Technical Integration and
Planning Meeting #2
In person planning meeting focused on engineering efforts for the
2018 TLA Reference Implementation, ISD efforts for the
curriculum/course development, and implementation guidance for
associated activities
July 21-24 Pilot Test In person testing of the 2018 TLA Reference Implementation with
representative sample of learning resources connected and
substantially complete
August 13-17 TLA Test and
Demonstration
One-week course provided to SWEG(A) students. Pretest and
surveys completed by students prior to the course. A post-test,
surveys, and focus groups were completed by students on the last
day of the course.
4.2 TLA CONTENT DEMONSTRATION
A key emphasis was placed upon the content demonstration aspects of this event in addition to the
instructional relevance of the 2018 TLA Test and Demonstration event. From this perspective, instructional
content had to be presented in a logical and intuitive way that maximized our ability collect, analyze and
display learning data to the relevant target audience (student, instructor, administrator). By instrumenting a
wide array of related content, the demonstration was expected to generate a large amount of inter-connected
human performance data (based on individual performance) that can be used to demonstrate the potential
of the TLA for observing learner progress across learning modalities.
45 The ADL Initiative – May FY19
The TLA is content agnostic; it should be able to
equally support any-and-all subject areas. However, to
enable empirical testing of the reference
implementation, the Combat Profiling domain as taught
in the CODIAC course shown in Figure 14 was
selected. Major work efforts included the
decomposition of an existing course into approximately
14 hours of instruction, building a related competency
framework, establishing levels of mastery, and the
alignment of learning resources with specific
competency objects. The intention of the working group
was to expand the aperture for the types of content the
TLA can utilize by going beyond traditional DL content
and making use of serious games and simulations, live
exercises, eBooks, and other adaptive content that
participants may encounter in their lifetime.
4.2.1 CODIAC Course Description
The CODIAC Program of Instruction (POI) includes systematic training designed to improve cognitive
skills, showing personnel how to read the human terrain, establish a baseline, detect an anomaly, and make
decisions “left of bang.” The CODIAC course is presented as a 200-hour course that includes nine modules
with the last module consisting primarily of scenario-based exercises. Of the eight remaining modules, each
has explicitly defined Enabling and Terminal Learning Objectives (ELOs/TLOs). Multiple courses of action
were discussed by the working group about how to structure the course for the 2018 TLA Test and
Demonstration. Initial thoughts were to find content that skimmed across all CODIAC modules to lightly
provide an understanding of the entire CODIAC POI. However, after further reviews and discussion, the
decision was made to provide a more robust curriculum around the entire course.
The 2018 TLA Test and Demonstration focus on delivering instructional content and learning activities that
support a competency-based approach to delivering the CODIAC program of instruction. This course has
well-defined learning objectives, instructional activities, and available learning content including structured
exercises, activities, and assessments. To successfully learn the required content contained within this
module, numerous learning objectives from the different CODIAC modules were analyzed. Supporting
knowledge, skills, and abilities were defined, cataloged, and decomposed into their own competency
network. Table 7 on the following page shows the how the ELOs and TLOs were structured for the original
course. These were augmented with ELOs and TLOs from other related courses. The modified CODIAC
course used for the 2018 demonstration was supported by other educational resources from across the
services including the following:
• CODIAC POI, including several hours of edited video – permission received from JSJ7
• Combat Hunter eLearning course – permission received from USMC
• PercepTS Small Unit Leader Kit – Permission received from U.S. Navy
Figure 14. CODIAC – Inspired by the US Marine
Corps’ Combat Hunter program, CODIAC was
used as a foundation for the 2018 TLA Test and
Demonstration.
46 The ADL Initiative – May FY19
Table 7. CODIAC ELOs and TLOs. These ELOs and TLOs derived from the existing course structure presented a
guide for identifying, reviewing, and aligning content with an overall CODIAC Competency Framework.
Module Name Description
TLO1 Left of Bang This module is about positioning to out-fight the enemy, specifically out-think them too. At the
end of this module, learners will explain the importance of operational tempo, of making sense
of the situation faster than our opponents to anticipate their actions and be able to act “left-ofbang.”
ELO1.1 Define Consequences Define consequences of being reactive versus proactive on the battlefield
ELO1.2 Incident Timeline Explain incident timeline and Left-of-Bang.
ELO1.3 Benefits Describe benefits.
ELO1.4 Response Describe when and how to respond Left-of-Bang
TLO2 The OODA Loop At the conclusion of this unit, trainees will be able to explain the cognitive processes associated
with the OODA-Loop, including how these processes are impaired by stress and how decisionmaking limitations can be mitigated, such as employing the five combat multipliers.
ELO2.1 Cognitive Processes Explain cognitive processes associated with the OODA loop.
ELO2.2 Define Steps Define steps in the OODA loop.
ELO2.3 Decision-making Given various scenarios, explain the process for OODA loop decision-making.
ELO2.4 Behavioral Cues Effect of behavioral cues on OODA loop and decision-making
TLO3 Baseline + Anomaly =
Decision
This module introduces the concept of baseline and anomaly, and how together they require a
decision. Upon completion of this module, learners will be able to describe baseline, an anomaly,
and how anomalies to the baseline require a decision.
ELO3.1 Describe Baseline Describe the concept of a baseline.
ELO3.2 Describe Anomaly Describe the concept of an anomaly.
ELO3.3 Explain Decision-making Explain why a decision is required when there is an anomaly
ELO3.4 Decision-making Process Given various scenarios which include anomalies, explain the process for decision-making.
TLO4 Reading the Human Terrain This unit introduces techniques for interpreting human behavior cues. At the conclusion of this
module, learners will be able to describe the six domains of profiling in detail and will be able to
demonstrate the six domains’ use.
ELO4.1 Biometrics Describe biometrics; list several biometric cues and their interpretations. There are 11 specific
biometric cues discussed. We may want to decompose further.
ELO4.1.1 Explain Biometrics Explain the biometrics domain.
ELO4.1.2 Interpret Biometric Cues Given specific biometric cues, explain what they might mean.
ELO4.1.3 Apply Biometrics Describe how to apply biometrics in the battlespace.
ELO4.2 Kinesics Describe Kinesics; list several kinesic cues and their interpretations. There are 8 specific kinesic
cues discussed; we may want to decompose further.
ELO4.2.1 Explain Kinesics Explain the kinesics domain.
ELO4.2.2 Interpret Kinesic Cues Given specific kinesic cues, explain what they might mean.
ELO4.2.3 Apply Kinesics Describe how to apply kinesics in the battlespace.
ELO4.3 Proxemics Describe proxemics; list several proxemic cues and their interpretations. There are 11 specific
proxemic cues (4 groups) discussed. We may want to decompose further.
ELO4.3.1 Explain Proxemics Explain the proxemics domain.
ELO4.3.2 Interpret Proxemic Cues Given specific proxemic cues, explain what they might mean.
ELO4.3.3 Apply proxemics Describe how to apply proxemics in the battlespace.
ELO4.4 Geographics Describe geographics; list several geographics cues and their interpretations. There are 5 specific
geographics cues discussed. We may want to decompose further.
ELO4.4.1 Explain Geographics Explain the geographics domain.
ELO4.4.2 Interpret Geographics Cues Given specific geographic cues, explain what they might mean.
ELO4.4.3 Apply Geographics Describe how to apply geographics in the battlespace.
ELO4.5 Atmospherics Describe atmospherics; list several atmospheric cues and their interpretations. There are 5
specific atmospheric cues (4 groups) discussed. We may want to decompose further.
ELO4.5.1 Explain Atmospherics Explain the atmospherics domain.
ELO4.5.2 Interpret Atmospherics Given specific atmospherics cues, explain what they might mean.
ELO4.5.3 Apply Atmospherics Describe how to apply atmospherics in the battlespace.
ELO4.6 Heuristics Discuss heuristics and explain how heuristic matches are made.
ELO4.6.1 Explain Heuristics Explain the heuristics domain.
ELO4.6.2 Explain Dangers of Heuristics Explain the dangers of the heuristics domain.
ELO4.7 Icons, Colors, and Symbols List and interpret the relevant icons, colors and symbols in one’s own area of operation. There
are 4 specific image-types discussed; with colors having 7 different significances. We may want
to decompose further.
ELO4.7.1 Explain Iconography and
Symbolism
Explain the importance of iconography and symbolism.
ELO4.7.2 Interpret Icons and Symbols Give examples of specific icons, color and symbols, and their meanings.
47 The ADL Initiative – May FY19
Figure 15. CODIAC Competency Framework – The CODIAC course structure provided the foundational ELOs
and TLOs that informed the creation of a related Competency Framework. This is an example Competency Framework
created around the Combat Tracking TLO.
4.2.2 CODIAC Competency Framework
As shown Figure 15, the competency framework developed to support the 2018 TLA Test and
Demonstration was largely defined by each competency’s many-to-many relationship with other
competency objects. This is complicated by the need to weight the inferred relationship between
competencies which are different depending on the context (e.g., job, role, and periodicity) of how they are
applied. The discussion highlighted some of the inherent challenges in migrating to a competency-based
education system.
The CODIAC ELOs and TLOs were written succinctly so they can be measured and assessed; however, it
was determined that the simple mapping of ELOs and TLOs to competencies did not adequately reflect the
entirety or potential of what a competency model should include in its definition. Competencies and
learning objectives are two related educational terms that can create confusion. Competencies define the
applied skills and knowledge that enable people to successfully perform their work while learning
objectives are specific to a course of instruction. Learning objectives describe what the learner should be
able to achieve at the end of a learning period. There may be more than one measurable learning objective
defined for a given competency. While competencies and learning objectives can be written similarly, a
competency is obtained or developed during the process of learning and is intended to describe the desired
knowledge, skills, and behaviors we want students to walk away with after taking the course.
Due to time constraints, three cross-cutting competency frameworks comprised of 234 interrelated
competency objects were created to illustrate how different competencies can be inferred from different
assessment activities. Three clusters of competencies were created to include Combat Awareness, Combat
Profiling, and Combat Tracking. As shown in Figure 16, CaSS was used to encode the frameworks. As
assertions are made from existing assessments, they are applied to the relevant competencies included in
these three frameworks via a unique identifier for each competency. For example, as a student evaluates an
image, the specific act of analyzing the image might demonstrate proficiency with the proxemics ELO that
traces back to both Combat Awareness and Combat Profiling.
48 The ADL Initiative – May FY19
Figure 16. Competency Framework – The CaSS Competency Framework Editor was used to encode the CODIAC
competency frameworks used within the 2018 TLA Test and Demonstration. CaSS was used to define the basic
elements of each competency and its relationships to other competencies. Each competency object has a unique
numerical identifier that was referenced using the LRMI alignment object.
4.2.3 CODIAC Content Discovery and Alignment
One goal of the 2018 TLA Test and Demonstration was to go beyond traditional DL content to demonstrate
the breadth of learning activities that can be connected into the TLA. As shown in Table 8, the 2018 TLA
Learning Record Providers span the range of learning activities from traditional eLearning content (e.g.,
SCORM courses) to emerging technologies like serious games and mobile learning applications.
After encoding the Competency Frameworks, the content obtained from CODIAC, Combat Hunter, and
PercepTS was reviewed, cataloged and aligned to each competency included in the framework. An initial
assessment revealed existing content that included video-based lectures, reading assignments, instructor
activities/handbooks, assessments/quizzes, interactive web-based training lessons, and written, scenariobased exercises. A gap analysis was performed to determine how many learning activities were available
to support each competency object. It was discovered that most of the available DL content and assessment
activities were appropriate to support knowledge and comprehension, but were not adequate for
demonstrating the application, analysis, or synthesis of the required knowledge, skills and abilities
associated with many of the competencies. These were necessary to support the novice, intermediate, and
advanced levels of mastery within each competency.
Further, to differentiate between recommendations, numerous learning activities and assessment activities
were required for each identified competency in the framework. A search was performed to identify
additional formal and informal learning activities including non-digital content (e.g., Combat Hunter card
game), serious gaming applications, and scenario-based live exercises. Other courses were also identified
from the Army Distance Learning Program (TADLP) and Joint Knowledge Online (JKO). These courses
ultimately were not used; however, the ADL Initiative staff did create six serious game vignettes using the
Unity3D game engine. Numerous reference materials including articles, handbooks, lessons learned, and
other materials were also collected and mapped to each competency.
49 The ADL Initiative – May FY19
Table 8. 2018 TLA Learning Record Providers – The 2018 learning activities included many different types of
learning content that were instrumented with xAPI statements that are then transmitted to an LRS.
To measure the transfer of learning, a pre-test and post-test were created. Each test consisted of multiplechoice questions, matching exercise, true/false questions, and free-response short answer questions. Other
assessment activities were integrated into the various activity providers, and in some cases, a single
assessment was aligned with multiple competencies. A minimum of six assessment activities were desired
for each competency with at least two of those assessment activities capable of evaluating the higher orders
of Bloom’s taxonomy. Situational Judgement Trainers and concept maps were created to allow learners to
demonstrate their understanding of different competencies. Live, scenario-driven exercises were performed
with an instructor as a capstone event to receive a badge for each of the three competency frameworks
(Combat Awareness, Combat Profiling, Combat Tracking).
4.2.4 Instrumenting CODIAC Content
CODIAC learning activities were initially aligned to
competencies using an Excel spreadsheet. Decisions had to be
made about which TLA Activity Providers will deliver which
content, where different pieces of content will reside, and how
each piece of content will be launched, tracked, and managed over
the course of the one-week test and demonstration.
A primary focus of the 2018 TLA Test and Demonstration was to
demonstrate the instrumentation, collection, aggregation,
TLA Learning
Record Provider
(LRP)
Short Description Related
Standards
Live Exercises Instructor-led exercises and group activities were used as Capstone
assessments for receiving a badge and to assess for other higherlevel competencies.
xAPI
Moodle™ LMS MarineNet Combat Hunter e-Learning Course: Three SCORM
lessons that consisted of a single large Flash® file that encapsulated
all interactions and behaviors. These files had very limited reporting
capabilities.
SCORM, xAPI
Moodle™ LMS Multiple choice and short answer quizzes implemented throughout
the program of instruction.
xAPI
PALMs Perceptual Adaptive Learning Modules. xAPI
PEBL eBook Content in the form of an e-publication has the characteristics of a
book but is augmented by the technologies included on each device.
xAPI,
ePub3/ADB
Sero! Concept Maps Open Source Concept Map toolkit used to create capstone exercises
to demonstrate understanding about key concepts. Used for
assessment.
xAPI,
Static Content
Viewer
An ADL Initiative-developed viewer that allows a learner to selfreport whether they’ve completed viewing/reading assigned
content. A big limitation of this platform is that it is only able to
collect initializes and completes data.
xAPI,
Unity3D Serious
Game
Situational Judgement Trainers: Content delivered in the context of
a digital, interactive judgement trainer where scenarios are
presented to each learner and they are assessed on their responses.
xAPI, Serious
Games xAPI
Profile, WebGL
xAPI Video Player Web-based utility that plays any number of video formats and tracks
viewer usage (e.g., pause, play, skip, focus, etc.)
xAPI
A working group was
originally established to
focus on this topic;
however, this working
group was absorbed by the
Content Demonstration
working group.
50 The ADL Initiative – May FY19
analysis, and visualization of learning data. To meet this goal, a data strategy was created to define how
each activity would be described using metadata and an xAPI profile was created to enable a shared
vocabulary for consistently tracking learner performance across all activities. Metadata is used by FLUENT
to help inform recommendations to the learner and the CaSS to help inform the meaning behind different
assertions being processed. Mandatory and optional xAPI statements were created for each media type used
in the 2018 demonstration. These were derived from the types of performance data that each media type
can generate, the roles of each user that needs to visualize this data, and the type of data that needs to be
generated to support each user.
Figure 17. xAPI Statement Mappings – The xAPI working group began identifying basic xAPI statements that will
be required across all media types. Additional statements will be needed as the different types of content/media are
confirmed for use in the 2018 demonstration.
As shown in Figure 17, numerous strategies were discussed for how and when to instrument the different
learning activities. Dashboards and data analytics strategies ranged from instrumenting learning activities
with xAPI statements to the fusion of data from other TLA sources. Other data types include competency
data and Learner Profile information created by CaSS, metadata for content/activities, and learner inference
data derived from FLUENT. A TLA data requirements document was authored to clearly define all activity
tracking requirements for each learning activity. This document provided guidance for those implementing
xAPI to ensure data interoperability across learning activities, but also provided a foundation to inform the
types of dashboards that would be available in the 2018 event.
An overarching xAPI profile was created for the 2018 TLA Test and Demonstration that was comprised of
numerous other xAPI profiles and a shared vocabulary that allowed disparate TLA services and content
providers to track data consistently, facilitating a common language for analytics and visualization services.
However, it was not implemented consistently across all 2018 activities and components. The xAPI actors
were defined by the participants UUID (assigned by Keycloak™). Statement templates were created for
specific learning activities and experiences that were delivered to learners. Client-based services like the
video player or PeBL integrated quickly; server-based services like PALMS, Unity3D, and Moodle™
required special helper services to facilitate full integration.
51 The ADL Initiative – May 2019
Figure 18. TLA Metadata Spreadsheet – An Excel spreadsheet captured the 23 fields used to describe each learning activity. The Activity Name and description
were used verbatim by the recommender’s interface to provide the learner information about how the activity applies to the competencies it is aligned with. As
learning activities were added to the spreadsheet, a script pulled data from the relevant cells and automated the process of creating an LRMI formatted JSON file.
The JSON file was uploaded to the Activity Index which acted as a common repository that all TLA components used to understand relevant information about
learning content and activities.
52 The ADL Initiative – May 2019
Using the LRMI format, the 2018 metadata schema included 23 properties describing the activity, including
its location, associated competency alignments, associated thumbnail, and a 140-character description of
the learning activity. The metadata files also included human-readable text fields (e.g., name and
description) that FLUENT used to display to the learners as part of the recommender interface. As shown
in Figure 18, the approach was a labor-intensive process that required human data entry into an Excel
spreadsheet that included all metadata fields.
This data was then exported and brought into a script that could generate JSON files that were conformant
with the LRMI specification. The JSON files were aggregated into an Activity Index which was resident
inside the FLUENT boundary. The Activity Index was used by FLUENT content filtering algorithms to
make basic recommendations. CASS also used the Activity Index each time it received an assertion from
an activity. CaSS would use the activities metadata to align to a specific competency and to inform mastery
estimates.
4.2.5 Sequencing and Delivery
While the original CODIAC course was an established program of instruction, the 2018 Test and
Demonstration took other courses such as Combat Hunter and PercepTS and merged this content with other
available content that was never considered in the original development of the CODIAC course. This
presents the challenge of reassembling the content in a way that makes pedagogical sense to learners and
instructors. Often, this is because the fundamental elements of instruction (objectives, content, instructional
events, activities, and assessments) are not aligned with sound learning principles or instructional strategies.
While the CODIAC course was being migrated into the TLA competency framework, instructional
designers reviewed the domain and found that the course contains both well-defined and ill-defined tasks.
Table 9 and Table 10 show a breakdown for how different instructional activities will be divided for the
2018 TLA Test and Demonstration.
Table 9. Well Structured Tasks. Instructional framework and sequencing of activities for well-structured tasks and
problems.
Activity Type Sample Activities
1. Presentation • Tell me (verbal information, concepts, procedures, principles)
• Show me (demonstration)
• Copy me (demonstration with active participation)
2. Practice • Locate, name, identify, list, match (information, concepts, procedures,
principles)followed by correct answer or corrective feedback
• Discuss verbal information, concepts, procedures, principles
• Do next or you do it with correct answer feedback
• Given conditions, predict consequences with correct answer feedback
• Given outcome, specify conditions with correct answer feedback
3. Support • Motivational, cognitive, and metacognitive adaptions
• Corrective feedback
• Explanation of what happened and why
• Discussion of verbal information, concepts, procedures, principles
53 The ADL Initiative – May FY19
Table 10. Ill-Defined Tasks. Problem-centered framework and sequencing of activities for ill-defined tasks and
problems.
Activity Type Sample Activities
1.Authentic
Problem
• Ill-structured learning tasks (e.g. CODIAC Unit 5, Lessons 5001-5006 – Meaning of cues,
image, 5008 – Profile environments)
• Practical experiences (e.g. CODIAC Unit 8, Lessons 8005, 8006, 8007, 8010, 8013)
• Practical Applications (e.g. CODIAC Unit 9, Lessons 9001-9010)
2.Demonstration • Show me (demonstration)
• Copy me (demonstration with active participation
3.Practice • Do next step or you do it with correct answer feedback
• Given conditions, predict consequences with correct answer feedback
• Given outcome, specify condition with correct answer feedback
4.Support • Tell me and Show me (verbal information, concepts, procedures, principles covered in
unit 1-7 and lessons 8001-8004, 8008, 8009, 8011, 8012)
• Locate, name, identify, list, match (information, concepts, procedures, principles)
followed by correct answer or corrective feedback
• Discuss (or videotaped discussion of) verbal information, concepts, procedures, and
principles covered in Unit 1-7, and lessons 8001-8004, 8008, 8009, 8011, 8012
• Motivational, cognitive, and metacognitive adaptions
• Corrective feedback and explanation of what happened and why
The FLUENT recommendation engine was developed to accommodate multiple instructional frameworks
to guide recommendations according to established instructional strategies that are commonly used today.
This allows future implementations to tailor the instruction according to the domain being taught and other
considerations where different instructional strategies are desired. For the 2018 TLA Test and
Demonstration, FLUENT was instrumented with two frameworks that were best suited to deliver
recommendations for which content a user should encounter next. Figure 19 and Figure 20 below show
how content was delivered to support well-structured and ill-defined tasks.
Figure 19. Well Structured Tasks – This represents an instructional framework for delivering content in support of
well-structured tasks. Content is presented in a linear fashion where knowledge, Skills, and Abilities (KSAs) are
structured hierarchically and the user cycles through learning content while using the checks on learning to progress
to each consecutive level.
54 The ADL Initiative – May FY19
Figure 20. Ill-Defined Tasks – This represents an instructional framework for delivering content in support of illdefined tasks such as understanding the meaning of cues inside different images. For these types of tasks, the user
cycles through content and a learner behavior model is updated and used to determine the appropriate content the
user should encounter next.
4.3 TECHNICAL INTEGRATION
After an initial registration with Keycloak™, the FLUENT recommendation engine provided the primary
interface to the learner through a browser based HTML5 application. Upon log-in, users were sent to the
home page seen below in Figure 21. This is the primary interface for the students and offered a few different
methods for accessing content. Basic functionality included the ability for each learner to set his or her
goals and visualize progress through an interactive skill tree. Based on selected goals, FLUENT would
direct learners to different LRPs including the Moodle™ LMS, PeBL, PALMs, Sero!, and live exercises.
Each LRP was capable of delivering different learning activities that were tracked and used to make future
recommendations.
Figure 21. Home Page of 2018 TLA Test and Demonstration FLUENT Interface.
55 The ADL Initiative – May FY19
Instructional content had to be presented in a logical and intuitive way that maximized our ability collect,
analyze and display learning data to the relevant target audience. By instrumenting an array of disparate
content, the 2018 TLA Test and Demonstration generated a large amount of inter-connected human
performance data (individuals and teams) that could be used to demonstrate the potential of the TLA for
observing learner progress across learning modalities. The recommended activities generated by the
FLUENT system were based on the goal selected by the user on the screen shown in Figure 22. For this
event, instructional content was organized into skill trees with three top-level nodes (badges): Combat
Awareness, Combat Profiling, and Combat Tracking. This structure was solely for content organization,
not to restrict users to a predefined path. Learners could begin at any level of a skill tree and there were no
restrictions on what goals learners could select or how often they could change goals.
Figure 22. FLUENT Interface Showing the CODIAC Skill Tree Structure and Goal Selection
As students worked through the content, xAPI statements were generated to record their interactions with
the system. These statements were captured in the Learning Locker LRS. One LRS was set up to capture
learner actions (e.g., answering a question), while another was set up to capture system actions (e.g., making
a competency assertion). Each TLA component published their system log files via xAPI statements to
store a ledger of what was going on internal to each system. xAPI system logs were housed in a separate
LRS and were used to provide additional insight into the computational resources and data dependencies
in the 2018 TLA Reference Implementation. In addition, non-xAPI learner inference data were stored in
the CaSS. This resulted in three data sets:
• Learner data: a 77 MB JSON file with 57,381 xAPI statements
• System log data: a 1.8 GB JSON file with 353,746 xAPI statements
• Learner inference data: a 90 MB JSON file (non-xAPI format)
xAPI Data Structure: The xAPI is a technical data specification that enables flexible and customizable
tracking of learner performance and behaviors across different learning activities. The xAPI statements are
constructed in a JavaScript Object Notation (JSON) format with three important properties: Actor, Verb,
and Object. In sequence, these pieces of data produce a clear recording of the learning experience. These
data can also be cross-referenced with performance data to map the training to performance. The xAPI
specification is open-source and maintained by the ADL Initiative. In addition to the three data fields
mentioned above, the following properties were also used in the demonstration:
56 The ADL Initiative – May FY19
• ID: a unique identifier for each xAPI statement
• Version: the statement’s associated xAPI version
• Timestamp: when the event described by the statement occurred
• Stored: when the statement was recorded by the LRS
• Result: further details representing a measured outcome (e.g., assessment score)
• Authority: agent or group behind assertion; verified by LRS via authentication
• Context: further details of recorded event (e.g., altitude of flight simulator)
Competencies and Badging: Badges represent a way of acknowledging achievements or skill acquisition
in a granular fashion. The CODIAC course was organized into skill trees with three top-level nodes:
• Combat Awareness
• Combat Profiling
• Combat Tracking
These nodes were deemed badges and arbitrarily given internal codes 1, 2, and 3, respectively. Expanding
each node revealed more layers of content, up to a maximum of five layers (e.g., competency code
3.1.3.B.3). Each competency’s unique ID was a URL that contained further text information about it,
including its weight, description, and internal code.
Learner Inferences: CaSS used each learner’s data to estimate their proficiency for each node of the skill
tree. This information was stored in JSON format, but it did not follow the xAPI structure. As such, these
data were not logged by an LRS but were instead stored locally in CaSS. These data included:
• Number of attempts and timestamp of last attempt for each activity
• Mastery estimates (novice, intermediate, expert) for each competency
• Mastery probabilities, assertion source, and timestamp for each competency (the CaSS assertion
processor maintains a list of systems that can provide assertions)
Activity Index: The Activity Index was stored as an LRMI-formatted JSON file located inside the
FLUENT application boundary. The metadata drives the filtering algorithms used in the prioritization of
recommended learning activities. The metadata model also included paradata, assertions, analytical data,
identities, and reputations that flow through the distribution network. CaSS also has a service that goes into
the Activity Index and retrieves a mapping from the xAPI assertion statement to the appropriate metadata
for the activity generating the statement. Each competency has a unique identifier in the competency
framework. This is referenced in the metadata through the LRMI Alignment Object.
Figure 23. SWEG(A) Classroom at Fort Bragg – The classrooms for the TLA 2018 Test and Demonstration were
reconfigurable and allowed the team to support all participants at once for completing the initial pre-test and surveys.
57 The ADL Initiative – May FY19
4.4 TLA TEST EVENT AT FORT BRAGG
Learners valued the content, the team collected an enormous quantity of data, event administration ran
smoothly and timely, no system outages occurred, and the vision for the future learning ecosystem was
realized to an enthusiastic audience. The event itself took place from August 13-17, 2018 at Fort Bragg in
Fayetteville, North Carolina. As shown in Figure 23, the 2018 TLA Test and Demonstration was delivered
in collaboration with the John F. Kennedy Special Warfare Center and School (JFKSWCS), with
participants from its Army SWEG(A). Roughly 60 volunteer active duty personnel from SWEG(A)
interacted with the TLA Reference Implementation for 14 hours over five days.
On the first day, participants were briefed on the purpose of the event and provided an overview of the tools
and technologies they would encounter throughout the week. Upon completion of the briefings, participants
signed the inform consent forms and began completing a survey that collected demographic information
about each participant. Upon completion of the surveys, each participant took a pre-test that was intended
to measure their current state of proficiency across the 234 competencies identified for use in this course.
The pre-test was graded, and the results were used to initialize the participant’s Learner Profile so that the
different TLA systems could tailor recommendations based on each individual’s current level of
understanding. The pre-test results were also compared to the results of a post-test taken after the event to
measure the immediate transfer of learning after taking the course.
As participants navigated through the course, observers continually collected data about what each
participant was working on, and several focus groups were held after the course to poll participants on
various aspects of their experience in the course. Using this data, coupled with the digital data collected
throughout the event, seven areas of the TLA were evaluated:
1. Functionality: TLA features and proposed benefits were documented, and the reference
implementation was assessed against the defined functional requirements. As participants
interacted with the system, basic system metrics, user behaviors, and system behavior were
gathered to determine functional performance and whether design goals were met. Some of the
measures used for assessing functionality included a comparison of systems actions taken as a result
of corresponding event data and the comparison of recommendations to competency data per user.
2. General System -ilities: Related to the functionality assessment, the system’s general technical
performance was captured; this includes criteria such as latency, composability, technical reliability,
and modularity. System interruptions, downtime, and stress load failures were also monitored.
3. Specific System -ilities: Thisinvolves documentation and assessment of the idiosyncratic technical
performance criteria, such as how varied learners’ trajectories are from one another (i.e., system
adaptability). System logs about operational performance were also reviewed to help understand
how and why different actions occurred.
4. Usability (End Users): This assessment included learners’ satisfaction, engagement, and
subjective experiences using the system. Data was captured using existing instruments (i.e., System
Usability Scale [Brooke, 1996] and User Experience Questionnaire [Turner, 2011]).
5. Usability (Developers): This assessment focused on the satisfaction and experience of those who
interact with the system for design and development purposes, such as content developers and
instructional designers; this involved both the reference implementation and the viability and
quality of its technical documentation in terms of clarity, functionality and diffusion.
58 The ADL Initiative – May FY19
6. Learning Outcomes: Although the transfer of learning was not a focus for this test, the system’s
learning potential was assessed to provide a baseline for future experiments. This was determined
primarily by measuring learning gains using a pre/post-test design.
7. Cost Factors: Finally, initial development cost data (to eventually inform return on investment
analyses) was captured from the system designers and developers. Data collection includes both
qualitative and quantitative data.
Upon completion of the course, all collected date, including the results from the surveys, pre and post-tests,
and the three primary TLA data stores were cleansed and packaged in support of a TLA data hack-a-thon.
This event invited engineers, data scientists, and others to get an in-depth understanding of what data was
collected and made available during the 2018 TLA Test and Demonstration event. The hack-a-thon
provided detailed results about the potential for how learning data may be used to power data-driven
dashboards that support a variety of different roles and purposes across the continuum of learning.
4.4.1 Logistics and Administration
Leading up to the August 2018 event, the ADL Initiative worked with SWEG(A) and IDA to create surveys,
orientation briefings, and coordinate the exchange of materials required to obtain all the various permissions
for holding the event. The ADL Initiative coordinated staffing, travel, hardware/software implementation,
and overall management of the event. IDA created the research plan, and SWEG(A) provided 30 desktop
computers, 3 classrooms, student participants, and instructor/observer participants as needed to help
facilitate the 2018 demonstration. Prior to the start of the demonstration, the ADL Initiative staff and
vendors spent the week with SWEG(A) staff preparing the classroom, computers, and devices for the
demonstration. The team partitioned the classroom into two areas: a large area for learners and their
working areas, and a smaller area for developers and support staff. A small meeting room was also provided
to support focus groups and other ad hoc meetings.
The Research Plan submitted through IDA followed all policies required for the protection of human
research subjects. With concurring approval by the Department of Defense Personnel and Readiness
Research Regulatory Oversight Office, this project received an exemption. Participants were divided into
groups of 30. This allowed for a 4-hour class in the morning and one in the afternoon. SWEG(A) confirmed
participant availability and provided all the primary points of contact for JFKSWCS and Fort Bragg policies,
permissions and access. Participants worked in a computer lab with Internet connected desktop computers,
iPad devices, and WIFI access for iPad Internet connectivity. Public Internet was accessed through Fort
Bragg’s LangNet. SWEG(A) reviewed all instructional content and ran scans on all network-connected
hardware to monitor ingoing/outgoing network traffic.
4.4.2 Hardware and Network Connectivity
The TLA’s AWS infrastructure met all requirements. With over 30 concurrent users streaming video,
reading PDFs, and using Moodle™, the team and learners observed no system outages. Networking,
memory, and CPU metrics were well below their limits, rarely reaching even 50% capacity. Rather than
configuring 30 laptops individually, the SWEG(A) engineers requested the team provide a single, preconfigured master hard drive that could be cloned across all classroom computers. SWEG(A) used their
Master Drive Cloning System to rapidly deploy preconfigured operating systems, applications, and settings
on each desktop computer. The lead time for this system is 30-60 days (ideal). SWEG(A) delivered a single
desktop computer at the end of June from the 25th to the 28th
. The ADL Initiative engineers received the
desktop and configured it to run the required software components.
59 The ADL Initiative – May FY19
With 30 concurrent learners using two devices each, network latency and bandwidth limits were a concern.
The team used two Netgear Nighthawk routers broadcasting two frequencies48 each, allowing up to 128
simultaneous connections – satisfying connectivity requirements for the learner devices and allowing
network access for developers and administrators. After setting up the routers, network connections for
each device were configured with fallbacks to the developer network if necessary. Devices typically
maintained their connection, but wireless settings required the team confirm connectivity each morning.
For security reasons, SWEG(A) did not broadcast the wireless networks, so the team established each
connection manually across devices. Any loss of connectivity was only temporary as the backup network
took effect, but a building-wide power outage did impair connectivity for roughly 30 minutes during one
session.
Computers, iPads, and peripheral hardware functioned without issue. The desktops were preconfigured with
the software preloaded through SWEG(A)’s cloning process. Minor changes were made to the software as
some TLA components were updated after the cloning process. On the desktops, very few issues arose.
System permissions from SWEG(A) prevented the team from resolving some issues with browser caching49
,
so the team changed the learners’ web browsers from Google Chrome to Mozilla’s Firefox during the
demonstration resolve this. Initial system usage on the first day (Tuesday) uncovered a performance issue
with the LRS responsible for collecting system logs. The server was upgraded and its corresponding DNS
registration by midnight. The machine resumed function Wednesday morning.
4.4.3 Pre-Tests, Post-Tests, Surveys, and Focus Groups
Upon arrival Tuesday morning, all students gathered to sit
through an orientation brief, signed their informed consent
forms, and completed a series of surveys. Upon entering
the classroom, SWEG(A) staff randomly assigned a UUID
to each participant to ensure future anonymity in analysis
and reporting. Log-in accounts had previously been created
with the UUIDs and only SWEG(A) and IDA had access
to Soldier identities. As shown in Figure 24, the
Orientation provided an overview of the multi-year
initiative and walked participants through the intended
purpose and justification for this research. The CODIAC
course was introduced and the various delivery platforms
were described to show participants how to use the system.
The first survey was a Basic Demographic Questionnaire consisted of 16 items including name, gender,
age, service, duty station, office, assignment, military grade, military occupational specialty code, education,
and background. The target populations for this study are U.S. Army active duty or reserve military
personnel with military grades spanning mid and senior enlisted, non-commissioned officers, warrant
officers, and non-field grade officers. They are stationed at Ft. Bragg and are all under the US Army Special
Operations Command (USASOC) with Psychological Operations and Special Forces military occupations
48 Each router broadcast its 2.4 GHz and 5 GHz frequencies.
49 Caching is a process employed by modern web browsers to reduce page load times. These browsers keep local
caches of certain files from webpages that they expect to rarely change (like style sheets and images). While this
improves user experience, clearing these caches often requires administrator permission on the machine itself.
Figure 24. Orientation Briefing – The 2018
TLA Test and Demonstration is the culmination
of the second spiral of Design Based Research.
60 The ADL Initiative – May FY19
represented. The age span is approximately 20-26 years with gender skewed mostly toward males. A User
Experience (UX) Questionnaire was also presented for collecting data about system usability and the user’s
experience interacting with the system. A 24-item UX Questionnaire was developed to collect data on
usability and user experience. These surveys helped evaluators understand more about the participants and
for stratification in analysis.
Researchers were also on site to collect observational data of the interactions in correlation with the
collection of system performance and behavioral data from TLA systems over the interaction period. These
data should enable a unit of analysis at the individual level as well as the group and provide descriptive data
on learner individualization and system level performance.
To capture initial and final knowledge of the subject matter, the team administered pre-tests and post-tests.
Pretest scores were used to seed the initial user competency values in the system. Though originally
planning to use Moodle™ for serving these assessments, the team utilized paper copies because the final
pre- and post-test included short answers and essay questions which required manual grading outside of
Moodle™. Two separate tests were developed; one for the pre-test and one for the post-test. All learners
went through and took their pre-tests at the same time Tuesday morning. Eight team members with
knowledge of the CODIAC material graded each test manually. As the experts finished grading these pretests, results were transferred to an Excel spreadsheet, and run through a script to import those results into
CaSS to initialize the participant’s TLA Learner Profile. Each paper test was digitally scanned upon
completion and again after grading was completed. The lack of automation required considerable resources.
To understand more about each participant’s context of use, insight into thoughts and feelings about their
experience, and other individual/group dynamics, post intervention focus groups were held. The focus
group protocols called for a short social and warm-up period, followed by the open-ended questions and
conversation, and finally a wrap up. Questions were created ahead of time and were intended to draw out
experiences and perceptions concerning expectations of learning, expectations of interaction, actual
experiences of interaction, contrasts between expectations and actual system performance, and participants’
feelings about use. Focus groups were closed between the researchers and the participants and will be
recorded for analysis.
Lastly, the research team recorded learner, administrator, and engineer feedback about the event through
surveys and focus groups. Small groups of learners were led through focus groups while other the ADL
Initiative staff and IDA discussed the event with SWEG(A) administrators and instructors. Interviews were
also held with the ADL Initiative and vendor engineers to document their experiences. In addition to faceto-face focus groups, participants also completed paper surveys for both administrative and engineering
efforts.
4.4.4 Learner Management and User Experience
Prior to the event, the ADL Initiative created 90 TLA accounts to be used by learners at Fort Bragg. Each
participant received login credentials for a UUID they kept and used throughout the event, allowing the
system to track their individual performance without collecting any Personally Identifiable Information
(PII). Learners were asked to preserve their login credentials between days, although these credentials could
be given to a learner upon request.
After login credentials were distributed and all in-processing activities were complete, instructions were
provided on how to access and use the system, learners logged into their designated accounts and began
using the FLUENT User Interface (UI). Learners will interact with the CODIAC content for approximately
61 The ADL Initiative – May FY19
four hours per day over a four-day period with the first and last day limited to two hours to allow for inprocessing and out-processing the participants. Fixed groups of 30 participants alternated at designated
times throughout the week. When using the FLUENT UI, learners had difficulty knowing which
competencies had not been achieved due to a scaling issue50. By default, the tree view expanded nodes with
a fixed font size and did not reconcile tightly grouped elements for visibility, making large groups of
activities unreadable.
As the TLA is designed to empower personalized learning, everyone experienced a different learning path
through the CODIAC content. Overall, the system and content kept learners engaged and allowed them to
progress through the content in a logical fashion. Since the learner was the sole pivot point for all
recommendations, they alone chose their learning goals and corresponding activities. While many learners
began with the SCORM content (presumably because its relationship to so many competencies almost
guaranteed its recommendation), their activity paths began diverging after those were completed.
Participants viewed the relevance of CODIAC content as a positive experience and especially called out
the live activities and capstone exercises as having value to their near-term learning goals. It is important
to note that the human readable descriptions included in the metadata for each activity and the custom
thumbnail images also contributed to a positive learner experience by explaining why each activity was
related to different competencies. The research team observed some learners intentionally change their
goals to seek out an interesting or fun activity being performed by their neighbor.
4.4.5 Data Collection and Management
Automated data collection efforts produced a significant quantity of data. The system collected over
450,000 xAPI statements during the 4-day event, which were anonymized and distributed for the iFEST
hack-a-thon. Approximately 400,000 of these were generated through the system log LRS and the other
50,000 were generated through learning activities to reflect participant behaviors. The “2018 TLA Data
Requirements” document defined the meaning, structure, and requirements for the xAPI statements
produced within the TLA. Other than xAPI statements, the system captured each learner’s final competency
profile, their activity history, and statistically significant pre-test and post-test differentials. Additionally,
the surveys and focus groups provided a wealth of objective information about TLA usage, user interface,
instructional content, value proposition, and overall user experiences. All data was collected and put under
positive control by the ADL Initiative and IDA at the end of the event.
An objective third party evaluation of the raw data by an ADL Initiative vendor that was not part of the
2018 event concluded that the dataset collected at Fort Bragg had limited value for the purpose of true
analytics. This is a result of many factors including inconsistent usage of terms due to inconsistent usage
of the TLA’s xAPI profile. Some of the content providers used different verbs or did not include the result
extensions as outlined in the Data Requirements document. The result of numerous misalignments between
data expectations, metadata intents, and statements produced resulted in performance measurement gaps
from some learning activities. In most instances, enough data was generated by other activities to minimize
the impact of the missing data.
50 The tree view did not utilize vertical screen real estate, staying confined to roughly the top third of the
page. This caused competencies with large numbers of activities to produce dense packs of associated
activities, making it difficult for learners to see what was being displayed.
62 The ADL Initiative – May FY19
4.4.6 Data Analytics and Visualization
Analytics is the discovery, interpretation, and communication of meaningful patterns in data. It relies on
the simultaneous application of statistics, computer programming and operations research to quantify
performance. A variety of visualization techniques are used to communicate insights into the data being
collected.
Native LRS dashboards were used to support real time visualization of the Activity Streams as they entered
into each LRS (system log and participant actions). These dashboards allowed non-technical users to
participate and understand the analytics process by compiling data and visualizing trends and occurrences.
The dashboards also provided participants an objective view of performance metrics and served as an
effective foundation for further dialogue. The 2018 TLA 2018 Reference Implementation exposed
numerous limitations with the point-to-point messaging that took place between TLA components. This
approach specifically presented challenges in providing real time dashboards because the data was being
passed between different localized data stores which required numerous listener services to collect and
aggregate the data.
Figure 25. Activity Levels of All Learners.
In addition to the LRS dashboards, a hack-a-thon was held that promoted further exploration of the data
generated throughout the week. Hack-a-thon participants performed an exploratory analysis on usage
patterns across all learners within the TLA 2018 Test and Demonstration. To this end, attendees created a
variety of charts to explore patterns in user behavior, new ways to use xAPI data, and methods to refine the
data collection process. As shown in Figure 25, each bar represents a learner, and the height of the bar
indicates the number of xAPI statements associated with that account. Besides one outlier at the top end
and many low-activity accounts at the bottom, the distribution is reasonably smooth.
63 The ADL Initiative – May FY19
The next step was to analyze usage patterns across competencies. Each competency is shown in Figure 26,
along with the number of xAPI statements associated with it. The color of the bar represents the badge that
the competency was placed under. Users were free to work on any competency at any time, but the chart
below suggests that many may have chosen to begin with the badge positioned at the top (Combat
Awareness), even though there was no order between the three separate skill trees.
Figure 26. Statements Generated per Learner. – Color Indicates the Specific Competency Framework (Badge)
Figure 27. Statements Generated per Competency – Color Indicates Depth.
In addition, the same data were examined but with respect to competency depth. Each skill tree in the
FLUENT UI included five level parent-child hierarchy that reflected the CODIAC competency frameworks
created for this course. Figure 27 provides a chart that shows that the higher-level nodes generated more
statements, perhaps because they were more easily accessible to learners. To reach a competency at the
deepest level, users had to click through several parent nodes, whereas the depth 1 competencies were
immediately visible upon loading the page.
64 The ADL Initiative – May FY19
Research Question: Were students engaged during their hours with the system?
Answer: One key advantage of xAPI over older standards is that learner activity can be tracked in much
greater detail. In addition to completions, successes, and failures, we can now also record actions such as
turning a page, indicating a preference, or pausing a video. As shown in Figure 28, this allows for a nearly
perfect reconstruction of a learner’s experience over any span of time. To illustrate this, a prolific user was
chosen and all associated xAPI statements plotted on a timeline. For each minute of the four-hour period
from noon to 4:00 PM on Friday, August 17th, 2018, we can see the count of statements generated for that
user. This is further broken down by color-coding to indicate the verb ID in each statement.
Figure 28. Statements per Minute – This chart shows statements by minute for an individual participant, along with
a color-coded breakdown of the verbs associated with these statements.
Research Question: Which assessment questions are the most difficult?
Answer: Another frequently discussed research topic is gauging the difficulty of each task. For longer
assessments with a wide variety of possible scores, a simple mean or median calculation may not provide
enough information. Instead, we can examine the distribution of scores to see how different learners have
performed. One way to do so is to plot the proportion of all learners who achieved a given score (i.e., a
percentile). As shown in Figure 29, 50% of learners earned a scaled score of 86% or less.
65 The ADL Initiative – May FY19
Figure 29. Empirical Distribution Function – For a chosen assessment activity.
Research Question: Can we visualize the learning pathway of a single learner over time? Can multiple
student pathways through the learning activities be determined, contrasted, and visualized?
Answer: As shown in Figure 30 the team generated a preliminary visualization for learner paths. Given
the short duration of the TLA demonstration, most participants were unable to make significant progress
toward all three badges, so this learning path comparison focused on pairs of students whose final
competency states were similar. To calculate this similarity, each user’s final competency state for each
badge and terminal learning objective (TLO) was extracted into a vector. Then a difference vector was
constructed for each pair for learners by subtracting one learner’s vector from the others. Finally, the norm
(length) of each pairwise difference vector was calculated (a smaller norm indicating less distance between
their final outcomes, hence greater similarity).
Figure 30. Competency States and Learner Paths – The top chart compares two participants in terms of their final
competency states. The bottom charts show the learning paths taken by the two users. The chosen users were the two
most similar ones in the demonstration.
66 The ADL Initiative – May FY19
4.4.7 Lessons Learned
Data quality is of utmost important for analytics. The 2018 TLA Reference Implementation exposed
numerous issues that inhibited analytics and data visualization, both with the data sets themselves and the
way the data travel through the TLA. The 2018 demonstration did not reach full compliance with the
guidelines laid out in the 2018 TLA Data Requirements document. Compliance with the established
standards is crucial for data consistency and component interoperability. Data issues included the following:
• Non-conformant usage of context Activities object. Parent and object were frequently the same
activity. This object should maintain the relationship between the content and its competency node.
• Reuse of statement IDs. It is crucial to have a unique identifier for each xAPI statement. Per the
specification, statements should not be overwritten, but instead voided if they must be invalidated.
• Lack of functionality for YouTube video pages. For video content hosted on YouTube, there was
no way to track the user’s platform, how the user was directed to video, or how the video linked to
the competency framework. This resulted in malformed xAPI statements.
• Inconsistency in actor, verb, object naming patterns. Many statements were missing fields, such
as the object activity definition.
• Non-unique IDs and/or other fields being used as unique identifiers. It is essential to have at
least one unique identifier for each entity type, and uniqueness must be maintained.
• Inconsistency in verb language. For example, some statements used “en” for the English
description of a verb, while others used “en-US.”
• No established timestamp format. Timestamps were recorded to varying levels of precision.
Some contained time zone information while others did not. As this can cause significant disparities
between components, a standard timestamp format should be established.
• Inconsistent labeling of competency levels. Internal competency data (micro, macro,
competency) did not match external documentation (ELO, TLO, badge), and varying sources
describe different relationships between these terms.
• Poorly formatted objects. Different objects followed different schemes for capitalization, space
padding between fields, and field order.
One of the biggest challenges that arose was that FLUENT object IDs used a 24-digit hex-identifier that
was non-conformant with the object identifiers used within an xAPI51 statement. This inhibited CaSS from
generating assertions for the xAPI statements and required modifications to the approach for processing
competency assertions from different activities. This had downstream effects as other systems tried to
access the nonconformant data (e.g., LRS to LRS communications). There is also concern that many of the
operations within FLUENT were updated in place (e.g., the Activity Index and metadata fields like
popularity and emotional rating) which resulted in the loss of valuable time-series data. Had these values
been incorporated into the xAPI statements, the statements would have served as an immutable record of
these changes and could have been the basis for iterative feedback loops where it would be possible to
report on the correlations between content properties, learner preferences, and performance.
51 an xAPI object ID needs to be a URI
67 The ADL Initiative – May FY19
Assessment: As an Activity Stream, xAPI has no rubric for rolling xAPI statements up into meaningful
assessment data. Future reference implementations should investigate how an xAPI profile might map to
HPML’s Human Performance Data Model as evidence of competency. The CASE™ rubric is also suited
for establishing the criteria and performance levels for earning Credentials. Both may be relevant to
different types of competencies.
Granularity of Data: The granularity of data being collected from different LRPs was inadequate and
often consisted of data like that being generated from a SCORM course delivered via an LMS. Increased
granularity will enable more robust analytics that each platform can deliver. The use of an xAPI Profile
Server will help create conformity through controlled libraries and extensions.
xAPI Data Storage: A separate LRS was used to store system log data due to performance concerns. The
second LRS mitigated the risk of high network traffic slowing down a single LRS. The use of two different
LRSs presented difficulties for real-time analytics because the two data streams must be merged before
analysis can be performed. When real time dashboards are required, all data used in those dashboards should
be stored in a single LRS.
Competency Organization: Instructional content was organized into skill trees. This allowed
competencies to be placed at any desired level of depth, but each node could only have one parent. This
prevented the many-to-many relationships that competencies are expected to have in a framework. Moving
forward, competency frameworks should be structured into acyclic directed graphs to preserve
directionality and eliminate the single parent constraint.
Learner Profiles: The Learner Profile was used to house mastery estimates for the competencies each
participant was obtaining throughout the week. It was closely coupled to CaSS but FLUENT used xAPI
assertions to write learner inference information to the Learner Profile. This approach proved successful
and warrants additional exploration as a mechanism for collecting assertions about a learner from
authoritative sources within the DoD.
Search: No search service was provided in 2018. Participants desired the ability to find the different pieces
of information they required to supplement the existing learning materials. Searches could be scripted to
occur through internal DoD database structures or across all content associated with a TLA instance.
Content Categorization: The number of required LRMI metadata fields used did not create enough
differentiation between activities for the recommender. For example, different types of text documents such
as peer-reviewed journal articles, field manuals, guide books, technical reports, and others were all listed
as a text document. Schema.org presents a vocabulary that can be utilized to add this specificity to the 2019
Reference Implementation.
Weighting: All content was treated the same and incremental increases in competency were assigned for
each completed activity. Weighting was included in the metadata generation spreadsheet; however,
weighting was not included in the 2018 TLA Data Requirements document so some systems were not able
to support this capability. Future implementations will require the weighting of evidence based on the
metadata.
Service Separation: Many distinct applications and services were run on the same AWS instance with no
separation. This resulted in undesirable side effects, such as resource contention. Services should be
decoupled.
68 The ADL Initiative – May FY19
Data Types: While many of the data elements were known, there were still many undefined data types
included within the 2018 TLA requirements document. Data types such as activity metadata, relations of
competency objects, and elements of the Learner Profile need to be better defined so that dashboard
requirements can be defined.
Messaging: Multiple messaging formats including REST and AMQP were used to pull and push messages
between TLA components. Messages were not optimized or organized which resulted in poor data
discipline by creating multiple instances of the same message. This had a secondary impact of creating
more network traffic than was necessary and impeded the ability of the 2018 architecture to scale.
Service Discovery: A service discovery mechanism used either a REST API or an admin portal to register
endpoints for each TLA service. The service was under-utilized and many TLA components in 2018 used
hard-coded IP addresses inside their software. This was problematic when trying to set up a development
environment that is separate from the live production servers after the production servers were locked down
prior to the event. For future instances of the TLA, a dynamic Service Discovery Protocol (SDP) will be
investigated to help reduce the configuration efforts required by TLA developers.
69 The ADL Initiative – May FY19
5. CONCLUSION AND NEXT STEPS
The 2018 TLA Test and Demonstration, coupled with the 2018 TLA Reference Implementation, the
CODIAC content, and the 2018 technical specifications proved successful in demonstrating the potential
of competency-based learning, the viability of existing technical standards, and the amount of relevant data
that can be generated through the TLA. The 2018 TLA Test and Demonstration also exposed technical gaps
in the way the TLA was implemented and revealed numerous lessons learned on the complexity involved
with migrating a single course from a linear program of instruction into a competency-based learning
system. One of the biggest challenges faced was the attempt to demonstrate the value of data analytics and
lifelong learning in the context of a single, one-week course.
Future events should better represent the complexities of different learning environments, different learning
activities, and student behavioral changes that occur throughout their career. One big revelation during this
year’s event was the reliance on the training and education community to generate evidence of competency
within CaSS. From the perspective of competency-based learning, evidence needs to be collected from
numerous sources including the operational work environment. The 2018 experiment showed this is
possible from a technical perspective, but that there are many policy issues that need to be changed to enable
this capability for the future.
Figure 31. 2019 Technical Specifications and Standards – This figure shows the preliminary specifications and
standards being investigated for inclusion in the 2019 TLA Reference Implementation. These show the breadth of
scale and the scope of impact in migrating away from simply collecting performance data from a learning activities
and places additional focus on the collection of evidentiary data from operational systems and mission outcomes that
include key performance indicators that are aligned to the myriad of different competencies in a framework.
70 The ADL Initiative – May FY19
As shown in Figure 31, the 2018 TLA Test and Demonstration informed the team on new potential
standards and specifications that should be explored and evaluated for potential inclusion in the future.
Many of these technical standards are not the atypical learning standards that the ADL Initiative has
traditionally worked with. Instead, they tie into other areas of the DoD including Failure Modes and Effects
analysis (FMEA), business rules and logistics, the JCIDS requirements development process, and even the
way we acquire learning tools, systems and content. Many of the candidate specifications shown above are
being investigated with the overall intention of positioning the xAPI as a Global Information Grid (GIG)
standard to enable it to be used to pull data about learner performance from the same work system being
used on the job.
The 2018 TLA Reference Implementation showed that while technically feasible to collect and process data
derived from any system, the policy issues are a limiting factor in the DoD today. One example is during a
hack-a-thon leading up to the event where a 911 database from NYC was instrumented with xAPI so all
calls over a specific time frame would be decomposed, quantified into meaningful data, and streamed to an
LRS for further review.
Another area of 2019 TLA Test and Demonstration research should focus on creating vocabularies and
classifications of competencies that can be attached as metadata to each competency object. This would
allow for commonality and reuse of competency objects and descriptors across organizations. It would
improve the ability to build assessment strategies that predict the influence of different activity types applied
to specific competency classifications. Metadata vocabularies might include descriptors that inform
whether a competency includes psycho-motor skills, fine motor skills, cognitive skills, skill decay, or
relevant environmental factors that impact or inform the description of a competency. Some competencies
are linked to the environment in which the competency is expressed, and others are motivated by objectives
(knowledge, skills, abilities). Table 11 below shows one approach for creating competency classifiers that
demonstrate how different competencies might be grouped together
The 2019 TLA Test and Demonstration research should also investigate approaches for formalizing how
assessment rubrics can be defined, managed, consolidated and shared. Many competency frameworks
include rubrics, performance levels, or other data that can be used to evaluate proficiency. The Human
Performance Data Model provides similar capabilities. A TLA Assessment service using common
vocabularies could provide a common approach for integrating these rubrics into the TLA.
Table 11. Competency Classifications – This table shows a small number of competency classifications pulled from
Wikipedia at https://en.wikipedia.org/wiki/Competence_(human_resources).
Classification Description
Organizational
Competency
These competencies describe the mission, vision, values, culture, and core competency of the
organization that sets the tone and/or context in which the work of the organization is carried
out (e.g., customer-driven, risk taking, cutting edge).
Core
Competency
Capabilities and/or technical expertise unique to an organization. These competencies
differentiate an organization from their competition (e.g., the technologies, methodologies,
strategies, or processes or that organization that create competitive advantage). An
organizational core competency is an organization’s strategic strength.
Technical
Competency
Depending on the position, both technical and performance capabilities should be weighed
carefully as employment decisions are made. For example, organizations that tend to hire or
promote solely based on technical skills, (i.e. to the exclusion of other competencies) may
experience an increase in performance-related issues (e.g. systems software designs versus
relationship management skills).
71 The ADL Initiative – May FY19
Classification Description
Behavioral
Competency
Individual performance competency is more specific than organizational competencies and
capabilities. As such, these competencies are defined in a measurable behavioral context in
order to validate applicability to the work environment and the degree of expertise.
Functional
Competency
Functional competency is job-specific competency that drives quality results for a given
position. They are often technical or operational in nature (e.g., backing up a database).
Management
Competency
Management competency identifies the specific attributes and capabilities that illustrate an
individual’s management potential. Unlike leadership characteristics, management
characteristics can be learned and developed with the proper training and resources.
To mitigate the limitations of the point to point architecture used in the 2018 TLA Reference
Implementation, a real-time stream-processing software platform is needed to handle the numerous data
feeds in the TLA. As shown in Figure 32, Apache’s open-source Kafka platform is one such solution. This
migration will provide consistency in how data is accessed and communicated throughout the TLA. TLA
components will subscribe to relevant TLA data streams (topics), pull from various data stores within the
TLA data lake, and publish new TLA data streams that other components will subscribe to. In the future,
this exploration may identify new capabilities and potential for managing data about learning.
Figure 32. 2019 Data Streaming Architecture – The 2019 TLA Reference Implementation is migrating into a more
modern data streaming architecture using Apache Kafka. Data producers are along the top and may include learning
activities, biometric feedback devices, mastery estimates, Learner Profile updates, or any other types of data that need
to be collected in support of education and training.
72 The ADL Initiative – May FY19
Using the streaming architecture, we will be able to stream numerous types of data in any number of
formalized data schemas that represent the numerous types of data that a typical learning ecosystem might
generate. The concept is to forward all data to a brokerage service that validates the messaging format and
sends the appropriate data topics to the appropriate endpoint in the data lake. This provides a single
collection point that all other TLA components can publish or subscribe to in real time while also providing
the ability to pull in historical data from the data lake.
The vast increase in the quantity of personal information that is being collected and retained, combined
with the increased ability to analyze it and combine it with other information, is creating valid concerns
about the ability of different TLA components to manage these unprecedented volumes of data responsibly.
There is an urgent need to strengthen the underlying systems, component products, and services that make
learning data meaningful. Data labeling will enable the implementation of artificial intelligence,
recommendation engines, and adaptive tutors by creating a uniform approach for labeling data within the
future learning ecosystem. In addition to a data labeling strategy, an exploration into simulated datasets will
present the opportunity to start exploring other relevant topics such as the data lifecycle, periodicity of
required updates, and pattern discovery techniques.
As previously mentioned, the 2019 TLA Reference Implementation will also instantiate the Learner Profile
as separate entity from CaSS. The promise of new TLA applications stems from the ability to create, collect,
transmit, process, and archive information on a massive scale. The vast increase in the quantity of personal
information that is being collected and retained, combined with the increased ability to analyze it and
combine it with other information, is creating valid concerns about the ability of different TLA components
to manage these quantities of data responsibly. This work will identify best practices for managing a Learner
Profile across the continuum of lifelong learning while also defining approaches, processes and
methodologies for creating, managing and storing Learner Profile data and describing lifecycle
considerations for each data type including authoritative sources where each data type can be discovered.
An Activity Registry is an approach to capturing, connecting and sharing data about learning resources
available to an organization. Key features include the ability to generate and manage content metadata,
manage taxonomies and ontologies, alignment of content with competencies, generate and manage paradata,
perform semantic search services, and create machine-actionable metadata for AI-based recommenders.
Organizations that consume a learning resources will also capture information about how a resource is used
including context, user feedback, user ranking, rating, annotations, etc. Over time, this paradata (usage data)
and third-party analytical data may become more valuable and more useful than curated cataloging
metadata for discovery and understanding which learning resources are effective.
A big effort as part of the 2019 TLA Reference Implementation work includes enhancing the metadata
strategy using the LRMI specification to build out required, recommended, and optional metadata attributes
for each learning activity. Additional research needs to identify an approach for describing various learning
activities a learner will encounter across the continuum of lifelong learning. Best practices will also be
identified for managing, updating, and versioning this information in an intuitive way that can also drive
the development of a common course catalog across the entire DoD. The Data Analytics and Visualization
Environment (DAVE) Framework provides an open-source means of creating domain-based learning data
dashboards that is extensible to new learning problems, instructional strategies, technological realities, and
metrics objectives. As shown in Figure 33, each topic in the DAVE Framework features a domain problem,
an algorithm designed to solve the problem, and a recommended data visualization.
73 The ADL Initiative – May FY19
Figure 33. 2019 Data Analytics and Visualization Environment – The migration to a Data Streaming architecture
in 2019 will enable a common approach for accessing TLA data through micro-services that publish and subscribe to
different TLA data streams. This approach allows TLA dashboards to pull data from the different streams and
transform that data into meaningful reports, visualizations, and analytics.
The migration to the new architecture also provides the ability to subscribe to data from any TLA
component to produce specific data-driven visualizations that represent data and/or analytic results in
concise, graphical ways. These analytics are intended to go beyond static charts and graphs to produce realtime reporting capabilities. While these reports can be used to provide an answer to a specific question,
they should also promote further inquiry and exploration of the vast amount of data collected about different
elements of training and education. Within the TLA, a data dashboard is expected to be an information
management tool that visually tracks, analyzes, and displays key performance indicators (KPI), metrics,
and key data points about an individual, class, a training event, or other TLA-related information. This
requires data to be aggregated from across TLA components. TLA dashboards will combine data from an
LRS with learner data, content paradata and metadata, competency frameworks and other relevant data to
provide insights and reports. |