Jon Mott recently published an article in EDUCAUSE Quarterly on Envisioning the Post-LMS Era. Jim Groom captures the reactions of individuals who have been exploring the link between learning management systems and personal learning environments. There is a sense – and I’ll admit I felt it as well in reading the article – that many long-time contributors to the discussion were not referenced in the article. In theory, the review process should draw attention to important omissions of literature. However, most reviewers would likely not see the spaces (blogs) where much of the conversation happens before it jumps into mainstream as good sources.
I’ve posted below that I wrote while at University of Manitoba addressing the LMS/PLE issue. I’m not sure how long an archive of their copy will exist, so posting it here might give it a bit more of an existence.
A Review of Learning Management System Reviews
October 6, 2006
Learning Technologies Centre, University of Manitoba
Learning management systems (WebCT, BlackBoard, Desire2Learn, Angel, Moodle) hold a position of first choice in learning technology adoption within higher education. Selecting a traditional Learning Management System (LMS) requires balancing learning and management. Theinitial intent of an LMS was to enable administrators and educators to manage the learning process. This mindset is reflected in the features typically promoted by vendors: ability to track student progress, manage content, roster students, and such. The learning experience takes a back seat to the management functions. Numerous reports (citing administrators, IT departments, and educators) laud the management functions of an LMS. To-date, student experiences and efficacy of the tools have been subjected to limited research. The position offered in this report encourages an organizational definition of learning as the starting point for selecting a technology platform for creating and delivery learning content. A clear definition of learning vision and desired future states, created through input from stakeholders (administrators, faculty, students, and information services) should provide the foundation for decision making, and the boundaries of platform selection. This report covers the typical decision-making criteria utilized by various organizations in selecting an enterprise LMS—most often with the intention of settling on a single, system-wide platform.
Introduction and Background
While virtual learning environments have been available in some capacity since 1960, “the PLATO system featured multiple roles, including students who could study assigned lessons and communicate with teachers through on-line notes, instructors, who could examine student progress data, as well as communicate and take lessons themselves, and authors, who could do all of the above, plus create new lessons” (Wikipedia, 2006a, 1960s section, ¶ 1). Learning management systems have only been available, in roughly their present form, since the 1990s (Vollmer , 2003), with Blackboard and WebCT being broadly adopted in universities and colleges by early 2000 (Online, 2006). Initial versions of an LMS focused on organizing and managing course content and learners. As with many organizations, higher education was unsure about the role of technology in the educational process.
Aggressive sales and state or province-wide licenses resulted in WebCT and Blackboard—now merged as one company (Blackboard, 2006a) cornering over 75% or the market (Mullin, 2005). The rapid penetration of learning management systems as key tools for learning occurs in a vacuum of solid research as to their effectiveness in increasing learning—or even indication of best practices for technology implementation. Pedagogy is generally a secondary consideration to student management; some researchers attempted to bridge research from face-to-face environments to technology spaces (Chickering & Ehrmann, 1996)—a practice that may be convenient, but errs in assuming that the online space is an extension of physical instruction, not an alternative medium with unique affordances. Learning management systems became the default starting point of technology enabled learning in an environment largely omitting faculty and learner needs.
Learning Circuits’ (n.d.) publication, A Field Guide to Learning Management Systems, revealed the nature of most LMS decisions at committee levels (an experience paralleled in academic environments): “an LMS should integrate with other enterprise application solutions used by HR and accounting, enabling management to measure the impact, effectiveness, and over all cost of training initiatives” (p. 1). The value of an LMS is ensconced in language of management and control—notions that most academics would perceive as antagonistic to the process of learning. Most LMS options, features, and comparisons (LMS Options, 2006) focus on tools included in a suite, not on how to foster and encourage learning in relation to an organization’s definition of “what it means to learn.” Discussions of features are divorced from emphasis on learning opportunities.
Current LMS Trends and Needs
After almost a decade of LMS experience, educators and administrators are beginning to question the prominence of an LMS. In a recent LMS governance report, Wise and Quealy (2006) stated “the educational significance of LMS is largely overemphasized and misunderstood …[suggesting it is critical for a university to] … understand itself—what it values, what it does well and how it does it, what it would like to do, and how it might do this” (p. 4).
In a previous publication (Siemens, 2004b), this report author has suggested that LMS in general are the wrong starting point for learning:
Learning Management Systems (LMS) are often viewed as being the starting point (or critical component) of any elearning or blended learning program. This perspective is valid from a management and control standpoint, but antithetical to the way in which most people learn today.
Learning management systems like WebCT, Blackboard, and Desire2Learn offer their greatest value to the organization by providing a means to sequence content and create a manageable structure for instructors/administration staff. The “management” aspect of a learning management system creates another problem: much like we used to measure “bums in seats” for program success, we now see statistics of “students enrolled in our LMS” and “number of page views by students” as an indication of success/progress. The underlying assumption is that if we just expose students to the content, learning will happen. (¶ 1-2)
Two broad approaches exist for learning technology implementation:
- The adoption of a centralized learning management approach. This may include development of a central learning support lab where new courses are developed in a team-based approach—consisting of subject matter expert, graphic designers, instructional designer, and programmers. This model can be effective for creation of new courses and programs receiving large sources of funding. Most likely, however, enterprise-wide adoption (standardizing on a single LMS) requires individual departments and faculty members to move courses online by themselves. Support may be provided for learning how to use the LMS, but moving content online is largely the responsibility of faculty. This model works well for environments where faculty have a high degree of autonomy, though it does cause varying levels of quality in online courses.
- Personal learning environments (PLEs) are a recent trend addressing the limitations of an LMS. Instead of a centralized model of design and deployment, individual departments select from a collage of tools—each intending to serve a particular function in the learning process. Instead of limited functionality, with highly centralized control and sequential delivery of learning, a PLE provides a more contextually appropriate toolset. The greater adaptability to differing learning approaches and environments afforded by PLEs is offset by the challenge of reduced structure in management and implementation of learning. This can present a significant challenge when organizations value traditional lecture learning models.
The two dramatically opposing approaches to elearning deployment require consideration of what learning means within an institutional context.
Reviews of LMS selection criteria fluctuated considerably within the cases reviewed, often reflecting a lack of clear focus on intentions of an LMS as a learning support tool. These criteria were generally considered important:
- Ease of use by faculty and students
- Integration with a learning object repository
- Functionality and tools available
- Transition ease and cost from existing tool
- Integration with other enterprise-wide tools
- Extendibility—configuration to the university or college
Learning Management Systems: A Review
In LMS: A Review, Hultin (In press) analyzed key criteria to consider when adopting an LMS, and offers various common platforms. LMS purchasing mistakes include:
- Skirting senior management
- Failing to spell out your needs
- Comparing apples and oranges
- Excluding IT from the process
- Focusing more on price than on value
- Overlooking scalability
- Ignoring LMS interoperability
- Overlooking vendor track records
- Selecting customization instead of configurability (pp. 4-5)
The report attended to divergent needs of different users (administrators, faculty, course developers, learners), context of use (internet connections), usability, and time required to learn the LMS. To meet the needs of various users, a learning environment was offered as a valuable aspect of LMS implementation, while learning environments in this context were linked to an LMS, they will be presented later as an alternative to an LMS:
An important aspect of the learning environments is that they don’t realize any pedagogical models or create learning for the individuals itself. It demands a context based on a pedagogical idea. The pedagogical idea can be realized and strengthened with appropriate learning environments. It is therefore important to integrate the possibilities with Internet based learning already in the idea—and production phase when developing course content. (p. 8 )
Learning environments were categorized as: (a) communication (asynchronous and synchronous), (b) distribution, (c) test and assessment, and (d) interaction (p. 9).
Over the last several years, specialized service providers (like Questionmark, CourseGenie, and Articulate) have offered enhanced testing and content development tools—replacing the tools included in many LMS. This trend is resulting in LMS vendors providing “partners” (Blackboard, 2006d) with priority status in developing and integrating third-party tools.
Melbourne-Monash Collaboration in Educational Technologies
Input from diverse stakeholders within the university environment was solicited during this report. Informal conversations—with individuals directly involved in LMS implementation, support, and administration—were combined with internal reports, meeting minutes, a literature review, and project management reports (Wise & Quealy, 2006). The report presented two broad approaches for LMS governance:
- Top-down, command-and-control: Adopt a system, mandate its use, provide support, identify needs and support through new tools as needed (p. 18), and
- Bottom-up, emergent: “moves governance into the unordered, ambiguous realm of social complexity” (p. 19) by offering support based on elements of use that emerge.
Governance styles must be aligned with the nature of intended learning. The adoption of technology for learning will differ based on faculty learning models and needs. Medical faculty will require different tools and approaches than Engineering or Arts faculties. To suppose on enterprise-wide model of LMS implementation and governance is to overwrite and obscure the multi-faceted nature of learning and knowledge acquisition (and creation).
The governance model utilized in the Melbourne-Monash report (Wise & Quealy, 2006) relied on ten key principles:
- Lay solid foundations for management and oversight
- Structure the board to add value
- Promote ethical and responsible decision making
- Safeguard integrity in financial reporting
- Make timely and balanced disclosure
- Respect the rights of shareholders
- Recognize and manage risk
- Encourage enhanced performance
- Remunerate fairly and responsibly
- Recognize the legitimate interests of stakeholders. (p. 24)
The inclusion of a structured process for LMS review, selection, and governance provides value to all stakeholders. A clear process of selection, preferably tied into the larger university vision of “what it means to learn, dialogue, reflect, and inquire,” ensures the selection process is not vendor-driven or focused on only one aspect of university operation (i.e., needs of the IT department, enrolment and registration, etc.). The needs and interests of learners, however, were not directly addressed in the Melbourne-Monash Report.
EDUCAUSE Center for Applied Research (ECAR)
Beyond merely defining a suite of tools, LMS evaluations should “focus on the processes that underlie creating, preparing, teaching, and taking a course” (Hanson & Robson, 2003, p. 2). Most selection reviews “have typically focused on comparisons of feature checklists and on costs, often narrowly defined as license fees” (p. 2). Additional consideration should also be given to the university’s definition of effective learning, pedagogical models, and larger visions for a changed society—contrast fostering critical thinking with developing learners for the workforce.
ECAR (Hanson & Robson, 2003) presented several guidelines, or steps, for selecting course management systems:
- Determine process benefits (p. 3). This step involves determining critical processes, benefits, and features. For example, synchronous communication tools may be deemed as critical for extended education departments, while collaborative spaces (like wikis) may be important to on-campus only departments.
- Assigning value to products and features (p. 4). Once learning processes have been defined, products and features are explored. Synchronous learning—in the above example—can be supported through a variety of tools—whiteboard, instant messaging, Skype (or other external voice over IP applications), or integrated tools such as
Elluminate and Horizon Wimba.
- Assigning costs (p. 6). Cost determination is complex. Due to established technology investments (for example, an existing LMS), costs involve more than determining license fees. Integration, support, and faculty training costs will comprise a significant part of the total investment.
Learning Management System Strategic Review
California State University (Adams et al., 2005) conducted an LMS review of Blackboard, WebCT Campus Edition, WebCT Vista, Desire2Learn, and open source systems Moodle and Sakai. After an initial review, all LMS were disqualified, except for WebCT Vista and Blackboard. WebCT Vista was ultimately selected.
Systems were disqualified for a variety of reasons including: previous scale of integration, incompatible with “campus data center standards” (Adams et al., 2005, p. 5), limited feature sets, limited ease of use, open source movement still in infancy, and lack of confidence in product support by an LMS vendor. Mention of learner/faculty concerns were largely ignored in the report. Brief mention was made of “ease of use,” eportfolios, and pedagogical flexibility, which is not defined (p. 7). Migration, training, history with vendor, and technical concerns formed the bulk of decision-making criteria.
Course Management System
University of Oklahoma (CMS Task Force, 2000b) expanded its search for an LMS by including a series of surveys from faculty and students. The survey questions focused on individuals selecting needed features to support learning. As with other surveys and assessments, learning remained vague, poorly defined, and disconnected from how the organization viewed teaching and learning. Faculty responses were particularly revealing of the emphasis on “what works for me” versus “how does this align with larger organizational learning objectives:”
- Please keep WebCT! I have hundreds of hours invested in WebCT.
- Most other universities in the Great Plains Consortium use WebCt so I have a preference for remaining with that system.
- I have been using the blackboard system for past three years. I really enjoyed this system which meets all my needs. I hope this system can be kept.
- Switching to new CMS is a time-consuming (and for some faculty) an overwhelming endeavour—so, please, please make this decision with the unconfident computer user in mind – not the power users.
- I only use WebCT because I have no choice. (Faculty Overall Comments section)
Evaluation of Learning Management System Software
This report focused on “the issues or consideration for online pedagogy that impact on the selection of an e-learning platform” (Wyles, 2004, p. 4). The focus on pedagogy raised important questions:
- What pedagogy will be used?
- Will the pedagogy work over the internet?
Emphasis for the evaluation of these questions is based on Chickering and Ehrmann’s (1996) paper Technology as Lever. As mentioned previously, this report assumed that many of the tasks and goals of classroom activity can simply be transferred online. The growth of alternative models of online engagement, as well as parallel conversations found through use of blogs and RRS feeds—such as social bookmarking, tagging, social networks—reveals a dynamic where end-user control grows in prominence. The principles provided by many face-to-face to online transfers of principles or practices does not account for the transformative elements of online learning.
Laying aside the criticism presented, Evaluation of Learning Management System Software (Wyles, 2004) was particularly effective in matching tools (email, bulletin boards, chat, quizzes, tutorials, wikis, etc.) with the work of Chickering, Ehrmann, and Gamson. A critical concept was expressed in the report summary: “Educational institutions need more flexibility and control over their e-learning environments to enable different schools, programmes, course, or instructors to select and deploy the most appropriate e-learning tools suited to the pedagogy” (p. 6). Any LMS selection process should involve a similar match of functionality with the organization’s definition of teaching and learning.
Commonwealth of Learning: LMS Open Source
Open source tools like Moodle and Sakai continue to attract broad interest. The prospects of cost savings in license fees (though fee savings at this level may result in additional investment in maintenance and support) and potential for customization are attractive to organizations.
Commonwealth of Learning (2003) reviewed two open source platforms: ATutor and ILIAS. The methodology used was similar to other reviews listed previously (though focused only on open source options):
- Develop evaluation criteria
- Identify open source candidates
- Filter candidates to produce a short list
- Systemic evaluation of features
- Systems evaluation of general criteria
- Recommendation. (p. 3)
Criteria for selection included:
- Features and functionality
- Cost of ownership
- Maintainability and ease of maintenance
- Usability and ease of use and user documentation
- Current user community
- Standards compliancy
- Integration capacity
- LOM integration
- Intellectual property security
- Hardware and software considerations
- Multilingual support. (pp. 4-6)
Absent from the selection list is a complaint levelled at other reviews: the act and process of teaching and learning are largely ignored in the pursuit of functions, features, integration, and a myriad of other organizational concerns. The very purpose for which an LMS should be selected seems to be a secondary concern in most evaluations of technology solutions. Obviously an LMS needs to be stable, effective (however that is defined), supported, and integrated with other tools. Yet the failure to first define organizational views of learning results in an unanchored and misplaced model of LMS selection.
Vendor lock-in is prominent in the LMS space. Lock-in is described as: “a situation in which a customer is so dependent on a vendor for products and services that he or she cannot move to another vendor without substantial switching costs, real and/or perceived” (Wikepedia, 2006, ¶ 1). Due to a combination of proprietary software, weak
standards-adherence, and lack of foresight by colleges and universities, organizations are placed in a position where existing tools are weighted more highly due to financial and procedural constraints, rather than an evaluation on tool effectiveness for teaching and learning. For education institutions focused on innovating course design and delivery to align with rapid societal changes, lock-in is a significant barrier to the diverse options required to “seed, select, and amplify” (Johnson, 2001, p. 42) approaches to innovation.
Learning management systems are still developing in functionality. The last several years has seen existing providers extend their toolset to include tools currently growing in popularity with many online learners: blogs, wikis, podcasts, and social networking. Blackboard (2006c) recently announced Blackboard Beyond Initiative to integrate Web 2.0 functionality to the system.
For many faculty members, the challenges of learning a new tool require a significant investment in time. Departments face challenges with the nature of content, often created to work within a certain LMS—standards
are generally loosely followed, and even where compliance exists, fine tuning is often required.
A Word of Caution
Educational institutions seeking to adopt an LMS should be wary of Blackboard and WebCT, which BB recently acquired. Blackboard (2006b) recently received patent approval for key components of an LMS and initiated a lawsuit against Desire2Learn. The anti-open competition stance has a potentially chilling effect on learning platforms and the development of the industry as a whole. The patent comes at a time when provosts (Jaschik, 2006) are increasingly acknowledging the value of open source and collaboration. The preservation of intellectual property is a cornerstone of academic advancement. Claiming the work of other researchers as one’s own is unacceptable in academic environments and should cause decision makers to reflect on the values and corporate commitment to the health of a discipline, by organizations seeking to close down innovations that have been publicly documented as collaborative in nature (Wikepedia, 2006).
Limitations of LMS Selection Models
The most prominent difficulty, or limitation of review models explored, was the lack of focus on, or connection to, broader organizational views of learning. Instead of learning driving the tool selected, the process of reviewing and selecting an LMS often resulting in a tool that served other organizational needs (student management, content creation, etc.) in advance of learning itself.
Numerous factors impact successful LMS implementation. Key stakeholders include: (a) administrators, (b) faculty, (c) IT and technical support, (d) learners, and (e) curriculum developers.
LMS reviews considered in this paper generally erred in selecting or attending to the needs of one stakeholder at the expense of others. In selecting an LMS., an argument could be made for the supremacy of learning and the quality of learning, as being the most significant element in technology-enabled education.
Within the span of a decade, an LMS has moved from a support tool to the learning process, to the guardrails of what is possible. For many institutions, management, not learning, has become the most prominent criteria in e-learning.
The enterprise-wide, controlled, centralized learning model serves a particular type of learning (often entry-level or foundational). As learners move beyond content consumption and into stages of critical thinking, collaboration, and content creation, LMS weaknesses become apparent. For this reason, the definition of a university’s learning philosophy is critical in guiding LMS activities.
Seeking Alternative Directions
Educator frustration with LMS views of learning is driving alternative views of learning. Instead of having the software define learning, organizations are beginning to first define learning, and then seek tools (and tool suites) to meet desired needs.
All learning management systems. are not alike, and they can be used in different ways. However, a common idea behind an LMS is that e-learning is organized and managed within an integrated system. Different tools are integrated in a single system which offers all necessary tools to run and manage an e-learning course. All learning activities and materials in a course are organized and managed by and within the system. Learning management systems typically offer discussion forums, file sharing, management of assignments, lesson plans, syllabus, chat, etc.
Recently, the emergence of social software has questioned the use of an integrated LMS. Today, only few social software tools are employed within existing learning management systems. The question is: Is the next step to integrate social software tools in LMS? Social software has initiated discussions about the extent to which tools should be separated or integrated in systems. (Dalsgaard, 2006, Integrating section, ¶ 1-2)
Koper (2004) described the allure and promise of alternative learning models not based on management, but based on increased learner control:
Self-organised learning networks provide a base for the establishment of a form of education that goes beyond course and curriculum centric models, and envisions a learner-centred and learner controlled model of lifelong learning. In such learning contexts learners have the same possibilities to act that teachers and other staff members have in regular, less learner-centred educational approaches. In addition these networks are designed to operate without increasing the workload for learners or staff members.
This model does not exclusively replace traditional learning approaches, but does provide greater alignment with the emerging work-life-learning triad. Instead of learning housed in content management systems, learning is embedded in rich networks and conversational spaces. The onus, again, falls on the university to define its views of learning.
Social Software and PLEs
Two key areas are gaining substantial attention: (a) social software, and (b) personal learning environments (PLEs). Social software and PLEs have recently gained attention as alternatives to the structured model of an LMS. PLEs are defined as: “systems that help learners take control of and manage their own learning” (van Harmelen, 2006, ¶ 1).
PLEs “are about articulating a conceptual shift that acknowledges the reality of distributed learning practices and the range of learner preference” (Fraser, 2006, ¶ 9). A variety of informal, socially-based tools comprise this space:
(c) social bookmarking sites,
(d) social networking sites (may be pure networking, or directed around an activity, 43 Things or flickr are examples),
(e) content aggregation through RSS or Atom,
(f) integrated tools, like elgg.net,
(g) podcast and video cast tools,
(h) search engines,
(i) email, and
(j) Voice over IP.
The shortcomings of these approaches rest in their lack of integration and the control required by many universities. The experience of many educators parallels my own—learners are very active with technology, but once in an LMS space, they seldom do more than the minimum required (a particular concern in courses where dialogue and theory are important to explore). This may be a function of students taking on “the student role”—defaulting to passive behaviour—once in an academic environment. It may also be due to the change in behaviour expected by educators—where learners must leave their tools behind and adopt tools with limited functionality. For an individual used to Skyping, blogging, tagging, creating podcasts, or collaboratively writing an online document, the transition to a learning management system is a step back in time (by several years).
Recommended Process Forward
Different types of learning require different approaches. As educators, our selection of tools is determined by how we answer the question: “What types of technologies best suits a particular learning context?” (Sessums, 2006, Abstract, ¶ 1). Tool selection in advance of context determination eviscerates subsequent use and adoption. Learning management systems have been effective in eliminating the challenges faced by educators in selecting and aligning particular tools with particular tasks. Unfortunately, these systems have begun to determine options available for faculty and an institution.
Bates and Poole (as cited in Sessums, 2006) listed six characteristics for determining appropriate selection of technology:
- Will selected technologies work in a variety of learning contexts?
- How does it impact strategic, institutional level and tactical, instructional level decisions?
- Do the selected technologies provide equal attention to educational and operational issues?
- Will it take into consideration the affect of different media and technologies enabling an appropriate mix for a given context?
- Are the selected technologies user-friendly, practical, and cost-effective?
- Will the selected technologies be quickly out-dated, or will they be flexible and accommodate new developments? (Conclusions section, ¶ 2)
Universities and colleges need to explore broad applications of technology—beyond simple LMS implementations. LMS may well continue to play an important role in education—but not as a critical centre. Diverse tools, serving different functionality, adhering to open guidelines, inline with tools learners currently use, may be the best option forward.
The challenges of LMS utilization is compounded with ongoing changes in technology. E-portfolios continue to grow in prominence (Siemens, 2004a). Informal, life-long learning—validated or certified by educational institutions in the form of prior learning assessment and recognition—is developing in tandem with a greater societal shift. The rapid development of information (Lyman & Varian, 2003) requires a model that sees learning less as a product (filling a learner with knowledge) and more of a process of continually staying current and connected (learning as a process of exploration, dialogue, and interaction).
While desired, it is unrealistic to expect universities to shift significantly from an LMS to a PLE. Yet the trends occurring online (in relation to social software and Web 2.0 technologies—resources that are single-focus, connected, and two-way) are beginning to impact learner expectation. Many educators in the K-12 sector are adopting learner content-creation tools like blogs, wikis, YouTube, podcasts, and tagging. As these learners enter higher education, they may not be content to sit and click through a series of online content pages with periodic contributions to a discussion forum.
The following steps are recommended for moving forward with a broad review of learning technologies:
- Involve all stakeholders (beyond simple surveys).
- Define the university’s view of learning.
- Critically evaluate the role of an LMS in relation to university views of learning and needs of all stakeholders.
- Promote an understanding that different learning needs and context require different approaches.
- Perform small-scale research projects utilizing alternative methods of learning.
- Foster communities where faculty can dialogue about personal experiences teaching with technology.
- Actively promote different learning technologies to faculty, so their unique needs—not technology—drives tools selected.
Create ongoing university teaching and learning technologies council to evaluate ongoing trends, successes, challenges, and needed adjustments to current path. Creating a vision for online learning requires sustained evaluation and monitoring to ensure the approaches to fulfilling the vision change as the context of implementation changes.
The complex process of teaching and learning requires complex, multi-faceted models of implementation. One tool will not meet all needs in all contexts. Changes impact and influence existing models—rendering yesterday’s solutions obsolete. In the field of learning, an adaptive model of technology selection and governance is required to ensure that all stakeholders’ needs are met. A solution today may not be accurate tomorrow. A sustained process needs to be enacted to align context changes with changes and approaches to learning methods and technologies available.
The university “must adapt, using technologies and models of understanding, in this case to reconcile teaching, research, IT, a changing environment, financial accountability and managerial models” (Wise & Quealy, 2006, p. 4). Learning management systems have a position in higher education (certain types of under-graduate level learning are more structured and focused on memorization or content exploration). To meet the needs of all learners in various stages of their education, a multi-faceted (holistic) view of learning must be considered. Increasingly, personal learning environments provide the tools and model to attend to the diverse learning needs of individuals
Adams, S., Banks, B., Evans, B., Gardiner, L., Geunter, C., Irving, J., et al. (2005, April). Learning management system (LMS) strategic review: A next generation learning management system for CSU, Chico. Retrieved October 11, 2006, from California State University, Chico Web site: http://www.csuchico.edu/tlp/LMS2/LMSStrategicReview.pdf
Blackboard. (2006a). Blackboard and WebCT merge. Retrieved October 1, 2006, from http://www.blackboard.com/webct
Blackboard. (2006b). Blackboard recently awarded patent on elearning technology. Retrieved October 12, 2006, from http://www.blackboard.com/company/press/release.aspx?id=887622
Blackboard. (2006c, March 1). Blackboard unveils Blackboard beyond initiative: Four bold inaugural projects will advance e-learning 2.0 vision. Retrieved October 12, 2006, from http://biz.yahoo.com/iw/060301/0111604.html
Blackboard. (2006d). Why work with Blackboard? Retrieved October 12, 2006, from http://www.blackboard.com/company/partners/whyworkwbb.htm
Chickering, A. W., & Ehrmann, S. C. (1996, October). Implementing the seven principles: Technology as lever. AAHE Bulletin, 3-6. Retrieved October 12, 2006, from http://www.tltgroup.org/programs/seven.html
CMS Task Force. (2006). Course management system. Retrieved October 12, 2006, from Oklahoma State University, Faculty Development Link Page Web site: http://fp.okstate.edu/fsc/cms/
Common Wealth of Learning. (2003, July). COL LMS open source report. Retrieved October 12, 2006, from http://www.uv.es/ticape/pdf/Evaluacion_de_plataformas_libres2003.pdf
Dalsgaard, C. (2006). Social software: E-learning beyond learning management systems. European Journal of Open Distance and E-Learning. Retrieved October 12, 2006, from http://www.eurodl.org/materials/contrib/2006/Christian_Dalsgaard.htm
Fraser, J. (2006). More PLE questions. EdTechUK. Retrieved October 12, 2006, from http://fraser.typepad.com/edtechuk/2006/10/more_ple_questi.html
Hanson, P., & Robson, R. (2003). An evaluation framework for course management technology [Electronic version]. Educause Centre for Applied Research, 14(Research bulleting). Retrieved October 12, 2006, from http://www.educause.edu/ir/library/pdf/ERB0314.pdf
van Harmelen, M. (2006). Personal learning environments. Retrieved October 12, 2006, from University of Manchester, School of Computer Sciences Web site: http://octette.cs.man.ac.uk/jitt/index.php/Personal_Learning_Environments
Hultin, J. (In press). Learning management systems (LMS): A review. Retrieved October 12, 2006, from http://se2.isn.ch:80/serviceengine/FileContent?serviceID=NATOADL&fileid=09FD0FDC-723F-5E63-CCDB-6AFCDA98BCC8&lng=en
Jaschik, S. (2006, July 28). Rallying behind open access. Retrieved October 12, 2006, from http://insidehighered.com/news/2006/07/28/provosts
Johnson, S. (2001). Emergence. New York: Scribner.
Koper, R. (2004). Increasing learner retention in a simulated learning network using indirect social interaction. Retrieved October 12, 2006, from http://dspace.ou.nl/bitstream/1820/249/4/preprint-LNsimulation.pdf
Learning Circuits. (n.d.). Field guide to learning management systems. Retrieved October 12, 2006, from http://www.learningcircuits.org/NR/rdonlyres/BFEC9F41-66C2-42EF-BE9D-E4FA0D3CE1CE/7304/LMS_fieldguide1.pdf
LMS options and comparisons. (2006). WebCT instructor community. Retrieved October 12, 2006, from http://ctlet.brocku.ca/webct/LMS_Options_and_Comparisons
Lyman, P., & Varian, H. R. (2003). How much information? Retrieved October 12, 2006, from http://www2.sims.berkeley.edu/research/projects/how-much-info-2003/
Mullin, S. (2005, December 7). On line learning market may be monopolized. Retrieved October 12, 2006, from http://www.antitrustlawblog.com/article-on-line-learning-market-may-be-monopolized.html
- some differing views exist on how market share is defined, but—for example a CMS (course management system) definition would exclude enterprise-wide vendors like SAP (who are offering their own learning platform), resulting in a WebCT/Blackboard market share in excess of 80%.
Online learning history. (2006). Moodledocs. Retrieved October 1, 2006,
Sessums, C. D. (2006, April 11). Revisioning the LMS: an examination of formal learning management systems and component-based learning environments. http://elgg.net/csessums/weblog/11712.html
Siemens, G. (2004a). ePortfolios. Elearnspace. Retrieved October 12, 2006, from http://www.elearnspace.org/Articles/eportfolios.htm
Siemens, G. (2004b). Learning management systems: The wrong place to start learning. Elearnspace. Retrieved October 12, 2006, from http://www.elearnspace.org/Articles/lms.htm
Vollmer, J. (2003). Debunking the LCMS myth. Retrieved October 1, 2006, from http://www.clomedia.com/content/templates/clo_fairfield.asp?articleid=223&zoneid=13
Wikipedia. (2006a). History of virtual learning environments. Retrieved October 11, 2006, http://en.wikipedia.org/wiki/History_of_virtual_learning_environments
Wikipedia. (2006b). Vendor lock-in. Retrieved October 12, 2006, from http://en.wikipedia.org/wiki/Vendor_lock-in
Wikipedia. (2006c). Virtual learning environment. Retrieved October 12, 2006, from http://en.wikipedia.org/wiki/Virtual_learning_environment
Wise, L., & Quealy, J. (2006). LMS governance project. Retrieved October 12, 2006, from University of Melbourne, Information Services Web site: http://www.infodiv.unimelb.edu.au/telars/talmet/melbmonash/media/LMSGovernanceFinalReport.pdf
Wyles, R. (2004). Evaluation of learning management system software. Retrieved October 12, 2006, from http://eduforge.org/docman/view.php/7/17/Evaluation%20of%20LMS%20-%20Part%20II.pdf