IIER logo 3
Issues In Educational Research, Vol 16, 2006
[ Contents Vol 16 ] [ IIER Home ]

A research methodology: The development of survey instruments for research into online learning in higher education

Lou Siragusa & Kathryn C. Dixon
Curtin University of Technology, Western Australia
As online learning continues to be utilised in diverse ways from courses being delivered fully online through to class websites used to supplement face-to-face classes, and as the technologies continue to develop, research into the effectiveness of this mode of online delivery needs to be an ongoing process. These research findings need to be fed back into the instructional design process for the design and development of pedagogically effective online learning environments. This paper describes a research methodology developed for a recently completed study which examined the instructional effectiveness of various online learning environments. This methodology has the potential to be adapted for future studies into online learning in higher education to assist with maintaining this research momentum. The survey utilised both quantitative and qualitative methods for collecting data. The students and lecturers of the online learning environments studied were asked to complete an online questionnaire and to participate in a semi-structured interview. The survey focused on elements of online learning including content, structure, motivation, feedback, interaction, learning strategies and the lecturer's role. This paper presents a sample of the literature which underpinned this study, describes the design and development process of the survey instruments and presents a sample of the survey findings.

Introduction

The current drive to integrate the Internet for learning in higher education courses is relentless. This diverse use of the Internet ranges from courses being delivered entirely online in distance education mode through to online learning being used to supplement face-to-face classes. Online learning technologies are also continually being developed and utilised in various ways including proprietary online learning management systems (eg, WebCT, Blackboard, etc.) and learning object technologies which have the potential to accommodate individual learning styles (Martinez, 2001). As online learning technologies continue to develop, the application of effective instructional design principles based upon sound learning theories must also be integral to the development and delivery of this mode of delivery (Ally, 2004; Bennett, Priest & Macpherson, 1999; Chen, 1998; Winfield, Mealy & Scheibel, 1998). Gaps currently exist between the bodies of knowledge relating to learning theories, instructional design principles and research into student learning in higher education and the application of these bodies of knowledge to the use of online learning technologies (Siragusa & Dixon, 2005). The development of web-based learning is often carried out without consideration for the instructional and pedagogical needs of the students (Weller, 2002).

With this wide variation of the use of the Internet and related technologies for learning, a significant part of the ongoing development of online learning environments should be to evaluate their pedagogical effectiveness. The following will describe the methodology and development of survey instruments for a recently completed study which examined the pedagogical effectiveness of web-based learning environments in higher education (Siragusa, 2005). The study involved the administration of a survey to students who were engaging in units that used web-based learning environments in the five major universities in Western Australia. A similar investigation was carried out with a sample group of lecturers from the same universities who were creating learning materials for delivery in online environments. The objectives of this research were to: 1) identify factors that impact upon effective instructional design in web-based learning; and 2) identify learning strategies that influence successful student learning in web-based learning environments. This paper describes the research methodology and the process used to develop the research instruments that were employed in achieving these objectives.

Background

Online learning technologies offer the potential for students to study within an environment in ways which is often not possible through the use of other forms of distance education technologies (Simonson, Smaldino, Albright & Zvacek, 2006). In addition to providing a complete multimedia environment in which learners can interact with the learning materials as with other media (eg, CD-ROM), online learning technologies also provide opportunities for students and their lecturers to interact in various ways with each other utilising a variety of online communication facilities. This accessibility to a variety of interaction has the potential to see the emergence of new teaching practices and student learning strategies which can enhance the learning experience. Therefore, there is a need to identify appropriate instructional design principles, based upon a sound theoretical framework, which can facilitate such teaching practices and student learning strategies suitable for online learning. In order to identify these instructional design principles, a research methodology was developed in a recently completed investigation into online learning (Siragusa, 2005). This methodology accommodated the distinct features of online learning which allows for student interaction with each other and their lecturer as well as with the learning materials. These features include, for example, online discussions, screen design and technical ability.

The study put forward a theoretical framework which underpinned the development of this research methodology for examining instructional design principles and student learning strategies that enhance online learning in higher education. This theoretical framework was derived from the following three main discipline areas which have relevance for online learning: 1) learning theories, philosophies and instructional design; 2) research into student learning in higher education; and 3) online learning technologies. An examination of the literature within these three main discipline areas revealed seven interrelated focus areas which were identified as having direct influence on the design of effective online learning environments. These seven key focus areas have been categorised as structure, content, motivation, feedback/help, interaction, learning strategies, and lecturer's role. The following is a sample of studies, categorised under these key focus areas, which examined students' perceptions of the effectiveness of their online learning environments.

1. Structure

A study by Reisetter and Boris (2004) which examined students' perception of effective elements of online learning revealed that 96% of the students surveyed (n = 59) believed that the structure and coherence of their online course was very or somewhat important and that expectations had to be explicit. Chandler and Maddux (1998) conducted a large scale study where the structure and appearance of their online learning environment was investigated. The notion of structure involves the way in which information is displayed and is a major factor in the successful utilisation of web-based programs by students. The research by Chandler and Maddux (1998, p.1059) asked students (n = 249) what features of the website were available and provided their responses as a percentage, which included graphics (58%), variety of colour (52%), sound (12%), animation (13%), lecturer notes (34%), homework assignments (69%), class schedule (77%), instructor's office hours (77%) and links to other relevant sites (49%).

Hughes, McAvinia and King's (2004) study asked students in focus groups to identify elements of websites that they liked. Predictably, the students unanimously agreed that too much text on a page was off-putting and that websites with more visual appeal were desirable. They also liked websites with good content that was easy to scan without having to read in detail. While the student respondents liked numerous links on websites, they wanted to know what they would find before clicking on the links. They liked websites with colour, but not too much "in-your-face" colour such as "blinking" images. The students also preferred simple web page layouts without cluttered information. They did not like annoying sounds and slow loading Flash applications.

2. Content

Schlough and Bhuripanyo (1998) examined the instructional effectiveness of their online learning environment's content which included the subject/course content, assignments, activities, case studies, lecturer/tutorial/laboratory notes, reading materials, tests, and so on. The content of the course unit that is to be delivered to students is a major focus of the instructional design phase of an online learning environment. The content is the result of research into the subject matter which the students actually need to learn as determined by the learning outcomes for that unit.

A study by Swan, Bowman, Holmes, Sylvie and Vargas (1998-99, pp.98-9) found that the characteristics of the World Wide Web "text" itself seemed to affect the students' reading behaviours more strongly than what might be expected with printed texts. Students were observed to have rejected pages because they were "just text." Embedded graphics and interesting page designs received favourable responses from the students. On the other hand, MacDonald and Mason's (1998, p.41) enquiry found that half of the students (n = 21) did not like reading course content from the screen, and nine students had difficulties finding relevant information on the Internet due to Internet connection costs and validity of the information found. However, Renn and Zeligman's (2005) study found that students' (n = 17) level of comfort with using their library website for finding research and professional literature had increased over the surveyed semester.

3. Motivation

Summerville (1998, pp.431-7) conducted a study to explore how the online learning environment and study materials could be made appealing and interesting for students, including the appearance of the website, the use of text and graphics, and the amount of materials presented on each page. Agarwal and Day (1998) investigated students' perception of their instructor's effectiveness in a class group using the Internet and another group not using the Internet (approximately 40 students in each group). The Internet group reported higher instructor's stimulation of interest, use of class time, communication of ideas, provision of feedback, and interest in the students. Everett (1998, pp.126-8) surveyed students (n = 290) studying online and found that the instructors needed to address students' apprehension, fears and coping mechanisms by introducing students to the online technology to allay their fears. In Reisetter and Boris' (2004) study, more than 50% of the surveyed students (n = 59) perceived that more effort was required in an online class than in a traditional face-to-face classes. Students could not rely upon the presence of a classmate or teacher for immediate answers to questions.

4. Feedback/help

According to Thomas, Carswell, Price and Petre (1998), feedback/help is an essential component of effective online learning environments. Feedback is critical in order for students to measure their progress. Web-based learning environments often provide students with a mechanism to submit their assignments and to receive prompt feedback. The study conducted by Reisetter and Boris (2004) showed that, although 42% of the students surveyed (n = 59) believed that the availability of technical assistance was very important, only 8% used it frequently or often. Thomas et al. (1998, p.155) discussed disadvantages of distance education. These disadvantages included students not knowing if an assignment had been received, lost assignments, turn-around time reducing the impact of remedial advice, and the variability of communications systems world-wide making it difficult for students in some countries to study effectively.

5. Interaction

Wagner's (1998, pp.418-20) critical investigation into the importance of interaction described different types of interactions categorised by focusing on what learners are to achieve as a result of an interaction. These dimensions included interaction for participation, communication, feedback, collaboration, learner self-regulation, motivation, negotiation, team-building, discovery, exploration, clarification, and disclosure. Brannan's (2005) study indicated that students felt that their instructors encouraged more interaction in technologically-mediated courses; the interaction ratings increased progressively from non-technological delivery in the class, to some technology delivery, through to courses delivered entirely via technology. Another study by Angeli, Bonk, Supplee, and Malikowski (1998, pp.104-9) was concerned with how discussions spur on other discussions and how peer responsiveness affects the depth of dialogue. In this computer conferencing system, students were sharing problems, asking for help, and offering advice.

6. Learning strategies

Smith and Ragan (2005, p.225) contended that through the process of thoughtful instructional design, effective learning strategies for the students can be developed. They also argued that if learners are using strategies which lead to successful and efficient learning, they should be encouraged to continue using them.

Hawkes, Cambre and Lewis (1998, p.7) examined teacher-perceived effectiveness of students' online learning and found that, of the 62 teachers who returned the questionnaire, 96% believed that students were better at working collaboratively with peers, 92% thought that students had greater access to current information and a wider base of information, 87% indicated that students took more responsibility for their own learning, and 81% believed that students applied themselves for longer periods of time. The majority of students in Reisetter and Boris' (2004) study emphasised the importance of self-directed learning and personal responsibility for success in online learning. Jewett's (1998, pp.15-20) study administered a survey to 105 students enrolled in an online delivered course designed to examine students' reaction to the technology, their opinions about its usefulness towards success in the course and their learning styles. Of the 105 students who responded to the questionnaire, 76% believed that online discussions were important to the learning process, and 42% indicated that they had more interaction with their peers in the online class than in other classes.

7. Lecturer's role

While the focus of the literature presented above is on elements relating to the design of effective instruction, the role of the lecturer is also important and needs to be examined. The lecturer's willingness to participate in online learning can influence students' motivation to successfully learn with this medium (Weller, 2002, p.51). A comprehensive study into online learning in schools by Ravitz (1998, pp.323-32) asked teachers questions relating to the conditions in which they facilitated online learning and how they perceived their roles and skills while teaching with this medium. Five major components of the online teacher's role were identified by Ravitz, which include: 1) the teacher's perception of the importance of the use of the Internet for student learning; 2) their skills and ability to utilise the online learning technologies; 3) the technical support provided to teachers and students and incentives for incorporating online learning into the teaching programme; 4) their involvement in the decision-making process regarding online learning development; and 5) their involvement in design and development of online learning environments. Segrave, Holt and Farmer (2005) identified six professional capabilities required for the design and delivery of online learning environments including: 1) designing for learning online; 2) communicating, collaborating and online community development; 3) assessing student learning online; 4) developing online learning resources; 5) experiential online learning; and 6) continuous online quality improvement.

The brief overview of the existing literature that has been presented here has provided a theoretical framework which has the potential to underpin the development of survey instruments for research into the pedagogical effectiveness of online learning systems. Based upon this overview, the following prescribes a methodology for identifying the effectiveness of online learning.

Methodology

Researchers have commonly argued that good research practice involves the use of multiple methods to enhance the reliability and validity of the research findings (Mathison, 1988). The term triangulation was first coined by Webb, Campbell, Schwartz and Sechrest (1966) to describe this use of multiple research methods. Methodological triangulation was described by Denzin (1970) as one of the six principle types of triangulation. This type of triangulation allows for different research methods to be employed on the same object of study as a check on validity (Cohen & Manion, 1994, pp.236-8).

Collecting students' perceptions of the effectiveness of their class website was an achievable and practical method of inquiry which was carried out without interfering with their study routines. Students within a learning environment are in a good position to evaluate instruction because of their experience with many other learning environments (Fraser, 1998). The quantitative data collected from students and lecturers through online questionnaires allowed for summary statistical indices of the phenomena studied. The qualitative data collected through semi-structured interviews and questionnaires assisted to clarify and support the quantitative data. After the data had been collected, an analysis leading to discussions about the instructional effectiveness of the students' online learning environments was carried out.

Design of survey instruments

As the survey involved the use of questionnaires and interviews, a bank (or collection) of question items suitable for answering the research questions was required. The design and development of the survey instruments took place in six distinct stages. Firstly, a definition of the instructional principles that informed the study was developed. This process involved the examination of an existing model of systematic instructional design. Secondly, essential items from the instructional design model was extrapolated and used as criteria for examining the effective instructional design principles for online learning. Thirdly, a further examination of the existing literature that reported on the use of questionnaires into online learning environments was carried out to extract appropriate questions. Fourthly, the criteria derived from the existing instructional design models and questionnaires was further developed into questions and organised into appropriate survey dimensions. Fifthly, the questionnaires and interview schedules to be administered to students and lecturers were designed and constructed. Finally, the questionnaires and interview schedules were piloted and revised.

An instructional design model

To achieve the current study's objectives, a clear definition of the term instructional design was required. The literature suggests that designing instruction involves a systematic process which translates principles and instruction into materials for activating and supporting learning (Gagné, Briggs & Wager, 1992, p.4; Smith & Ragan, 2005, p.6). In the case of developing materials for online learning, the instructional designer needs to systematically examine the learning materials provided by content experts. Usually the content material is in printed form or produced electronically from a word processor needing translation into a form that is suitable for placement on the Internet. While learning materials must be suitable for viewing on the Internet, the materials must also effectively activate and support the learning of the students. Consideration of learning theories and philosophies also need to be integrated into the instructional design process.

Numerous models of instructional design have been developed for the purpose of analysis, strategy development and evaluation of course and lesson design (Smith & Ragan, 2005, p.10). A widely accepted model is the Dick, Carey and Carey (2005) Systems Approach Model for Designing Instruction which outlines each stage of the instructional design process including analysis of the learners, the learning context and the instructional goals, instructional strategy development, evaluation and revision. This process can accommodate nearly all instructional needs and modes of delivery.

Development of question items from an existing instructional design model

Through the identification of the essential criteria of each stage of Dick, Carey and Carey's model, question items could be developed. The task, therefore, was to create questions that solicited students' perceptions of how well instructional design principles had been applied to their online learning environment. The questions had to avoid specific instructional design discipline terminology while asking them to determine the effectiveness of the online instructional materials that were delivered to them.

From the criteria outlined for each stage of Dick, Carey and Carey's instructional design model (Dick et al., 2005, pp.6-8), questions were developed. These questions assisted with collecting students' perceptions of the effectiveness of each stage of the instructional design process. As the questionnaire predominately used a five-point Likert-scale style format, the majority of questions were written as attitude statements. This format allows students to place themselves on an attitude continuum for each statement - ranging from 'strongly agree', 'agree', 'uncertain', 'disagree' to 'strongly disagree' (Oppenheim, 1992, p.195). Students were asked to respond to items which explored their reaction to a number of elements concerning online learning. Students in the sample were asked, for example, to indicate whether online components helped with their learning, whether their prior knowledge was adequate in order to deal with the requirements of the online learning environment, whether the unit objectives were clearly indicated, whether they had clearly understood the pre-requisites, whether the learning materials were well organised, whether they had participated in activities such as group discussions, whether adequate consideration was made with the development of the online instructional materials , whether there were difficulties with learning the new information, what were the strengths and weakness of the content and whether it was easier to study with a unit with online support.

Literature review informing survey dimensions

The Background section of this paper provided a sample of the literature relating to studies carried out to determine the effectiveness of online learning. Questions from these studies were then utilised in the development of the survey instruments for the current research enquiry. The literature review assisted with the creation of the dimensions used in the survey instruments administered to students and lecturers. The dimensions which were developed for use in the student survey included: structure, content, motivation, feedback, interaction and learning strategies. The lecturer survey contained dimensions derived from the literature review which included: importance, abilities, support, decision-making and development activities.

These dimensions were further reduced into individual elements. These elements represent the draft form of the questions that eventually appeared in the survey. These questions, in combination with the questions derived from instructional design models, formed the bank of questions used in this research.

Dimensions and elements for survey questions

The elements presented in the following are grouped within the survey dimensions which appeared in the survey. With each of the dimensions and their elements identified, the questionnaires and interview schedules were then created.

Survey dimensions and elements (student)

The structure dimension included elements such as: general look and feel, colour scheme, textual styles, amount of text on each screen, readability, graphics, sound, animation, visual clues, location of content, organisation of the material, sequence of the material, navigation, and student directed or teacher directed.

The elements for the content dimension included: clearly stated learning objectives and assessment criteria, enough information provided to cover the learning objectives and assessment criteria, understanding what is required from students, clarity, relevance, accuracy, completeness, spelling/grammar, content found on the Internet, and enough information to complete the assignments.

Within the motivation dimension, the elements included: pleasing appearance, information is understandable, graphics clarifies the topics, amount of new material, relates to previous knowledge, information presented is meaningful, amount of printing required, time needed to figure out how to use the online environment, promotes confidence in the use of the Internet, provision of online help, promotes learning, self-paced, timely presentation of the materials, meeting students' expectations and needs, provision of online help, online collaboration with other students, interest expressed by online lecturer, and flexibility (anywhere, anytime).

The elements within the feedback dimension included: provision of online support, submission of assignments, receipt of assignments notification, prompt feedback of assignments, assistance from online lecturer, and feedback on course performance from online lecturer.

Within the interaction dimension, the elements included: encouraged to ask questions with online lecturer and with other class members, discussion with other class members and online lecturer, sharing of ideas, reading and responding to other students' answers to discussion questions, value of communication with class members and online lecturer, use of email and bulletin board, use of IRC (chat) and other communication tools, adequate amount of communication to assist with succeeding in the unit, and communications with online lecturer and class members to increase motivation to succeed.

The elements for the learning strategies dimension included: transfer of previous knowledge, integrate real-life situations, students taking responsibility for their own learning, sharing problems with other students, finding information on the Internet, search strategies for finding relevant information, learning through self-exploration, online discussions used to think about the unit topics, having time to think about what is learnt, mental imagery tactics for recognising problem types or solution types, methods for providing external storage to deal with memory limitations and specific strategies for presenting problems or retrieving solutions for the particular domain.

Survey dimensions and elements (lecturer)

The elements within the importance dimension included: using the Internet prepares students for life long learning, online teaching increases students motivation, online skills learned are transferable to other units, online learning makes students active class members, new teaching practices have come from online learning, online learning influences and supports larger departmental changes and directions, and use of the Internet overcomes problems of remoteness for external students.

Within the abilities dimension, the elements included: confidence with creating simple and complex web pages, confidence with use of HTML authoring and graphics software, confidence with use of online learning management systems, and time spent learning how to develop online learning materials.

The support dimension included the following elements: availability of technical support, availability of training for online learning delivery skills, availability of online curriculum resources, availability of help for integrating online activities into the curriculum, availability of release time for online development activities, and recognition of leadership and for helping other lecturers use the Internet for learning.

Within the decision-making dimension, the elements included: availability of colleague to discuss online learning issues, opportunity exists to voice concerns regarding online learning issues, opinions are sought after regarding decision making for online learning development, availability of finance for continuing online learning development, and high priority for online development within department.

The elements with the development activities dimension included: involvement in selecting and purchasing hardware/software products for online learning, involvement with integrating online learning into the curriculum, involvement with online learning planning committees, involvement in development of online learning products, involvement in development of policies for online learning, involvement in seeking instructional design advice, involvement in the development of online learning strategies, others are involved in the development of the online learning, students provide feedback about the online learning, and which learning strategies are encouraged.

Questionnaire format and design

The student and lecturer questionnaires were designed to be administered online and were, therefore, created as HTML forms using the Dreamweaver software (http://adobe.com/). The questionnaire forms were placed on a Curtin web server for students and lecturers to access through the Internet. A PHP (Hypertext Preprocessor, http://www.php.net/) script was used to allow the results of the questionnaires to be submitted to the researcher via email in the required format. After answering the questions, the respondents submitted the questionnaire by clicking on the Submit the questionnaire button at the end of the questionnaire which enabled the PHP script to send the results. The questionnaire results were transferred into a Microsoft Excel spreadsheet ready for analysis.

The quantitative nature of this research required that the data be collected in a format which allowed for analysis through software such as SPSS (http://www.spss.com/) and LERTAP (http://www.lertap.curtin.edu.au/). Many of the questions were in a Likert-type five point format that asked if the respondents 'Strongly disagree' (1), 'Disagree' (2), 'Uncertain' (3), 'Agree' (4) or 'Strongly agree' (5) with the given statements. There were spaces available for students and lecturers to answer open-ended type questions. Multiple choice type questions were also provided on the form.

Survey piloting

As the student questionnaire was to be administered to a large group of students to collect a significantly large amount of data, piloting was an important procedure in the development of the questionnaire. Students from the Humanities Division in one of the universities in the sample were selected to pilot the student questionnaire. Three lecturers agreed to ask their students to participate in the questionnaire pilot. A total of 29 of these students returned the questionnaire with their comments. There were students from three traditional face-to-face class groups who participated. These classes were supported with a class website to provide online access to the unit materials and communication devices (email and bulletin board). These class groups were considered typical samples of the overall population to which the survey was to be administered. The pilot was announced to students during classes and some students were given time to complete the questionnaire during their scheduled class.

The comments from the students about the questionnaire were generally positive. They appeared to respond well to the online format of the questionnaire and felt that the questions were well targeted to online learning. The single major concern students raised regarded the length of the questionnaire. The piloted questionnaire contained 139 question items. Although some students commented that it was a quick process to complete the online questionnaire by clicking on radio buttons and checkboxes, it was clear from other responses that the number of items in the questionnaire needed to be reduced. The final version of the questionnaire contained 80 items in total. There were only a small number of comments about the questions presented. Some questions were considered repetitive and not suitable for the Likert-type options. As the questionnaire needed its length to be reduced, many of these types of questions were omitted in the final version. The final version of the student questionnaires can be found at the following URL: http://members.optushome.com.au/~lsiragusa/pages/resources.htm.

Three lecturers from the Division of Humanities at Curtin University completed the pilot version of the online lecturer questionnaire. These lecturers had taught numerous classes which were supported with a class website and were, therefore, considered a typical sample of lecturers exposed to and using online learning in higher education. Generally, the questionnaire was well received without difficulty in using the online format. The pilot questionnaire contained 65 items. There were a few comments from the participants relating to the wording of questions which may have had ambiguous meanings, as well as the comments relating to the possible repetitiveness of some questions. Questions which were seen as repetitive were removed from the questionnaire and ambiguous questions were reworded. The final version of the lecturer questionnaire contained a total of 55 question items. The final version of the lecturer questionnaires can be found at the following URL: http://members.optushome.com.au/~lsiragusa/pages/resources.htm.

The semi-structured interviews for this survey were conducted via telephone, allowing the respondents to expand upon any part of the online learning experience they encountered. Therefore, it did not seem necessary to go through a formal pilot phase for the interview schedules. As the amount of volunteering participants for the interview was limited, it did not seem warranted to use these willing participants for piloting. The final question on both the student and lecturer online questionnaire asked if they would participate in a follow-up interview. By the time the interviews were administered, the students and lecturers had already completed the online questionnaires and were, therefore, familiar with the types of questions being asked in the interviews. After conducting one or two student and lecturer interviews, minor changes to the interview schedules were made. Refer to the following URL to view the final version of the student and lecturer interview schedules: http://members.optushome.com.au/~lsiragusa/pages/resources.htm.

Survey administration and findings

The completed survey was designed to include as many students and lecturers studying and working with class websites from all five major universities in Western Australia. The student online questionnaire was administered during semester two 2001 and semester one 2002. The final estimated response rate was approximately 20% of approximately 1350 students across these universities who were specifically asked to participate. This response rate was disappointing and needs to be acknowledged as a limitation of this study. Only students who expressed a willingness to participate were involved in the study, which had the potential to introduce bias in the sample. Another possible limitation is that this survey explored online learning in its broadest form to accommodate the wide uses of this media in higher education; this has the potential to limit some of the findings and conclusions that could be drawn.

The lecturers, ten in total, from these class groups who participated in the survey completed the lecturer online questionnaire. A total of 25 students who completed the student online questionnaire participated in a semi-structured telephone interview, and five of the lecturers who completed the lecturer online questionnaire also participated in a semi-structured interview. The following presents the major findings from the student and lecturer survey.

Student survey findings

The responses to the Likert-type questions for the structure, content, motivation, feedback, interaction and learning strategies dimensions follow a typical pattern. Most students selected the Agree option, some selected Strongly Agree, the rest dropped off towards Disagree, and very few selected the Strongly Disagree option. This seemed to indicate that, generally, students were reasonably satisfied with their websites. A closer look at the questions revealed specific areas of weaknesses. For example, 51% of students agreed that the learning was provided just when it was needed, while 49% of students indicated that they disagreed with or were uncertain about this statement. Students also appeared equally divided about whether or not they had received adequate prompts, feedback or assistance from their lecturer and whether or not they had found the online discussions to be valuable. These responses indicate areas needing attention and a need for further investigation into the variety of weakness and strengths of the class websites students were using. Other areas of weakness included the inappropriate use of graphics and icons, lack of use of the Internet for finding additional information and many of the students not being able to work collaboratively with other students. Approximately 78% of students indicated that they enjoyed learning through their own experiences, while 65% agreed that they enjoyed learning with other students. Approximately 50% of the class websites were used as a repository of information for the students to access, while the remaining 50% completed interactive learning activities on the Internet.

The qualitative data collected from the student questionnaire and the semi-structured telephone interviews highlighted what factors contributed towards students' satisfaction and dissatisfaction with online learning. Examples of these factors include technical problems with making initial access to the class website, technical problems with the electronic assignment submission system, employment of sound web page and instructional design principles, access to unit information such as unit outline and assignment requirements, allowing students to study when and where they choose, allowing students to study at their own pace, effective use of online communication facilities such as the class bulletin board and email, feelings of isolation for distance students, providing students with prompt notification of receipt of submitted assignments, and providing students with adequate feedback on their returned assignments.

Lecturer survey findings

Although most (75%) lecturers appeared to be reasonably positive towards using the Internet for teaching, their responses to the Likert-type questions for importance, abilities, support, decision-making and development activities highlighted several areas of concern. For example, when asked if they found teaching with the Internet rewarding and enjoyable, there was a rather low score (mean = 2.90). Items relating to exploration of new teaching practices for online learning, the availability of curriculum resources for using the Internet for teaching, the perception that teaching with the Internet was rewarding and enjoyable, recognition of assisting other lecturers to use the Internet for teaching, and the availability of a significant budget for online learning development all scored rather low. This may indicate that these lecturers were pressed to develop materials for online learning with little support and recognition for their efforts. The majority of lecturers, however, were reasonably positive about the importance of keeping up with current technologies and with their abilities to create simple web pages. Approximately 70% of lecturers were reasonably positive about keeping up with current computing and Internet technologies, while at least 80% had access to training for the development of their online learning skills and access to a trusted colleague for voicing their concerns.

The lecturer questionnaire and semi-structured interview revealed what elements of online learning lecturers consider to be important. Examples of these elements include exploring how online learning may be effectively utilised for their students' learning experiences (eg, automating routine tasks such as administering quizzes), accommodating various student learning styles through encouraging appropriate learning strategies, providing equity of computing and Internet access, providing additional time and financial support to compensate for increased workloads for developing and delivering online learning, ability to create and edit simple HTML pages, familiarity with the features within the online LMS for enhancing student online learning, employment of effective teaching methods which facilitate appropriate online learning strategies, recognition for lecturers involved in effective online learning development and for supporting other lecturers. Other elements included the provision of online LMS technical support assistance at school or departmental level, coordination of online learning development activities within schools or departments, awareness of the decision-making process and funding opportunities within the university. Lecturers also considered encouraging effective learning strategies allowing students to provide feedback about the unit's content and the assignments, encouraging lecturers involved in innovative online learning development activities to conduct forums to display exemplary samples of online learning, allowing lecturers to explore the possibilities of online learning development and delivery without being pressured, and making lecturers aware of the discipline relating to instructional design and how it is applied to creating pedagogically sound online learning environments as important elements.

Conclusion

This paper has argued that, with the varied use of web-based learning in higher education courses, constant monitoring to determine its pedagogical effectiveness is needed. The use of online learning technologies in higher education provides opportunities for the development of teaching practices and student learning strategies which are often not possible through the use of other forms of distance education technologies. In order to identify these practices and strategies, a research methodology that is grounded in online learning as a mode of course delivery is needed which differs from other research methodologies for examining the effectiveness of other modes of delivery. This study has developed a methodology for examining online learning which occurs in various forms in higher education. The methodology clarifies factors that contribute to effective instructional design principles and learning strategies that influence successful student learning in web-based environments (Siragusa, 2005). Methods of investigating the nature of and structures involved in online learning environments as opposed to more traditional means of distance learning are essential. Rapid advancements in current technology have meant that flexible delivery has advanced equally rapidly, offering students wide choices in learning methods. This paper has examined amongst other things, the challenges for facilitating online learning and the ground level components for a model which clearly indicates a sound approach for techniques which can be used to assist students to learn in this unique environment. The survey dimensions that have been developed in this research, which provide the foundation of a model for teaching and learning, are unique to the online environment as they have been tested and developed in this instance with those working in online contexts. The student and lecturer surveys that were developed from these dimensions are sufficiently balanced so that they may be utilised together for evaluating online learning and teaching effectiveness as well as providing a concise check list for unit design purposes.

Students who engage in online learning environments face issues and contexts that do not concern those who are studying in more traditional paper-based distance education. As online learning continues to expand and technologies continue to be developed, these issues require ongoing investigation which employ appropriate research methodologies including the methodology presented in this paper. Other researchers are continuing to develop specific methodologies into the examination of effective online learning such as the investigation of students' perceptions of online learning environments (Trinidad, Aldridge & Fraser, 2005), academics' professional capacities to educate online (Segrave, Holt & Farmer, 2005), online collaboration and community building (Jackson & Schaverien, 2005), learning through online discussions (Hammond & Wiriyapinit, 2005), authentic and deep learning in mixed mode environments (Campbell, Frost & Logan, 2005), and so forth. While issues surrounding technologies and technology use are dramatically altering all areas of education and training in Australia, online delivery is still in a relatively embryonic stage. Future research will further develop the instructional design model for online learning as distinct from distance education per se and will therefore enhance and facilitate appropriate pedagogical approaches for the innovative use of web-based learning in higher education and the overall maturation of online delivery of learning into the twenty first century.

References

Agarwal, R. & Day, E. (1998). The impact of the internet on economic education. The Journal of Economic Education, 29(2), 99-110.

Ally, M. (2004). Foundations of educational theory for online learning. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning. Athabasca, Canada: Creative Commons: Athabasca University. http://cde.athabascau.ca/online_book/

Angeli, C., Bonk, C. J., Supplee, L. & Malikowski, S. (1998). A case based electronic learning environment for preservice teacher education. Paper presented at the SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceeding, U.S., Indiana. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED421156

Bennett, S., Priest, A. & Macpherson, C. (1999). Learning about online learning: An approach to staff development for university teachers. Australian Journal of Educational Technology, 15(3), 207-221. http://www.ascilite.org.au/ajet/ajet15/bennett.html

Brannan, T. A. (2005). Learner interactivity in higher education: Comparing face-to-face, hybrid, and online instruction. Distance Learning, 2(2), 1-8.

Campbell, M., Frost, D. & Logan, J. (2005). Mixed-mode learning for students of school counselling. Peer reviewed paper presented at the Australian Association for Research in Education (AARE) 2005 Education Research - Creative Dissent: Constructive Solutions, New South Wales, Parramatta. http://www.aare.edu.au/05pap/cam05609.pdf

Chandler, B. & Maddux, C. D. (1998). Student use of instructors' Web sites. Paper presented at the SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceedings. U.S., Nevada, Washington, DC. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED421152

Chen, L. L. (1998). Web-based distance instruction: Design and implications of a cybercourse model. Paper presented at the Proceedings of Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT) Sponsored by the Research and Theory Division (20th, St. Louis, MO, February 18-22, 1998), U.S., Michigan. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED423828

Cohen, L. & Manion, L. (1994). Research methods in education (4th ed.). London: Routledge.

Denzin, N. K. (1970). The research act in sociology: A theoretical introduction to sociological method. London: The Butterworth Group.

Dick, W., Carey, L. & Carey, J. O. (2005). The systematic design of instruction (6th ed.). New York: Pearson Allyn & Bacon.

Everett, D. R. (1998). Taking instruction online: The art of delivery. Paper presented at the SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceedings, U.S., Kentucky. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED421094

Fraser, B. J. (1998). Classroom environment instruments: Development, validity and applications. Learning Environments Research, 1, 7-33.

Gagné, R. M., Briggs, L. J. & Wager, W. W. (1992). Principles of instructional design (4th ed.). Fort Worth: Holt, Rinehart & Winston, Inc. Harcourt Brace College Publishers.

Hammond, M. & Wiriyapinit, M. (2005). Learning through online discussion: A case of triangulation in research. Australasian Journal of Educational Technology, 21(3), 283-302. http://www.ascilite.org.au/ajet/ajet21/hammond.html

Hawkes, M., Cambre, M. & Lewis, M. (1998). The Ohio SchoolNet telecommunity evaluation. Paper presented at the North Central Regional Educational Lab., Oak Brook, IL. U.S., Illinois, Oak Brook, IL. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED416830

Hughes, J., McAvinia, C. & King, T. (2004). What really makes students like a web site? What are the implications for designing web-based language learning sites? ReCALL, 16(1), 85-102.

Jackson, K. & Schaverien, L. (2005). Developing research designs and methodologies for investigating learning in postgraduate e-learning contexts. Peer reviewed paper presented at the Australian Association for Research in Education (AARE) 2005 Education Research - Creative Dissent: Constructive Solutions, New South Wales, Parramatta. http://www.aare.edu.au/05pap/jac05488.pdf

Jewett, F. (1998). Course restructuring and the instructional development initiative at Virginia Polytechnic Institute and State University: A benefit cost study. Paper presented at the Case Studies in Evaluating the Benefits and Costs of Mediated Instruction and Distributed Learning. Virginia Polytechnic Institute and State University (California State University, 1998), California State University. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED423802

MacDonald, J. & Mason, R. (1998). Information skills and resource-based learning in an Open University course. The Journal of Open and Distance Learning, 13(1), 38-42.

Martinez, M. (2001). Designing learning objects to personalize learning. Retrieved 11th July, 2002, from http://reusability.org/read/chapters/martinez.doc

Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13-17.

Oppenheim, A. N. (1992). Questionnaire design, interviewing and attitude measurement (New ed.). London: Continuum.

Ravitz, J. (1998). Conditions that facilitate teachers' internet use in schools with high internet connectivity: Preliminary findings. Paper presented at the Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT). Sponsored by the Research and Theory Division (20th, St. Louis, MO, February 18-22, 1998), U.S., Massachusetts. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED423855

Reisetter, M. & Boris, G. (2004). What works: Student perceptions of effective elements in online learning. Quarterly Review of Distance Education, 5(4), 277-291.

Renn, K. A. & Zeligman, D. M. (2005). Learning about technology and student affairs: Outcomes of an online immersion. Journal of College Student Development, 46(5), 547-555.

Schlough, S. & Bhuripanyo, S. (1998). The development and evaluation of the internet delivery of the course "Task analysis". Paper presented at the SITE 98: Society for Information Technology & Teacher Education International Conference (9th, Washington, DC, March 10-14, 1998). Proceedings, U.S., Wisconsin. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED421089

Segrave, S., Holt, D. & Farmer, J. (2005). The power of the 6three model for enhancing academic teachers' capacities for effective online teaching and learning: Benefits, initiatives and future directions. Australasian Journal of Educational Technology, 21(1), 118-135. http://www.ascilite.org.au/ajet/ajet21/segrave.html

Simonson, M., Smaldino, S., Albright, M. & Zvacek, C. (2006). Teaching and learning at a distance: Foundations of distance learning (3rd ed.). Upper Saddle River, New Jersey: Pearson Education, Inc.

Siragusa, L. (2005). Identification of effective instructional design principles and learning strategies for students studying in Web-based learning environments in higher education. Unpublished PhD thesis, Curtin University of Technology, Perth. http://adt.curtin.edu.au/theses/available/adt-WCU20051005.143925/

Siragusa, L. & Dixon, K. (2005). Closing the gap between pedagogical theory and online instructional design: A bridge too far? In G. Chiazzese, M. Allegra, A. Chifari & S. Ottaviano (Eds.), Methods and Technologies for Learning. Southampton: WIT Press.

Smith, P. L. & Ragan, T. J. (2005). Instructional design (3rd ed.). Hoboken, NJ: John Wiley & Sons, Inc.

Summerville, J. (1998). The role of awareness of cognitive style in hypermedia. Paper presented at the Proceedings of Selected Research and Development Presentations at the National Convention of the Association for Educational Communications and Technology (AECT) Sponsored by the Research and Theory Division (20th, St. Louis, MO, February 18-22, 1998), U.S., Kansas. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED423865

Swan, K., Bowman, J., Holmes, A., Sylvie, S. & Vargas, J. (1998-99). "Reading" the Web: Making sense on the information superhighway. Journal of Educational Technology Systems, 27(2), 95-104.

Thomas, P., Carswell, L., Price, B. & Petre, M. (1998). A holistic approach to supporting distance learning using the internet: Transformation, not translation. British Journal of Educational Technology. Open University (Great-Britain), 29(2), 149-161.

Trinidad, S., Aldridge, J. & Fraser, B. (2005). Development, validation and use of the online learning environment survey. Australasian Journal of Educational Technology, 21(1), 60-81. http://www.ascilite.org.au/ajet/ajet21/trinidad.html

Wagner, E. D. (1998). Interaction strategies for online training designs. Paper presented at the Distance Learning '98. Proceedings of the Annual Conference on Distance Teaching & Learning (14th, Madison, WI, August 5-7, 1998), U.S., California. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED422881

Webb, E. J., Campbell, D. T., Schwartz, R. D. & Sechrest, L. (1966). Inobtrusive measures. Chicago: Rand McNally.

Weller, M. (2002). Delivering learning on the Net: The why, what & how of online education. London: Kogan Page.

Winfield, W., Mealy, M. & Scheibel, P. (1998). Design considerations for enhancing confidence and participation in web based courses. Paper presented at the Distance Learning '98: Proceedings of the Annual Conference on Distance Teaching & Learning (14th, Madison, WI, August 5-7, 1998), U.S., Wisconsin. http://eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED422885

Authors: Dr Lou Siragusa lectures in the Training and Development Program and Learning Technologies area in the Department of Education at Curtin University of Technology. Email: l.Siragusa@curtin.edu.au

Dr Kathryn Dixon is the Coordinator of the Training and Development Program in the Faculty of Education at Curtin University of Technology. Email: k.dixon@curtin.edu.au

Please cite as: Siragusa, L. & Dixon, K. C. (2006). A research methodology: The development of survey instruments for research into online learning in higher education. Issues In Educational Research, 16(2), 206-225. http://www.iier.org.au/iier16/siragusa.html

[ Contents Vol 16 ] [ IIER Home ]
© 2006 Issues In Educational Research. This URL: http://www.iier.org.au/iier16/siragusa.html
Created 14 Oct 2006. Last revision: 14 Oct 2006.
HTML: Clare McBeath [c.mcbeath@bigpond.com] and Roger Atkinson [rjatkinson@bigpond.com]