Reliability and Validity of a Questionnaire to Estimate Satisfaction Level of Cardiac Critical Care Nursing Students in with Virtual Education COVID-19 Pandemic Era

Behshid Ghadrdoost¹, Parham Sadeghipour², Ata Firouzi², Farshad Shakerian², Reza Kiani², Alireza Rashidinejad², Zahra Hosseini², Saeideh Mazloomzadeh¹, Fidan Shabani¹, Bahram Mohebbi^2,3*

¹Rajaie Cardiovascular Medical and Research Center, Iran University of Medical Sciences, Tehran, Iran

²Cardiovascular Intervention Research Center, Rajaie Cardiovascular Medical and Research Center, Iran University of Medical Sciences, Tehran, Iran

³Cardio-oncology Research Center, Rajaie Cardiovascular Medical and Research Center, Iran University of Medical Sciences, Tehran, Iran

*Corresponding Author: Bahram Mohebbi, Cardiovascular Intervention Research Center, Rajaie Cardiovascular Medical and Research Center, Iran University of Medical Sciences, Tehran, Iran, E-mail: roodbar@yahoo.com

Received date: July 23, 2021; Accepted date: August 17, 2021; Published date: August 24, 2021

Citation: Ghadrdoost B, Sadeghipour P, Firouzi A, Shakerian F, Kianix R, et al. (2021) Reliability and Validity of a Questionnaire to Estimate Satisfaction Level of Cardiac Critical Care Nursing Students in with Virtual Education COVID-19 Pandemic Era. J Card Cardi Sur. 1(2):09.

Copyright: © 2021 Ghadrdoost B. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Keywords

COVID-19 Pandemic; Reliability; Validity; Questionnaire; Nursing student; Virtual education

Abstract

Background: COVID-19 pandemic has impacted the way things are done in walks of life including education of students of critical care nursing and has shifted it to virtual education. The development of valid and reliable questionnaire to estimate nursing students’ satisfaction with e-learning can be useful for medical educational system. So, the purpose of this study is to design and validate a questionnaire on the level of satisfaction of students of critical care nursing with virtual learning in the COVID-19 pandemic.

Methods: A self-administered questionnaire which was prepared by the nursing staff and faculty members of Rajaie cardiovascular medical and research center was assessed in terms of face and content validity, internal consistency by Cronbach’s alpha. Principal component factor analysis was done to determine the domains of the designed questionnaire.

Results: Content validity of the questionnaire was measured using CVR (0.97) and CVI (0.93). To determine the content validity index, the criteria of relevance, clarity and simplicity of each item were evaluated and they were acceptable. The study results demonstrated that Cronbach's alpha based on internal consistency ranged from 0.79 to 0.93. In order to determine the test-retest reliability, the correlation between two round of evaluation was more than 89 (r>0.89 p>0.001) and correlation intra-class coefficient was. 097(ICC:0.97, p>0.001).

Conclusion: Results of this study confirmed reliability and validity of questionnaire to evaluate the level of satisfaction of students of critical care nursing with virtual learning in the COVID-19 pandemic.

Introduction

COVID-19 as a global crisis affects several aspects of life, including education in all levels from primary school to university, or from teaching different skills to medical education. Nursing education was greatly affected by the pandemic because of its specific circumstances such as face-to-face classes, clinical skills laboratories and the clinical placement of students. The intensity and volume of the COVID-19 pandemic has forced nurses to change the way they learn and stay on track for completion [1].

However, nursing students and teachers do not always feel ready to act seamlessly in this pandemic scenario. The main weaknesses of nursing students are: Lack of personal protective equipment; adequate development of all skills required to complete the course and access to internet and communication technology. In relation to the teacher, the issue of using and mastering information and communication technology in a short period of time is particularly significant [2,3].

But the movement of teaching programs to the virtual education has created some new benefits. Increase productivity, unlimited educational space, cost reduction, no time and place restrictions, no limit on the number of audiences and users, more harmony with the taste of the new generation [4].

Nursing students’ satisfaction is an index for evaluating medical education, but there are few researches measuring this factor in COVID-19 era. Since the nursing students’ satisfaction has been associated with their professional attitudes and commitment, professional education faculties should be concerned with their satisfaction as an outcome of the educational process [5]. The development and use of valid and reliable questionnaire that report nursing students’ feedback about virtual education, especially in the period of coronavirus outbreak seems necessary. So, the purpose of this study is to design and validate a questionnaire on the level of satisfaction of master degree of nursing students with virtual learning in the COVID-19 pandemic.

Methods

In current study we evaluate the reliability and validity of a Persian version questionnaire of satisfaction and usefulness of master degree of nursing students regarding virtual education during the COVID-19 pandemic in Rajaie cardiovascular medical and research center which is a main referral cardiovascular medical center in Tehran, Iran. The study was approved by the Ethics Committee and obtained ethic code.

This questionnaire was prepared by the 10 members of the faculty of nursing and circulatory technology (perfusionist) of Rajaie cardiovascular medical and research center based on expert opinion and our previous experiences in creating questionnaire for cardiology residents and faculty members in this context. Before creating a questionnaire, research objectives were defined. Then, based on the most convenient method and selecting question types, questions were written as succinctly as possible. Questionnaires had a combination of question types and they were clear and unambiguous and not use language that is inappropriate for the respondents. In creation the questionnaire, the use of negative and double negatives was avoided and it was ensured that in multiple choice questions and rating scales that all categorizes are mutually exclusive. After determining the content of the questions, question wording was developed and questions put into a meaningful order and format.

Before explaining in detail how to determine the validity and reliability of the present questionnaire, the steps after its creation are summarized as follows: Establish face validity, Use principal component analysis, Check the internal consistency of questions loading onto the same factors, Revise the questionnaire based on information

This questionnaire consisted of three parts: The first part of the questionnaire has 15 questions that evaluate the participant's attitude towards various factors involved in virtual training such as adequacy of teaching hours, the quality and content of virtual education, interaction between teacher and students and internet facilities. This part of questionnaire neither evaluates the above areas by typical five-level Likert item including strongly disagree, disagree, neither agree nor disagree. The second part is 3 questions that the respondent must choose between two options: Face-to-face or virtual, and the third part includes two visual scales in which the participants, their level of satisfaction with the face-to-face methods and virtual education are specified in the form of a continuum from zero (minimum satisfaction) to 10 (maximum satisfaction).

Also, to check the qualitative validity, the questions were given to 10 nursing faculty members to apply their opinion about grammar, sentence structure and placement of phrases in the appropriate place. At this stage, according to experts, three questions changed the structure and grammar and one question changed the concept. In order to assess all steps of validity and reliability of tool, all expert were asked to answered in person while the questionnaire were anonymous and confidential.

To evaluate content validity, after design (identifying areas, producing items and instrument formation), the next step of content validity process includes evaluation, quantification of the validity indicators of each question (appropriateness, transparency) along with the validity indicators of the whole tool (appropriateness, transparency and comprehensiveness) which was done by a certain number of experts. The consensus of experts was to remove one question in this step. To quantify the content validity, Content Validity Ratio (CVR) coefficients and Content Validity Index (CVI) were used. In order to evaluate the content validity ratio, the expert group was asked to rate each question on a three-part 'essential' scale; Useful but unnecessary; Unnecessary 'Check that finally. The higher score indicates further agreement of members of experts on the necessity of an item in the questionnaire. The formula of content validity ratio is CVR=(Ne-N/2)/(N/2), in which the Ne is the number of experts indicating "essential" and N is the total number of experts. In this study, acceptable CVR was considered 0.49 [6]. The CVI was calculated by aggregating the scores for each item that scored "relevant but in need of review" and "fully relevant" divided by the total number of professionals.

In this study, to determine the external reliability, the questionnaire was marked by 10 students of master degree of nursing which they answered in person and was assessed using test-retest method to measure the reproducibility of the answers to the questions at 10-day intervals. These students were studying for a master's degree of nursing, but in educational centers other than our center but they were working in our hospital (medical wards). Then the correlation between the results of the two tests was calculated and correlation more than 0.7 was considered as a reliable feature. Also, the internal consistency coefficient (Cronbach's alpha) was used to calculate the internal reliability of the questionnaire, which varies from zero to one. Scale with an internal consistency coefficient greater than 0.7 was considered as an acceptable result [7].

Principle component factor analysis for investigating the construct validity was used to determine the domains of the designed questionnaire in multiple choice questions. Factor analysis was applied to statistical processes to simplify the relevant measurements and discover a pattern of a group of variables. This method was mainly used to find the easiest way to interpret the observed data. For this purpose, a factor analysis using varimax rotation was conducted and Kaiser-Meyer-Olkin measure of sampling adequacy (KMO) that is a proportion of variance among variables was used. It that might be vary from zero to one, in which zero is inadequate, while close to one is adequate. In order to optimize interpretation, a promax rotation was performed. Bartlett’s sphericity test was also did for factor analysis to compares the observed correlation matrix to the identity matrix. Overall, KMO values above 0.50 and p<0.05 for Bartlett’s sphericity test are considered acceptable [8,9].

Statistics

The internal consistency of the questionnaire was examined through determination of the Cronbach's alpha value, inter-correlation between the scales which evaluated by Pearson correlation coefficient and was determined by measuring the Intra-class Correlation Coefficient (ICC).

In factor analysis to investigate the construct validity, Kaiser-Meyer-Olkin (KMO) test and Bartlett’s sphericity test was used. The statistical analyses were performed with the SPSS software, version 15, for Windows (SPSS Inc, Chicago, Illinois).

Results

The demographic characteristics of all participants are shown in Table 1. Content validity of the questionnaire was measured using CVR and CVI. The CVI=0.93 and CVR=0.97 were calculated. To determine the content validity index, the criteria of relevance, clarity and simplicity of each item were evaluated and they were acceptable.

Demographic Variable			Critical care nursing students (n=20)	Circulatory technology students (n=11)
Sex		Female	14 (70%)	8 (73%)
		Male	6 (30%)	3 (27%)
Marital status		Single	8 (40%)	7 (64%)
		Married	12 (60%)	4 (36%)
Age			30.75 ± 3.44	30.99 ± 4.25

Table 1: The demographic characteristics of all participants.

The study results demonstrated that Cronbach's alpha based on internal consistency for each question in part one of questionnaire ranged from 0.79 to 0.93 and the total alpha value of this part of questionnaire was 0.96 that was indicative of the goodness of overall reliability of the questionnaire.

In order to determine the test-retest reliability, the correlation between two round of evaluation was more than 89 (r>0.89 p>0.001) and correlation intra-class coefficient was. 097(ICC: 0.97, p>0.001) which demonstrate relatively good stability for the questionnaire.

Appropriateness of the tool for evaluating the intended purpose, difficulty level of the tool for the target group (master degree of nursing students) and inadequacy in the questionnaire questions which are parameters that shows content validity of questionnaire had high score from the point of view of the respondents.

Factor analysis for investigating the construct validity was used to determine the domains of the designed questionnaire in multiple choice questions. Kaiser’s criterion (Eigenvalue=1.0) was considered to determine main factors by which 2 domains were found. Kaiser-Meyer-Olkin measure of sampling adequacy (KMO) was 0.93, proposed that the degree of common variance among the 15 items was acceptable. In order to optimize interpretation, a promax rotation was performed. The results are shown in Table 2. The first domain included ten items belonged to quality and content of virtual education. The second domain consisted of five items presented facilities required for e-learning and suitability of days/ hours of the week to for virtual lessons.

Description		Rotated Factor Loadings Matrix
		Factor-1	Factor-2
1	The rate of achievement of educational goals appropriate to each lesson with virtual education	0.81
2	Good quality of virtual education	0.82
4	Satisfaction with the method of education	0.88
5	Satisfaction with the scientific quality of education	0.90
6	Possibility of communication with the teacher	0.69
7	Ability to provide appropriate feedback by the teacher	0.70
10	Quality of course content (interactivity-multimedia-audio)	0.63
11	Possibility of accurate evaluation of residents by teacher	0.64
14	face-to-face meetings to solve academic problems	0.76
15	Meet the expectations of virtual education	0.76
3	Suitability of days of the week to for virtual lessons		0.84
8	Suitability of hours of the day to for virtual lessons		0.76
9	Ease of internet access		0.67
12	The possibility of reviewing previous class by the system		0.75
13	Having the necessary technical support from the system in case of problems		0.79

Table 2: Factor analysis on questionnaire of satisfaction and usefulness of virtual education from the perspective of master of nursing students of cardiac critical care during the COVID-19 Pandemic.

Discussion

The purpose of this study was to valid and reliable questionnaire to evaluate the level of satisfaction of master degree of critical care nursing students with virtual education in the COVID-19 pandemic.

In public health pandemic of COVID-19 era, several teaching methods have been considered for master degree of nursing students, such as the use of blended learning method to enhance clinical skills, and to reduce the clinical exposure for them; however, gaps have been identified in finding an effective method of attaining clinical skills competency with virtual teaching [1,10]. It seems that integrating clinical skills and the development of required competences have become the greatest curricular challenges in nursing education [11]. But the satisfaction of these students from changing the teaching method in the hospital as its main users is not clear. Therefore, due to the importance of measuring satisfaction, the need for a valid tool to measure this concept in, we designed a questionnaire to assess satisfaction of nursing students of cardiac critical care with virtual education.

Our prepared questionnaire in current study shows has good reliability and validity. The Cronbach's α coefficients were>0.9, indicating internal uniformity. It shows that participants had almost the same perception of the questions. The retest of this questionnaire after 10 days indicates a low level of reliability and indicates the reproducibility, stability and internal coherence of the instrument.

The validity and reliability tests of new designed tools in research are so important and gave both literary and technical meanings of these tests [12]. Validity expresses the degree to which a measurement measures what it purports to measure including internal and external validities. Reliability is related to the degree of replications of obtained results [13].

Factor analysis revealed clinically relevant two domains that shows construct validity including quality and content of virtual education and facilities required for e-learning and suitability of days/ hours of the week to for virtual lessons. All the factor loadings were in 0.63 to 0.90 range which is acceptable range in reference’s studies [14].

Conclusion

Results of this study confirmed reliability and validity of questionnaire to evaluate the level of satisfaction of master degree of critical care nursing students with e-learning COVID-19 pandemic era. A credible questionnaire that reports the feedback of master degree of nursing students of intensive care about e-learning during the coronavirus outbreak can help the education system select the best methods for their more effective training. These methods may be practical and effective for nursing education after the end of the COVID-19 pandemic.

Acknowledgments

We thank all counselors and also all staff of Rajaie cardiovascular medical and research center in this study.

Conflict of Interest

The authors have no conflicts of interest to declare.

Funding

None.

Authors’ Contribution

• Study concept and design: Behshid Ghadrdoost, Bahram Mohebbi
• Technical and material support, conducting the project: Bahram Mohebbi, Behshid Ghadrdoost, Parham Sadeghipour
• Revising the manuscript: Behshid Ghadrdoost, Bahram Mohebbi, Parham Sadeghipour, Saeideh Mazloomzadeh
• Collecting the data and review of literature: Parham Sadeghipour, Ata Firouzi, Farshad Shakerian, Reza Kiani, Alireza Rashidinejad, Zahra Hosseini, Saeideh Mazloomzadeh, Fidan Shabani
• Data analysis and preparing the manuscript: Behshid Ghadrdoost, Reza Kiani, Alireza Rashidinejad, Zahra Hosseini, Farshad Shakerian

Ethics Approval

This study was approved by the Ethics Committee of Rajaie Cardiovascular, Medical, and Research Center, Tehran, Iran. The ettical code is: IR.RHC.REC.1399.108

References

01. Agu CF, Stewart J, Mcfarlane-Stewart N, Rae T. COVID-19 pandemic effects on nursing education: Looking through the lens of a developing country. Int Nurs Rev. 68(2): 153-158 (2021).

02. Lira A, Adamy EK, Teixeira E, Silva FV. Nursing education: Challenges and perspectives in times of the COVID-19 pandemic. Rev Bras Enferm. 73(S2): e20200683 (2020).

03. Ghadrdoost B, Sadeghipour P, Amin A, Bakhshandeh H, Noohi F, et al. Validity and reliability of a virtual education satisfaction questionnaire from the perspective of cardiology residents during the COVID‑19 pandemic. J Edu Health Promot. (2021).

04. Schneider SL, Council ML. Distance learning in the era of COVID‑19. Arch Dermatol Res. (2020).

05. Ziaee V, Ahmadinejad Z, Morravedji AR. An evaluation on medical student satisfaction with clinical education and its effective factors. Med Educ Online. 9(1): 4365 (2004).

06. Ashtiani AF, Afzali SM, Ebadi A, Hassanabadi H. Design and assessment of psychometric features of life skills inventory. J Edu Health Promot. 7: 86 (2018).

07. Stone AT, Bransford RJ, Lee MJ, Vilela MD, Bellabarba C, et al. Reliability of classification systems for subaxial cervical injuries. Evid Based Spine Care J. 1(3): 19-26 (2010).

08. de Oliveira Santos R, Gorgulho BM, de Castro MA, Fisberg RM, Marchioni DM. Principal component analysis and factor analysis: Differences and similarities in nutritional epidemiology application. Rev Bras Epidemiol. 22: e190041 (2019).

09. Masaeli N, Omranifard V, Maracy MR, Kheirabadi GR, Khedri A. Validity, reliability and factor analysis of Persian version of schizophrenia quality of life scale. J Edu Health Promot. 5: 10 (2016).

10. Honey M. Undergraduate student nurses’ use of information and communication technology in their education. Stud Health Technol Inform. 250: 37-40 (2017).

11. McCutcheon K, Lohan M, Traynor M, Martin DA. A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs. 71(2): 255-270 (2015).

12. Bolarinwa OA. Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Niger Postgrad Med J. 22(4): 195-201 (2015).

13. Wong KL, Ong SF, Kuek TY. Constructing a survey questionnaire to collect data on service quality of business academics. Eur J Soc Sci. 29: 209‑21 (2012).

14. Sadeghi-Gandomani H, Alavi NM, Afshar M. Psychometric testing of the Persian version of the conditions of work effectiveness Questionnaire-II (CWEQ-II-PV). J Edu Health Promot. 8: 32 (2019).