Research Paper Based On Questionnaire Sample

Abstract

Survey research is sometimes regarded as an easy research approach. However, as with any other research approach and method, it is easy to conduct a survey of poor quality rather than one of high quality and real value. This paper provides a checklist of good practice in the conduct and reporting of survey research. Its purpose is to assist the novice researcher to produce survey work to a high standard, meaning a standard at which the results will be regarded as credible. The paper first provides an overview of the approach and then guides the reader step-by-step through the processes of data collection, data analysis, and reporting. It is not intended to provide a manual of how to conduct a survey, but rather to identify common pitfalls and oversights to be avoided by researchers if their work is to be valid and credible.

data reporting , health care surveys , methodology , questionnaires , research design , survey methods , surveys

What is survey research?

Survey research is common in studies of health and health services, although its roots lie in the social surveys conducted in Victorian Britain by social reformers to collect information on poverty and working class life (e.g. Charles Booth [1] and Joseph Rowntree [2]), and indeed survey research remains most used in applied social research. The term ‘survey’ is used in a variety of ways, but generally refers to the selection of a relatively large sample of people from a pre-determined population (the ‘population of interest’; this is the wider group of people in whom the researcher is interested in a particular study), followed by the collection of a relatively small amount of data from those individuals. The researcher therefore uses information from a sample of individuals to make some inference about the wider population.

Data are collected in a standardized form. This is usually, but not necessarily, done by means of a questionnaire or interview. Surveys are designed to provide a ‘snapshot of how things are at a specific time’ [3]. There is no attempt to control conditions or manipulate variables; surveys do not allocate participants into groups or vary the treatment they receive. Surveys are well suited to descriptive studies, but can also be used to explore aspects of a situation, or to seek explanation and provide data for testing hypotheses. It is important to recognize that ‘the survey approach is a research strategy, not a research method’ [3]. As with any research approach, a choice of methods is available and the one most appropriate to the individual project should be used. This paper will discuss the most popular methods employed in survey research, with an emphasis upon difficulties commonly encountered when using these methods.

Descriptive research

Descriptive research is a most basic type of enquiry that aims to observe (gather information on) certain phenomena, typically at a single point in time: the ‘cross-sectional’ survey. The aim is to examine a situation by describing important factors associated with that situation, such as demographic, socio-economic, and health characteristics, events, behaviours, attitudes, experiences, and knowledge. Descriptive studies are used to estimate specific parameters in a population (e.g. the prevalence of infant breast feeding) and to describe associations (e.g. the association between infant breast feeding and maternal age).

Analytical studies

Analytical studies go beyond simple description; their intention is to illuminate a specific problem through focused data analysis, typically by looking at the effect of one set of variables upon another set. These are longitudinal studies, in which data are collected at more than one point in time with the aim of illuminating the direction of observed associations. Data may be collected from the same sample on each occasion (cohort or panel studies) or from a different sample at each point in time (trend studies).

Evaluation research

This form of research collects data to ascertain the effects of a planned change.

Advantages and disadvantages of survey research

Advantages:

  • The research produces data based on real-world observations (empirical data).

  • The breadth of coverage of many people or events means that it is more likely than some other approaches to obtain data based on a representative sample, and can therefore be generalizable to a population.

  • Surveys can produce a large amount of data in a short time for a fairly low cost. Researchers can therefore set a finite time-span for a project, which can assist in planning and delivering end results.

Disadvantages:

  • The significance of the data can become neglected if the researcher focuses too much on the range of coverage to the exclusion of an adequate account of the implications of those data for relevant issues, problems, or theories.

  • The data that are produced are likely to lack details or depth on the topic being investigated.

  • Securing a high response rate to a survey can be hard to control, particularly when it is carried out by post, but is also difficult when the survey is carried out face-to-face or over the telephone.

Essential steps in survey research

Research question

Good research has the characteristic that its purpose is to address a single clear and explicit research question; conversely, the end product of a study that aims to answer a number of diverse questions is often weak. Weakest of all, however, are those studies that have no research question at all and whose design simply is to collect a wide range of data and then to ‘trawl’ the data looking for ‘interesting’ or ‘significant’ associations. This is a trap novice researchers in particular fall into. Therefore, in developing a research question, the following aspects should be considered [4]:

  • Be knowledgeable about the area you wish to research.

  • Widen the base of your experience, explore related areas, and talk to other researchers and practitioners in the field you are surveying.

  • Consider using techniques for enhancing creativity, for example brainstorming ideas.

  • Avoid the pitfalls of: allowing a decision regarding methods to decide the questions to be asked; posing research questions that cannot be answered; asking questions that have already been answered satisfactorily.

Research methods

The survey approach can employ a range of methods to answer the research question. Common survey methods include postal questionnaires, face-to-face interviews, and telephone interviews.

Postal questionnaires

This method involves sending questionnaires to a large sample of people covering a wide geographical area. Postal questionnaires are usually received ‘cold’, without any previous contact between researcher and respondent. The response rate for this type of method is usually low, ∼20%, depending on the content and length of the questionnaire. As response rates are low, a large sample is required when using postal questionnaires, for two main reasons: first, to ensure that the demographic profile of survey respondents reflects that of the survey population; and secondly, to provide a sufficiently large data set for analysis.

Face-to-face interviews

Face-to-face interviews involve the researcher approaching respondents personally, either in the street or by calling at people’s homes. The researcher then asks the respondent a series of questions and notes their responses. The response rate is often higher than that of postal questionnaires as the researcher has the opportunity to sell the research to a potential respondent. Face-to-face interviewing is a more costly and time-consuming method than the postal survey, however the researcher can select the sample of respondents in order to balance the demographic profile of the sample.

Telephone interviews

Telephone surveys, like face-to-face interviews, allow a two-way interaction between researcher and respondent. Telephone surveys are quicker and cheaper than face-to-face interviewing. Whilst resulting in a higher response rate than postal surveys, telephone surveys often attract a higher level of refusals than face-to-face interviews as people feel less inhibited about refusing to take part when approached over the telephone.

Designing the research tool

Whether using a postal questionnaire or interview method, the questions asked have to be carefully planned and piloted. The design, wording, form, and order of questions can affect the type of responses obtained, and careful design is needed to minimize bias in results. When designing a questionnaire or question route for interviewing, the following issues should be considered: (1) planning the content of a research tool; (2) questionnaire layout; (3) interview questions; (4) piloting; and (5) covering letter.

Planning the content of a research tool

The topics of interest should be carefully planned and relate clearly to the research question. It is often useful to involve experts in the field, colleagues, and members of the target population in question design in order to ensure the validity of the coverage of questions included in the tool (content validity).

Researchers should conduct a literature search to identify existing, psychometrically tested questionnaires. A well designed research tool is simple, appropriate for the intended use, acceptable to respondents, and should include a clear and interpretable scoring system. A research tool must also demonstrate the psychometric properties of reliability (consistency from one measurement to the next), validity (accurate measurement of the concept), and, if a longitudinal study, responsiveness to change [5]. The development of research tools, such as attitude scales, is a lengthy and costly process. It is important that researchers recognize that the development of the research tool is equal in importance—and deserves equal attention—to data collection. If a research instrument has not undergone a robust process of development and testing, the credibility of the research findings themselves may legitimately be called into question and may even be completely disregarded. Surveys of patient satisfaction and similar are commonly weak in this respect; one review found that only 6% of patient satisfaction studies used an instrument that had undergone even rudimentary testing [6]. Researchers who are unable or unwilling to undertake this process are strongly advised to consider adopting an existing, robust research tool.

Questionnaire layout

Questionnaires used in survey research should be clear and well presented. The use of capital (upper case) letters only should be avoided, as this format is hard to read. Questions should be numbered and clearly grouped by subject. Clear instructions should be given and headings included to make the questionnaire easier to follow.

The researcher must think about the form of the questions, avoiding ‘double-barrelled’ questions (two or more questions in one, e.g. ‘How satisfied were you with your personal nurse and the nurses in general?’), questions containing double negatives, and leading or ambiguous questions. Questions may be open (where the respondent composes the reply) or closed (where pre-coded response options are available, e.g. multiple-choice questions). Closed questions with pre-coded response options are most suitable for topics where the possible responses are known. Closed questions are quick to administer and can be easily coded and analysed. Open questions should be used where possible replies are unknown or too numerous to pre-code. Open questions are more demanding for respondents but if well answered can provide useful insight into a topic. Open questions, however, can be time consuming to administer and difficult to analyse. Whether using open or closed questions, researchers should plan clearly how answers will be analysed.

Interview questions

Open questions are used more frequently in unstructured interviews, whereas closed questions typically appear in structured interview schedules. A structured interview is like a questionnaire that is administered face to face with the respondent. When designing the questions for a structured interview, the researcher should consider the points highlighted above regarding questionnaires. The interviewer should have a standardized list of questions, each respondent being asked the same questions in the same order. If closed questions are used the interviewer should also have a range of pre-coded responses available.

If carrying out a semi-structured interview, the researcher should have a clear, well thought out set of questions; however, the questions may take an open form and the researcher may vary the order in which topics are considered.

Piloting

A research tool should be tested on a pilot sample of members of the target population. This process will allow the researcher to identify whether respondents understand the questions and instructions, and whether the meaning of questions is the same for all respondents. Where closed questions are used, piloting will highlight whether sufficient response categories are available, and whether any questions are systematically missed by respondents.

When conducting a pilot, the same procedure as as that to be used in the main survey should be followed; this will highlight potential problems such as poor response.

Covering letter

All participants should be given a covering letter including information such as the organization behind the study, including the contact name and address of the researcher, details of how and why the respondent was selected, the aims of the study, any potential benefits or harm resulting from the study, and what will happen to the information provided. The covering letter should both encourage the respondent to participate in the study and also meet the requirements of informed consent (see below).

Sample and sampling

The concept of sample is intrinsic to survey research. Usually, it is impractical and uneconomical to collect data from every single person in a given population; a sample of the population has to be selected [7]. This is illustrated in the following hypothetical example. A hospital wants to conduct a satisfaction survey of the 1000 patients discharged in the previous month; however, as it is too costly to survey each patient, a sample has to be selected. In this example, the researcher will have a list of the population members to be surveyed (sampling frame). It is important to ensure that this list is both up-to date and has been obtained from a reliable source.

The method by which the sample is selected from a sampling frame is integral to the external validity of a survey: the sample has to be representative of the larger population to obtain a composite profile of that population [8].

There are methodological factors to consider when deciding who will be in a sample: How will the sample be selected? What is the optimal sample size to minimize sampling error? How can response rates be maximized?

The survey methods discussed below influence how a sample is selected and the size of the sample. There are two categories of sampling: random and non-random sampling, with a number of sampling selection techniques contained within the two categories. The principal techniques are described here [9].

Random sampling

Generally, random sampling is employed when quantitative methods are used to collect data (e.g. questionnaires). Random sampling allows the results to be generalized to the larger population and statistical analysis performed if appropriate. The most stringent technique is simple random sampling. Using this technique, each individual within the chosen population is selected by chance and is equally as likely to be picked as anyone else. Referring back to the hypothetical example, each patient is given a serial identifier and then an appropriate number of the 1000 population members are randomly selected. This is best done using a random number table, which can be generated using computer software (a free on-line randomizer can be found at http://www.randomizer.org/index.htm).

Alternative random sampling techniques are briefly described. In systematic sampling, individuals to be included in the sample are chosen at equal intervals from the population; using the earlier example, every fifth patient discharged from hospital would be included in the survey. Stratified sampling selects a specific group and then a random sample is selected. Using our example, the hospital may decide only to survey older surgical patients. Bigger surveys may employ cluster sampling, which randomly assigns groups from a large population and then surveys everyone within the groups, a technique often used in national-scale studies.

Non-random sampling

Non-random sampling is commonly applied when qualitative methods (e.g. focus groups and interviews) are used to collect data, and is typically used for exploratory work. Non-random sampling deliberately targets individuals within a population. There are three main techniques. (1) purposive sampling: a specific population is identified and only its members are included in the survey; using our example above, the hospital may decide to survey only patients who had an appendectomy. (2) Convenience sampling: the sample is made up of the individuals who are the easiest to recruit. Finally, (3) snowballing: the sample is identified as the survey progresses; as one individual is surveyed he or she is invited to recommend others to be surveyed.

It is important to use the right method of sampling and to be aware of the limitations and statistical implications of each. The need to ensure that the sample is representative of the larger population was highlighted earlier and, alongside the sampling method, the degree of sampling error should be considered. Sampling error is the probability that any one sample is not completely representative of the population from which it has been drawn [9]. Although sampling error cannot be eliminated entirely, the sampling technique chosen will influence the extent of the error. Simple random sampling will give a closer estimate of the population than a convenience sample of individuals who just happened to be in the right place at the right time.

Sample size

What sample size is required for a survey? There is no definitive answer to this question: large samples with rigorous selection are more powerful as they will yield more accurate results, but data collection and analysis will be proportionately more time consuming and expensive. Essentially, the target sample size for a survey depends on three main factors: the resources available, the aim of the study, and the statistical quality needed for the survey. For ‘qualitative’ surveys using focus groups or interviews, the sample size needed will be smaller than if quantitative data is collected by questionnaire. If statistical analysis is to be performed on the data then sample size calculations should be conducted. This can be done using computer packages such as G*Power [10]; however, those with little statistical knowledge should consult a statistician. For practical recommendations on sample size, the set of survey guidelines developed by the UK Department of Health [11] should be consulted.

Larger samples give a better estimate of the population but it can be difficult to obtain an adequate number of responses. It is rare that everyone asked to participate in the survey will reply. To ensure a sufficient number of responses, include an estimated non-response rate in the sample size calculations.

Response rates are a potential source of bias. The results from a survey with a large non-response rate could be misleading and only representative of those who replied. French [12] reported that non-responders to patient satisfaction surveys are less likely to be satisfied than people who reply. It is unwise to define a level above which a response rate is acceptable, as this depends on many local factors; however, an achievable and acceptable rate is ∼75% for interviews and 65% for self-completion postal questionnaires [9,13]. In any study, the final response rate should be reported with the results; potential differences between the respondents and non-respondents should be explicitly explored and their implications discussed.

There are techniques to increase response rates. A questionnaire must be concise and easy to understand, reminders should be sent out, and method of recruitment should be carefully considered. Sitzia and Wood [13] found that participants recruited by mail or who had to respond by mail had a lower mean response rate (67%) than participants who were recruited personally (mean response 76.7%). A most useful review of methods to maximize response rates in postal surveys has recently been published [14].

Data collection

Researchers should approach data collection in a rigorous and ethical manner. The following information must be clearly recorded:

  • How, where, how many times, and by whom potential respondents were contacted.

  • How many people were approached and how many of those agreed to participate.

  • How did those who agreed to participate differ from those who refused with regard to characteristics of interest in the study, for example how were they identified, where were they approached, and what was their gender, age, and features of their illness or health care.

  • How was the survey administered (e.g. telephone interview).

  • What was the response rate (i.e. the number of usable data sets as a proportion of the number of people approached).

Data analysis

The purpose of all analyses is to summarize data so that it is easily understood and provides the answers to our original questions: ‘In order to do this researchers must carefully examine their data; they should become friends with their data’ [15]. Researchers must prepare to spend substantial time on the data analysis phase of a survey (and this should be built into the project plan). When analysis is rushed, often important aspects of the data are missed and sometimes the wrong analyses are conducted, leading to both inaccurate results and misleading conclusions [16]. However, and this point cannot be stressed strongly enough, researchers must not engage in data dredging, a practice that can arise especially in studies in which large numbers of dependent variables can be related to large numbers of independent variables (outcomes). When large numbers of possible associations in a dataset are reviewed at P < 0.05, one in 20 of the associations by chance will appear ‘statistically significant’; in datasets where only a few real associations exist, testing at this significance level will result in the large majority of findings still being false positives [17].

The method of data analysis will depend on the design of the survey and should have been carefully considered in the planning stages of the survey. Data collected by qualitative methods should be analysed using established methods such as content analysis [18], and where quantitative methods have been used appropriate statistical tests can be applied. Describing methods of analysis here would be unproductive as a multitude of introductory textbooks and on-line resources are available to help with simple analyses of data (e.g. [19, 20]). For advanced analysis a statistician should be consulted.

Reporting

When reporting survey research, it is essential that a number of key points are covered (though the length and depth of reporting will be dependent upon journal style). These key points are presented as a ‘checklist’ below:

  1. Explain the purpose or aim of the research, with the explicit identification of the research question.

  2. Explain why the research was necessary and place the study in context, drawing upon previous work in relevant fields (the literature review).

  3. Describe in (proportionate) detail how the research was done.

    • State the chosen research method or methods, and justify why this method was chosen.

    • Describe the research tool. If an existing tool is used, briefly state its psychometric properties and provide references to the original development work. If a new tool is used, you should include an entire section describing the steps undertaken to develop and test the tool, including results of psychometric testing.

    • Describe how the sample was selected and how data were collected, including:

    • How were potential subjects identified?

    • How many and what type of attempts were made to contact subjects?

    • Who approached potential subjects?

    • Where were potential subjects approached?

    • How was informed consent obtained?

    • How many agreed to participate?

    • How did those who agreed differ from those who did not agree?

    • What was the response rate?

  4. Describe and justify the methods and tests used for data analysis.

  5. Present the results of the research. The results section should be clear, factual, and concise.

  6. Interpret and discuss the findings. This ‘discussion’ section should not simply reiterate results; it should provide the author’s critical reflection upon both the results and the processes of data collection. The discussion should assess how well the study met the research question, should describe the problems encountered in the research, and should honestly judge the limitations of the work.

  7. Present conclusions and recommendations.

The researcher needs to tailor the research report to meet:

  • The expectations of the specific audience for whom the work is being written.

  • The conventions that operate at a general level with respect to the production of reports on research in the social sciences.

Ethics

Anyone involved in collecting data from patients has an ethical duty to respect each individual participant’s autonomy. Any survey should be conducted in an ethical manner and one that accords with best research practice. Two important ethical issues to adhere to when conducting a survey are confidentiality and informed consent.

The respondent’s right to confidentiality should always be respected and any legal requirements on data protection adhered to. In the majority of surveys, the patient should be fully informed about the aims of the survey, and the patient’s consent to participate in the survey must be obtained and recorded.

The professional bodies listed below, among many others, provide guidance on the ethical conduct of research and surveys.

Conclusion

Survey research demands the same standards in research practice as any other research approach, and journal editors and the broader research community will judge a report of survey research with the same level of rigour as any other research report. This is not to say that survey research need be particularly difficult or complex; the point to emphasize is that researchers should be aware of the steps required in survey research, and should be systematic and thoughtful in the planning, execution, and reporting of the project. Above all, survey research should not be seen as an easy, ‘quick and dirty’ option; such work may adequately fulfil local needs (e.g. a quick survey of hospital staff satisfaction), but will not stand up to academic scrutiny and will not be regarded as having much value as a contribution to knowledge.

References

Vernon A.

A Quaker Businessman: Biography of Joseph Rowntree (1836–1925)

 . London: Allen & Unwin,

1958

.

Google Scholar

Denscombe M.

The Good Research Guide: For Small-scale Social Research Projects

 . Buckingham: Open University Press,

1998

.

Google Scholar

Robson C.

Real World Research: A Resource for Social Scientists and Practitioner-researchers

 . Oxford: Blackwell Publishers,

1993

.

Google Scholar

Streiner DL, Norman GR.

Health Measurement Scales: A Practical Guide to their Development and Use

 . Oxford: Oxford University Press,

1995

.

Google Scholar

Sitzia J. How valid and reliable are patient satisfaction data? An analysis of 195 studies.

Int J Qual Health Care

1999

;

11:

319

–328.

Google Scholar

Bowling A.

Research Methods in Health. Investigating Health and Health Services

 . Buckingham: Open University Press,

2002

.

Google Scholar

Arber S. Designing samples. In: Gilbert N, ed.

Researching Social Life

 . London: SAGE Publications,

2001

.

Google Scholar

French K. Methodological considerations in hospital patient opinion surveys.

Int J Nurs Stud

1981

;

18:

7

–32.

Google Scholar

Sitzia J, Wood N. Response rate in patient satisfaction research: an analysis of 210 published studies.

Int J Qual Health Care

1998

;

10:

311

–317.

Google Scholar

Edwards P, Roberts I, Clarke M et al. Increasing response rates to postal questionnaires: systematic review.

Br Med J

2002

;

324:

1183

.

Google Scholar

Wright DB. Making friends with our data: improving how statistical results are reported.

Br J Educ Psychol

2003

; in press.

Google Scholar

Wright DB, Kelley K. Analysing and reporting data. In: Michie S, Abraham C, eds.

Health Psychology in Practice

 . London: SAGE Publications,

2003

; in press.

Google Scholar

Davey Smith G, Ebrahim S. Data dredging, bias, or confounding.

Br Med J

2002

;

325:

1437

–1438.

Google Scholar

Morse JM, Field PA.

Nursing Research: The Application of Qualitative Approaches

 . London: Chapman and Hall,

1996

.

Google Scholar

Wright DB.

Understanding Statistics: An Introduction for the Social Sciences

 . London: SAGE Publications,

1997

.

Google Scholar

International Journal for Quality in Health Care 15(3) © International Society for Quality in Health Care and Oxford University Press 2003; all rights reserved

Address reprint requests to John Sitzia, Research Department, Worthing Hospital, Lyndhurst Road, Worthing BN11 2DH, West Sussex, UK. E-mail: john.sitzia@wash.nhs.uk

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix D Sample Questionnaires {These questionnaires are subject to further review and revision.) 1. Institutional Questionnaire 2. Program Questionnaire 3. Faculty Questionnaire 4. Student Questionnaires a. Questionnaire for Acimittecl-to-Cancliclacy Doctoral Students b. Questionnaire for Program Gracluates 105

106 Institutional Questionnaire To the institutional coordinator: This questionnaire is intended to collect data about university-provided resources that are available to all doctoral programs. Typically, the ideal respondent will be in the university's office of institutional research. Most of the questions apply to all programs. One, on laboratory space, applies only to the sciences (including some social sciences). In listing programs, please refer to the attached taxonomy and answer for those programs that are present at your institution. I. For the libraries at your institution: (Please enter the average over the past three years) a. What is the average size of your professional library staff in total FTEs? b. What is the average annual library budget? c. What is the average annual budget for acquisition of books? d. A, ~ What is the average annual budget for acquisition of: print journals electronic journals ? What is the average annual budget for microprint and electronic databases? 2. Is health care insurance available to graduate students uncler an institutional plan? Yes No a. If available, health care insurance is made available to: ~ Students only ~ Students end faculty b. If available, what is the level of institutional support? (Check all that apply) Institution covers premium costs for: Teaching assistants ~ Research assistants ~ All other full-time graduate students ~ Al] graduate students Institution covers partial premium costs for: Teaching assistants ~ Research assistants ~ All other full-time graduate students ~ All graduate students No institutional contribution for: ~ Teaching assistants ~ Research assistants ~ Other graduate students 3. Does the university provide childcare facilities that are available to graduate students? O Yes ~1 No a. If yes, is the cost subsidized by the institution? ~ Yes :] No b. If not, does the institution provide a listing of childcare providers to graduate students? O Yes ~ No 4. Is university-subsidized student housing available to doctoral students? :] Yes ~ No APPENDIX D

APPENDIX D If so, what is the percentage of the doctoral students who live in university-provided housing? 5. Are graduate students are unionized on your campus? ~ Yes ~ No If yes, ~ Some students ~ All students If yes, are teaching assistants unionized? ~ Yes ~ No If yes, ~ Some teaching assistants ~ All teaching assistants If yes, are research assistants unionized? ~ Yes ~ No If yes, ~ Some research assistants ~ All research assistants? 6. Which of the following apply to the doctoral program at the institutional level? a. The institution confers awards to honor graduate students for teaching and/or research. ~ Yes ~ No b. Awards are given to faculty for mentoring or other activities that promote scholarship of doctoral students. Yes ~ No c. The institution provides some form of travel support for doctoral students to attend professional meetings. ~ Yes ~ No d. There is an organized program at the institutional level to help doctoral students improve their teaching skills. ~ Yes ~ No e. The institution provides an office that assists doctoral students in learning about employment opportunities. ~ Yes ~ No 7. For the information displayed in the following table, please provide in a file sent by small to rdpilof~as~ed~ For the each doctoral program in science (including the social sciences) and engineering at your institution, what is the net assignable square feet (NASF) of research space dedicated to the program (exclude space that is used only for undergraduates)? Please use the same definitions for NASF and research space that are used in the NSF Survey of Scientific and Engineering Research Facilities. See "Taxonomy] for a list of the program iEelds in the study, and provide the information in the Emai! i ile for only those doctoral programs that are offered at your institution. 107

APPENDIX D Program #3 108 Program Research space NASF Shared space with other programs (Y/N) Program #1 Pro cram #2

APPENDIX D Background Information Program Questionnaire This information will enable the National Research Council to contact you if there are any questions about the data. It will also permit us to contact faculty for the purpose of administering a questionnaire to elicit reputational ratings and background! data ant! to contact students to obtain information about their perceptions of the practices and offerings ~ ~ ~ 1 of the doctoral program. Please note that in addition to the web questionnaire, we would like lists of faculty and previous employers to be sent to us via e-mail. Please indicate the doctoral program to which the following information applies 1. Please provide the name and e-mai! address of the program respondent who will serve as the primary contact with the graduate oro cram. Name: Title: E-mail address: Mailing Address: State Zip Cocle- If this is an interdisciplinary program, please list the departments affiliated with the program. For each individual identified in questions 2 and 3, please provide in a file sent by emai! to rdpilot~)nas.~du the information displayed in the table for the question. Program Faculty: For each faculty member or senior research fellow or associate who participates in your doctoral program by directing theses, serving on doctoral committees, or teaching graduate courses, please provide the following information. Name Rank Highest Gender Race/ US Citizen or Tenure E-mail l | Degree | (M or F) | Ethn city | Permanent | Status | Addres (Y/N) 1 1 1 1 1 1 1 = Faculty Employment History: For each faculty member listed in Question 2 who joined your program within the past five years, please provide the institution, company, or organization where he or she was employed immediately before joining your institution. pros

110 Name Prior employer Position at that employer 4. For the doctoral students in your program, please provide the number of students that fall into each of the following categories. a. Total number of students: b. Status: Full-time Part-time Unknown c. Gentler: Male Female Unknown d. Citizenship: U.S. Permanent Resilient Temporary Visa Unknown cI. Race/Ethnicity (if U.S. citizen or Permanent Residents) American Indian or Alaskan Native Asian or Pacific Islander Black White Hispanic Mexican American Puerto Rican Other Multiracial Unknown e. Percentage of doctoral students with master's degree Program Information 5. Does your program have a mission statement? If so, what is the mission statement? (50 words or less) ~ Yes :] No If there are particular areas of research emphasis in your doctoral program, please choose from the subfields in ETaxonomy]: APPENDIX D

APPENDIX D 6. How many Ph.D.s have been awarded in the program in each of the past five years? (Note: Years span from July ~ to June 30) 2001-02 2000-01 1999-00 1998-99 1997-98- 7. For each of the academic years listed in the following table, enter the number of students who entered the program in the year and the number who completed their degrees in 4, 6, or 8, years or are still in the program. (Note: Years span from July 1 to June 30) Entering Number Student of Academic Entering Year Cohort Doctoral Students 1992 1993 1993-1994 1994-1995 1995-1996 1996-1997 1997-1998 1998-1999 1999-2000 2000-2001 2001-2002 Number of Students admitted to candidacy by the end of the 4th year of enrollment Of those admitted to candidacy, number who complete within 4 years Of those admitted to candidacy, number who complete within 6 years Of those admitted to candidacy, number who complete within 8 years Of those admitted to candidacy, how many are still enrolled after 8 years? . I_ 7a. Averaged over the past three years, what percent of entering students withdrew from the program before completing two years of study? % 7b. Averaged over the past three years, what has been the median time to degree for those who completed the program? (Note: the median time is the number of years it takes half of the number of students from the same entering year who are admitted to candidacy to complete their degree.) 8. Is a master's degree required of students prior to admission to your program? ~ Yes ~ No 9. What proportion of your full-time first-year doctoral students receive full support throughout their first year (tuition and an adequate living allowance provided as stipend or salary in program related work (TA or RA)?

112 10. How many years of full financial support could students entering your doctoral program expect to receive from your institution or an external source? 1. Over the past five years approximately what fraction of the first-year students in your program received financial support either from your institution or from extramural grants or fellowships? Tuition only Tuition and stipend- Stipend only- 12. What proportion of currently enrolled doctoral students in your program (included in multiple categories if appropriate) are currently supported by: Externally funded fellowships: Externally funded traineeships: Externally funded research assistantships: University funded teaching assistantships: University funded research assistantships: University funded tuition waivers, fellowships, or stipends: 13. Averaged over the past three years, what are the average and minimum GRE scores for students accepted into the program? Average Verbal GRE: Minimum Verbal GRE: Average Quantitative GRE: Minimum Quantitative GRE: Do you require GRE subject scores for all students entering the program? ~ Yes ~ No 14. In each of the last three academic years, how many students did you accept into your doctoral program, and how many enrolled? 2000-2001 2001 -2002 2002-2003 Accepted Enrolled 15. What percentage of the doctoral students in your program have individually assigned workspaces for their exclusive use? TAs RAs All students 16. On average, how many courses per term is each graduate teaching assistant in the program expected to teach or assist a faculty member in teaching? With sole responsibility As an Assistant to a faculty member 17. Which of the following apply to your doctoral program? APPENDIX D

APPENDIX D a. The program confers awards to honor graduate students for teaching and/or research. ~ Yes ~1 No b. Awards are given to faculty for mentoring or other activities that promote scholarship of doctoral students. ~ Yes :] No The program provides some form of travel support for doctoral students to attend professional meetings. ~ Yes ~ No d. There is an organized program to help doctoral students improve their teaching skills. ~ Yes n No e. The program provides organized assistance to help doctoral students explore employment opportunities. Yes ~1 No 8. List up to 5 institutions with which your program normally competes for graduate students: Institution # 1 Institution #2- Institution #3 Institution #4 Institution #5 1 9. Does your program collect data about employment outcomes for your graduates? ~ Yes ~ No If yes, do you provide potential applicants with this information? ~ Yes ~ No 20. Please list those interdisciplinary centers in which doctoral students from your program participate (conduct research or teach). ~3

114 Faculty Questionnaire This questionnaire is part of the National Research Council's Pilot Test of the Assessment of Research Doctoral Programs. Your university has volunteered to participate in this pilot test to assist the National Research Council's study of the methodology used to assess doctoral programs. Further information about the methodology study may be found at www7.nationalacademies.org/resdoc/index.html You have been selected to receive this questionnaire because you are a member of the faculty who participates in the education of doctoral students at your university. This means that you either teach courses to doctoral students or supervise their dissertations. If this is not the case, please indicate that in question 1. The assessment of research doctoral programs is conducted approximately every ten years and consists of a reputational survey of doctoral programs and the collection of data about doctoral faculty and students in f~fty-seven areas of study. This questionnaire provides information that will assist the study in a number of ways: licit will help us construct a pool from which to select raters for the reputational survey; 2)it will provide us enough information about you that we can collect data on grants, citations, and publications from other sources; and Hit will permit a statistical description of the faculty in the graduate program or programs with which you are affiliated. Your answers will be treated as completely confidential by the National Research Council and will only be released as part of a statistical analysis. I. Program Identification a. Do you supervise dissertations, serve on doctoral committees, or teach graduate courses in a doctoral program? ~ Yes ~ No If your answer was "No", you do not need to complete the rest of the questionnaire. b. From the pulldown list, please choose the program of your primary affiliation/appointment tPull Down List of Res-Doc Programs] If you have difficulty locating your program on the list, please refer to the "Taxonomy] list with fields and subfields Please list all programs in which you supervise dissertations, serve on dissertation committees, or teach graduate courses and the average percentage of your time during the past year that you spent in all activities for each program with which you are associated. (Do not list programs where you are an outside reader.) Program Supervise dissertations Teach courses Serve on Percent of time spent in all (YIN) (YIN) dissertation activities for this program committees (YIN) ~ (total= IJ0%) ~ d. For the articles and books that you have published in the past five years, please list what fields you have published in Table 1. If you have a single publication that spans multiple fields, please indicate them and their fields in Table 2. APPENDIX D

APPENDIX D Table 1: Books and articles in a single field published in the past 3 years Field(see Taxonomy) ~ Articles ~ Books 1 1 ~ ' 1 1 1 Table 2: Books and articles in multiple fields published in the past 3 years Field (Enter all that apply) Articles Books II. Current Employment a. Department affiliation: b. Rank: ~ Instructor ~ Assistant Professor ~ Associate Professor ~ FullProfessor ~ Other c. Tenure status: ~ Tenure-track, not tenured Tenured ~ Non-tenure-track d. Year first employed at current institution: tIf employment was not continuous, please list year of most recent appointment at this institution.] Have you received an extramural grant or contract support in the past year? Yes ~ No f. Subfields of current research interest (refer to "Taxonomy] with subfields): Subfield # 1: Subfield #2: Subfield #3: g. Do you consider part of your research to be interdisciplinary? ~ Yes ~ No If so, what is the area of that research? h. Under what names or variants of your name have you published books or articles? III. Prior Experience What was your status prior to your current position? ~ Student ~ Postdoc ~ Faculty. ~ Other: Previous employer: Address: 115

116 IV. Educational Background City Title: Employment Sector: Industry (for profit) National laboratory State or local government Federal government agency International agency 4-year college or university 2-year college K- 12 school Hospital or clinic Foundation or nonprofit Military Other (specify: State/Country Zip Code- a. Highest degree earned: ~ Bachelor's ~ Master's ~ Ph.D. ~ Professional (M.D., J.D., D.V.M., for example) b. Institution that conferred highest degree: c. Field of highest degree: Other: d. Year of highest degree: tPulldown List] To what extent does the field of your current research, teaching, or professional activities differ from the field of your highest degree? ~ Very similar ~ Somewhat similar ~ Very different V. Demographic Information a. Date of birth: b. Gender: c. Citizenship Male Female U.S. Permanent Resident Temporary Visa (mmlddlyy) d. Race/Ethnicity (if U.S. citizen or permanent resident) American Indian or Alaskan Native Asian or Pacific Islander Black White APPENDIX D

APPENDIX D Hispanic (I Mexican American, ~ Puerto Rican, ~ Other) ~ Multiracial VI. Please provide your preferred e-mai! address (where you can be reached if there are questions.) Thank you for your time. ~7

118 Questionnaire for Admitted-to-Candidacy Doctoral Students This questionnaire is part of the National Research Council's Pilot Test of the Assessment of Research Doctoral Programs. Your university has volunteered to participate in this pilot test to assist the National Research Council's study of the methodology used to assess doctoral programs. One innovation we are considering is adding student responses about the educational processes of the program. We believe that students' input is important to improving the quality of the educational experience. Further information about the methodology study may be found at www7.nationalacademies.org/resdoc/index.htm! You have been selected to receive this questionnaire because you are a student who has completed over half of your doctoral program. If this is not the case, please indicate that in question 1. The assessment of research doctoral programs is conducted approximately every ten years and consists of a reputational survey of doctoral programs and the collection of data about doctoral faculty and students in fifty-four areas of study. This questionnaire will provide information that will assist the study in a number of ways: 1) it will provide a statistical description of students in your program; 2) it will provide information about practices in your program; and 3) it will help future students in the selection of graduate programs. Your answers will be treated as completely confidential by the National Research Council and will only be released as part of a statistical analysis. Individual answers will not be shared with faculty or administrators of your doctoral program except in aggregated form. Institution: Doctoral Program: Educational Program A. Year of enrollment in this doctoral program: B. Year you expect to receive your doctorate: C. Did you (or will you) receive a master's degree before this doctorate? ~ Yes ~ No D. Did you (or will you) receive a master's degree in your doctoral field as part of your training? ~ Yes ~ No Ifyes,didyouwritea master's thesis? ~ Yes ~ No E. During the course of your study for the Ph.D. will you also receive any of the following as part of a joint, concurrent, or combined degree program: Professional doctorate (e.g., MD, DDS, OD, JD)? ~ Yes Professional master's (e.g., MBA, MPA, MPH)? ~ Yes No ~ No F. During the course of your study for the Ph.D. will you also receive a certificate in another field? ~ Yes ~ No APPENDIX D

APPENDIX D G. What were your career goals at the time you entered graduate school? Check all that apply] U.S. Employment: Industry ~ Government ~ Nonprofit ~ University ~ 2-yr. college ~ 4-yr. college Other: Non-U.S. Employment: Industry ~ Government ~ Nonprofit ~ University 2-yr. college ~ 4-yr. college Other: ~ Unknown H. What are your current career plans? tcheck all that apply] U.S. Employment: Industry ~ Government ~ Nonprofit ~ University ~ 2-yr. college ~ 4-yr. college Other: Non-U.S. Employment: Industry ~ Government ~ Nonprofit ~ University 2-yr. college ~ 4-yr. college Other: ~ Unknown I. Of the following sources of support, which have been your primary sources during your doctoral studies? (Check the three largest) I. ~ Personal/family funds 2. ~ Research Assistant (RA) 3. ~ Teaching Assistant (TA) 4. ~ Training grant 5. ~ Fellowship 6. ~ Loans 7. ~ Concurrent employment related to your degree 8. ~ Concurrent employment unrelated to your degree 2. Program Characteristics A. Professional Development I. During your doctoral program have you received (or will you receive) instruction, practice or professional development training in: a. Oral communication and presentation skills: ~ Yes ~ No b. Writing proposals for funding: ~ Yes ~ No c. Preparing articles for publication: ~ Yes ~ No d. Working in collaborative groups: ~ Yes ~ No Conducting independent research/scholarship:~ Yes ~ No f. Project management ~ Yes ~ No g. Research / professional ethics ~ Yes ~ No h. Speaking to nonacademic audiences ~ Yes n No 119 Yes Yes Yes Yes Yes Yes

120 2. In your doctoral program did you have an opportunity to obtain teaching experience? Check the typets) of teaching experience you have had: a. mentoring a high school student b. mentoring an undergraduate student c. grading papers for undergraduate or graduate courses d. leading discussion sections of undergraduate or graduate courses e. leading laboratory sections of undergraduate or graduate courses f. lecturing in undergraduate or graduate courses g. tutoring undergraduates If you have had teaching experience, please answer the following, h. ~ received formal instruction in leaching. ~ Yes ~ No i. {received formal supervision end evaluation. ~ Yes ~ No j. ~ had opportunities to teach in a variety of academic environments. ~ Yes ~ No B. Program Environment 1. Does your program provide an annual or more frequent assessment of your progress? 2. Do you receive timely feedback on your research! 1 - - - - - _ Yes ~ No ~ Yes ~ No 3. Do you have access to career advice covering a variety of employment sectors? Yes ~ No ~ Yes ~ No a. If yes, are you encouraged to use it? 4. Do you have one or more faculty members at your institution that you consider mentors (i.e., individuals from whom you seek advice about your education, career development, and other matters of concern to you as a graduate student)? ~ Yes ~ No 5. How would you rate the quality of teaching by faculty in your program? ~ Excellent ~ Good ~ Fair ~ Poor 6. How would you rate the quality of your research experience? Excellent ~ Good ~ Fair ~ Poor 7. How would YOU rate the curriculum of your Ph.D. program? ~ Excellent ~ Good S. How would you rate the overall quality of your program _ ~ O ~ Fair ~ Poor ~ , , ~ ~ Excellent ~ Good ' ' ~ ~ ~ Lair ~ Poor 9. How would YOU rate the intellectual liveliness of your pro cram? ~ Excellent ~ Good 10. Considering the overall intellectual environment of your university, how much do you fee! you have benei ited from it? ~ A lot ~ Some APPENDIX D , — - - ~ o n Fair n Poor ~ A little ~ Not at all

APPENDIX D C. Infrastructure I. Does your program give you access to: a. Your own personal work space b. Computer facilities Yes ~ No ~ Yes ~ No c. Other research facilities; if so, describe: 2. Does your program provide adequate space for interaction among students? C] Yes O No 3. Are the library resources available to you adequate to support your research and education? ~ Yes C] No D. Research productivity I. How many research presentations (including poster presentations) have you made at research conferences a. on your campus? b. at national or regional meetings? 2. How many research publications have you authored or co-authored during your cloctoral studies (include pieces accepted for publication but not yet published)? a. Refereed articles b. Book chapters c. Reviews d. Books or edited volumes 3. Background information A. Date of birth: (mm/~/yy) B. Gender: ~ Male n Female C. Citizenship U.S. Permanent Resident Temporary Visa D. Race/Ethnicity (if U.S. citizen) American Indian or Alaskan Native Asian or Pacific Islander Black White Hispanic Mexican American, ~ Puerto Rican, ~ Other) ~ Multiracial E. Dependent care responsibilities: 1. Number of children living with you: Age 6 or under Over age 6 3. Parents or other dependents ~ Yes ~ No 121

122 APPENDIX D G. Marital Status: Do you have a spouse or partner who lives with you? ~ Yes ~ No F. Level of Parents' Education: Mother Father High school diploma or less Some college/Bachelor's degree Advanced degree

APPENDIX D Five-Seven Years Post-Ph.D Questionnaire This questionnaire is part of the National Research Council's Pilot Test of the Assessment of Research Doctoral Programs. Your university has volunteered to participate in this pilot test to assist the National Research Council's study of the methodology used to assess doctoral programs. One innovation that we are considering is to add student responses to questions about the educational process of the program. Further information about the methodology study may be found at www7.nationalacademies. org/resdoc/index.html You have been selected to receive this questionnaire because you are a student who has received a Ph.D. from this program five to seven years ago. If this is not the case, please indicate that in question 1. ~ 4, , I, The assessment of research doctoral programs is conducted approximately every ten years and consists of a reputational survey of doctoral programs and the collection of data about doctoral faculty and students in fifty-four areas of study. This questionnaire provides information that will assist the study in a number of ways: 1) it will help us learn whether a high enough percentage of students respond so that we can add student observations to the larger study; 2) it will provide us enough information about practices in your program that we can compare the practices of graduate programs in your field at different universities; and 3) it will permit a statistical description of the f~rst-year students in the graduate program. Your answers will be treated as completely confidential by the National Research Council and will only be released as part of a statistical analysis. Individual answers will not be shared with faculty or administrators of your former doctoral program except in aggregated form. Educational Program a. Name of the program where you received your Ph.D. degree: b. Year of enrollment in the above Ph.D. program: c. Year you received your Ph.D.: d. Did you receive a master's degree at this institution before this Ph.D.? ~ Yes ~ No e. Were you enrolled as a full-time student throughout your Ph.D. program? ~ Yes ~ No f. Did you attend graduate school prior to enrollment in the above Ph.D. program? ~ Yes ~ No If so, what degrees or certificates, if any, do you hold? ~ Certificate ~ Master's ~ Doctoral ~ Professional g. What was your career goal when you completed your Ph.D.? U.S. Employment: Industry ~ Government ~ Nonprofit ~ University 2-yr. college ~ 4-yr. college Other: 123

124 Non-U.S. Employment: Industry ~ Government ~ Nonprofit ~ University 2-yr. college ~ 4-yr. college Other: ~ Unknown h. Have your career goals changed since you received your Ph.D.? ~ Yes ~ No i. During your Ph.D. program, were you supported by funds from outside the institution? ~ Yes ~ No (Check all that apply) Type: ~ Fellowship ~ Training Grant ~ Research Grant ~ Your employer ~ Other(Specify: ! J. Did you receive institutional support? ~ Yes ~ No (Check all that apply) Type: ~ Teaching Assistantship ~ Research Assistantship ~ Fellowship ~ Tuition scholarship or waiver only ~ Loan ~ None ~ Other(Specify: ! 2. Employment and Career Status a. First employer or place of postdoctoral study after Ph.D. completion: Name: Address: City State/Country Zip Code- Title: b. Employment Sector: Industry (for profit) National laboratory State or local government Federal government agency International agency University 4-year college 2-year college K-12 school Hospital or clinic Foundation or nonprofit Military Other (specify) APPENDIX D

APPENDIX D c. If you hold or have held a postdoctoral position or positions, how many , and at what institutions, companies or government agencies were they located? List chronologically starting with the most recent. Position # 1: Position#2: Position # 3: Position#4: Dates: Dates: Dates: Dates: d. Current employer: Name: Address: City State/Country Zip Code- Title: e. Current Employment Sector: Industry (for profit) National laboratory State or local government Federal government agency International agency University 4-year college 2-year college K-12 school Hospital or clinic Foundation or nonprofit Military ~ Other (specify) 3. Ph.D. Program Characteristics a. During your Ph.D. education, in which of the following areas was training PROVIDED, which skills or experiences have you USED since graduation, and which area do you wish you had learned MORE about? (check all that apply) 1) Teaching experiemce 2) Oral communication; presentation skills 3) Writing proposals for funding 4) Manuscript preparation Provided Provided Provided Provided Experience working in collaborative groups ~ Provided 6) Critical analysis 7) Locating and applying information 125 Used ~ More Used ~ More Used ~ More Used ~ More Provided Provided Used ~ More Used ~ More Used ~ More

26 8) Experience working with people of varied educational levels ~ Provided ~ Used ~ More 9) Experience working with people from diverse backgrounds ~ Provided ~ Used ~ More 10) Experience working in teams b. Research Productivity Provided ~ Used ~ More How many books or edited books have you published or are currently accepted for publication? 2) How many articles or book chapters have you published or are currently accepted for publication? 3) How many books or articles have you reviewed for publication? 4) How many reviews, enumerated in 3), have been or will be published? 5) How many refereed papers have you or a coauthor presented at professional conferences? How many awards have you received? (Respond to all categories.) a) For teaching: b) For research: From professional societies: From your institution or employer: 7) How many patents or licenses have you received? 8) How many grants have you received from your employer or institution? 9) How many grants have you received from extramural funding agencies? 4. Background Information a. Date of birth: b. Gender: c. Citizenship APPENDIX D Male Female U.S. Permanent Resident Temporary Visa (mmlddlyy)

APPENDIX D 127 d. Race/Ethnicity (ifU.S. citizen) American Indian or Alaskan Native Asian Pacific Islander Black White Hispanic (~ Mexican American, ~ Multiracial e. Martial Status ~ Married ~ Single f. Number of Children: Age 6 and under Over age 6 g. Level of Parents' Education: Less than high school High school diploma Some college Bachelor's degree Master's degree Professional degree Doctoral degree h. Is English your first language? Mother Yes ~ No Puerto Rican, ~ Other) Father

Next: Appendix E: Taxonomy of Fields and Their Subfields »

One thought on “Research Paper Based On Questionnaire Sample

Leave a Reply

Your email address will not be published. Required fields are marked *