CSCI 848: Empirical Methods in Software Engineering

Formulating Research Questions (RQs) Refer EXERCISE # 1 SELECTING EMPIRICAL METHODS FOR SE RESEARCH Despite widespread interest in empirical software engineering, there is little guidance on Which research methods are suitable for which research problems? Software engineering is a multi-disciplinary field

Because of the importance of human activities in software development, research methods are drawn from other disciplines 2 CLASSES OF RESEARCH METHODS IN SE Controlled experiments (including Quasi-Experiments) Case Studies Survey Research 3 HOW TO SELECT A RESEARCH METHOD? (START BY FORMULATING RESEARCH QUESTIONS)

To illustrate the steps involved in deciding which method or methods to use, we present two guiding examples. Two fictional software engineering researchers, Joe and Jane, will explore how the various research methods can be applied to their work: Some of the examples will be used in the remainder of the slides. The other examples will be used for EXERCISE #1 4 WHAT RESEARCH QUESTIONS ARE YOU ASKING?

The first step in choosing an appropriate research method is to clarify the research question. Asking questions + systematic process of obtaining valid answers Problem Statement RQ and Hypothesis 5 RESEARCH QUESTION Often the most obvious question is not the best choice for starting point. Janes first attempt to formulate her RQ is: Is a fisheye-view file navigator more

efficient than the traditional view for file navigator Joes first attempt to formulate her RQ is: How widely are UML diagrams used as collaborative shared artifacts during design? 6 RESEARCH QUESTIONS Both questions are vague, because they make assumptions about the phenomenon (to be studied) and the kinds of situations (in which these phenomenon occur). For example: Is a fisheye-view file navigator more efficient than the traditional view for file navigator

Janes question only makes sense if we already know that some people (who?) need to do file navigation (whatever that is?), under some circumstances (which are?), and that efficiency (measured how?) is a relevant goal for these people (how would we know that?) 7 RESEARCH QUESTIONS For example: How widely are UML diagrams used as collaborative shared artifacts during design? Joes question presupposes that we know what a collaborative shared artifact is, and can reliably identify one, and even reliably say which things are

UML diagrams Defining the precise meaning of terms is a crucial part of empirical research, and is closely tied with the idea of developing (or selecting) an appropriate theory 8 RESEARCH QUESTIONS In early stages of a research program, we need to ask exploratory questions Suitable research methods for exploratory questions helps us to build tentative theories Unless Jane and Joe are building on existing work, they

need to formulate exploratory questions, such Existence Question : Does X Exist? Description and Classification Questions: What is X like? Descriptive- Comparative questions: How does X differ from Y? 9 RESEARCH QUESTIONS: EXISTENCE QUESTIONS Existence questions of the form, Does X exist? X is a thing, attribute, phenomenon, behavior, ability, condition, state of affairs etc.

Is there a programmer who can write 200k lines per year? Jane might need to ask, Is file navigation something that (certain types of programmers) actually do? and, Is efficiency actually a problem in file navigation? What might JOE need to ask? EXERCISE # 1 10 RESEARCH QUESTIONS: DESCRIPTION/CLASSIFICATION QUESTIONS Description and Classification questions such as, What is X like?,

What are its properties?, How can it be categorized?, How can we measure it?, What is its purpose?, What are its components?, How do the components relate to one another?, What are all the types of X? Jane might ask, How can we measure efficiency for file navigation? What might JOE ask? EXERCISE # 1 11

RESEARCH QUESTIONS: DESCRIPTIVE-COMPARATIVE QUESTIONS Descriptive-Comparative questions of the form, How does X differ from Y? investigate similarities and differences between two or more phenomena. Jane might ask, How do fisheye views differ from conventional views? What might JOE ask? EXERCISE # 1 12 WHY FORMULATE EXPLORATORY RESEARCH

QUESTIONS? The answers to these questions result in a clearer understanding of the phenomena, including more precise definitions of the theoretical terms, evidence that we can measure them, and that the measures are valid. In exploring these questions, Jane and Joe will refine their ideas about the nature of the phenomena they are studying. It is possible that there are already good answers to these questions in the published literature. A literature survey, instead of an empirical study, may answer them. Once we have a clearer understanding of the phenomena, we may need to ask base-rate questions about the normal patterns of occurrence of the phenomena

Helps evaluate whether situation is normal or abnormal? 13 BASE-RATE QUESTIONS Example Base-Rate Questions include: Frequency & Distribution Questions: How often does X occur? Descriptive Process Questions: How does X normally work? 14

RESEARCH QUESTIONS: FREQUENCY AND DISTRIBUTION QUESTIONS Frequency and distribution questions such as, How often does X occur? and, What is an average amount of X? Joes original question appears to be a frequency question, but there he can formulate it more precisely. For example, he might ask, How many distinct UML diagrams are created in software development projects in large software companies?

He might discover the results follow some standard statistical distribution. What might JANE ask? EXERCISE # 1 15 RESEARCH QUESTIONS: DESCRIPTIVE PROCESS QUESTIONS Descriptive-Process questions such as, How does X normally work? What is the process by which X happens? In what sequence do the events of X occur?, What are the steps X goes through as it evolves?, How does X achieve its purpose?.

For example, Jane might ask, How do programmers navigate files using existing tools? What might JOE ask? EXERCISE # 1 16 RESEARCH QUESTIONS: RELATIONSHIP QUESTIONS Often, we are interested in relationship questions between two phenomenon, specifically whether the occurrence of one is related to the occurrence of the other. Hence we need to formulate: Relationship questions: Are X and Y related?

Do occurrence of X correlate with occurrence of Y? 17 RESEARCH QUESTIONS: RELATIONSHIP QUESTIONS Relationship questions such as, Are X and Y Related? Do occurrences of X correlate with the occurrences of Y? Jane might ask, Does efficiency in file navigation correlate with the programmers familiarity with the programming environment?

What Joe might ask, EXERCISE # 1 18 NEXT STEP Once we have established that a relationship exists between two phenomena, it is natural to try to explain why the relationship holds by attempting to identify a cause and effect. It is a common mistake to confuse correlation with causality. In general it is much harder to demonstrate causality than to show that two variables are correlated. If high values of X correlate with high values of Y, it may be because X causes Y, or because Y causes X.

But it is also possible that X and Y share some common cause and neither causes the other. Or perhaps they co-evolve in complex ways so that there is no clear cause-and-effect. 19 RESEARCH QUESTIONS: CAUSALITY QUESTIONS Descriptive-Process questions such as, Does X cause Y? Does X prevent Y? Plus the more general forms: What causes Y?, What are all the factors that cause Y?, What effect does X have on Y?

For example, Jane might ask, Do fisheye-views cause an improvement in efficiency for file navigation? What JOE might ask? Exercise # 1 20 RESEARCH QUESTIONS: CAUSALITY-COMPARATIVE QUESTIONS Causality-Comparative questions investigates the relationship between different causes Does X cause more Y than does Z? Is X better at preventing Y than is Z? Unless Jane has good base-rate data for existing file

navigation tools, Janes causality question would be better formulated as Do fisheye-views cause programmers to be more efficient at file navigation than conventional views? What JOE might ask? Exercise # 1 21 RESEARCH QUESTIONS: CAUSALITY-COMPARATIVE INTERACTION Causality- Comparative Interaction questions investigate how context affects a cause-effect relationship: Does X or Z cause more Y under one condition but not others?

If Janes initial studies reveal a factor (e.g., distractions) that affects causality, she might ask Do fisheye-views cause programmers to be more efficient at file navigation than conventional views when programmers are distracted, but not otherwise? What JOE might ask? Exercise # 1 22 EMPIRICAL VS. NON-EMPIRICAL RESEARCH QUESTIONS The classes of research question above are all knowledge questions focused on the way the world is. In contrast, most non-empirical research in software

engineering focuses on a very different type of question concerned with designing better ways to do software engineering 23 DESIGN QUESTIONS Design questions of the form, Whats an effective way to achieve X? or, What strategies help to achieve X? For example, Joes research might lead him to ask, What is an effective way for teams to represent design knowledge to improve coordination? These types of question are necessary when the goal is to design better procedures and tools for carrying out some activity Such questions presuppose that the associated knowledge questions

have already been addressed so that we have enough information about the nature of the design problem to be solved. In practice, a long term software engineering research program involves a mix of design questions and knowledge questions 24

Recently Viewed Presentations

  • The Second Amendment

    The Second Amendment

    Arguments for Gun Control The Constitution is an antiquated document The Constitution is a "living" document No need for a gun except hunting and target shooting today Substance abuse related killings Crimes of passion - temptation People simply cannot be...
  • What is a Sensor Web ? Abhinav Roongta

    What is a Sensor Web ? Abhinav Roongta

    What is a Sensor Web ? Abhinav Roongta Wireless Information Networking Group University of Florida March 3, 2004 Outline Sensor Web Definition Basics Organization Potential Current applications Introduction Definition According to NASA New Technology Report on Sensor Webs, " The...
  • Microsoft Robotics Studio: A Robot In Every Home

    Microsoft Robotics Studio: A Robot In Every Home

    Microsoft Robotics Studio: A Robot In Every Home and Business ... Runtime Extensible Platform Enables users to transfer learning skills Third-parties provide additional services to extend functionality Allows communication despite geographic location Visual Programming Tool Makes it easier to ...
  • Linearity - University of Texas at Austin

    Linearity - University of Texas at Austin

    Using the definition requires a contour integration in the complex z-plane Use Cauchy residue theorem (from complex analysis) OR Use transform tables and transform pairs? ... * Z-transform Properties Linearity Right shift (delay) Second property used in solving difference equations...
  • Kista Rongga Mulut - Fk Uwks 2012 C

    Kista Rongga Mulut - Fk Uwks 2012 C

    Kramer, Pindborg dan Shear, 1992. Dimodifikasi oleh Shear) - Kista pada rahang. Epitelial. Pertumbuhan (a) Odontogenik. ... Varian dari kista odontogenik dan tumor, misalnya: keratosis odontogenik, kista radikular, kista periodontal lateral.
  • PMACS Operations Portal (POP)

    PMACS Operations Portal (POP)

    PMACS Operations Portal (POP) Kevin Lux, Senior Systems Programmer. Penn Medicine. [email protected] @luxk on Slack. Overview. Background info. How POP addresses the challenges. Design of POP and an abbreviated feature list. Current production use cases of POP.
  • Mountain Park Christ Centered Fitness

    Mountain Park Christ Centered Fitness

    Christ centered fitness offers our students the opportunity to become healthy in mind, body, and spirit. We are free from the judgment, social/media pressures, and the negative stereotyping common in secular gyms.
  • Corso di Informatica per Giurisprudenza

    Corso di Informatica per Giurisprudenza

    SHOPPING Jewelry The perfect gift for Valentine's Day Valentine's Day Shop in the name of love at neiman marcus.com Gift Ideas Diamond Bracelets. Free overnight shipping! Flowers & Gifts Proflowers roses direct from the grower at $39.95 Travel Book Travel...