empiricism

What is empiricism?

Empiricism is a philosophical theory applicable in many disciplines, including science and software development, that human knowledge comes predominantly from experiences gathered through the five senses.

In empiricism, knowledge is spoken of as a posteriori, or "from the latter," meaning gained from experience. Simply put, empiricism is the idea that all learning comes from only experience and observations.

The term empiricism comes from the Greek word for experience: empeiria. The theory of empiricism attempts to explain how human beings acquire knowledge and improve their conceptual understanding of the world.

In science, empiricism heavily emphasizes the use of experiments and observation to collect evidence and draw conclusions. The goal of such experimentation is to apply theories to real-world observations, record the findings in the form of empirical data and present them to the relevant audience.

Some other illustrative real-world examples of empiricism are the following:

• measurement
• sensors
• hypothesis formed with rational thought
• correlation causation
• data dredging

Empiricism vs. rationalism

The idea of empiricism contrasts with the idea of rationalism. While empiricism is about improving knowledge through experimentation and firsthand experience, rationalism -- also known as intellectualism -- says that knowledge can also be developed by exploring concepts and through deduction, intuition and revelation.

In rationalism, deductions based on intuition can create knowledge without prerequisite sensory experience. Since knowledge is gained prior to experience, rationalism is associated with the term a priori and, therefore, also known as apriorism. This differentiates rationalism from empiricism, which is always associated with the idea of a posteriori or knowledge after experience. However, empiricists sometimes argue that all these mental processes also come from primary experiences, at least initially.

Empiricism in software development and project management

Empiricism is an important concept in IT, particularly in the following:

• software development
• data analytics
• project management

In these areas, empiricism refers to an evidence-based approach that uses real-world data, metrics and results to build knowledge and prove hypotheses, rather than simply relying on unproved theories and concepts.

Both Agile software development and Agile project management are considered empirical approaches. In these areas, work is completed in small sections called iterations. At the end of each iteration, the results are reviewed and critiqued by the project team and/or other stakeholders.

The decision about the next step is based on the experience of the previous iteration and its review. Thus, in the Agile retrospective, each team member observes the work done so far and answers two important questions:

1. What worked well for us?
2. What did not work well?

Based on the answers to these questions, they get guidance to answer a third question:

3. What actions can/should we take to improve our process going forward?

Empirical research

Empirical research is driven by the idea that "I will believe it when I see it." In this type of research, conclusions are drawn from verifiable evidence that's obtained either by observation or by scientific data collection methods. Verifiable evidence is also known as empirical evidence.

Empirical evidence can be collected in two ways:

1. Quantitative methods
   • experiments;
   • surveys;
   • polls;
   • causal-comparative research, which is aimed at finding a cause-effect relationship between two or more groups;
   • correlational research, which examines relationships between two or more variables, without the researcher manipulating them;
   • cross-sectional research, which looks at data from a population at a particular point in time; and
   • longitudinal study in which data is collected repeatedly for the same subjects over time.
2. Qualitative methods
   • interviews;
   • observation;
   • textual analysis;
   • case studies; and
   • focus groups, e.g., in market research.

Qualitative empirical research methods are often unstructured or semistructured, and they are usually used to discover subjects' opinions or feedback. Quantitative methods are almost always structured and used when there is a need to quantify one or more defined variables.

The method chosen would depend on the data sample, that is, whether the data is numerical and quantifiable or non-numerical and, therefore, unquantifiable. In some cases, both methods are used to gather empirical evidence.

Need for and benefits of empirical research

Most empirical research projects incorporate these features or characteristics:

• research questions;
• definition of the research variables;
• description of the research methodology (design, processes, tools);
• research outline; and
• findings.

A hypothesis is also an important element of empirical research. The researcher runs experiments or sets up a way to observe conditions to prove or disprove the hypothesis and thus add to the human knowledge base.

Empirical research is crucial to strengthen traditional or nonempirical research practices with experiments, observations and tangible results. Since it is based on verifiable facts and actual experiences, it adds authenticity and believability to a research project.

With empirical research, researchers can analyze the dynamic changes that happen and modify the research strategy accordingly. They also have greater control over research variables, giving them higher control over the experiment itself.

Empirical research process

Since empirical research is primarily based on observation and experimentation, it's important to conduct it in a systematic and step-by-step manner. By laying out clear steps, researchers can resolve challenges that may arise during research before they can cause further problems later in the process.

The key steps involved in empirical research include the following:

• Define the purpose of the project, and create the problem statement.
• Find supporting theories.
• Do a literature review -- if available and relevant.
• Create a hypothesis.
• Set up variables.
• Define the research methodology and strategy.
• Identify the tools required for research.
• Collect data.
• Analyze data.
• Report results/findings.
• Conclude the experiment.

In general, the empirical research cycle consists of five phases:

1. Observation. Gather empirical data.
2. Induction. Frame a general conclusion from the gathered data.
3. Deduction. Create a conclusion from the experiment.
4. Testing. Analyze and validate collected data with appropriate statistical methods.
5. Evaluation. Present the gathered data and the conclusions of the experiment to the relevant audience or stakeholders.

Biases and quantitative fallacies in empirical research

Empirical research and analysis are prone to many biases and fallacies that may invalidate the findings. For example, outcome reporting bias refers to a study's author only including positive results in the report, while leaving out results that appear negative. Similarly, spin bias is the use of language to make negative or statistically insignificant results appear positive.

A quantitative fallacy in empirical research refers to the belief that something is true because it is supported by tangible numbers. For instance, people might not believe a study that says, "Organizations that focus on customer experience earn more revenues." However, they are likely to believe a study author who says, "81% of organizations that prioritize customer experience earn 47% higher revenues" -- even if the numbers themselves are manipulated or fake.

See also: Scrum, data collection, data mining, data cleansing, data curation, data validation and big data.