Qualitative and quantitative research: comparison of different research methods
Anna Kozerawska
Elizabeth Kovatcheva
The research problem
Since the computer-related technologies became widely available, companies started to employ the new technologies seeking efficiency and effectiveness benefits. We found it very interesting to do some research into the issue, trying to find scientific evidence for association between IT capability and a company's competitive advantage. The question we posed was: does a company benefit from employing IT? If so, how? To get an answer (and to comply with the assumptions of this exercise), we selected three peer reviewed scientific papers that dealt with the issue and that provided scientific evidence using three different research methods.
Paper 1 - Research Method: Statistical Tests
A resource-based perspective on IT capability and firm performance: An empirical investigation by A.S. Bharadwaj, MIS Quarterly, March 2000
In this paper IT in a company is studied from the point of resource-based view of company, which attributes superior financial performance to organizational resources and capabilities. It develops the concept of IT as an organizational capability and empirically examines the association between IT capability and firm performance.
The research method employed in this paper is statistical testing of hypotheses. The author comes up with two main hypotheses:
1. Superior IT capability will be associated with significantly higher profit ratios.
2. Superior IT capability will be associated with significantly lower cost ratios.
The examination was conducted through statistical matched-sample comparison group methodology. Two sets of companies were created: treatment sample and control sample. To form the treatment sample, rankings of InformationWeek from 1991 till 1994 were used. That resulted in a treatment sample of 56 firms. The matching set of control firms was drawn from the Compustat database. It was assured that none of the control firms were ranked as an IT leader and each equalled to the one from treatment sample by industry group and size.
For this particular study the Wilcoxon Rank Sum Test was used, which is intended to evaluate the differences between target variables for the two groups of companies. It was chosen among others because it’s more resistant to departures from normality and is considered more powerful than the pairwise t-test.
The test results were reported as Wilcoxon Rank Sum Z-statistics. As was hypothesized, all the profit ratios in each of the 4 years were significantly higher for the IT leaders when compared to the control sample of firms. As for cost ratios, total operating expenses to sales (OEXP/S) was lower for the IT leaders sample in all 4 years. The cost of goods to sales (COG/S) was also lower for the IT leaders in all 4 years with significance (at 10% level) in 2 of the 4 years. In contrast to expectations, the results for selling and administrative expenses to sales ratio (SGA/S) appeared to be higher for IT leaders than for control sample.
Empirical findings of this study indicate that IT capability is rent generating resource that is not easily imitated and sustituted. Isolating mechanisms such as time compression diseconomies, connectedness of resources, and social complexity allow firms with high IT capability to achieve and sustain superior performance.
Paper 2 - Research Method: Single In-Depth Case Study
Implementing supply chain management: Lessons learned at Becton Dickinson by J.J. Kanet & A.R. Cannon, Production and Inventory Mangement Journal, 2nd Q 2000
This paper deals with implementation of the IT technology in the form of supply chain management (SCM), in this particular case two elements of SCM: distribution requirements planning (DRF) and continuous replenishment process (CRP), at a major manufacturer of medical equipment and supplies. The focus of this paper is on the question, whether launching of new technology did improve company’s performance.
Becton Dickinson is a major American supplier to the healthcare industry. Provides products to physicians, hospital clinics and labs, public health labs, blood banks and research institutions. It competes in two segments: diagnostic systems and medical supplies and devices, from which it generates equal revenues. It is also considered a global company, as half of its sales are made to overseas markets.
The object of this single in-depth case study was the company's strategy, which involved three steps:
Implementation of the Supply Chain Management Integration Software – a logistics planning system that included Distribution Requirements Planning System, which enabled to forecast individual product demand and Continuous Replenishment Processing System for calculating order points and appropriate safety stock levels. It also enabled efficient allocation and deployment of existing stocks between warehouses.
Implementation of EDI (Electronic Data Interchange), which allowed electronic communication of supply and demand transactions between Beckton Dickinson and its network of distributors as well as efficient creation and handling of new orders and cost-efficient shipments to distribution centers.
Creating a new division - BDSCS (Supply Chain Services) with responsibility for order fulfillment in every market segment.
The implementation procedure involved creation of seven teams with different set of goals and responsibilities. The teams were composed of individuals from functions that had critical roles in implementation process. The activities involved:
1. Pilot testing of new system (single delivery partner for initial piloting)
2. Personnel training
3. Creation of a technical reference library housing to operate new computer systems
4. PC-equipped facility for hands-on training in the new system
5. Advertising to launch new system at all divisions of the Becton Dickinson.
6. Early assessments to check the reduction of the logistics chain
7. Creation of cross-functional teams to ensure adoption and acceptance of the new technology.
The results of the study carried out between 1994 and 1999 showed that DRP/CRP system made it possible to predict the upcoming production requirements with greater accuracy, and thus enabled inventory reductions. With respect to receivables and required base of fixed assets the income per finished goods inventory improved by and average of 9,5%, also income per plant, property and equipment improved by an average of 8% and the ratio of revenues to receivables fell by an average of 0,69%. Overall, the results showed general effectiveness of Becton Dickinson's implementation of a new approach to SCM. The contribution of the case study was that the strategy could be used at other organizations in consideration of its individual characteristics.
Paper 3 - Research Method: Multiple Case Study
Approaches to the development of multi-dimensional database: Lessons from four case studies by H. Hasan, P. Hyland, D. Dodds, R. Veeraraghavan, Database for Advances in Information Systems, Summer 2000
The paper explores the manner in which an organization’s data can be effectively utilized to assist an organization to achieve its business objectives. It presents Multi-Dimensional Data Bases (MDDB) as a new tool of making a more efficient use of company's data and formulates a more useful approach to build an MDDB that suits the information needs of a company better. The research paper is also aiming at making a contribution to the knowledge of MDDBs, as there is a need for more objective and theoretical analysis in the field.
The research method employed in this paper was a combination of classic case study and implementation of a prototype. Four case studies were carried out. The first two cases involved implementation of MDDBs that were developed using two opposite approaches: top-down, i.e. with main focus on business planning, and bottom-up, i.e. with main focus on existing systems and databases. Successful aspects of the two developments were used to create a new ”middle-out” approach – a combination of the previous two approaches. Two prototype MDDBs were developed using the middle-out approach and implemented in two subsequent case studies.
The analysis of the four case studies intended to explore suitability of different approaches to system development. It involved development of MDDB applications related to a specific business problem in an organization, using Gentia tool, with the author’s actively participating in all four development and implementation procedures.
Qualitative data were collected as a source of well-grounded, rich descriptions and explanations of processes in identifiable local contexts. Interpretation of that data was carried out by the research team and analyzed through group discussions in order to determine accurate descriptions of the case studies.
1st Case Study
The MDDB was developed to register, track and monitor problems dealt with by an IT help-desk in a large multinational company. The top-down approach was applied with a positive result: the manager positively evaluated the MDDB system as satisfying his information needs.
2nd Case Study
The MDDB was developed to monitor and analyze student data in a large university. This time an opposite, bottom-up, approach was applied, focusing not on the business planning needs and objectives, but on existing systems and databases. The result was that the manager showed some interest in the system, but it did not fully correspond with his information needs.
3rd and 4th Case Study
In both cases the middle-out approach was used to develop the MDDB, i.e. taking into consideration both the management’s or user’s information needs as well as the available data sources and systems. In the third case study an MDDB was developed to assists managers in a university to collect and track research publication data. The result was presented to the users and their suggestions were taken, as to improvements that could be made two the MDDB systems. Some of the proposed changes were seen as viable and could provide significant improvement to the system with little effort. Other deficiencies however proved not worth of correcting, as the effort would far outweigh the benefits. The fourth case study involved development of MDDB with a goal to provide statistical data from external sources to company executives and market analysts. The results showed that a combination of top-down and bottom-up approaches has produced an acceptable, general purpose MDDB system.
Overall, the four case studies have proved that the dimensional view of data provides users with an effective means of making sense of large quantities of data. Compared to RDB, the simplified yet meaningful structure of data in MDDB matches the information needs of decision makers: senior managers who regularly need summarized and comparative information on performance towards business goals. In most cases that data must be extracted from the underlying relational structure, which is more useful for processing day-to-day business transactions. The two are thought to be complementary, not competing database architectures. At the same time the study raised several issues, which should be the subject of further research, such as development methodologies used with both MOLAP and ROLAP systems, dealing with problems that would arise when transferring the data to an MDDB structure, how to handle changes of MDDB structure over time effectively, to what extent such systems should be tailored to needs of individual managers. The paper also points out that there is a need for some standard tools for MDDB.
Qualitative versus Quantitative Research Methods
Studying the different research methods: statistical testing of hypotheses, single in-depth case study and a multiple case study with elements of prototype implementation, used in the three above summarized papers, we came to the following conclusions regarding the advantages and drawbacks of each research method:
Statistical analysis is helpful when there is a need to determine certain facts, or correlations between facts. It provides an answer to what? question. In paper 1 statistical testing of hypotheses showed that there was a correlation between the IT capability and a company’s performance, however it failed to answer the how? question, it did not provide any explanatory data regarding what is the direction and nature of the cause-effect sequence of the correlated variables. That question was better answered by the case studies method, which provided evidence for at least one direction of the causality between the correlated events: connection between introducing IT and benefiting from it.
Statistical methods are usually applied based on a model simplifying reality. It therefore depends on the quality of the model how closely the results of statistical tests are reflecting reality. The qualitative research methods, like case studies, are by their nature reflecting reality. The limitation of this method is of a different kind, it lies in the possibility of making errors of miscommunication and interpretation when describing and analyzing reality.
On the other hand, statistical methods, thanks to the simplifications of their models, are helpful, when conducting a research on a broader scale. Results obtained through a well conducted statistical testing are safer to generalize, whereas results of case studies can depict the reality better, but are limited when it comes to making conclusions about more general population.
Case studies' advantage is that they show how processes work in practice and therefore can be more useful for practitioners than for theorists. For example, the studies conducted in papers 2 and 3 could greatly benefit other companies seeking to implement given type of IT (SCM, MDDB) as to information about products, methods, processes, etc. whereas the results of statistical testing from paper 1 would contribute the information, that they can benefit from employing IT, it would however not provide any further information on how to do it.
Summing up, both qualitative and quantitative research methods can provide valuable contribution to scientific knowledge. It is only the nature of their contribution that is different. They should be regarded as complementary, not competing methods of generating scientific knowledge and should be chosen depending on which method is more likely to provide a more comprehensive, clearer, more complete and above all more descriptive of reality answer to the research question we are stating.