Theme 4 - Quantitative research

Which quantitative method or methods are used in the paper? Which are the benefits and limitations of using these methods?

I have read the paper “Channeling Science Information Seekers’ Attention? A Content Analysis of Top-Ranked vs. Lower-Ranked Sites in Google” by Nan Li, Ashley A. Anderson, Dominique Brossard, Dietram A. Scheufele (2013).

The purpose of the paper was to determine the emphasis search engines has on search results, how biased those results will be depending on popularity. They wanted to look at how skewed the information would become when big, popular sites diminished the visibility of smaller, less popular ones. Could this contribute to an underrepresentation of minority voices, they ask. To measure this, the authors of the paper created a bot that empirically crawled across over 200 000 sites to look for root words regarding a certain subject. They then ranked these results based on the frequency of these words.

This is a pure quantitative method of gathering data, it’s consistent but lacks validity. However, they did also conduct some qualitative research methods by using data from a structured panel discussion about the subject to, together with qualitative data from a public opinion survey, formulate search terms which they used to get a better sense of what the user would see when searching for the same subject as the bot and to extract links related to the subject. Which later was used in the bot’s algorithm.

Because they used a bot to conduct research, to collect data samples, they performed a huge number of requests. This in turn yielded tons of data with pretty good reliability, distributed over months. This gives them a pretty good image, with not so much effort put in in the long term. They also got a lot of easy to use data, with numbers they could get a good grasp on the state of the problem without having to analyze different inputs. At the same time the data the bot gathered were of pretty poor quality. The way they measured what should have been higher up in the search results was questionable because the algorithm the bot used could have been more extensively developed, to consider a broader variety of factors than frequency across a set number of root words. 

What did you learn about quantitative methods from reading the paper?

I learned to not only rely on the top results when conducting research, other than that i wouldn’t say that this paper brought something new to the table than what I already knew. Maybe that using some sort of multi-method would be better to get a more comprehensive picture of the problem. They really only used a quantitative method to gather data, how they would use qualitative methods to look for search rankings I have no idea how to accomplish.

Which are the main methodological problems of the study? How could the use of the quantitative method or methods have been improved?

I think this goes hand in hand with what i wrote above, about what they measure. Can you really measure the quality of a text based on how often certain words show up and compare that to another text? I think they would need a more advanced bot to conduct research with some form of AI, to analyze the content if they wanted more “true” results. They drew graphs based on the word-frequency to see the state of the research more easy by comparing different root word searchterms, which I think contributed to generalized, saturated results. The bot malfunctioned under two months which made them lose about ⅙ of the data which I am sure made the results somewhat skewed. 

Which are the benefits and limitations of using quantitative methods?

They make it easy to conduct surveys, structured interviews and observations. Quantitative methods gives a less biased, more objective point of view primarily based on numbers which lays the ground for easy to generalize, statistical tests. If the measurement device is objective and stable, the data is valid and reliable. The data that comes out is easy to analyze but the time expenditure of creating the quantitative method is heavy. 

Which are the benefits and limitations of using qualitative methods?

Qualitative methods are based on focus groups, in-depth interviews and document reviews. Unlike quantitative methods this is more subjective, the condition described are from the point of view of the one experiencing it. The information gathered is often unstructured but more in-depth than quantitative methods and consists primarily of text rather than numbers, which makes it pretty bad for statistical tests and harder to generalize. Using qualitative methods the time expenditure lies on the analyzis phase rather than the planning.