Identifying bias in samples and surveys

It's important to identify potential sources of bias when planning a sample survey.

When we say there's potential bias, we should also be able to argue if the results will probably be an overestimate or an underestimate.

Try to identify the source of bias in each scenario, and speculate on the direction of the bias (overestimate or underestimate).

David hosts a podcast and he is curious how much his listeners like his show. He decides to start with an online poll. He asks his listeners to visit his website and participate in the poll.

The poll shows that $89\mathrm{\%}$‍ of the $200$‍ respondents "love" his show.

David hosts a podcast and he is curious how much his listeners like his show. He decides to poll the next $100$‍ listeners who send him fan emails.

They don't all respond, but $94$‍ of the $97$‍ listeners who responded said they "loved" his show.

A senator wanted to know about how people in her state felt about internet privacy issues. She conducted a poll by calling $100$‍ people whose names were randomly sampled from the phone book (note that mobile phones and unlisted numbers aren't in phone books). The senator's office called those numbers until they got a response from all $100$‍ people chosen.

The poll showed that $42\mathrm{\%}$‍ of respondents were "very concerned" about internet privacy.

A senator wanted to know about how people in her state felt about internet privacy issues. She conducted a poll by calling people using random digit dialing, where computers randomly generate phone numbers so unlisted and mobile numbers can still be reached. They called over $\mathrm{1,000}$‍ random phone numbers—most people didn't answer—until they had reached $100$‍ respondents.

The poll showed that $46\mathrm{\%}$‍ of respondents were "very concerned" about internet privacy.

A high school wanted to know what percent of its students smoke cigarettes. During the week when students visited the counselors to schedule classes, they asked every student in person if they smoked cigarettes or not.

The data showed that $5\mathrm{\%}$‍ of students smoked cigarettes.

A high school wanted to know what percent of its students smoke cigarettes. Counselors selected a random sample of students to take a survey on drug use. One of the questions reads, "If you are under the age of $18$‍ years, do you illegally smoke cigarettes?"

The data showed that $5\mathrm{\%}$‍ of students smoked cigarettes.