Lateral Research Evaluation Strategies

Is a lateral scan of review opinions useful?

• If their are complaints in high numbers about a product or service...it should be taken into consideration.
• However:
  • Each source of review or conclusions should (ideally) be evaluated for reputation of objectivity and accuracy.
  • If the competition is high, positive and negatives reviews can be planted or purchased.
  • Paper mills /AI generated content can generate fake content that matches the perceived expectation, instead of being based on real experience.

There are many reasons conclusions may be contradictory:

• The issue may be more complex than the variables studied.

  • Many system components do not function in isolation, so studying them in isolation does not give you an accurate estimate of how the components act in a natural environment.  

• Re-proving results is difficult, and is not apt to lead to publication, so are not high priority.

  • If the experiments are not done under the same environmental conditions (ingredients in media, temperature, light intensity, humidity, barometric pressure, speed of stirring, etc.), the results may vary.

• You never know if a non-considered variable is more important than the variables examined.

  • Whether eggs or yogurt are good for you may be the conclusion, but coincidence does not equal causation.  Genetics or lifestyle may be the main difference, not diet.

• The accuracy of data collection and interpretation may vary.

  • Data collection is susceptible to experimenter bias.  Best if people collecting data do not now what is being studied.
  • Data interpretation is susceptible to experimenter bias.  Best if people interpreting data do not now what is being studied.
  • If there is no control population, or the population studied is too small, then the results or conclusions may only apply to that particular study.

Note:  When you find content that is duplicated, or very similar, sometimes it is because it is copied.  Many websites use text from Wikipedia as content bait to bring people to their website, then they provide you with ads they hope you click (so they get paid).

Why would a website try and discourage you from using their service or product?

Would you expect a website to claim:

• We are not very good at this.  We are innacurate and sloppy.  We are a waste of money, but we look professional, so why not give us a try?

Would you expect a website to admit:

• We sponsored a bunch of studies, but only allowed the ones showing us in a positive light to e published.  Therefore you can see proof published that our product is good for 2 years but you will not read anything about the carcinogenic breakdown products present after three years.

Looking for comments in the world wide web might seem like you are scanning for gossip

which you are. Some places are more fruitful to look, such as discussion groups that are frequented by people who seem to care about the topic, and seem to have personal experience. 

Any one of the posts may be fake.  There are people who create a fake accounts (sock puppets) to support their point of view.  If you already know a lot about a subject, it makes it easier to judge responses.

If you don't know whether you can trust a comment, look for more collaboration or disagreement.  

Sometimes there is more than one valid answer. 

This is called divergent thinking.  Expecting one "correct" answer is convergent thinking. 

• Convergent thinking is good when you are learning sciences like chemistry and maths. 
• Divergent thinking is more useful for the complexity of advanced sciences and socio-psychological concepts.

1. Multiple independent sources - should have similar conclusions. 
2. Methods used to come to conclusions should be explained. References provided for claims.
3. Biases, including confirmation bias, should be considered.
4. Sources of information should have a reputation to risk.  
5. Verify, then trust.  Be skeptical.

“We learn wisdom from failure much more than from success. We often discover what will do, by finding out what will not do; and probably he who never made a mistake never made a discovery.” ― Samuel Smiles

"Science is a process of trying to figure out why things happen. Current theories are based on current information and are accepted as long as that is the most defensible answer or conclusion. When new information is found, theories may change. A change in understanding does not mean all the previous study was a waste of time. You will learn more from mistakes than from successes."

― Science teachers everywhere.