Curious to learn more? Check out the related webinar in the Online Learning Center!

It seems that every day there is some new technology available. You go online and see an app that promises to write your reports, or a robot that will teach social skills, or maybe a software program that conducts assessments. How do you decide which of these to embrace and which to pass on? Sugai and Horner (2006) established four criteria for evaluating interventions. These can also be used to make informed decisions on purchasing and using new technologies. The four criteria are effectiveness (useful), efficiency (worthwhile), relevancy (applicable), and duration (maintainable.

Effectiveness

The first criterion is effectiveness, which examines how well something works. In the case of both interventions and technology, ideally this criterion is evaluated based on findings published in peer-reviewed research. Unfortunately, since new tools often come and go without ever being subjected to independent research, it is common that peer-reviewed research studies do not exist for technology.

This may mean that there is no ideal way to measure the effectiveness of a new technology. Instead, we can only do our best to meet this criterion in other ways.

• Read the product’s website. Ideally, they will have some in-house research that you can read published on their webpages. Although potentially helpful, these in-house published materials should be viewed with a critical eye.
• Go online to look for other people’s reviews of the products (e.g., Yelp, Amazon, technology popular press). Be aware that these online reviews may not be verified, meaning that the people writing them may not have ever actually used the product, or they might be the sellers of the product.
• Connect directly with individuals who have used the technology. You can ask the sellers to connect you with someone who has bought the product. Ideally, you would talk directly with these users and not just read their statements provided by the publishers.

Although not perfect, these sources of data provide some estimates of perceived effectiveness.

Efficiency

The next criterion for technology adoption is efficiency. Efficiency is essentially a cost-benefit analysis. Considered costs should include the amount of money, time, and effort needed to successfully implement and use a new technology. For example, something that requires a high per-pupil cost may not be considered efficient. However, if this product is highly effective in solving a problem within the school and the school is located within a high-income district, then a high price tag might not be a problem. Therefore, the perceived financial burden of any technology may change from school –to school. Similar to financial costs, investments of time and money may vary, but even high levels of investment may be justified if they bring high levels of benefit. Therefore, when determining efficiency, one must consider the impact of both the cost and the benefit.

Relevancy

Relevancy is the most personal of the four criteria. This criterion really focuses on if the technology will fit your needs. What might be a relevant piece of technology for one practitioner may not be for another. For example, if you aren’t a technologically savvy person, then a software program that requires a lot of coding would not be for you. In another example, if you are teaching high school, and the program is for elementary students, it might not be relevant to your students. Relevancy issues must be evaluated prior to purchase with any stakeholders involved before the adoption of any new technology.

Durability

Finally, durability must be evaluated. When you hear the term durability, you might assume that the criterion is evaluating how breakable the item is. This is only partially true for technology consideration. When working with technology, reasons other than breakage can cause a piece of technology to gather dust on a shelf, such as:

• In the case of a physical item (e.g., robot, tablet, projector), durability may focus on how fragile or hard to use that item is.
• A software program may constantly change so it does not continue to work if the computer you are using becomes obsolete.
• Older items might be replaced when a cheaper and easier version of the product is released.
• The producer of the materials may go out of business or stop supporting that technology.

Therefore, when considering durability in technology, it is mostly a prediction of if the product will continue to be usable over time. As with efficiency and relevancy, the determination of durability may differ from user to user. For example, if a school has a qualified I.T. department that knows how to fix these devices, then it may not matter if the tools are fragile or if a software program needs frequent updating.

These four criteria (effectiveness, efficiency, relevancy, and durability) provide a structure for evaluating any new technology. Prior to deciding to purchase a new technology, it is important to ecologically evaluate the unique situation of all interested parties to make data-based decisions about technology use.

References

Sugai, G., & Horner, R. R. (2006). A promising approach for expanding and sustaining school-wide positive behavior support. School Psychology Review, 35(2), 245–259. https://doi.org/10.1080/02796015.2006.12087989