Dr. Kevin McGrew

 Psychometrician

Dr. Kevin McGrew is the Director and Founder of the Institute for Applied Psychometrics (IAP) and has a master’s degree in School Psychology from Moorhead State University and his doctoral degree in Educational Psychology from the University of Minnesota.

He was a practicing school psychologist for 12 years, spent 10 years as a Professor of Applied Psychology at St. Cloud State University. Dr. McGrew is currently a Visiting Professor in Educational Psychology at the University of Minnesota and serves as the Research Director for the Woodcock-Muñoz Foundation, Associate Director for Measurement Learning Consultants, and Research and Science Director for Interactive Metronome.

He is a co-author of the Woodcock-Johnson III Battery of cognitive and achievement tests.  He has published over 60 journal articles and books, primarily in the area of intelligence theories and testing. He maintains three professional blogs—IQ’s Corner, The Brain Clock, and Intellectual Competence and the Death Penalty.

Can you describe the field of psychology you are in?

Educational and school psychology are the broad fields, but approximately half of my work is in the subspecialty of applied psychometrics.

Why did you decide to become a cognitive psychometrician?

I always enjoyed the measurement aspect of psychological testing when I was a practicing school psychologist for 12 years.  I was very interested in learning about the technical aspects of the intelligence tests I was administering.

Through a set of serendipitous events, I connected with one of the leading test developers of clinical (one-on-one testing – not group testing) cognitive and achievement tests – Dr. Richard Woodcock.  I shared some applied research I was conducting on his 1977 Woodcock-Johnson battery of tests and we eventually started communicating regularly.

He mentored me as I was learning research skills.  He then had a task that required the calculation of test norms.  He asked if I would be willing to learn how to calculate the norms and complete the task under his supervision.

I visited him for a week and he tutored/mentored me on the procedures.  This is only a small part of what an applied psychometrician works on, but I loved the work.  This began a life-long mentoring relationship where he taught me the art and science of applied psychometrics.  After a while, I realized I wanted to learn more, and realized the need to acquire more formal, doctoral-level training in educational and psychological measurement.

What does your job as a psychometrician entail?

This will vary from position to position.  If a person is with a large testing company there will likely be a team of junior and senior psychometricians, and thus the roles will be specialized.

For myself, as primary applied psychometrician for one set of instruments (Woodcock-Johnson family of tests), as well as a co-author of the tests, I am involved in all aspects of test development.

This includes: item writing; item analysis; developing norm sampling plans; conducting item tryout data analysis; completing item analysis and scale development; calculating norms; completing all statistical analysis for the final test technical manual; and writing the technical manual and other special reports or articles.

What kind of education did you need in order to pursue and or be successful in your career field?

You need a PhD in psychology or educational psychology from a program that offers an emphasis in psychometrics or educational and psychological measurement.  Prior experience in clinical testing is not required, but can be very useful for understanding many of the practical issues.

The formal learning one receives with a PhD is only the beginning.  An internship or strong mentoring relationship with an experienced psychometrician or staff of psychometricians is critical.  There is a big jump between the theory of psychometrics and the art and science of doing it in the real world.

A post-doc internship or a strong mentoring relationship with a competent psychometric person or staff is critical.

Can you describe an average day in your line of work?

There is no such thing as an average day.  From inception to completion, the development of tests can often take up to five years.  Thus, what I do during the initial stages of a project will vary greatly from what I will be working on during the middle and near the end.

The process is much more non-linear than what one learns from textbooks or classes.  A person needs the ability to stop work on a current task and quickly switch to a different statistical analysis or technical problem that emerges elsewhere in the project.  You are not working alone and are typically working with a team of people who are working on different aspects of a very large project.

Thus, an issue someone else is working on may become the top priority out of the blue and you have to stop your current task work and switch modes to tackle a new problem.  Working in this field requires a high degree of tolerance for ambiguity and the ability to manage a constantly changing set of work priorities and tasks.

What are your favorite things about being a psychometrician?

The initial and final stages of a project are my favorite parts.  During the initial stages one is dealing with exciting new ideas, concepts, innovative test ideas, etc. Basically work that gets the creative juices flowing.

The middle of a 4-5 year project tends to be less interesting as it is completing repetitive tasks and involves less creative work.  Then, as the statistical analysis for the technical manual starts, one focuses on the statistical analysis that demonstrates how well the instrument is performing.

Although certain traditional analyses are required to be completed, you frequently discover statistical results that reveal unanticipated findings.  This stimulates creative thinking in order to understand why a particular score from a test battery is behaving in a new and potentially exciting manner.

Finally, the putting together of all the psychometric technical information in the technical manual, plus helping with the final building of the publication version of the test, produces an immense feeling of accomplishment.

If you could name 2 things you don't like about your career what would they be?

The need to track down apparent errors in a data file, which often requires concentrated focus on computer data files filled with raw data.  Also, no project ever flows as planned as unanticipated problems in data collection and analysis occur.

The time spent doing the detective work to identify and remedy the problem is at times frustrating.  However, probably the biggest drawback is that no matter how well planned a project is, I have never seen a project stay on schedule.  Thus, near the end, there are typically strong pressures to complete the primary psychometric work in less time than was originally planned.

This often results in long work days and the need to work during evenings and weekends.  Test publishers typically have a set date for publication and that date is critical for revenue forecasts.  Strong pressures are placed on those working on the "back-end" of a 4-5 year project, and this is where applied psychometricians are doing the bulk of their hard work on a project (calculating the norms and doing all the statistical analysis for the final manuals).

Working with a compressed "drop-dead" deadline work schedule near the end of a 4-5 year project can be very stressful.

What challenges should they expect?

Challenges during schooling depend on a person's degree of quantitative abilities.  The core psychometric courses and readings are often heavy on equations and formulas.  They are essential to learn so you understand the foundations of the field.

However, once you enter the applied world of test development, most companies have statistical software that completes these tasks.  So, one challenge is to not get frustrated with the occasionally heavy statistical/mathematical coursework during school. It is more important to have a conceptual grasp of the theory, concepts and methods of the field.

Once employed, an important challenge is to hook up with experienced psychometricians who can teach the implicit knowledge and skills not taught during graduate work.  There is a big jump from the theoretical/empirical coursework during the PhD program and the nuts-and-bolts of implementing that work during an actual test development project.

There is considerable implicit or tacit knowledge you must learn.  Having access to a group of experienced fellow psychometricians, or a strong mentoring relationship with an experienced psychometrician, is very important and will make "on-the-job" challenges and learning easier.

Finally, one must be prepared for unexpected challenges during a 4-5 year test development project.  There is no such thing as a perfectly executed test development plan.  You will frequently need to learn some new techniques mid-project to address a unique data analysis issue.

What characteristics should they have or acquire?

A person needs to have a logical, scientific mind and good quantitative thinking abilities.  A person needs to have the ability to work both independently and cooperatively with a diverse group of staff working on different parts of the same project.  A person needs the ability to deal with non-linear task management.

In other words, a person must be able to juggle multiple tasks and be flexible when issues arise that require shifting focus to a new task.  A person needs to have an attentive eye for details and perfection, as there is little room for error in the psychometric and statistical analysis.  The final product must be error-free.

What is the average starting salary for someone in this career field?

Salary depends on where one works and in what type of testing organization.  Individuals working for private testing companies tend to make more than those working in education (e.g., working on a state's educational testing program).  I have seen entry level salaries ranging from $35- to-70,000 and senior staff earning $100-to-150,000.

Do you see the field of psychometrics changing, and what should we expect in the future?

The field has a core set of well-established methods, principles and procedures that will remain the constant for the near future.  However, innovative advancements in statistical procedures and specialized software are frequent and require an orientation to life-long career learning.

Currently the most noticeable change is the increasing development of web-based computerized assessments.  Applied psychometricians need to learn to apply their knowledge, skills and abilities to a new delivery platform.

Web-based delivery of tests also results in the ability to constantly "fine-tune" an online battery of tests – so instead of completing a test development project and moving to the next one, psychometricians are likely going to remain involved in test projects for longer periods of time (as new online data suggests the need for changes, revisions, and new innovations that can be implemented "on-the-fly").

What kind of demand for psychometricians do you see right now and in the future?

The demand is always high. This is a very specialized field and there is always a shortage of qualified individuals.  The ongoing push for educational accountability has dramatically increased the demand for psychometricians to work on state-level educational accountability and testing programs.

The need for psychometricians to work on large-scale testing programs has spanned the past decade and continues.  More and more businesses are pushing to have previous paper-and-pencil testing put on the web or other platforms (e.g., iPads).  The demand for qualified psychometricians will likely stay high for many years to come.

Do you have any advice for future students looking to become a psychometrician?

Carefully investigate the focus of the doctoral programs you are considering.  Some are more oriented towards the theory and principles of psychometrics. Graduates from those programs are the most qualified to be future academics who teach psychometrics or who complete work on developing new statistical procedures and methods.

These programs focus less on the "art and science" of "applied" psychometrics and may not provide the experiences necessary to work in the actual test development industry.  If a person wants to be an applied psychometrician and work on test development products that are delivered to consumers (e.g., schools, parents, businesses, etc.), then be sure to select a doctoral program that has more of an emphasis on the application of psychometric methods and not just the study of the methods themselves.

Finally, it is critical individuals new to the field realize that coursework only provides one-half of what they need to know to be successful in the field.  There is considerable implicit or tacit knowledge that one can only learn by working on actual projects.

It is critical to work in an environment where there is an experienced team of senior staff who can teach you the ropes of the real world reality of applied psychometrics.  Developing a post-doctoral mentoring relationship with one or more experienced psychometricians is critical.

If you would like to learn more about Dr. McGrew and his career visit his webpage at www.themindhub.com and on Twitter via @iqmobile. And for more information about what exactly a psychometrician does and how to become one, please feel free to visit our Psychometrician page in the careers section of our website.