More important than years of experience
What are the key factors in estimating a programmer's ability?
When software engineers and designers register with a job search site, they may be asked to submit a skills sheet: how many years of Java, how many years of MySQL, how many years of Photoshop, and so on.
On the other hand, engineers with many years of experience may not be able to write well when it comes to slightly more difficult tasks. The discrepancy between the expected value of years of experience and actual work ability can be observed.
Is the number of years of experience a useful indicator for estimating the ability of programmers?
In the ESEC/FSE 2022 adopted paper Correlates of Programmer Efficacy and Their Link to Experience: A Combined EEG and Eye-Tracking Study, I found an interesting comparison.
Programmer's ability to understand code per unit of time
In the paper, 37 subjects were asked to solve tasks to understand code in one hour, and the results were compared to see if there was a correlation with measures such as years of experience.
The programmer's EFFICIENCY under certain conditions is defined as the number of code comprehension tasks solved within a completion time.
The actual questions used is available. It seems to be a task of reading Java code and choosing from a list of options what you think the return value of a certain function will be.
The subjects are 31/37 males. With ± standard deviation statistics, their age is 25.95 ± 6.76 years, and they have 3.55 ± 4.30 years of professional programming experience.
Off topic, but it is interesting to note that professional engineers spend 25 hours per week on software engineering, of which 11 hours are spent on programming, suggesting that 60% of their time is not spent writing code.
Programmer competence is not measured by years of experience
Results are compared to correlations with pre-survey items based on results 𝜌 ≥ 0.3, interpreted as a medium to strong correlation.
Conclusions are below.
Competence (EFFICACY) of programmers under certain conditions does not correlate with commonly used measures such as years of experience. Self-estimation and measures of willingness to learn are at least moderately correlated.
It has been stated that.
Here are the detailed data.
Here they are, sorted from the top.
- self comparison with peers 0.59
- time spent on code review 0.47
- number of programming languages 0.42
- self comparison with programmers with 10 years of experience 0.38
- time spent writing tests 0.35
- experience with imperative paradigm 0.32
- time mentoring 0.32
- time spent learning 0.32
Self-comparison with peers was the item with the highest correlation. Time spent learning, adopt a learning animal, is an old saying.
Conversely, sorted from the bottom, it looks like this
- years of working experience -0.06
- experience with Java 0.04
- time spent programming on the job 0.06
- experience with logical paradigm 0.07
- time spent deploying 0.10
- years programming at work 0.14
- Years programming at work 0.15
- time spent in meetings 0.16
There is talk of AI taking over programming tasks, but the paper also emphasizes that there is in fact no correlation between time spent coding and comprehension.
I'd argue that more than half of the skills needed for back-end development are other than language skills. Actually, in spite of the fact that it's a Java issue, years of Java experience doesn't seem to have any more impact on comprehension than years of programming.
Questions to ask in an interview
I made a list of questions that should be asked in an interview for the items that correlated well with ability.
- Q: If the average of your colleagues' abilities (engineers with 10 years of experience) is 100, where do you think you rank?
- Q: In your everyday work, how do you tend to work within a team? (Are you in a reviewing or mentoring position?)
- Q: Can you tell us about the languages you have touched? How many do you have? (Average in papers is 5.1)
- Q: What percentage of the time do you usually write tests when implementing a task?
- Q: How do you usually keep up with information? Do you participate in study groups or do you do personal development?
Eye tracking and brainwave data are also available
I focused on the experience years part this time, but the paper also discusses the behavior and cognitive load when reading using eye tracking and EEG data as well. It is also interesting, so please read it.
