EVALUATION

Evaluation

Two metrics will be used: the average Root Mean Squared Error (RMSE) as well as the Pearson Product-Moment Correlation (PC) between the participant’s software scores and the ground-truth scores.

Participants results

Two metrics are shown for each trait: RMSE / PC. Best measures per trait are highlighted in bold and underlined. Below the table, descriptive statistics are provided. At the bottom of the table, our baselines: (a) a bag of character 3-grams with frequency weight; (2) an approach that always predicts the mean value observed in the training data.

Team Run Neuroticism Extroversion Openness Agreeableness Conscientiousness
besumich 1 10.69 /  0.05 9.00 /  0.14 8.58 / -0.33 9.38 / -0.09 8.89 / -0.14
2 10.69 /  0.05 9.00 /  0.14 8.58 / -0.33 9.38 / -0.09 8.89 / -0.14
3 10.53 /  0.05 9.05 /  0.10 8.43 / -0.33 9.32 / -0.07 8.88 / -0.17
4 10.53 /  0.05 9.05 /  0.10 8.43 / -0.33 9.32 / -0.07 8.88 / -0.17
5 10.83 /  0.10 8.60 /  0.38 9.06 / -0.31 9.66 / -0.10 8.77 / -0.06
bilan 1 10.42 /  0.04 8.96 /  0.16 7.54 /  0.10 9.16 /  0.04 8.61 /  0.07
2 10.28 /  0.14 9.55 / -0.10 7.25 /  0.29 9.17 / -0.12 8.83 / -0.31
3 10.77 / -0.12 9.35 / -0.07 7.19 /  0.36 8.84 /  0.21 8.99 / -0.11
4 12.06 / -0.04 11.18 / -0.35 7.50 /  0.35 10.89 / -0.05 8.90 /  0.16
5 11.95 /  0.06 11.69 / -0.37 7.46 /  0.37 11.19 / -0.05 9.10 /  0.11
castellanos 1 11.83 /  0.05 9.54 /  0.11 8.14 /  0.28 10.48 / -0.08 8.39 / -0.09
2 10.31 /  0.02 9.06 /  0.00 7.27 /  0.29 9.61 / -0.11 8.47 / -0.16
3 10.24 /  0.03 9.01 /  0.01 7.34 /  0.30 9.36 /  0.01 9.99 / -0.25
delair 1 19.07 /  0.20 25.22 /  0.08 23.62 / 0.62 21.47 / -0.15 22.05 / 0.33
2 26.36 /  0.19 16.67 / -0.02 15.97 /  0.19 23.11 / -0.13 21.72 /  0.10
3 18.75 /  0.20 25.22 /  0.08 20.28 /  0.54 21.47 / -0.15 22.05 / 0.33
4 17.55 /  0.29 20.34 / -0.26 16.74 /  0.27 21.10 / -0.06 20.90 /  0.14
5 26.72 /  0.18 23.41 / -0.11 16.25 /  0.13 27.78 / -0.19 15.53 /  0.27
doval 1 11.99 / -0.01 11.18 /  0.09 12.27 / -0.05 10.31 /  0.20 8.85 /  0.02
2 12.63 / -0.18 11.81 /  0.21 8.19 / -0.02 12.69 / -0.01 9.91 / -0.30
3 10.37 /  0.14 12.50 /  0.00 9.25 /  0.11 11.66 / -0.14 8.89 /  0.15
4 29.44 / -0.24 28.80 / 0.47 27.81 / -0.14 25.53 / 0.38 14.69 /  0.32
5 11.34 /  0.05 11.71 /  0.19 10.93 /  0.12 10.52 / -0.07 10.78 / -0.12
gimenez 1 10.67 / -0.22 8.75 /  0.31 7.85 / -0.12 9.29 /  0.03 9.02 / -0.23
2 10.46 / -0.07 8.79 /  0.28 7.67 /  0.05 9.36 /  0.00 8.99 / -0.19
3 10.22 / 0.09 9.00 /  0.18 7.57 /  0.03 8.79 /  0.33 8.69 / -0.12
4 10.73 / -0.15 8.69 /  0.28 7.81 / -0.05 9.62 / -0.03 8.86 / -0.09
5 10.65 / -0.16 8.65 /  0.30 7.79 / -0.02 9.71 / -0.06 8.89 / -0.12
hhu 1 11.65 /  0.05 14.28 / -0.31 7.42 /  0.29 12.29 / -0.28 8.56 /  0.13
2 9.97 /  0.23 9.60 / -0.10 8.01 /  0.02 11.91 / -0.30 8.38 /  0.19
3 11.65 /  0.05 14.28 / -0.31 7.42 /  0.29 11.50 / -0.32 8.56 /  0.13
4 9.97 /  0.23 9.22 / -0.20 7.84 /  0.07 11.50 / -0.32 8.38 /  0.19
5 10.36 /  0.13 9.60 / -0.10 8.01 /  0.02 11.91 / -0.30 8.73 / -0.05
6 13.91 / -0.10 25.63 / -0.05 33.53 /  0.24 12.29 / -0.28 14.31 /  0.16
kumar 1 10.22 / 0.36 8.60 /  0.35 7.16 /  0.33 9.60 /  0.09 9.99 / -0.20
2 10.04 /  0.27 10.17 /  0.04 7.36 /  0.27 9.55 /  0.11 10.16 / -0.13
lee 1 10.19 /  0.10 9.08 /  0.00 8.43 /  0.00 9.39 /  0.06 8.59 /  0.00
2 12.93 / -0.18 9.26 /  0.26 9.58 / -0.06 9.93 / -0.02 9.18 /  0.21
3 9.78 /  0.31 8.8 /  0.25 8.21 / -0.36 8.83 /  0.24 9.11 /  0.05
4 12.20 / -0.19 8.98 /  0.31 8.82 / -0.04 9.77 /  0.07 9.03 /  0.26
5 12.38 / -0.16 8.80 /  0.31 9.22 / -0.15 9.70 /  0.02 9.05 /  0.31
montejo 1 24.16 /  0.10 27.39 /  0.10 22.57 /  0.27 28.63 /  0.21 22.36 / -0.11
uaemex 1 11.54 / -0.29 11.08 / -0.14 6.95 /  0.45 8.98 /  0.22 8.53 /  0.11
2 11.10 / -0.14 12.23 / -0.15 9.72 /  0.04 9.94 /  0.19 9.86 / -0.30
3 9.84 /  0.35 12.69 / -0.10 7.34 /  0.28 9.56 /  0.33 11.36 / -0.01
4 10.67 /  0.04 9.49 / -0.04 8.14 /  0.10 8.97 /  0.29 8.82 /  0.07
5 10.25 /  0.00 9.85 /  0.00 9.84 /  0.00 9.42 /  0.00 10.50 / -0.29
6 10.86 /  0.13 9.85 /  0.00 7.57 /  0.00 9.42 /  0.00 8.53 /  0.00
min 9.78 / -0.29 8.60 / -0.37 6.95 / -0.36 8.79 / -0.32 8.38 / -0.31
q1 10.36 / -0.08 9.00 / -0.10 7.54 / -0.05 9.38 / -0.11 8.77 / -0.14
median 10.77 /  0.05 9.55 /  0.08 8.14 /  0.07 9.71 / -0.03 8.99 /  -0.01
mean 12.75 /  0.04 12.27 /  0.06 10.49 /  0.09 12.07 / -0.01 10.74 / -0.01
q3 12.20 /  0.14 12.23 /  0.21 9.58 /  0.28 11.66 /  0.07 9.99 /  0.14
max 29.44 /  0.36 28.80 /  0.47 33.53 /  0.62 28.63 /  0.38 22.36 /  0.33
Neuroticism Extroversion Openness Agreeableness Conscientiousness
baseline bow 10.29 /  0.06 9.06 / 0.12  7.74 / -0.17 9.00 /  0.20 8.47 /  0.17
baseline mean 10.26 /  0.00 9.06 / 0.00  7.57 /  0.00 9.04 /  0.00 8.54 /  0.00

Author profiling consists of predicting an author’s demographics (e.g. age, gender, personality) from her writing. In the PR-SOCO shared task we will address the problem of predicting an author’s personality from her source code. Personality traits influence most, if not all, of the human activities, such as the way people write (Celli et al., 2014), (Rangel et al., 2015), interact with others, and the way people make decisions, for instance in the case of developers the criteria they consider when selecting a software project they want to participate (Paruma-Parbón et al., 2016), or the way they write and structure their source code.