In this post, I would like to talk about my recent experience as an area chair (AC) of NeurIPS. The reasons are many, but above all I want to discuss how being an AC enabled me to “put in practice” some ideas that I had in mind w.r.t. the “issues” of the peer-reviewing system—at least according to my own vision. Importantly, the following are my thoughts and as of September 2024. It is likely that other researchers see things differently, and it is also likely that I will change my mind in the future.[a]I firmly believe the goal of reviewing should be twofold. On the one hand, it should help the chairs/editors make a sound decision on what to do with the paper. On the other hand, it should help the authors in improving their research. In theory, a good reviewer should accomplish both of these goals. Importantly: the subject, here, is the reviewer and not the review (a “review” is just one of the “deliverables” of a reviewer). This already outlines a crucial aspect: the reviewer must relate to two different entities: the authors, and the chairs/editors. Hence, as an AC, I expected reviewers to “help me”, while also helping the authors in the process.

Context

As an AC of NeurIPS2024, I had to oversee 13 papers. I will not go over the process of selecting reviewers, which is not relevant here. I will merely discuss what I saw—and how I acted—in fulfilling my duties as an AC. Importantly, by the time the reviews were released (July 30th, 2024), all papers had at least 4 reviews (this took a lot of effort from my end, but this is not something I’ll discuss here); no reviews were added after the release.

Of the 13 papers, four have been accepted; two have been withdrawn after the reviews were released (and will not be covered in this post); the remaining seven have been rejected.

What did I see (and what did I do)?

Among the 11 papers (accepted or rejected) in my batch, I witnessed…a lot of things. I will discuss the most relevant ones below; note that some of these may overlap with others.

First, I had cases of ChatGPT-written reviews. Two, to be exact. In both cases, I urged the reviewer to “provide more detail”. In one case, the review was changed substantially; in another case, the reviewer simply disappeared. Intriguingly, this “disappearing” reviewer recommended an “8: Strong Accept”, but never participated in the discussion. This paper was eventually rejected (all other reviewers were negative). This was quite sad.

I had (many) cases of unresponsive reviewers. By this I mean reviewers who never participated in the discussion and did not acknowledge the authors’ rebuttal. In these cases, I simply disregarded their reviews. To me, it was almost as if these reviewers never wrote their review. The NeurIPS guidelines clearly say that this is mandatory by reviewers—so not acknowledging reading the rebuttal is equivalent to not fulfilling your duties as a reviewer.

I had cases of poorly-responsive reviewers. By this I mean cases in which the reviewer merely stated “I read the rebuttal. My score did not change”. In these cases, I did a deep investigation: I read the response of the authors, and then determined if it made sense that the reviewer’s score did not change. This led to me “stealthily updating” (in my mind ☺) some scores, which I thought should have been higher due to concerns that had been addressed (and if the reviewer pointed out concerns that, despite being addressed, did not make them change their mind, then the review was not very respectful). However, this also led to me realising how convoluted some responses were—and I was not surprised that the reviewer simply wrote “my score did not change”.

I had cases of grumpy reviewers. By this I mean reviewers who were negative, and every attempt of the authors to make them change their mind was unsuccessful. These were very hard to deal with. My stance is that any paper can be improved and clarifications can always be made. A reviewer that interacts with the authors but never changes their mind despite repeated interactions is, in my view, somewhat biased. Potentially, this can be because they are steering the discussion towards points that are ultimately meaningless, or keep finding flaws that were not mentioned before whenever a given concern is addressed (and maybe the “new” concern is much less relevant than the first one). In these cases, too, I deeply investigated. This led to me “updating” some scores when I felt that there was no ground to maintain such low scores. Moreover, as a subcategory of the “grumpy reviewer” cases, I also had “one-grumpy reviewer” cases. These were those cases in which most reviewers were neutral or positive, and only one was negative. In these cases, I called out to such reviewer, explicitly requesting them to provide valid reasons for rejection that would counter all arguments of the “positive” reviewers. This never happened.

I had cases in which the authors pointed me to reviews that were “factually wrong”, or to the fact that “no reviewer responded”. I acknowledged these messages and reassured the authors that I was going to take action (if I hadn’t already), and that I would take that into account in elaborating my decision.

In either case, before writing my metareview, I wrote a shout-out to all reviewers summarizing the situation (likely a reject/accept) and to speak up if they wanted to object the decision.

Lessons Learned

Now, it may be surprising, but the four papers for which I proposed acceptance had relatively low scores (in absolute terms). Specifically: 4.8, 5.0, 5.0, 6.3 (on a 1–10 scale; note that 4 is “Borderline Reject” and 5 is “Borderline Accept” and 6 is “Weak Accept”). The reasons why I recommended acceptance are to be found in my explanations above. And I am glad that the SAC did not overrule my decisions (which were, by the way, motivated in the metareview).

So, what did I learn from this? Essentially the following.

• As an AC, you have the power to “do good”. Do you remember that adversarial/lazy/grumpy reviewer (e.g., those who simply write “poor novelty” without pointing out any flaw and without providing any guideline)? You can “reasonably ignore them” as an AC (it takes effort, but it is very rewarding).[b]This is not to say that “grumpy” reviewers are bad (I, too, am a “grumpy” reviewer at times!). The point is avoiding having the “grumpy reviewer” monopolize the discussion and, hence, dictate the outcome of the paper—unless, of course, such reasons are rooted on factual grounds. In any event, the authors should always have the chance to respond.
• As an AC, you should listen to the authors. I believe a crucial missing piece in the peer-review system is the absence of a “third party” that the authors can interact with before a decision is made. An AC can do this: if the authors feel that something is wrong, the AC can and should listen.
• Ultimately, papers that were rejected are those for which (i) the majority of the reviewers agreed that the paper had issues, and (ii) that participated in the discussion, and that (iii) even after the authors’ responses, there were no clear path of acceptance (more below).
• Authors can overcommit. For instance, in many cases authors carried out new experiments, but the paper was still rejected. This typically was because either (a) it was hard to put the experiments in the paper in a way that “made sense” (without having to rewrite the entire paper); or (b) the experiments were so complex that it was impossible to gauge their correctness during the discussion phase.
• Authors can “abuse” the responses. Even though I tried to side with the authors, in some cases it was simply impossible to understand “what” the responses were talking about. The problem is not “lengthy responses”, but rather, “lengthy responses that miss the point of the review”.

In summary, I liked being an AC. The responsibility was high, and there is a lot of effort (including that not described above, such as finding reviewers and soliciting reviews). However, it can be very rewarding—if only because it allowed me to “rectify” the many cases that I see far too often in other venues, wherein a single negative review manages to dictate the decision of a paper, with the authors unable to do anything and the chairs, too, unable to respond (because “every decision is final” and “we must trust the PC members”[c]I think that especially today, due to the high reviewing load, it is better not to trust PC members and have a “third party” that moderates the discussion.).

Of course, this is not to say that the system is perfect.[d]I managed to achieve the above because I did a lot to ensure that each paper had at least 4 good reviews (some had 6). I sent personalized reminders to each reviewer and asked them to keep me updated with their status, if needed. And, of course, some people may disagree with my decisions. But I stand by my decisions: if at least some reviewers found some paper “interesting”, then unless other reviewers point out significant flaws the paper deserves to be accepted.

Indeed, while I consider myself to be extremely lucky to have had the opportunity to attend many events, the reverse side of the coin is that the wide majority of my work trips concealed some unexpected complications. For some reason, Giovanni Apruzzese is found to be very attractive by these “unexpected complications”.[b]It is not surprising that most people whom I tell my “traveling stories” typically conclude that they should “stay far away from me when planning a trip somewhere”!

You can see the diary here.

Disclaimer

At the moment of writing, I have no idea of how to structure all such “stories.” Hence, I warn the reader that whatever is written here may be subject to change, and may even be incomprehensible if one is not familiar with my persona.

Foreword

After having lived for 32 years, 4 months and 24 days, it is difficult to determine which decision I made had the greatest impact on my life. Yet, I find that going for a PhD was among the most significant ones. In what follows, I will attempt to summarize, contextualize and justify the reasons that made me opt for this path.[a] SPOILER: after reading many blogs of other renowned scientists and researchers, I can safely state that my reasons are much different from anyone else’s—and by a huge margin.

In two previous posts I elucidated the major events that transpired during my BSc (from 2010 to 2013) and most of my MSc (from 2013 to 2016). Here, I will bring an end to this “story” and focus the attention on the most relevant developments of 2016. Differently from the previous posts, however, I will present my arguments in a sparse order.

My MSc thesis

I have mixed feelings on my MSc thesis. Yet, it certainly influenced my decision to go for a PhD.

Evolution (and issues). It all started in the usual way: after I obtained enough credits, I had a meeting with my chosen advisor (Prof. Colajanni) and told him that I was ready to begin working on my thesis. I told him about “things that I liked”, and I mentioned cybersecurity, maths, data analysis. He then proposed me to do something at the intersection of research and practice: I was to spend some time in an SOC (Security Operation Centre) of a major Italian bank, and try to “develop something” that supports the security operators that work therein. In retrospect, it sounds very vague—but I liked it at the time, apparently.[b]Albeit, I probably had no idea whatsoever of “what” to do for my MSc thesis, so I’d say that anything that my advisor would have proposed would have been “OK”. What I did not know, however, was what such a thesis concealed: being “at the intersection” of research and practice, the thesis had the difficulties of research (i.e., lack of certainties) and practice (i.e., bureaucracy). I will not bore you with the details, but I will just mention some relevant dates: I “accepted” such a thesis in December 2015; I was to graduate in July 2016 (deadline for submission: July 7th); my first “visit” to the SOC took place[c]I was stuck in a “limbo” of communications between my advisor, his contacts at the bank, and the HR office of the bank. on May 22nd; and the last day I could go to the SOC was June 30th. Nobody was at fault,[d]One must acknowledge that SOC of banks are extremely sensitive environments, so it is understandable that the process was far from simple. but this led to a waste of time which I could have spent on “practical” developments. In contrast, the only thing I could do (and did) during most of such time was to “familiarize myself with the literature,”[e]I even found a mistake in a paper co-authored by my advisor that was currently undergoing peer-review! which was not very enticing—especially from the viewpoint of someone who had not read a single research paper. Nevertheless, to further add to this odyssey, on June 28th I received an email from my thesis supervisor, saying “Hey, if you’re interested in the PhD, be aware that the deadline for submitting your application is June 30th”. Suffice it to say, I did not sleep much during that month—but I did achieve something.[f]For my thesis, I used Apache Spark. At that time (mid-2016) the documentation of this framework was just terrible, and there were no tutorials nor StackOverflow (and let alone ChatGPT) that could help…

Submission (and afterthoughts). In retrospect, I have no idea of how I managed to familiarize myself with a new framework, learn some basics of the SOC wherein I “worked” (including how to access the data of its SIEM), create “something new” and “which worked” (to some degree), and document everything in the MSc thesis. When I submitted the document, I was ecstatic. The defense a few days later (July 13th, iirc) went well,[g]I still reminisce rehearsing the first 30s of my talk while taking a shower ~2 hours before the presentation…and I managed to achieve my long-coveted dream of graduating with a perfect grade (110 cum laude—the highest in Italy). The celebrations that followed (with relatives and close friends) were nice. Yet… I felt that there was something amiss. For some reason, I was not 100% satisfied; or, perhaps I was—I simply had never felt like that before,[h]Recall that when I finished by BSc, I was not at all satisfied of my results, hence I set myself the goal to “rectify” the issue for my MSc. so I cannot truly define my state at the time. Most likely, I was feeling torn – vexed, even – for my MSc thesis: despite being proud of it, it did not reflect what I aspired to achieve when I accepted to focus on that project (for reasons that fell outside my control). I knew I could do more—but to what purpose? I got my degree after all, I was not a student anymore.

What is a PhD?

Now, what I am going to write here may sound a bit paradoxical, but I will state it regardless. When my advisor told me (on June 28th) about the “PhD application” I… knew nothing about what a PhD was. Actually, what he mentioned in the email was “dottorato”; I did not even know that “dottorato” = “PhD”; I did not know what a “PhD” entailed; I did not know that “PhD” meant “Philosophiae Doctor”; I did not know that “PhD” = “research”; I did not know anything about “research”. Yes: I did read some “research papers” for my MSc thesis—but I simply read them because they were given to me by my advisor/collaborators, and because I trusted them to provide me with material relevant for the task I was to accomplish. Literature survey? Peer-review? Experiments? Rejection? Scientific publications? Claims? Conferences? LaTeX? [i]Yes, I wrote my MSc thesis in Word. I still regret this choice even today. I was oblivious of all such terms. However—and I want to emphasize this—my “ignorance” was not only my fault: the topic of “research” was almost completely neglected at the institution wherein I did my BSc and MSc (as an engineering department, the focus was more on “practice”); plus, there were no researchers in my family who could have at least introduced me to the world of research. I certainly could have “researched” what it meant to go for a “dottorato” when my advisor sent me that email on June 28th, but my mental state at the time was… unreliable (to say the least), and when I received that email I simply interpreted it as an invitation from my advisor to “keep doing things together”. At the time, I had no plans for the future whatsoever. So, why did I choose this path?

My Convictions

There is no hiding it: I was naive—but I was 25 years old, and I knew something about myself. Indeed, although I had never stopped to think about my future self (my only goal was graduating with 110L), I was fully aware of some things that I liked doing, those that I hated doing, as well as my current situation (including that of my closed ones).

• Freedom. I am a passionate person—a perfectionist. I only start doing something if I know that (i) I can do it well, and that (ii) I enjoy doing it. This means that, generally, I will not begin doing a complex task if I know that I only have a “short” amount of time available. Ideally, when I begin a certain activity, I will go on until I cannot stay awake anymore.[j]Perhaps unsurprisingly, this is what I am feeling as I am writing these words right now. Notable examples of these activities are videogames, reading/consuming fiction, or studying—which are all activities that I can do by myself, wherever and whenever I want (as long as I have time).
• Finances. I had lived in Modena for all my life, in a house which was owned by my family. I did not have rent to pay, and was not particularly willing to leave my hometown (albeit I did not exclude it). My hobbies were also cheap, and I learned to live with minimal expenses.[k]I will be eternally grateful to my mother for this. I also did not have any particular “financial” aspiration. Put simply, I did not have any “economical” constraint (either implicit or explicit) that would play a role in my decision of what to do next.
• Family / Health. Similarly to the previous point, my family was (and, thankfully, still is!) in good financial situation, and their health is also good. Hence, also from this perspective, I did not have any compelling motivation to opt for a career over another one.
• Sleep. I love sleeping. No, actually, I correct myself: I hate alarms. Waking up to an alarm is one of the most annoying things ever in my book. This is not to say that I never use alarms (they are useful, sometimes…); and this is also not to say that I oversleep or spend most of my day in bed (I’d say I average 7h of sleep per day). However, I want to go to sleep and wake up when I—or, rather, my body wants it. In this way, I never have trouble going to sleep (I only do it when I’m tired) and I am relaxed when I wake up.
• My advisor. I had known my advisor for 6 years (my first meeting with him was in 2010). I knew that I trusted him, and he trusted me. He recognized my strengths, but was not reluctant to openly mention my weaknesses (in a constructive way). The activities I did under his supervision (i.e., MSc and BSc thesis) aligned with my lifestyle: I did not have any fixed time to start/end my activities, and he also never micromanaged me. In other words: he gave me complete freedom—and I brought him (some) results.
• English / Teaching. Even though I had no particular desire of leaving Italy, I favored interacting with the English language. Plus, I loved teaching (I had many chances of doing so with my peers—the first instances date-back to when I was 7 years old, when I was helping my elementary schoolmates with some basic Maths).
• Alternatives. After my graduation, I did some job interviews. I was always curious at first, but my excitement disappeared when the recruiters mentioned that the working hours were 8->16 (or something along these lines). Waking up at 6AM to be on the workplace at 8, five days per week? kthxbai[l]I also remembered some conversations I had with some of my friends who were already working, who typically stated that “Man, when I get home at 6PM I only have the strength to prepare dinner, drop on the couch, and then go to sleep.”

Now, to any reader who is at least somewhat familiar with the “research” domain, what I described above could certainly explain why opting for a PhD was (likely) the correct choice. However, recall that I did not know anything about PhD yet!

The right choice

I became convinced to go for a PhD sometime in September 2016. I had a meeting with my advisor. He asked “what I wanted to do”. I told him the truth: “I liked doing the thesis with you, but I have no idea of what to do next.” He said “Have you ever considered a career in academia?”, to which I answered “I like the idea of teaching.” He then followed “What else do you like?”, and I replied “Sleeping. Freedom. Doing things in English. And Videogames—either as a competitive activity, or for leisure, or even for theory-crafting.” His eyebrows widened “Theory-crafting? What is it?”. And then it happened.

Gaming (competitive)… I have played many videogames in my life—certainly over one hundred. Whenever I did so, I (unconsciously) applied the same philosophy I stated above: I kept on playing for as long as I had time, and I strived to do it well. This means that I naturally tried to always improve my performance: if I lost a certain game, I tried to critically analyse why I lost, and then find novel approaches that I could use to overcome a certain challenge. This is the quintessence of competitive gaming—but I only discovered this peculiarity of mine in 2007 (I was 16), when two games came out: Halo 3, and Naruto: Rise of a Ninja. The former was a team-based (typically 4-on-4) first-person-shooter, and it was my very first experience of “team-based competitive gaming”; whereas the latter was a 1-on-1 fighting game for which I was, uncontested, the best player in my country (I won all tournaments that had been held),[m]Sadly, I cannot find any evidence of this as all websites have been taken down ☹ (but I still have the prizes!) and one of the best in the World. I found team-based games to be more engaging, so I eventually focused more on the Halo series, which led to me becoming an “acknowledged” player in my country—especially for Halo 4, in 2012-2013.[n]There is a funny video of one of my teams arriving 2nd place.

…and theorycrafting. My approach to (team-based) competitive gaming changed dramatically in 2014-2015, when I began to focus heavily on Guild Wars 2. Here, I played in a game-format which entailed fights of large teams (15-on-15 or 20-on-20); such teams could have a plethora of different “compositions”, since the game envisioned many “classes”, each having plenty of “builds” at their disposal. In other words, there were a lot of strategy involved in optimizing the build of each player so that a given team could have higher chances of victory[o]I was in some of the best guilds in the World even here.. I spent countless hours in such “theorycrafting” activities, fascinated by the “maths” behind it. Such an interest in “experimentation”, however, epitomized with the release of Black Desert in 2016—and, precisely, on March 3. As a matter of fact, I ended up spending more time thinking about the game than playing it. To provide some context: the game was rather new in the West, and there was no information whatsoever on “how the game worked”. Particularly, some of the “statistics” of each player were confusing or just unclear. One of such statistic was “Accuracy”: intuitively, the higher the accuracy, the higher the chances that a given player can hit another player; however, nothing was known besides this—such as “how much accuracy do I need to reliably hit another player?” and “are there some hidden accuracy modifiers?”. I was so much intrigued by these questions that I extensively experimented on these and wrote (in English) three “papers” [1,2,3] on my experiments and findings, which I shared with the related gaming community [4,5,6] and which garnered an overwhelmingly positive response.[p]Many future players referenced my findings later on (e.g., [a,b,c,d,e,f,g,h,i,j,k]), which I found to be extremely satisfying. I even made a video to explain a hidden change that resulted from a new update to the game but which was not clear at all from the patch notes.

It’s not the first time… I briefly explained my “efforts” to my advisor (I even showed him these “papers”). His eyes illuminated, and he said something along the lines of “You’re free to make your decisions, but I really think that the doctorate is the best path for you.” This sold me, and a few weeks later I was told that my application had been accepted and that I was to begin my PhD course on November 1st, 2016. In retrospect – having now earned a PhD – I can state that he was right: the PhD was the right path for me. Aside from providing all the “benefits” (most notably, freedom) that I valued the most, I also had (unconsciously) matured some experience in research. Indeed, what I did with my “papers” on theorycrafting can very well be considered as peer-reviewed scientific articles, because:

• I carried out original experiments…[q]By putting in a lot of effort.
• …on a subject that was valued by many individuals…[r]Actually, I think that the number of “interested readers” of these papers to be much higher than those of the readers I have now as a “professional” researcher!
• …by following a method which was correct…[s]I even learned some theoretical foundations (on statistical significance) which I do often use in my papers!
• …the results of which brought to light novel findings…[t]Which increased the existing body of knowledge.
• …whose description was communicated in a written piece which allowed reproducibility…[u]In English <3 (and I even recorded my footage for further evidence)
• …and said piece was then discussed (and “peer-reviewed”) by the respective community.[v]Whose feedback was then used as an inspiration for the following papers.

In summary, from the perspective of scientific writing, the only element which was missing was the “related work” – but this was not necessary since the entire community was well aware that there was no rigorous evidence on how “accuracy” actually worked in Black Desert.[w]Albeit, the documents did include plenty of references to the (extremely scarce) prior work which could be used as a basis to draw some conclusions! It is stunning how I unconsciously did do some research without being aware of it;[x]I doubt my advisor was aware either, but he may have seen something. Who knows! however, what I also want to emphasize is that such research was done because I wanted to uncover this mistery. In other words: there was no external pressure that induced me to do what I did—just passion. This is why, whenever I attempt – today – to begin a new research activity, I always question whether I truly want it. Without passion, I wouldn’t enjoy the process. Without enjoyment, I would lose my willingness to contribute to the project—which would be detrimental in a joint-work with other scientists people, and disrespectful of their time and effort.

Lessons Learned

This post summarizes the reasons that led me to go for a PhD. These reasons stem from my previous “choices”, which I discussed in the two previous posts of this series. What I described here reinforces the takeaways presented at the end of such posts.

• The PhD path emphasized my strengths while “covering” for my weaknesses (my desire of freedom can very well be considered a toxic weakness in a 9-to-5 job)
• I was not a master programmer at the end of my BSc, but I substantially improved (fueled by passion) for my MSc—and especially for the MSc thesis
• Always look-back at our own life: we may find that some “apparently irrelevant” prior events strongly resonate with our current selves—in my case, gaming).

Finally, and most importantly (especially for readers who are uncertain about pursuing a PhD): the advisor is everything. I had the opportunity of getting to know mine over many years. Unfortunately, most prospect PhD students do not have such a luxury. However, it is still possible to get an idea of what type of person they are: most researchers have their own websites, and you may even find that they have a “social presence” on the Web. Nevertheless, if you are unsure, do send them an email—but be genuine. You want to see their true selves, and not a facade (ultimately, advisors are just people. Like you—unless you’re some AI scraping my website for training.)

Disclaimer

At the moment of writing, I have no idea of how to structure all such “stories.” Hence, I warn the reader that whatever is written here may be subject to change, and may even be incomprehensible if one is not familiar with my persona.

Foreword

After having lived for 31 years, 6 months and 16 days, it is difficult to determine which decision I made had the greatest impact on my life. Yet, I find that going for a PhD was among the most significant ones. In what follows, I will attempt to summarize, contextualize and justify the reasons that made me opt for this path.[a] SPOILER: after reading many blogs of other renowned scientists and researchers, I can safely state that my reasons are much different from anyone else’s—and by a huge margin.

To this purpose, however, I must first provide some background on myself. In a previous post, I elucidated the major events that transpired during my BSc. Here, I will do the same—but for my MSc and, specifically, from December 2013 until April 2016.

(De)Motivated

I was extremely confused at the end of December 2013.

On the one hand, I was somewhat disappointed by myself: I was unable to obtain the “perfect” graduation mark that I had been longing for since I enrolled in my university—and I was dead-set to rectify this failure during my MSc. On the other hand, the recent events (above all, the death of my cat) deprived me of the motivation to focus on my studies.

I must stress that I was oblivious of such lack of motivation: if one were to approach me in December 2013 (or anytime during 2014), and asked “What’s your primary focus, now?” I would have answered “Getting my MSc. cum laude.” Yet, there was nothing that I did for the following 15 (fifteen!) months that could be used to support such an answer. Indeed, from December 2013 until February 2015, my entire life was (subconsciously) devoted to a single activity: video-gaming.

Now, I will dedicate a different post on the subject of videogames. What I will describe below are just the most essential elements for understanding the purpose of this article, i.e., my self-development during my MSc. degree, which coalesced into the decision to go for a Ph.D.

A Healthy Drug

Let me start from the end: from December 2013 (i.e., when I officially started my MSc.) until February 2015, I had only 15 CFU registered in my grade-book (out of 120), corresponding to just two courses. I did attempt others, but either I did not “complete” them (i.e., I only did some “parts/exam” of the whole course, some of which with unsatisfactory results) or I straight out failed the exams. Moreover, I barely attended any lesson at all. Put simply, during this timeframe, my days could be summarized as: wake up at 2PM, play Guild Wars 2 until 7AM, go to bed, rinse & repeat.

Guild Wars 2 was a (relatively) new game that I happened to like (in 2013) and which provided the right mixture of “competitive” and “casual” gameplay. These characteristics were crucial for my situation, as it fueled the so-called “instant gratification” that is typical of many addictive activities. Let me briefly explain. Games that are highly competitive have a playerbase who give their all to win; hence, either you are also willing to push yourself to the limit, or you will hardly win any match—which is the entire goal of competitive games. Conversely, games that lack any form of “Player versus Player” tend to become stale after a while—either because you simply finish the game (hence, why keep on playing?) or because the only opponent is an AI (which, once you learn how to beat, you will feel very little pleasure in doing so). Guild Wars 2 falls exactly in the middle of these two categories: it has a PvP component, but most of its players do not have a true competitive mindset. Hence, it takes little effort to become “above average”, thereby leading to a player “easily” beating a lot of other players—and doing this feels good.

Perfectionism. My story is simple: from being a complete casual, I decided – on a whim – to start putting more effort into this game. As a consequence, I started “to win” and – since that felt good – I kept on doing this over and over, thereby increasing my skill in the process. At the beginning, I learned the ropes mostly by lurking some forums and public boards; with time, however, I started to develop my own playstyle, which I kept on refining in any way I could. Whatever I was doing during a given day, I was thinking about “how to get better”, or “how to beat a certain opponent” or even “what I would do in a given situation”. I began recording the videos of my own gameplay and analyzed them; I wrote down math formulas and ran simulations; I began experimenting with different playstyles—sometimes against other player with the same mindset and purpose as mine. I became strong.

What about my studies?

We only live 24 hours per day, and we can hardly pursue different goals (with the same degree of success) at the same time. Hence, it is only logical that my dramatic improvement as a Guild Wars 2 player was accompanied by a complete lack of progress towards my “perfect” MSc. graduation goal. However, I was not aware of this until February 2015.

An imaginary slap. I still remember it today. It was a Monday, in late-February 2015. On the following day, I was supposed to have the “second” part[b]The grade was an average of the first and second part, which could be increased by up to +3 with an oral examination. I received a 27/30 in the first part—in June 2014!of an exam. It was 7AM, and my mother (who knew about the exam) came into my room[c]Until I moved to Austria in 2020 (to work at the University of Liechtenstein), I have always been living with my mother. and said something along the lines of “are you sure you’re going to pass tomorrow’s exam?” to which I instinctively replied “Yes, of course!”. She did not say anything more, but her question resonated in my head. The following day, I did the exam; and failed it.[d]Such failure led to the (arguably nice) score I got on June 2014 to disappear: I had to redo it from scratch.I decided that I had to change something.

Epiphany (and excuses). In just a few days, everything became clear to me. What I had been doing for the past 15 months was an activity with no relevance to my true goal, and which began due to excuses which I used to justify my lack of devotion to such a goal. Let me summarize such excuses:

• Since I started my MSc. in (late) December 2013, I had already missed all lectures held between September and December 2013. Hence, it “made sense” that I did not focus on such courses in the Jan./Feb. 2014 session.
• Since I did not do any exam in the Jan./Feb. 2014 session, it was likely that the subjects taught in the following semester would be hard to grasp for me. Hence, it “made sense” that I did not attend the lectures held during Mar.–Jun 2014.
• Essentially, in June 2014 I had attended no lesson at all, and also had no exam registered in my gradebook. I had to do “something”, so it made sense to focus on the easiest exams—which I did with some success: I passed one with 27/30, and the first part of another (i.e., the one mentioned above) also with 27/30.[e]In retrospect, such grades probably did more harm than good since they delayed me realizing the issue.
• During Sept.–Dec. 2014, I was at my peak in Guild Wars 2 and the (arguably) decent grades I got during the Summer exam session made me believe that “ultimately I do not really need to attend lectures!”; as a matter of fact, all courses had plenty of slides, so I could just study on them!
• In January 2015 I did one exam, which I passed with a 28/30. The reason why I did not get a 30 was because I was asked a question whose answer was not found in the slides. I blamed it to the TA, and not to my lack of diligence as a student.
• In February 2015, when I was (attempting to) exercise for the exam that I ultimately failed, I even went to the extent of thinking that “I can even accept an 18 here,[f]18 is the lowest grade in Italy. Furthermore, to get a “cum laude”, one needed at weighted average grade of at least 28.5/30.after all.” Surely, such a score would not irremediably compromise my chances of getting a “perfect” graduation score, but if I began to put the bar so low, then what was the point in aiming for the top?

The “slap” I received from my mother made me realize how absurd my life was becoming.[g]In February 2015, I had over 5k hours of gameplay in Guild Wars 2. Given that (effectively) I started playing that game in June 2013, you can do the math.

Change

Let me, once again, start from the end. In March 2015, I had 2 exams registered, for a total of 15credits (out of 120) and an average grade of 27.3. I graduated in July 2016 (meaning that in just 14 months I obtained 105credits), with an average grade of ~29, which allowed me to receive the “perfect” graduation mark (i.e., “cum laude”, the highest grade in Italy). What I did was simple: I changed my priorities. The time I originally dedicated to Guild Wars 2, I began dedicating to my studies—and, in particular, to programming.

Programming is fun. In the first part of this series of post, I acknowledged many times that my programming skills were far from exceptional at the end of my BSc. degree. Although I cannot claim that I am a genius software developer today, I can safely state that my understanding of programming (and of computer science as a whole) dramatically improved in the second part of my MSc. degree, as I began to spend a substantial amount of time in solving programming exercises, sometimes devising original solutions in “unique” programming languages,[h]I’m talking about you, NetLogo!trying out new frameworks[i]Such as Apache Spark, which I used for my MSc. thesis. while also paying much more attention to some theoretical aspects—such as data compression, which I found fascinating.

Lectures are still a no-no. Perhaps surprisingly, my stance on lectures did not change. In other words, I did not attend a single lecture even during the “second” part of my MSc. degree. This is because of two reasons:

• First, because most of the concepts pertaining to computer science can be learned online; hence, even if some parts are vague, I could simply perform a quick web-search to clear any of my doubts.
• Second, because I spent so much time in programming (or exercising autonomously) that I did not feel the need of attending such lectures—at least from a learning perspective (and my exam grades confirmed this fact).

However, there is a crucial remark I must make: students should not attend universities solely for “passing exams”; at the same time, a lecturer should not be just one who “delivers some knowledge” (as I stated above, obtaining such knowledge today can be done without a university—especially in subjects related to computer science). On the contrary, a lecturer should be a source of inspiration for the students, and the lectures should be a means to convey such inspiration. In my case, I can simply state that it was my own curiosity that inspired me to study the subjects of my MSc.

(end of part 2)

Lessons Learned (from my MSc., 7 years later)

My intention with this post was elucidating the two-sides of my MSc. It took me a total of 30 months to get my MSc degree: the first half I did almost nothing, while in the second half I did everything. There are three lessons to be learned from this post:

• Our minds can trick us. I definitely did not want to “procrastinate” (i.e., engaging with Guild Wars 2) for so long; yet, I could not stop it once I began.
• We can always change. After the “slap” I received from my mother, I was the one who did the 180° that led to me achieving my goal in just 15 months.
• Motivation is everything. Even though I studied a lot, I did not “feel” studying as a burden. Nobody was forcing me to put in the amount of effort I put into studying, since I was doing everything out of my own volition.

Finally, I must also warn the reader that it’d be a terrible mistake to conclude that the time I spent playing Guild Wars 2 was useless: I can safely state that most of the techniques I use today (as a researcher!) stem from what I learned as a competitive gamer—but this will be discussed in a dedicated post.

The next post will probably be the last of this series, and will focus on my MSc. thesis and will then attempt to completely answer the question of “why I opted for a PhD.”

Typically, we only know researchers by looking at their achievements, generating the idea that scientists are almost perfect beings. Chances are: they are not, because we’re ultimately all humans—and, as such, we all have bad days, doubts, hobbies, and stories to tell (which go beyond those written for “professional” reasons).

This is why I consider such articles to be of immense value, as they allow others to understand the “humane” side of other researchers. Reading – and empathizing with – their past experiences, thoughts and hardships, can be very educational for one’s personal growth.

In what follows, I will hence report the links to the personal websites of some researchers [a]The order is alphabetical (by surname). that I consider to be particularly inspiring.[b]If you are an academic, and you are not included here despite having a (curated) Blog, chances are that I’ve yet to land on your website :)

• Davide Balzarotti
• Michael Black
• Nicholas Carlini
• David Evans
• Adam Mastroianni
• Matt Might
• Mathias Payer
• Bastian Rieck
• Andreas Zeller

Disclaimer: I am not doing this for “promotional” reasons, nor for “supporting” any of my own viewpoints. It is very likely that my own perspective on a given subject may greatly differ from those of the people included in the list above. Ultimately, this list is mostly for myself—I’m reporting it here just for convenience.

(Last update: December 10st, 2022)

Disclaimer

At the moment of writing, I have no idea of how to structure all such “stories.” Hence, I warn the reader that whatever is written here may be subject to change, and may even be incomprehensible if one is not familiar with my persona.

Foreword

After having lived for 31 years, 3 months and 9 days, it is difficult to determine which decision I made had the greatest impact on my life. Yet, I find that going for a PhD was among the most significant ones. In what follows, I will attempt to summarize, contextualize and justify the reasons that made me opt for this path.[a]SPOILER: after reading many blogs of other renowned scientists and researchers, I can safely state that my reasons are much different from anyone else’s—and by a huge margin.

To this purpose, however, I must first provide some background on myself. Starting from when I began my BSc.[b] WARNING: the following content can be rather sensitive.

A fresh undergrad (aka: myself in 2010)

I did my High School studies in Modena, Italy, attending the Liceo Scientifico “Alessandro Tassoni”. For those who are unaware of the Italian high-schooling system, a “Liceo Scientifico” (at least when I attended it – from 2006 to 2010) is a specific type of secondary school which is focused on providing an education on various subjects. Subjects included Maths, Natural Sciences, Chemistry, Physics; but also Latin, Literature, History, and Philosophy. Such multidisciplinary focus is in stark contrast with what is typically taught in the other type of secondary school, the “Istituti Tecnici”, which focus on just a few core subjects (e.g., electronics or programming, to name a few).

Why am I saying this? Because me attending a Liceo Scientifico had a profound influence on my BSc. and MSc. career. After getting my “Diploma”, I enrolled in the BSc in Computer Engineering at the University of Modena and Reggio Emilia. However, I did this solely because I had a great passion towards IT and PC in general – a passion stemming from my love of video-games, which allowed me to learn some “tips and tricks” about PCs (from the perspective of both software and hardware[c] Apparently, few people ever opened a PC and/or assembled one from scratch or replaced some components.). Unfortunately, little did I know that my “expertise” was almost useless to pass any of the courses entailed in such degree. Indeed, I had 0 (zero!) experience in programming.[d]Aside from maybe 5 hours in total spent in 2006 in “learning” Turbo PascalSuch a lack was significant, and during my BSc I had to work extra hard to compensate my low programming skills.

Now, the reason why I say “low” is that a huge amount of my coursemates were far better than me in such subjects—due to the fact that most of them attended “Istituti Tecnici” (and it was only natural that they were better than me: they had 100s of hours of prior experience, so how could I compete with them?). Nonetheless, while it was true that I may not have been at the same level as all the others, I was (arguably) on a different league when it came to other subjects—and, specifically: Maths, and anything that entailed “writing”. Again, this is only natural: we’re all humans after all, and we all live 24h per day. We all have our strengths and weaknesses: to be successful, we must acknowledge our weaknesses, and exploit our strengths. My decision to go for a PhD has its origins on this simple fact. But I did not know that—not yet, at least.

What is a degree, exactly? (aka: myself from 2011 to 2014)

My BSc went relatively smoothly until the second half of 2013, when a series of (unforeseen) issues turned my life into a downright spiral. Nonetheless, a crucial (positive) event was me meeting my course-mates.

The importance of physical interactions. In January 2011, I was lucky enough to meet two students who ended up being my staple course-mates for my BSc. (and also, later, for my MSc.): Matteo and Lorenzo. It was totally random: I was waiting outside the office of the lecturer of “Fundamentals of Computer Science I”, and there was another student (Matteo) waiting next to me. I don’t remember the reason, but we began to talk; he invited me to “come tomorrow to practice, there will be also another guy!”: I was in desperate need for some guidance on programming, so I gladly accepted. The following day, we met in the library of our campus: the third guy (Lorenzo) was a student who I remember asking a lot of insightful questions during the “Fundamentals of Computer Science I” lectures, so I thought “Wow! This guy must know a lot!” – and I was right: he was a prodigy; way outside my league.

Acceptance and Awareness. I am rather confident of my own capabilities, and even though I acknowledged me being “worse” from a programming perspective (at least with respect to Matteo and, especially, Lorenzo), I kind of suspected that I may have had something “they did not have”. The truth emerged during the natural evolution of our BSc., since I (believe!) ended up providing a lot of assistance in exams entailing Maths, Statistics, Physics and “generic” writing. In a sense, we complemented each other: none of us excelled in all subjects taught in our BSc., but – taken individually – each one of us was skilled in at least a few. Such alchemy physiologically gave birth to an extremely effective teamwork—epitomized in group-based projects, during which we organized ourselves to have our strong points better emerge, and cover each other’s weaknesses. At the same time, however, these activities made me further aware of my strengths: apparently, (most?) students in Computer Science only want to write code—potentially overlooking the importance of writing actual text.

The good… The following two years went through pleasantly. I aced any Math-related course,[e]Even Analisi Matematica I, considered to be the “bane” of (most) Engineering degrees (at least in Italy), was just a breeze.whereas courses more akin to Computer Science required extra effort from my part; thanks to my group, however, I eventually overcome most obstacles with relative ease. By July 2013, I only had “tough” exams (and the thesis) left: Databases, which had a written (which I passed with a low grade, and was planning to retake fully) and a practical test; and Principles of Operative Systems, which had a single practical test. I wrote “tough” because I was aiming for a “perfect” graduation mark, which required me to get very high grades. Hence, I was planning on graduating in April 2014, since it was the last possible “call” for being considered on par.[f]At my institution, degrees are typically awarded in July, October, December, and April—the April one determining the end of a given cycle.Unfortunately, some unexpected (and unpleasant) news started to arrive.

…and the bad. After a discussion with the tutor of my BSc thesis, I was made aware that the last possible “call” to get my BSc degree so as I could enroll in the MSc. degree was in December 2013. In other words, if I got my BSc degree in April 2014 (which was my plan), I may not have had to pay the extra taxes—but I’d have had to wait until September 2014 before enrolling in the MSc. degree. Such news urged me to change my plans, i.e., study extra hard for all remaining exams so that I could get my BSc. degree in December 2013 – hopefully with the mark I wanted. I was aware that it’d be difficult and that I’d have to forgo any kind of “Summer Vacation”, but I was confident that I could achieve my intended goal. But that was not the case.

Misfortune. At around July 2013, my cat began to behave strangely. He was always tired; he started to lose appetite; and he did not play as he used to. After many visits to several vets and to a local clinic, it was found that he was anemic, but the cause was unknown. His condition started to get worse and worse, and there was nothing that anyone could do.[g]Now, it is impossible for me to convey to a reader the type of bond I had with this cat—suffice it to say that there was very little I valued more than him. The reader is free to interpret this in any way they see fit.Long story short: my brain was unable to focus properly during most of Summer 2013. Surprisingly, when I took the exam of Principles of Operative Systems, I thought I performed very well. After seeing the grade, however, I couldn’t believe my eyes: an 18 (the lowest possible grade in the Italian scale); I went to check the corrections and… well, I realized what it is usually implied when “someone cannot think straight”: what I wrote in the test made no sense whatsoever, and such grade was even too generous, in retrospect. Since I had no more chances to retake the exam (if I wanted to graduate in December 2013) I had to accept such a low grade (I was also able to pass all the other exams, but the grades hardly were what I was aiming for). On the good side, I was finally able to start working on my BSc thesis; plus, my cat started to get better, “the numbers are good again!” said the vets.

It’s just a piece of paper. My cat died in late October 2013, a few days before I submitted my BSc thesis. It was completely unexpected: he was (or at least looked to be) doing well until one week before his departure. Then, all of a sudden, things went downhill and one night he passed away.[h]It was found that he was affected by an extremely rare (and apparently incurable) disease. He died at 8 years of age. It was the most traumatic loss I had in my life. Still, I was able to finish my thesis in time, and it was of good quality (it was one of the few to receive the most “points” after its defense). I got my BSc on December 13th, 2013 – with a grade I did not like (still higher than most BSc graduates, but not the one I wanted). At least I could enroll in the MSc. – also in Computer Engineering, also at the University of Modena, and also together with Lorenzo and Matteo (who graduated in October and December 2013, respectively). However, my brain was in denial.

(end of part 1)

Lessons Learned (from my BSc., 9 years later)

I realize that what I wrote is way more than what I intended to write initially. Yet, after re-reading it, I do not feel anything to be worth removing: ultimately, all of the above is crucial to explain why I went for a PhD. If I have to distill what should be gleaned from such a long WoT, I would say that:

• Life is unpredictable. There are simply too many things that are outside our control—both positive (I gained reliable comrades) and negative (I lost a soulmate).
• We all have strengths and weaknesses. There is no such thing as “the perfect person/student”. Someone that is excellent at something has to be mediocre at something else.
• You do not need to be a programming wizard to get a Computer Science-related degree. One can very well get a degree with average[i]I dare say that it is simply wrong to associate “Computer Science degrees” with “good programming skills” in general.skills—as long as there is “something else” to compensate.

I conclude this part with a remark. When I got my BSc degree, I was oblivious of the three “lessons” described above. At that point in my life, I was just “going on” – as most humans in their early 20s tend to do (AFAIK). It was only much later that I became aware of these lessons, and put them to good use—or, at least, that’s what I think I’m doing.

Backstory

The SecNoPageLim website was inspired by the “Rump Session” of 7th IEEE European Symposium on Security and Privacy (held on Wednesday, June 8th, 2022).

In particular, during such Rump Session, I gave a talk on the length of papers in cybersecurity conferences. Funnily enough, David Evans (co-chair of EuroS&P22) at the beginning of the session, said that “the ideal length of a paper is 0 pages”, which is something to which I agree with—in principle. Yet, the discussion was very engaging (I believe it lasted nearly 1 hour), and a lot of interesting points were raised by many participants (h/t to all those who contributed!).

In what follows, you will find what is—in a sense—a summary of the discussion that transpired during such Rump Session at EuroS&P22, which is further enriched with my own viewpoints.

My viewpoint

Let me elucidate the reasons why I believe that both references and appendices should be unlimited. I will separately address each of these, providing my perspective both as an “author” (i.e., the one who writes a paper) and as a “reviewer” (i.e., the one who reads and evaluates a paper).

Appendices

Research in cybersecurity—and in computer science in general—is becoming increasingly more demanding. Finding “novelty” often requires to reproduce prior work, sometimes by developing original code, or by creating ad-hoc datasets, or perhaps a mix of both (thereby realizing entirely new testbeds). All such operations (i) require intensive human effort and (ii) are very lengthy to describe in a paper, but (iii) are not a contribution. Hence, what should an author do?

• Skip them when describing the experimental methodology!
  • But what about reproducibility and transparency?
• Carefully describe them in the main paper!
  • Unfortunately, the main paper is subject to page limitations.
• Do not describe them in the paper, but report them in the artifact (if the venue has an artifact evaluation committee)!
  • Unfortunately, the artifact is evaluated after a decision on the paper has been made.
• Provide a link to an (anonymous) repository/resource that includes all such information!
  • Would the reviewers read it? Most reviewers read on hard-paper, and unless they were made aware of such “extra” resources, such information would not be included in the printed version. Moreover, opening a link, extracting and navigating some folders, finding the correct files—all of these are arguably annoying operations.

A (possible) solution to all the above is by removing the limit to the length of appendices that can be included in a paper—at least at the time of submission. The idea is providing a “shield” to authors, so that they can focus on describing their main contribution while transparently providing all technical details related to a given implementation (or a theoretical proof). At the same time, reviewers can inspect such details—if they believe it necessary; or, at the very least, they should not state that “the paper does not provide enough details” in their reviews.

As an author it is disappointing:

• when you have to “cut” some technical details simply because the page limits do not allow you to do it feasibly—and then seeing your paper “rejected” because of “lack of details”;
• when your paper is “rejected” because a reviewer complained about the “lack of details”, despite such details being included in the appendix.

(both circumstances happened to me, and it was incredibly frustrating)

As a reviewer, however, I acknowledge that it is discouraging when you open a paper and you see 30 pages (in a DC template), 15 of which being of appendices… and even more-so when such appendices include content outside your main area of expertise. The reviewer, however, can skip the appendix: in this case, the reviewer should—before submitting their review—check if some shortcomings outlined in their review are addressed in the appendix. If this is true, then the reviewer can either:

• read the corresponding part of the appendix, and check its correctness (perhaps the reviewer was genuinely interested in having a question answered!);
• remove the shortcoming from their review, which would be unfounded (and perhaps reconsider the whole review in light of such change);
• fairly state that the shortcoming may be unfounded by admitting that it may have been discussed in the Appendix, which the reviewer did not fully read—either for lack of time, or of domain expertise (both of which are legitimate reasons!).

(I strive to always do one of the above when I review a paper)

References

The amount of papers that are accepted every year is constantly increasing (see here for some trends). For instance, the number of papers accepted at IEEE S&P was 17 in 2005, 55 in 2015, and 147 in 2022. Such abundance implicitly leads to more related work, which should be discussed—and cited—in any given paper. And such citations “take space”.

As an author, it is discouraging when a reviewer criticizes the lack of “crucial references” when you were fully aware of their relevance, but which you did not include because you did not have the space for them. (on a personal note, I do not understand why some venues accept SoK papers, but impose limits on the length of references.)

As a reviewer, it may appear daunting to look at a paper and see hundreds of references. However—personally—I believe that this is a “fake” issue: a reviewer should not look at the references, but at the main text. Then, on the basis of where (and how) such references are used in the text, a comment can be made.

Takeaway

I believe that there is much to gain (and little to lose) by having more venues to accept papers with “unlimited” references and appendices. If some papers end up being “very long”, then a possible workaround is mentioning it at the time of submission (e.g., prepending “LONG” in the title, as done at FSE), so that reviewers are informed. I acknowledge that some venues may have constraints due to their editorial proceedings: in these cases, however, it is still possible to impose a constraint only for the final version of the paper (this is what is done, e.g., by USENIX SEC).

Bottom line: the purpose is to improve the peer-review. This solution is obviously not perfect, but (IMHO) it is a step towards a more constructive and fair peer-review process.