1. the biggest mistake I made and what did I learn from it?
This biggest mistake I made was a lapse in fixing a stability problem of our online ranking model’s daily update pipeline. It resulted in a significant financial loss due to this problem.
This happened shortly after I joined the team at the beginning of 2024. Since I had made a minor modification from the training pipeline, the responsibility for the baseline model was temporarily under my account. During the Chinese Spring Festival holiday, the training framework became unstable and the model stopped updating. I received the automated alert. Instead of handling it myself, I handled it off to a newer college in US time zone because he should be at work in that time. He didn’t realize this is a serious problem. And the problem persisted undetected until my supervisor was alerted. With the help of my supervisor, the problem had been solved very quickly.
This incident led to several detailed case study meetings and it was a pivotal learning experience. System stability and robust monitoring has already been my number one priority after this case. I now dedicate time every day to verify the health and status of the entire training pipeline and when my team members ask me what is the most important part in the work, the answer is system stability.
