The article considers the twenty-year history and experience of creating automated systems for testing knowledge and skills in programming at the Novosibirsk State University. An analysis of the effectiveness or ineffectiveness of methods and means in the knowledge testing systems in use is given. The modern state of the architecture of the NSUts system is described.