List.ayy.fi server that administers AYY’s mailing lists failed to operate on 20 May and has been restored after that from very old backup copies from August and December 2012. The service is scheduled to reopen between 27-28 May.

Effects:

• Recipient lists, passwords and administrators of the lists are according to information from 1 Aug 2012. List administrators will have to update this information. In case of forgotten passwords, you can get help from AYY’s IT sector as usual at tietotekniikka@ayy.fi.

• Due to old recipient information, all lists have emergency moderation set on. Because of this, all messages must be separately released from the administrative interface of the lists. When the list recipients have been updated, emergency moderation can be removed from the administrative interface at https://list.ayy.fi/mailman/admin/<listanimi>, at General Options -> Additional Settings -> Emergency moderation of all list traffic.

• Lists that were added during the lost period will have to be recreated. Lists requested via e-mail at tietotekniikka@ayy.fi will be recreated during Monday.

• Mailing list archives were lost from the period between 1 Dec 2012–20 May 2013. If you need the archives, it is now advisable to ask the list recipients if someone has saved them.

Further information and help from AYY’s IT sector at tietotekniikka@ayy.fi.

—

A broader description of the incident:

List.ayy.fi is a blade server with two mirrored 300 GB disks. Backup copying of the server has been reduced due to the problems encountered and because of its long duration. The problems result from the large mailing list archives of list.ayy.fi, poor backup tape library and average backup software.

The other disk of the disk mirror of list.ayy.fi failed to operate on Thursday, 16 May 2013. The replacement disk was received on Monday, 20 May 2013. Before the exchange of the disk, the intention was to take ”the last backup” to another server. However, this was not done due to time pressure, as we relied on the fact that normally these replacements of broken disks run smoothly.

This time, the exchange of the disk led to the erasing of the intact disk and there were no other options than recovery from backups.

The latest functioning backup was from December 2012. In addition to the backup, disks were sent to Ibas for examination. However, Ibas was not able to restore the files and we will have to make do with the old backups.

Our company apologises for the inconvenience caused and the resulting downtime of list services, as well as the data loss of list archives from the period of six months.

In order to prevent a similar problem in the future, we have decided to renew the backup system completely without leaving any old components left.

Tommi Saranpää, Administrator of Trinet