2019 Server Upgrade – addendum

Nov 6, 2019

We like to think SOBS is a nice stable application that is always available. In general this is true, although like all IT companies we “suffer the slings and arrows” of the upgrade process.

In the last three weeks we have had a number of problems appearing, and we are very sorry if any of these have impacted your work. We have been working long hours to get everything working smoothly again.

We really appreciate the time you take to notify us when something in SOBS isn’t working correctly. Often we won’t know SOBS is broken until you send us an email or give us a phone call, so don’t be shy – get in touch.

Below is a brief look at the problems we have been dealing with.

Accounting system

We have used the same accounting system for 10 years now, and eventually the technology gets old, to the point where the required libraries are simply no longer available. This is what happened to our accounting system three weeks ago.

Getting a new accounting system is easy enough, however we have a significant integration with the accounting system for generating sales orders, receiving payments and updating customer details between SOBS and the accounting system. It took me 13 days to complete this integration!

Resource Booking

Occasionally we have one of those intermittent problems that are difficult to identify, requiring specific situations or event sequences in order to recreate the problem. We had this with Resource Booking (version 2) – it seems to happens between 8:30 – 9:30 (the morning rush) – effectively bumping users back to the login page.

We tracked this down to a performance issue – the new server is too fast. It appears to be tripping over its own feet as it rushes to complete each request. When the second request attempts to start before the first request has completed this problem occurs.

The simple solution is to upgrade to version 3. You can upgrade here

Student Attendance

Similar to the Resource Booking bug above a second instance of the problem appeared when processing the absence triggers. In this instance we rewrote the code around this problem and it is now resolved.

Email

Based on expert advice we upgraded our mail server from EXIM4 to POSTFIX. The advice wasn’t wrong, it was the correct move, however it means that some things will operate differently. There were a couple of post migration issues sending emails, however these were relatively easy to fix once we were aware of them.

We also took some additional efforts to tighten the security around our mail processing. This may sound simple, but I assure you it is not a simple process. We support the option for schools to send email via their own email servers, and this involves a number of different email systems. In theory the systems all support a standard connection convention, in practice however when combined with a tightened security model there are some differences.

Calendar – duplicating year records

It was a surprise to us when we went to view a calendar for a school and found 50+ records for 2019 where there should only be a single record. We hadn’t changed any code around this so why is it suddenly a problem?

We found the problem – we were passing the current year as a value to the library that looked for a matching record in the database, but the library was actually expecting a full date (not just the year). While we were passing the wrong value to the library, it was actually working, now it doesn’t. As soon as we passed the correct date to the library it all started working again normally.

Database field lengths

For the past 12 years we have been able to pass a long text value to the database and it would store as much as it could, truncating the characters that didn’t fit.

The latest version of the database is now issuing an exception when this happens. Consequently there have been a few places where we have extended the length of text fields to cope with the values being entered.

This happened on the SMS messages and responses. Booking description, and approval text in Resource Booking. The absence reason in Student Attendance. The short-code value in staff records (has a maximum of 10 characters).

There may well be some other areas where we need to increase the field size, or at least validate the maximum content length. Please do let us know if you come across something like this.

A slight variation of this is the notice content in School Notices. Not so much the length of the content but characters that aren’t considered regular text. This can happen when you copy and paste from a Word document or other non-text source. We now pass the notice content through an additional text validation routine.

Kamar Import

This school is in New Zealand (currently 3 hours ahead of Brisbane) and when we imported the bookings they were appearing in the database with the effective Brisbane times.

Initially I was suspicious of the ICS parsing library, however they claimed no responsibility. I’m now guessing that the latest version of PHP has corrected an issue with the construction of a date/time from the booking data – it’s the only possibility.

We now construct the date/time and apply the schools timezone offset before storing the booking.

Importing Booking Data

The latest version of the database once again caused havoc with the import process. Previously where a table had 10 fields (or columns) we were able to load 9 fields and the database would assume the last field had no value.

The current version now complains if you don’t import all 10 fields. So we now specify all 10 fields on the import, for each of the seven time tabling products we support.