Issue Summary
On Thursday, the 27th of March, at 14:50 UTC, an update was rolled out to all TrakIT installations. This update caused an issue when work orders were reset. The problem caused some events and data fields in the work order to be replicated in other work orders where the same fields were empty. The issue was rectified, after which the extra events and data fields with incorrect values were cleared. No data was lost, and no other modules were affected.
The majority of TrakIT installations were not affected. Only those installations where one or more work orders were reset during this time were affected. If your installation was affected, our support team has been working with you to restore the affected data.
We want to sincerely apologize for this issue. We understand that this may have disrupted your workflow or affected your experience, and we deeply regret any inconvenience this may have caused.
Issue Details
An update was made to a stored procedure that resets a work order. This procedure is called when a work order is reset manually or when the workflow and/or services are changed. It can also be called with an action after an event or data field is changed.
The Reset procedure performs five steps:
- All events & data are copied into a temporary table.
- Extra events & data are removed from the work order. In case of a workflow change, all events & data fields are removed.
- Events & data that are missing are added to the work order. In case of workflow change, all events & data fields are added as per the workflow set-up based on the services selected, job type, and customer selection.
- In case of workflow change, restore the temporary table into the event & data fields that match the code of the original fields.
- Reset KPIs to match the workflow definition.
In step 4, an issue was identified that was preventing the restoration of events and data fields after the change of the workflow. The update issued on Thursday attempted to fix this issue. While the issue was fixed, events and data were also restored on other unrelated work orders on empty matching fields. It is important to note that only empty events and data fields were populated. No existing events or data were overwritten at any time.
Upon discovery of the issue, the cause was identified and fixed at about 09:30 UTC on the 28th of March. Subsequently, affected events & data rows were identified and cleared using available backups as a reference.
Root Cause
While the update to the stored procedure was tested extensively on work orders by changing workflows in test scenarios, the test cases only tested the work orders on which the updates were performed and failed to notice the full impact of the modified procedure.
Recurrence Prevention
Updates to stored procedures that involve bulk updates or insertions shall be verified against expected row counts. Expected run times shall also be measured and verified when testing stored procedures. A more careful impact analysis shall be performed on such updates.
Please be rest assured that we take this matter seriously. We work diligently to identify and resolve such problems, and we will implement measures to ensure that similar issues are prevented in the future.
We truly value your trust and patience as we strive to improve. If you have any concerns, questions, or feedback, please do not hesitate to reach out to us at support@trakit.in. Your insights are invaluable to us and help us serve you better.
Thank you for your understanding and continued support.
