Captain’s log: The Journey from Project to Product

Published in

CYBR Notes

7 min readMay 8, 2021

This is the Captain’s log, containing a tale of teamwork, failure, and success. Along the way, we as a team faced unforeseen challenges. So we had to learn how to navigate this ocean of uncertainty on our own. It not only made us stronger but a more efficient crew. So grab a cup of coffee and come with us on a journey to making a Product we have all come to love.

Chapter One: The Proposal from Hell

This was the time before the pandemic. I was sitting in the office doing regular work. I remember the cup of coffee (now I‘m a green tea man) on my table when Umar came and told me about a new project. He said, “A company wants us to develop a ticketing and entry management system for an amusement park. They are in an emergency and they need things as soon as possible”. “Well what are the requirements?” I asked. “They will give us details tomorrow”, he replied. The next day around 9:00 am I receive massive paperwork. Just skimming through it took me three hours.

In requirements, it was mentioned that they need access doors, ticket issuing systems, and software with a long list of features. I was surprised why they are asking us to make it from scratch. Because we don’t sell those machines and that software is also in the market. We usually work on custom hardware and software projects instead of deploying off-the-shelf solutions available in the market. We are a small team of engineers and they need a team of technicians (which they themselves were). I requested a meeting with the client.

The Pandemic

Days went by and the pandemic arrived. Now major businesses were shutting down. And we had heard nothing from the client. One by one our team members started to leave and in the end, only Umar and I were left. The projects we had disappeared one by one and finally the revenue stopped flowing into the company. We managed to hold our ground for a few months, thankfully businesses started to reopen again. New and old clients started reaching out to us again. We also began to expand the team once again.

Chapter Two: The Lying Paperwork

At this point, things were looking up, the client started discussions for the ticketing system again. I was still not sure if the requirements document was really scary or maybe it was just me. I had questions: Why do they want us to develop this system from the ground up? Why do they want to reinvent the wheel? Aren’t there enough off-the-shelves solutions available to get the job done? What exactly needs to be done in this project? My understanding was that they need a ticketing system for their theme park, which will contain access control systems. Just one meeting on call and the requirements were clear.

The client had a ticket management system and the requirement was to build a custom controller which will control the access doors from their software. There was only one company that used to make these controllers which went bankrupt. Now they want us to build those custom controllers, which is exactly what we do.

“Documents are liars. They are written by those who don’t know their audience. They are shared with those, who aren’t their audience. When requirements change no one wants to update them and they get bloated with multiple layers of lies.”

A simple zoom call could have saved the client from writing those redundant and liar documents. And also could have saved me from a lot of overthinking.

404: Specs not Found

Now comes the challenging part; We had no information about the turnstile hardware our custom controller was going to interact with. It was especially annoying to find no datasheets of the hardware our client was going to use.

A technician’s job is to install equipment, connect a bunch of wires and do some mounting and everything starts working and they are good to go. They are not concerned with what's under the hood, what kind of signals are flowing, etc. But this was bad news for us. Our client only deploys off-the-shelves solutions. The information they had wasn’t rich enough to proceed with custom hardware development.

Adding fuel to the fire, the client was in another city, so we didn't have physical access to their equipment. Now we need to fill this information gap. We did this by studying similar turnstile hardware and scanners. And we selected some of them for in-house testing and development.

“Sometimes when you are making a decision the vital part of the information is missing. It’s your job as a professional to fill that information void and if it can’t be filled then it’s your job as a professional to measure the risk.”

Chapter Three: All hands on Deck

Now here’s where things start getting intense. Picture this; Deadline is just around the corner and the engineering team is testing prototypes, Everyone is under pressure. Only a few days to go for the Final Deployment.

Products like these usually take months and even years to complete and perfect. In this stressful time, our engineers did what they are good at doing. They applied hacks and made everything work. Quickly prototyped the system and made it ready to ship (no pun intended). Now our engineering team had to coordinate remotely with the client to make the deployment a success.

A Fall from Grace

Due to the huge knowledge gap between engineers and technicians, things were not working out. And the deployment was plagued with delays. We had no choice but to physically go and deploy the system ourselves. We requested the client to arrange our visit to the deployment site. They refused this request because this was going to cost a lot. Things kept on dragging and finally, news arrived that the controllers were stuck or crashing, and a few of the turnstiles broke down. This happened because the technician used the wrong wires to connect the turnstile with our controller.

The client requested a new and improved batch of controllers be made, but we were one step ahead this time, the engineering team had fixed all the hiccups in the hardware and the firmware quality was improved. But when we asked for more time for testing, they were not happy. They were furious because the deployment was delayed and unsuccessful. They just did not want to wait.

Chapter Four: The Journeys End

Everyone was trying their best to meet the deadline. And looking at the work quality of the team. I found two major issues:

The interface of the system was not intuitive to understand for a technician. This is not easy to fix and requires time to perfect.
There was no logging in the controller, the next day is the deadline, and the engineering team has decided to ship the next batch.

I intervened and stopped the shipment of the next batch. Rehman, who was looking at all the engineering was angry and stressed. He was trying to meet the deadline and got more stressed about this decision. I felt guilty to do this to him and wasn’t sure how to convince him but I am very thankful that he listened to me at that time and stopped the shipment, we added the logger, and did all the testing again. This delayed us for a day.

This time coordination with the deployment team was better but still took several days, and still, deployment took longer than the development and testing. And after deployment, we started getting complaints that the system fails on a random ticket. The client was yet again furious with us and lost all confidence in our system. This time the engineering team insisted to go onsite and fix the issue themselves. Flight tickets were arranged for Umar and Rehman to leave for the mission.

The next day our onsite team faced a lot of criticism and was blamed for the faults of the entire system. Issued tickets would not work with our turnstile controller, some tickets would work and others would randomly stop. Rehman told me that “When we were debugging the system I was badly puzzled and thought maybe we have made some serious blunder. Because the problem was not easily diagnosable. Then I remembered that you insisted on adding a logger”. He fetched the logs and sent us for analysis. Huzaifah (our trusty crewmate) checked the logs and communicated the issue back.

Suddenly the tables were turned, people who were angry at Rehman and Umar were running around to fix the issue. The problem was in their ticketing software and not our hardware; random tickets were printed wrong. Rehman told me that he was trying to help the software team fix the Software bug in the ticketing system. I always admire this type of supportive behavior. The deployment was a success. When the team came back and were discussing this issue over tea. They told me that adding the logger was a lifesaving decision.

“If an external error is occurring your system should point out who’s fault it is so that they can fix it.
If an internal error is occurring your system should point out the source of the fault so that you can fix it.”

Professionalism is all about ownership

Meanwhile, we were improving the system and firmware quality. Some more hiccups with the hardware were identified. The controllers were getting damaged due to external electrostatic noise. These issues were fixed one by one. And a more reliable and efficient batch was ready for deployment. The project we started a month ago as a hacky prototype now took the shape of a robust product. We now have more clients asking for the same thing.