by Aamir Attaa and Syed Talal
Pakistan voted yesterday. Then it waited for results. And waited. And is still waiting.
With theories abound about the possible reasons behind the delay in declaring election results, Pakistan has seemingly divided into gloating PTI supporters and everyone else crying foul.
In this report, we aim to break down exactly what went wrong.
What is RTS?
For the first time in election history, Pakistan used a Results Transmissions System (RTS) — an online system to relay results from 85,000 polling stations around the country to the central headquarters in Election Commission of Pakistan.
Here’s how it (should have) worked. The Presiding Officers (PO) were supposed to use an app to enter results and take a picture of Form 45. The data would then be transferred to the ECP server… and that’s it. Really.
What seems rather basic from a technical point of view became a point of contention as RTS went down at 9 p.m, delaying results.
RTS Was Far From a Perfect System
There were question marks about the system even before the elections. According to local media reports, Returning Officers (RO) from Faisalabad, Okara and other cities wrote to the ECP and expressed concerns about RTS.
According to the Express Tribune, complaint letters were received from multiple ROs.
An RO said:
I have the apprehension that in such circumstances your RTS system will not perform on the poll day.
An unnamed NADRA source cited by the publication claimed:
Due to possible failure of the RTS system, serious issues related to transparency may arise on the night of poll day.
RTS Was Designed by NADRA
RTS was developed and its usage was made mandatory while PML-N was in power, through a law that was passed by the previous government.
It was developed by NADRA, who have expertise with large-scale apps. This is not the first time the institution has developed apps on such a scale so it’s unlikely that technical abilities were a factor in the failure of the system.
Not to mention, ECP and NADRA knew the system scope from the very start: 85,000 endpoints would be submitting data and that makes a developer’s job easy.
If they know the load, they can test and optimize for it. This means that all possible outcomes were known to the developers even before they started working on RTS. It’s usually a simple task to design a system where all variables and constraints are known, compared to a system that has to deal with unknowns.
Why Hasn’t the System Come Back Online?
The worst happened and the system went down. Fine. But does it really take hours to restore it? Even if the system was designed poorly and the load became unpredictable, absolutely not.
In the time since the system went down, completely new server-side solutions could have been made from scratch and launched – considering the nature, emergency and scale of the problem.
If an entirely new solution could be designed, developed and launched from scratch in the time RTS has been offline, you can imagine how much time would be required to restart services and put the system back online: merely a few minutes.
The lack of a contingency plan on part of ECP and NADRA is simply baffling, if not downright criminal.
So what happened to RTS? There are a few possibilities here:
- People using the system were unfamiliar with it to the point that they didn’t know what to do in case of failure
- The system was targeted by a malicious entity with a DDoS attack
- It wasn’t brought back online on purpose
We should mention here that mitigating for DDoS attacks is quite easy these days. ProPakistani suffers DDoS attacks from within Pakistan, India and abroad and with a limited budget, we suffer near zero downtime.
Even if all 85,000 end-points were connected and submitting data at the same time, it should not have caused the system to collapse. We are aware of websites, apps, and servers within the country which serve hundreds of thousands of concurrent connections so technical failure simply does not make sense.
And even if it did collapse, it does not take so much time for it to be restored.
Possible Scenarios in Light of Failure of RTS
So what now? The solution is quite simple: conduct an audit.
An audit for a mobile app and a system like RTS isn’t a complex task. If ProPakistani were given access to the servers, it could determine what went wrong within hours, if not minutes. It’s not a testament to our aptitude; rather, it really is that simple.
We want to emphasize that with the government’s resources, it’s child’s play to determine why RTS failed.
After the audit, there could be three possibilities.
1 – RTS Failure was Intentional
The failure was intentional and the system was sabotaged by the outgoing government in case of PTI taking the majority.
Remember, this is the first time RTS has been used in Pakistan. It was designed, developed and made mandatory when PML-N was in power.
As soon as a clear lead was established, the system was shut down to create controversy and unite the losing parties against Imran Khan.
d
2 – RTS Went Offline Due to High Load
RTS went down unintentionally and if that’s the case, then it is gross incompetence so severe that it can be classified as a crime against Pakistan.
The responsible people and departments should be investigated, tried and put in jail for messing up a system so critical to the country’s future.
3 – Outside Involvement in Putting RTS Offline
Considering Indian involvement and stakes in Pakistan election, we would not want to rule out the possibility of outside interference in elections via attacking RTS through a DDoS attack.
But then again, it’s not that hard to plan against DDoS attacks. There are websites (such as banks, social media websites and so on) which are routinely attacked and never fail.
If it’s an outside job, the people at the core of app design and deployment should be fired immediately with heavy fines at the least.
Finding Out Who is Responsible
If you think that a JIT and several months are required to determine responsibility for the catastrophic RTS failure, we have good news for you.
Unlike other fields, debugging computer systems and determining the cause of failure is straightforward. A single expert (or more if you want to be completely certain), can determine the cause within minutes, if not hours.
Here’s how it can be done:
Let us know your thoughts in the comments below!
Can you confirm if the RTS was on public domain?
test
“The failure was intentional and the system was sabotaged by the outgoing government in case of PTI taking the majority.”
How an offline site can be used to manipulate data? If they have to do, all they could do is to take a mirror of existing db and put it offline for manipulation and let the front app keep sending data to originally connected db.
Pleas learn a bit of things before you say a word.
They didn’t have to manipulate the data. Just delaying the results could have sabotaged the process enough to be termed the whole election as rigged.
Are you suggesting that a server sends data in delay is ALWAYS engineered? Think again. Accept it was a technical failure, a project endorsed by PML-N despite of reservations by Dr Saif. Read his post.
I think you are getting it wrong. Delaying relay by just shutting down the system is a possibility that we discussed above.
We are proposing is — based on how things happened — system was shut down on purpose and intentionally. Other possible scenarios are also discussed for your reading.
Shouldn’t Dr. Saif himself be investigated ? When can we get an independent Punjab IT Board audit?
And Dr. Saif has a long standing grudge against NADRA; in 2012 or so his PMLN nomination to head NADRA failed due to background concerns- Google yourself !
The pot calling the kettle black.
The great doc has admitted his role in undermining our democracy. He [as well as khadim ala] knew RTS was going to fail. He kept mum and didnt say anything until today-the right time to malign NADRA+EC. Why didnt he soeak up before election? 1 tweet would have done it.
I must wonder, why is he so eagre to head NADRA?
1.NADRA leaks
2.PITB Leaks
3.And now we have EC leaks
Please have mercy.
Add to this:
4. Our hero quietly receiving $200K from Google as PITB chairman without due disclosure for a “data research” project.
A. Whose data?
B. Who owns it?
C. Which relevant GoP agency was informed?
Reminds me of another 2011 data collection incident.
Actually not all Presiding Officers had Smartphones with them. And those who has were supposed to use their own sim with self purchased data to upload. 85,000 presiding officers did not all have smart phones and not all were willing to use their own data package. this was ECP responsibility.
Good post… it should be investigated.
Being so close to the whole procedure, here are my findings.
1) Poor training of the RTS system by NADRA officials. Being a master trainer of ECP, I witnessed this.
2) No backup of electricity. POs were supposed to reach polling station a day before elections but were required to send data after preparation of result. Most of the POs had 0% charging left in their smartphones because most of the Polling stations haven’t electricity access. Power-banks should have been given.
3) The app was compatible with marshmallow and newer versions and selective lollipop devices. Most of the POs had to bring someone else phone for the task. The app hadn’t any compression module to compress the photos for faster uploading. High end devices with good cameras took longer time to upload and in most of the cased ate-up all the data but couldn’t successfully upload the data.
4) NADRA server wasn’t responding after 10pm. Many POs tried to transmit data after capturing but app crashed.
Presiding officers were given 1000Rs to bring their own smartphone with data package to upload data. Most of the presiding officers were already dissatisfied with the system due to clause 1, 2 and 3. Even NADRA officials knew the issues and acknowledged them.
Only guy with facts ?
All valid points .
1) and 3) The trainers from NADRA were not very clear about the Android version as well.
2) Having been on the election duty myself, I had to bring my own personal power bank as back up
We were just discussing the app side of the solution. I am sure there must be issues (other than app itself, such as trainings). So to debug the issues on the app side could be determined by a simple approach (such as one mentioned above). More issues can be investigated and brought to the front my the concerning authorities.
Also to note, we couldn’t say that NADRA never wanted this solution to be successful. They showed ZERO interest for reasons beyond our comprehension. But we can’t say this in a report since we couldn’t obtain proof of such a statement.
Another thing in your Diagram, that mentions ROs. RTS was being used by presiding officers and not Returning officers.
I believe. only (4) is the main cause here that cause the anarchy :)
I am system architect who also happens to manage Dev Ops for my company’s Solution.
I dont know about the training and effective manual delivery for RTS to ROs and POs.Thats not my domain but software is.
(In the time since the system went down, completely new server-side solutions could have been made from scratch and launched – considering the nature, emergency and scale of the problem).
It is such a blatant statement, its not even funny.And I am so confident if you can really do that, call me 03137727142. I will hire your at the moment. Please Please Please avoid yar. For god’s sake Its okay.I will not hate proPakistani if it didnt knew what was going on.Its understandable.But imagining systems, there problems and then fixing those imaginary problems. Thats not very responsible journalism.
Assalamo Allikum Aamir Bhai and Talal !
I must say that for the firs time even ProPaksitani said something unjust and can’t help agreeing with Mr. Muhammad AbuBakar and Yes need to revise your specific statement above.
Now come to some otherthings that though RTS was developed in PMLN government but who( department ) was having the technical and administrative ownership of the app ??
then need clarification that if Form 45 etc( results details ) data was also entered manually through app or only form uploading was the only thing needed to be uploaded.
then needed to know as i was told by a presiding officer that total of 7 forms mean 2 triple copies each ( best of 3 will be selected later once uploaded) ( 3 X 2 (1 NA + PA ) ) in addition of single form 46/47( ballot counts details etc ). Then here comes the thing 7 forms ( images of yet unknown sizes X 85,000 polling stations concurrently is big question on the working principal of RTS.
I agree completely with your statement Abubakar here. But what was the real problem? Do you think we will be able to hear trustable official version?
Sadly in such cases.You cant pretty much know anything without being on the ground.
Sir jee, I know a new app can’t be developed in such a short time — but a well tested app also doesn’t remain offline for hours after hours of going down.
The anomaly of developing new app from scratch was referential only to point out that a system that went down at 10PM couldn’t be brought back online even after 12 hours (when above report was published).
About the app, we have some understanding of what it was suppose to do: Upload an image to Google Drive. That’s it.
+1 as being a Solution architect in industry for more than 12 years.
The issues raised by ProPakistani are quite lame. I believe, they have no idea about the Software Development Life Cycle (SDLC), deployment and common issues that can arise on a software that goes live for the first time.
There are several faults in this post.
1. Debugging is easy — Well, I bet you never did it and when it comes to production server with such a high load. No-one can give a deadline to locate the issue.
2. “Even if all 85,000 end-points were connected and submitting data at the same time, it should not have caused the system to collapse. ” — I am sure, you never experienced even 20,000 concurrent connections. If you had, you would not have said it. Technically, it depends on the infrastructure where remote application is hosted.
3. “RTS went down unintentionally and if that’s the case, then it is gross incompetence so severe that it can be classified as a crime against Pakistan.” — I agree to the first part of incompetence but in the field of IT, system fails, just like an electrical bulb that stop work just like that. Again, I am not saying that those who developed it should not be punished but “crime against Pakistan” is a big word to say.
There are few others for which I do not have time to respond on.
IMHO, if we assume that it was unintentional, I think, system was not tested with such a load that cause the server to crash. Uploading a file takes some time to upload while on a mobile internet connection, Uploading is a task that keeps the server engaged all the time until it is fully upload. Imagine, 30,000+ users uploading 5+ Mb of file it will overflow the web server’s connection queue. In such scenarios, a scalable server with load balancer should be used, there are many other things to consider to ensure it will bear the load.
ITNI ENGLISH MAR RAHAY HO…. SYSTEM BAND HOWA YA KIA GAYA…. DHANDLA AKHIR HI KIA GAYA, JAB TAK PTI 110 + NAI HOI TAB TAK, MANU YA NA MANU KISI KO KOI FARAQ NAI PARTA, KISI MEDIA MEIN HIMMAT NAI OPENLY ESTABLISHMENT K KHILAF BOLNE KI, THIS SELECTION 2018 = MUSHARAF 2002
Dear Aamir & Talal,
as you know in govt. sector mostly many things developed in “THANDAY KAMRAY” air conditioned room / environment and all things available or consider in ideal status.
in RTS case the first question is that is this system tested in field esp. in remote areas in any by-elections for next preparation.
2ndly Sir you believe that it should be handled this way n that way. I report you here I belong to a remote district and there is not all area even covered by 2G and this app designed for at least 3G. you know better then me that how much area is covered by 3G by All operators till now.
Many POs have one sim or connection of that operator that is not available at their duty place so the RTS failed.
There is no exercise made for detailed data about cell operators of their services in all 85000 polling stations areas.
If ECP want to do this type of option for Results he should first purchase some esp. devices that have this RTS type app and all type of connection then he should exercise in by-elections then in General Elections.
Mere bhaion Simple si baat ha..Whatever had gone gone( jo ho gya so hogya).. System ko behtar bnane ki baat karo.. purani qabrein khodne se koi faida tu hoga nahi.. I am a common man and working in a simple normal company and i have normal salary and all. All to sum up is “I am a Common Normal Man”. Hum cheezon ko simple way mein q nahi lete, hum khud mushkil bnate hain. Normal baat keh raha hun with my normal brain.. jo har admi ki samajh mein a sakti ha qk yeh simple c baat ha..agar samajhna chaho tu!
How will system improve..that’s where we need to focus!
Simple sa solution ha..May be ap sab jante bhi hu.. Humare fingerprints tu milte nahi kisi se not even our twinsjis tarha bank mein ya ATM mein ap fingerprint use karte hu same aise process hona chaiye for casting the vote..jese hum mobile k liye sim lene jate hain tu sim ki verification k liye angotha lgate hain..same goes for the vote process..is se fair election bhi ho jaenge aur kisi ko koi pareshani bhi nahi hogi.. Is se rigging wala chakkar he khatam.. we can say e-polling
Sabse pehle ap election ka koi day fix na karein jis se load barhay..ap date announce kar dein k is week mein apko vote dena ha.. Nadra have our database and fingerprints (right people). Secondly, jaga ki halls ki koi kami nahi ha..wahan devices lga dein ya malls hain wahan lga dein for voting..first step mein hum jab finger lgayen tu humara data ajaega k naam pata etc etc aur phir next karein tu agli window ayegi parties symbols, name etc jo bhi ha..apne jo choose karna us par click kia aur wo save hogya next msg ayega thank you for your vote.. your national duty is done is type ka.. i think ye koi rocket science nahi ha..banks mein jis tarha token system chalta ha ya ATM machines with biometrics ya hum jahan job karte hain ha waha attendance system ha na apka jo thumb pe chalta ha..
I hope everyone understand what i said. Its my opinion (sirf meri zaati opinion ha) kisi ka agree na hona ya hona uski apni marzi ha..
I now rarely visit propakistani mainly because of these type of posts along with other reasons like irrelevant material and masala posts.
This whole (unnecessarily long) post has not a single useful or correct information that shows authors ability to understand how multi-tasked, multi-threading applications work and what factors impact or cause such scenarios, this is not a matter of opening task manager and viewing stats for cpu etc it’s far more complicated with variety of reasons that can cause such scenarios.
I’m also surprised to read proposed fallback scenario of developing something from scratch during downtime, what a laughable suggestion for an application of this criticality and large number of users etc.
Lastly the claim to troubleshoot root cause in hours if not minutes :D seeing the above “technical competence” one can only laugh on such tall claims.
Respect your opinion. The anomaly of developing new app from scratch was referential only to point out that a system that went down at 10PM couldn’t be brought back online even after 12 hours (when above report was published).
About multi-tasked, multi-threading apps, this app in question was uploading the data (image of voting result paper) to a Google Drive and that’s it. You can well imagine the complexity of its operations.
Yes, finding out the failure of such a basic app shouldn’t be an issue — I still stand by my claims, still.
This is not RTS…. This is ECP website
Imran got his facts wrong. The system that allegedly got down was RTS and it was hosted by NADRA themselves. No tender was issued for hosting this platform by ECP. At the same, the system which requisitioned through tender is still up and running.