July 3, 2018

Episode 3: Tickets, JavaScript and suppliers. What really happened in Ticketmaster data breach?

Tune in to understand what really happened with Ticketmaster databreach as Nikoloz looks at this event from a different perspective, provides insights and shares lessons-learned.

Episode 3: Tickets, JavaScript and suppliers. What really happened in Ticketmaster data breach?

While the world was busy reading through popular media about Ticketmaster data breach, playing with their muscles by stomping down an “evil third party” and not trying to look in to the causes of this incident, I was eager to understand what went wrong and what went right. I will be diving in to what really happened with Ticketmaster and share lessons learned.

Other ways to listen:

Anchor    Apple Podcasts (iTunes)    Google Podcasts    Breaker    CastBox    Pocket Casts    Radio Public    Spotify    Stitcher    RSS

Transcript

TICKETMASTER

Last Thursday on June 28th, customers of ticket sales and distribution company Ticketmaster, have received an alarming email from the company informing about what seemed to be a data breach. Let’s analyze that email and see how story unfolded.

The email stated that

On Saturday, June 23, 2018, Ticketmaster UK identified malicious software on a customer support product hosted by Inbenta Technologies, an external third-party supplier to Ticketmaster.

Note, that they informed customers five days after they allegedly discovered the data breach. And right in the first line of the email they take your attention to the fact that “hey it was a third party, it was a supplier”. This way company immediately tries to hide their responsibility.

Than Ticketmaster mentions that they have disabled the supplier’s product on all websites as soon as the malicious software was discovered.

As a result of Inbenta’s product running on Ticketmaster International websites, some of our customers’ personal or payment information may have been accessed by an unknown third-party.  

Ticketmaster again emphasizes that it is because of Inbenta’s products that hackers got their hands on you personal or payment information. The word ‘or’ is an interesting bit here, does this mean that they are not sure what was accessed? Does this mean that they have no capability to investigate the full extent of the data breach? I don’t know, maybe we can find it later

The email continues by stating that customers are being informed because they purchased or attempted to purchase tickets between September 2017 and June 23 2018, so whole 10 months. Another interesting bit is the part where they say “Whilst we have no evidence to suggest your data has been compromised, we are notifying you out of an abundance of caution”. Another sign that Ticketmaster is probably not aware about the extent of the attack, a hack or a data breach which I hope we will see in today’s podcast.

Forensic teams and security experts are working around the clock to understand how the data was compromised.

At the time of this email it has already been minimum 5 days, whilst trying to understand the attack vectors.

In the last part of the email Ticketmaster gives its customers an insight of what they are doing, and apparently, they have established a dedicated webpage for this breach security.ticketmaster.co.uk, they did a password reset for all customers so customers would need to reset passwords upon next authentication,

So, what I see from this email is the following “Hey guys, we are really sorry but our provider had a malicious software which we were running for at least 10 months and we decided to let you know that your sensitive information, which might be personal data or financial data, has been stolen. But has it? Well, we are not quite sure what happened so that’s why we want you to inform you, just in case you know, that this was discovered 5 days ago. And guys, guys… It was not us okay, it was them!”

What I liked about this communication

  • Not much, but good thing is that they provided information about what they are doing and they also created a dedicated website for the data breach with little bit more details I hope. We will have a look at the website in a minute.

Now let’s see what I did not like:

I think the communication lacked the leadership, and owning up to the mistakes. In every single part of the email they try to tell us, that it is because of the third party, its their fault, we did not do anything, and we are trying to clean their mess. It’s seems to me a little bit unethical and frankly a communication from a company who knows they have f*-ed up and are afraid to clearly state that. Even though we do not yet know what happened exactly, telling your customer that their personal or financial data has been continuously compromised for at least past 10 months through your application, service or website and shifting blame to the third party with every opportunity, leaves me with a weak and negative impression about company and their security leadership. Also, it gives me a valid hint, that first, company might not have clearly defined procedures in place and second they might not have risk assessments, vulnerability scans, code reviews and regular penetration tests in place.

Okay, let’s see what information does a data breach web page provide. According to the webpage security.ticketmaster.co.uk “less than 5% of global customer base has been affected by this incident. Customers in North America have not been affected.” And interestingly enough they are open to provide customers with 12 months free identity monitoring service. Hmm, so I hover over the link which should lead to the identity monitoring service provider and the url says a.pgtb.me, what? And when trying to visit pgtb.me is not accessible. What? After a search through duckduckgo I found out that the domain is reserved by a company names short stack, which is an email marketing company. Okay, so, a company that just has been breached provides customers with a service on a domain name that would most probably be better used for phishing attacks. And interestingly enough, by following the link we will end up on a signup form page asking for FirstName, Last Name, Email and location. No the best way to reinstate trust in your customers.

Back on the data breach webpage we have additional information about what happened. According to the page Ticketmaster UK identified that malicious software on a customer support product hosted by Inbenta Technologies, an external third-party supplier to Ticketmaster, was exporting UK customers' data to an unknown third-party. As soon as we discovered the malicious software, we disabled the Inbenta product across all Ticketmaster websites.

So it was not a traditional misconfigured database issue or a one time hack, hackers were literally exporting information continuously for whole 10 months.

And that’s it, no more information from Ticketmaster and many unanswered questions. Why did a support product have any kind of access to personal or especially financial data? Does this incident mean that the data that this data was not encrypted? What is the level of compromise? What means of security was ticket master using?
Furthermore, many customers reported that they were not asked to change their passwords, demanding answers on their hot questions, but no reaction from Ticketmaster, at least as far as I could see from their emails, blogs and social media.

Currently National Crime Agency of United Kingdom is also helping Ticketmaster with this incident. According to the statement they are actually managing the ongoing investigation and are working with Ticketmaster and partners to secure evidence as well as provide mitigation advice and guidance.

Of course this incident did not hide behind the scenes and media from around the world was speaking about it. And all of them were accusing or blaming the third party Inbenta, even cyber security experts were emphasizing how this incident clearly showed the vulnerabilities in supply chain. BBC also published a short post titled “Ticketmaster admits personal data stolen in hack attack”, but frankly I have no idea what is hack attack? Is this a specific type of cyber-attack? Maybe I am behind the time, I do not know.

So while the world was busy reading through media, playing with their muscles by stomping down an evil third party and not trying to look in to the cause and reasons.  I was very interested to understand all this buzz around Inbenta technologies, what is it, who are they and maybe they had something to say?

I jumped straight to their website and found a link straight on a home page labeled “Inbenta and the Ticketmaster Data Breach”. Following the link, I came across a message from Jordi Torras, from the CEO of Inbenta. Message begins with “By now, you may be aware of the security breach related to Inbenta technology that may have impacted some Ticketmaster customers. As the CEO of Inbenta, I’m writing you to convey (1) the full scope of the breach, and (2) how we have worked to ensure the issue is resolved:”.  In the end he mentions that “We’re truly sorry that the use of our technology resulted in a violation of Ticketmaster users’ privacy. The privacy of our users is one of our core values, and we will continue to take every measure in our own power to safeguard the personal information of all users who interact with our products. “

You already see the difference in communication, this is not some auto generated message, it is allegedly from the CEO of the company, so the leader comes out to speak with the people, and customers of their customers.

Upon further investigation by both parties, it has been confirmed that the source of the data breach was a single piece of JavaScript code, that was customized by Inbenta to meet Ticketmaster’s particular requirements. This code is not part of any of Inbenta’s products or present in any of our other implementations.

Ticketmaster directly applied the script to its payments page, without notifying our team. Had we known that the customized script was being used this way, we would have advised against it, as it incurs greater risk for vulnerability. The attacker(s) located, modified, and used this script to extract the payment information of Ticketmaster customers processed between February and June 2018.

To sum it up according to Inbenta the attackers used the JavaScript code, that was inserted in to the payments page by Ticketmaster and without notifying the supplier.

I guess we see the difference between this communication and the one from Ticketmaster:

First of all, this is a message from the CEO, a company leader acknowledges, is sorry, gives us more details about the attack vector and discusses what went wrong. He does not use every opportunity to attack Ticketmaster, rather he provides insights and details. I think I got more information about what actually happened from this message rather than from the email or the FAQ page of Ticketmaster combined.

Interestingly enough Inbenta has also created a webpage dedicated to this incident and guess what? They provide much clearer view of what they learned from this incident and what will they do to prevent it in the future. While Ticketmaster did not provide any information of the FAQ about how will they prevent such incidents, what did they learn, Ticketmaster is not openly speaking with their customers and leave them with a sense that they might be hiding something.

And I guess you will not be surprised if I told you that customers are sometimes right. So, there is this company Monzo, which is a digital bank and they have published an interesting blog post right on the day this data breach was announced.

Monzo claims that due to transparency reasons they want to inform people that they have spotted signs of this breach back in early April and made Ticketmaster aware about this topic multiple times.

So, from their perspective the timeline looks something like this:

April 6ths – Monzo notices a pattern of fraudulent transactions on cards used previously at Ticketmaster. And they replace affected cards

Between April 7 and 12 – Monzo see attempted transactions on more customer cards, all of which had been used at Ticketmaster

April 12th – Ticketmaster visits Monzo office to gather information

April 19th – Monzo identify 9 more affected cards. The disclose potential breach to Mastercard an replace around 6000 cards that have been used at Ticketmaster. Ticketmaster in return informs Monzo that they have found no evidence of a breach.

June 21 – 2 months after previous events Mastercard issues an account data compromise alert to all banks

June 27 – Is they day when Ticketmaster publicly disclosed the breach

So according to Monzo (who do not have any interest of hiding something in this story), Ticketmaster did some sort of checks for seven days straight and did not find any sort of a sign of a data breach.

I would recommend to go through this blog post as well, for obvious reasons I will not be fully reading it on the podcast, you can visit monzo.com/blog.

This post gives us more information about how Ticketmaster is structured, what limited capabilities they have and how they were most probably ignorant to first notifying evil third party that they have applied a modified javascript code in to the payment page, second to conducting a proper investigation about this fraud and third notifying customers in time.

Personally, I learned a lot from this story and I want to share this with you:

Lesson 1: Trustworthy companies don’t pass the bucket of blame but own up to their mistakes.

Lesson 2: Do proper third-party assessments and make sure that you are aware of the risks, conduct continuous vulnerability assessments, third party penetration tests and code reviews.

Lesson 3: If you want to modify a software product that you are using, first check with the supplier and do a proper analysis

Lesson 4: Change management is the key and it only works if procedures are clear, simple and known to everyone. Change management requires proper change reviews and approvals.

Lesson 5: If you are told on multiple occasions about fraud in your systems and services, at least hire highly skilled and dedicated personnel who can really look into the matter.

Lesson 6: Strong leaders do not hide behind curtains, are transparent and reassuring.

Lesson 7: During the incident talk with your customers, show them that you are trustworthy, explain what happened, explain that you messed up, it’s okay everyone can be hacked, but not anyone can learn from their own mistakes. Tell your costumers what you learned from this disaster and tell them how will you use this knowledge!

That was it for today, Thanks for tuning in and I hope it was interesting for you.  Let me know what you think, hit me up on twitter and LinkedIn, send your voice messages via anchor.fm and subscribe on your favorite podcast platform or at cypherowl.com. Talk to you in the next one!

Don't forget to subscribe below, stay up-to-date with latest podcasts and other developments!