Rendered at 12:04:55 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
colonCapitalDee 16 hours ago [-]
Prompt and competent response, as expected from bunny.
"Last week, one of our infrastructure providers reported unauthorized access involving third-party server management software. One of our CDN nodes, located in Armenia, was hosted on a server supplied by this provider.
After receiving the alert from the affected provider, our engineering team took immediate action. Within 20 minutes, we stopped routing traffic through the server. We then reviewed and preserved the relevant security logs for investigation, securely wiped the server, and revoked its access to our other systems. The affected server has been permanently removed from our infrastructure."
ptsneves 16 hours ago [-]
It looked like competent,but some of the events look really bad for the company. Especially the fact that TLS certificates that customers entrusted bunny now need to be revoked by the customers themselves. This detail was at the very last of the email I got and it was the only part I did not like, because it means customer secrets were indeed compromised. I hope they followed up with all the customers on revoking those certificates.
solyin 16 hours ago [-]
> No action is required from customers using managed certificates. We will post a further update on our status page once this process is complete.
> If you manage your own TLS certificates, we recommend revoking and reissuing them with new private keys as a precaution.
Only if you manage them yourself. If Bunny is managing them for you, they'll roll them for you.
boredpudding 15 hours ago [-]
> because it means customer secrets were indeed compromised.
It does not mean secrets were compromised. It means: If there is a risk customer secrets were compromised, the right thing to do is to assume they were and refresh everything. Even if there's no evidence of it.
quickthrowman 12 hours ago [-]
> Especially the fact that TLS certificates that customers entrusted bunny now need to be revoked by the customers themselves. This detail was at the very last of the email I got and it was the only part I did not like, because it means customer secrets were indeed compromised.
It does not mean that at all, it’s sane risk management. There was an opportunity for someone to steal customer secrets so the safe thing to do is assume everything was compromised and revoke/reissue certificates even if nothing was stolen.
benatkin 6 hours ago [-]
Someone who never should have been their vendor, due to closeness to Moscow. It's too bad, but I will probably look elsewhere.
it looks like they are doing automatic renewal of certificates, the only ones that customers have to do themselves is the ones the customers uploaded themselves
abofh 15 hours ago [-]
Which reads like the CA key got compromised - the customer's keys are fine
jeremyjh 14 hours ago [-]
Bunny is not a CA and doesn’t have those keys.
freecodeio 15 hours ago [-]
no it reads as if all individual private keys got compromised, hence requesting custom ssls to be updated
philipallstar 14 hours ago [-]
> Prompt and competent response, as expected from bunny.
Call me pathologically unbiased, but I doubt this same article written by Cloudflare would get this response.
motionlessveloc 13 hours ago [-]
Cloudflare's incident response usually gets high praise from HN. Look at the remarkable lack of hate here:
At least the statement is not competently written.
They write that TLS certificates have potentially be exposed. That doesn't make sense. TLS certifcates are public. Probably they mean private server keys have been potentially be exposed. And after that the certificates are not only useless but dangerous (if the private is really in the hands of the attacker).
They write they have been revoked, OK. But they continue to be served until new ones are in place. Hmm? Well, no browser checks revocation anyway I guess.
eraclide 16 hours ago [-]
Bunny used Gcore in Armenia. Gcore themselves got hacked, not Bunny. Bunny was just an affected downstream, in only one of their POPs.
It's not really misleading. A lot of people here use Bunny and this is a relevant security incident. It's not a major failing but rather a "hey shit happened come make sure if it applies to you that you respond in kind".
NGL it's the exact wording I'd personally like to see from a product that had an infra provider have an incident just so that I can mentally tune in enough to figure out if I need to care.
wbl 15 hours ago [-]
Do I need to post the Rickover responsibility copypasta again? Simply put service providers own their outsourcing decisions and those consequences. Getting popped by shitty hosting services is a risk of using those services.
Bunny is doing the right thing here and owning it.
peri-cl 12 hours ago [-]
> "Bunny used Gcore in Armenia."
I'm unsure, but I have to ask: isn't this company Russian-controlled (with Luxembourg and Cyprus shell-company fronts)? Their Wikipedia entry[0] features a controversy involving RT (Russia Today); and a number of their top employees (i.e. Vsevolod Vayner) seem to live in Russian cities; and Moscow was their 2nd cloud location, after Luxembourg[1].
I'm not sure if I've got this right (most of the primary sources are Russian websites that block me and also block Western internet archives; I'm gleaning some things from Google preview caches). But: is it possible the #1 "European sovereign CDN" has been acting as a white-label reseller for Moscow? Is it possible they willingly trusted all of their customers' TLS certificates to a contractor operating from Russia?
Based on what I know, Gcore split it's Russian business into "Edgecenter" after 2022. What that means in reality? Probably not much.
peri-cl 11 hours ago [-]
I found an additional source of interest (translated from German):
> "An employee appears to be offering Medvedev access to intelligence information" [...] "This recording shows Michael Shurygin, then-head of GCore Russia, apparently discussing plans with former Russian President Dmitry Medvedev in April 2021."
> [...] "The group laughs. Shurygin had made a pun that only works in Russian: “Introduced”—that is, внедрять (vnedryat’)—is ambiguous. It can also be understood to mean “to penetrate” or “to infiltrate.” So they would have infiltrated Luxembourg. Medvedev says, “I hope on behalf of the state.” And they laugh again."
Being affected downstream means it's a security incident regardless? I don't understand why you think this is misleading, incomplete context in the title? Sure.
mmooss 13 hours ago [-]
Last week, after spending 30 minutes placing one order in Uber Eats, I finally hit the last confirm and it told me the restaurant was no longer taking orders that night. I called the restaurant and they said they couldn't help: they only take orders over Uber Eats and they can't control UE's system.
Yes, they absolutely could help. They could take an order over the phone. They could choose a subcontractor who delivered good service to the restaurant's customers. The restaurant is 100% responsible.
If I used a subcontractor for a client, not in a million years would I tell the client, 'the subcontractor's failure isn't my problem; I can't control them'. I would look like a joke, entirely unreliable and clueless.
bobtheborg 16 hours ago [-]
While neither investigation has found any indication of unauthorized access to our server, we cannot completely rule out the possibility.
If unauthorized access to our CDN node did occur, the potentially exposed information would have been: temporarily cached CDN content, Pull Zone configurations, and TLS certificates stored on that server.
As an additional precaution, we have revoked and reissued all TLS certificates for bunny. net platform domains (b-cdn. net and bunny. run). Reissuance of managed TLS certificates for custom domains is ongoing and is expected to be completed within the next 24 hours. No action is required from customers using managed certificates. We will post a further update on our status page once this process is complete.
If you manage your own TLS certificates, we recommend revoking and reissuing them with new private keys as a precaution.
No other customer data, including bunny. net account data, was stored on the server in question, and no internal systems were affected.
so it seems like the underlying provider potentially allowed, for instance, disk read, that above is what bunny says might have been on disk in this case? Am I reading that right?
evan_a_a 14 hours ago [-]
> After receiving the alert from the affected provider, our engineering team took immediate action. Within 20 minutes, we stopped routing traffic through the server. We then reviewed and preserved the relevant security logs for investigation, securely wiped the server, and revoked its access to our other systems. The affected server has been permanently removed from our infrastructure.
When responding to a potential breach, the correct protocol is to contain the incident and PRESERVE EVERYTHING for investigation. Do not try to be selective at what data you keep or don't keep. They've unfortunately likely destroyed evidence and will not be able to determine if the attacker actually breached the server or not, and if so, how. This speaks to them having a poor or non-existent incident response plan.
bitlad 15 hours ago [-]
It feels like theres an incident every hour.
esseph 15 hours ago [-]
Bigger the vault, better the treasure
MikeNotThePope 14 hours ago [-]
Bigger the vault, the more room there is to lock your enemies inside.
bcvvvy 14 hours ago [-]
Cloud bad! Vault big!
asimovDev 15 hours ago [-]
php.net is on Bunny to my knowledge and so is PECL for example. Great potential for compromised packages if this hadn't been caught , but I don't know the blast radius since the victim would need to connect to this particular CDN
ameliaquining 15 hours ago [-]
And so it comes full circle.
lackoftactics 14 hours ago [-]
is it only me or those security incidents on front page are kinda out of control?
maybe it's my brain just primed or attacks picked up pace recently
elAhmo 14 hours ago [-]
It is not just you. I wasn't actively reading for a few days, came back from a short trip and there is like 5 open security breaches on the top page right now.
arjie 13 hours ago [-]
No, the number of serious incidents is rising. The new -cyber models are incredibly capable.
freecodeio 14 hours ago [-]
result of years of negligent ceos and "eh nothing will happen" vibes finally emerging as LLMs turn script kiddies into dangerous hackers
whoanoway 14 hours ago [-]
[dead]
gtirloni 14 hours ago [-]
> Last week, one of our infrastructure providers reported unauthorized access involving third-party server management software. One of our CDN nodes, located in Armenia, was hosted on a server supplied by this provider
Are they talking about physical security? Someone grabbed the server, etc?
Colocation providers usually don't care about security beyond that, or has that changed?
traceroute66 16 hours ago [-]
Sadly this is entirely predictable.
Some time ago my colleague and I were doing some due-diligence on Bunny as a potential service provider.
We asked them some basic security questions, nothing low-level, just some basic high-level overview type of stuff to roughly check if they were following well-known best-practices.
Their response was a consistent and constant "cannot tell you due to security reasons".
That rang alarm bells. Any provider such as them who is not sufficiently confident in their security to be willing and able to discuss security at a high-level perspective is not worthy of trust as far as I am concerned.
everfrustrated 15 hours ago [-]
It's fairly well known that Bunny don't own the servers in their cdn pops so they will always be subject to the security of the people they lease etc from. If you wanted good security you gotta pay for it. Bunny are very much at the budget end of the market segment - servers cost money no way around it.
TL;DR you get what you pay for.
r_lee 6 hours ago [-]
they're not really a "proper" CDN
they use random providers for their cheap bandwidth so they don't have much control over their stuff, thus I'd only really use them as a cache for bandwidth heavy stuff
freecodeio 15 hours ago [-]
care to share what you asked? just curious
0xy 16 hours ago [-]
[flagged]
Wilder7977 15 hours ago [-]
One Armenian server provider got compromised, Bunny assumed (rightfully) compromise of their servers in that one POP and issued a timely set of responses.
As a customer, I couldn't ask anything more. No ambiguity, no downplaying, no word salads. If anything this incident increased my confidence in the company.
freecodeio 16 hours ago [-]
yes they should have just used railway and paid invoices with their binance wallets
pelcg 16 hours ago [-]
Its not just Europe, there are breaches at US companies too.
traceroute66 16 hours ago [-]
> American big tech only to get obliterated by slop security in Europe
Oh right because American tech companies are not equally capable of shitty security.
Remind me, oh yeah ... it was just TODAY we saw that Azure exploit providing full cross-tenant compromise.
A big part of why European companies are turning away from American companies is the sort of MAGA bullshit you just demonstrated.
josephtrump 14 hours ago [-]
Meanwhile this Bunny incident: "we found no evidence that anything happened, nothing probably happened, but rotate your keys to be certain"
nickv 15 hours ago [-]
Fortunately there have been zero security incidents involving American companies this past month…
"Last week, one of our infrastructure providers reported unauthorized access involving third-party server management software. One of our CDN nodes, located in Armenia, was hosted on a server supplied by this provider.
After receiving the alert from the affected provider, our engineering team took immediate action. Within 20 minutes, we stopped routing traffic through the server. We then reviewed and preserved the relevant security logs for investigation, securely wiped the server, and revoked its access to our other systems. The affected server has been permanently removed from our infrastructure."
> If you manage your own TLS certificates, we recommend revoking and reissuing them with new private keys as a precaution.
Only if you manage them yourself. If Bunny is managing them for you, they'll roll them for you.
It does not mean secrets were compromised. It means: If there is a risk customer secrets were compromised, the right thing to do is to assume they were and refresh everything. Even if there's no evidence of it.
It does not mean that at all, it’s sane risk management. There was an opportunity for someone to steal customer secrets so the safe thing to do is assume everything was compromised and revoke/reissue certificates even if nothing was stolen.
https://news.ycombinator.com/item?id=50311862
Call me pathologically unbiased, but I doubt this same article written by Cloudflare would get this response.
https://news.ycombinator.com/item?id=45973709
They write that TLS certificates have potentially be exposed. That doesn't make sense. TLS certifcates are public. Probably they mean private server keys have been potentially be exposed. And after that the certificates are not only useless but dangerous (if the private is really in the hands of the attacker).
They write they have been revoked, OK. But they continue to be served until new ones are in place. Hmm? Well, no browser checks revocation anyway I guess.
https://lowendtalk.com/discussion/222315/gcore-compromised-v...
The title is heavily misleading.
NGL it's the exact wording I'd personally like to see from a product that had an infra provider have an incident just so that I can mentally tune in enough to figure out if I need to care.
Bunny is doing the right thing here and owning it.
I'm unsure, but I have to ask: isn't this company Russian-controlled (with Luxembourg and Cyprus shell-company fronts)? Their Wikipedia entry[0] features a controversy involving RT (Russia Today); and a number of their top employees (i.e. Vsevolod Vayner) seem to live in Russian cities; and Moscow was their 2nd cloud location, after Luxembourg[1].
I'm not sure if I've got this right (most of the primary sources are Russian websites that block me and also block Western internet archives; I'm gleaning some things from Google preview caches). But: is it possible the #1 "European sovereign CDN" has been acting as a white-label reseller for Moscow? Is it possible they willingly trusted all of their customers' TLS certificates to a contractor operating from Russia?
Someone kindly fact-check me.
[0] https://en.wikipedia.org/wiki/Gcore#Controversies
[1] https://web.archive.org/web/20260420215140/https://www.strea...
> "An employee appears to be offering Medvedev access to intelligence information" [...] "This recording shows Michael Shurygin, then-head of GCore Russia, apparently discussing plans with former Russian President Dmitry Medvedev in April 2021."
> [...] "The group laughs. Shurygin had made a pun that only works in Russian: “Introduced”—that is, внедрять (vnedryat’)—is ambiguous. It can also be understood to mean “to penetrate” or “to infiltrate.” So they would have infiltrated Luxembourg. Medvedev says, “I hope on behalf of the state.” And they laugh again."
https://correctiv.org/aktuelles/russland-ukraine-2/2023/02/1...
(Who?) https://en.wikipedia.org/wiki/Correctiv
Yes, they absolutely could help. They could take an order over the phone. They could choose a subcontractor who delivered good service to the restaurant's customers. The restaurant is 100% responsible.
If I used a subcontractor for a client, not in a million years would I tell the client, 'the subcontractor's failure isn't my problem; I can't control them'. I would look like a joke, entirely unreliable and clueless.
When responding to a potential breach, the correct protocol is to contain the incident and PRESERVE EVERYTHING for investigation. Do not try to be selective at what data you keep or don't keep. They've unfortunately likely destroyed evidence and will not be able to determine if the attacker actually breached the server or not, and if so, how. This speaks to them having a poor or non-existent incident response plan.
maybe it's my brain just primed or attacks picked up pace recently
Are they talking about physical security? Someone grabbed the server, etc?
Colocation providers usually don't care about security beyond that, or has that changed?
Some time ago my colleague and I were doing some due-diligence on Bunny as a potential service provider.
We asked them some basic security questions, nothing low-level, just some basic high-level overview type of stuff to roughly check if they were following well-known best-practices.
Their response was a consistent and constant "cannot tell you due to security reasons".
That rang alarm bells. Any provider such as them who is not sufficiently confident in their security to be willing and able to discuss security at a high-level perspective is not worthy of trust as far as I am concerned.
TL;DR you get what you pay for.
they use random providers for their cheap bandwidth so they don't have much control over their stuff, thus I'd only really use them as a cache for bandwidth heavy stuff
As a customer, I couldn't ask anything more. No ambiguity, no downplaying, no word salads. If anything this incident increased my confidence in the company.
Oh right because American tech companies are not equally capable of shitty security.
Remind me, oh yeah ... it was just TODAY we saw that Azure exploit providing full cross-tenant compromise.
A big part of why European companies are turning away from American companies is the sort of MAGA bullshit you just demonstrated.
Oh wait.