Hi ​@j.spiess 

Same here.

Mews Status Page shows a spike in response times both for App as well as API, and has just switched to downgraded performance.

Best, Marc

 

Hey, we are also facing problems during the check out right now.

They said now on https://status.mews.com/ Downgraded Performance

The best time to check out is primetime on Sunday at 11 am. Nothing works anymore. :(

Same here - seems like the second to third time in almost 2 weeks? 
do you guys have similar experiences?

It's particularly annoying today, as check-out time for all weekend departures is at 11am and it's already been 1.5 hours there's nothing we can do. Very frustrating.

Same thing. Nothing is working: guest couldnt pay their extras, cant check out guests, housekeeping cant mark rooms as cleaned. And it started exactly at the check out time.

Same here!

Yes IT IS ANOYING!

We can’t even make a single booking or book any extras..

Same here, bad performance

but for the first time. ​@jhaug 

Very frustrating, still not work

Hi
Yes, we have the same problem here in switzerland
 

Yes, terrible. Just switched to Mews 4 weeks ago and already three downgrades… doesn't make me very positive...

Seems like the same problem around the world. Already over 4 hours… very questionable.

Very difficult to run a hotel when your PMS is this unresponsive. However, I would like to recommend that everyone set up hourly reports for check-ins, check-outs, and stayovers. This is the ONLY reason we are able to work at all today. 

I hope this helps someone in the future!

https://help.mews.com/s/article/schedule-report-exports?language=en_US

According to the latest happenings, the question is whether the system won't run straight into the next downgrade as soon as every Mews customer now follows your recommendation and activates hourly exports...

A little cynicism should be allowed of this situation

Well, I’d go so far to argue that MEWS should by default setup a basic backup report schedule during onboarding, or at least provide some wizard to set it up during onboarding or anytime later...

There is even a separate help page for just that purpose: https://help.mews.com/s/article/Emergency-reports?language=en_US

That beeing said, we did setup backup reports according to that help article, not hourly as that would be overkill for us, but anyway for us the report intervall is setup to cover only up to 1 day in advance (as recommended), as we anticipated only short disruptions, like a couple of hours or so. At the moment we are still coping, but we are not prepared to suffer from a sustained outage…

To be fair, in the past year from the top of my head I can only remember 2 outages that really negatively affected our daily operations for a significant time. So far, that is manageable…

Yours,

JP.

It´s working fellows ;-)

Yeahh!

I notices that most channel manager reservations from today have not been pushed into MEWS (yet?), even though CM had them marked ad deliverd - so keep monitoring that situation and take availability with a grain of salt at the moment...

Still Not working for me in Bavaria....

MEWS states that the background processing is over, but the CM reservations that had not been delivered were still not integrated. Only manually resending them to MEWS from the ChannelManager Extranet allowed them to be integrated after all… So be sure to check your Channelmanger for new reservations that are not yet integrated.  A manual push of availability to CM can’t hurt, too .. :-)

We apologize for the disruption caused by an unexpected outage on the Mews platform yesterday. Outages are inconvenient and we are really sorry about the impact this may have had on your operations over the weekend.    

To answer your questions that you raised in this thread please see below. Additional communication will be shared with all Mews users today:
 

What caused the outage? Is this an isolated incident?  

The issue was an isolated incident with our database replication (which provides high resilience and availability for our customers).  

This is an isolated incident and we’re taking action to mitigate anything like this happening in the future.  

What was the timeline / response time?  

Incident summary on 17th November 2024:

• 11:09 CET: Initial signs of performance degradation were observed which we began investigating, highlighted publicly on: Mews Status  

• 12:00–15:25 CET: A technical issue with database replication was identified. Our team worked to stabilize the platform. While improvements were noted during this period, background processing, including channel manager updates, was delayed. 

• 15:25 CET: Normal platform performance was restored. 

Was this the result of a cyberattack or a technical issue?  

This was a technical issue to do with the Mews database. This in no way was malicious or a cyberattack from external perpetrators.  
 

During the outage, I couldn’t contact Mews via live chat. How should we contact you for updates? 

To check the status of the Mews platform, you should view the status page where we will provide updates during an ongoing outage: Mews Status   

Will there be any delays or disruptions in existing operations / automations that I need to check?  

 No, all automation is retried within Mews. There is a possible impact of data from outside that we might not have received but commonly 3rd parties retry these as well.

How can I verify that all my data is intact and accurate?  

 That ultimately depends on your setup, but it’s worth checking platforms that integrate with Mews which are outside of our sphere of influence.

What are you doing to prevent this from happening again?  

Our engineering team is conducting a full post-mortem to prevent similar issues in the future. We are also enhancing our monitoring and failover processes to ensure greater resilience and faster response times. 

Thank you for your understanding and support.

Hello ​@Sarah.Masterton-Brown !

Thank you for your detailed follow up!

Let me add some comment regarding the reservation delivery during the down time.

At least for D-EDGE we obeserved that reservations that arrived during the down time had been considered delivered by D-EDGE. The interfaces marks them as delivered once MEWS interface responds to reservation delivery with a success message and a reservation id. So it seems like there was some chance to contact the API during the problematic period, and MEWS systems indeed provided a success message, even though the reservations where not committed to the database yet, obviously (since we did NOT have them after the incident, not even later in the evening). I had to manually push them to MEWS again.

Since we very recently had a very similar observation - see support case 00364977 - I suspect that there might be a very basic design issue in the underlying system. Why else can a reservations delivery success message be fed back to the CM before the reservation is actually committed to the mews database. Or was the database somehow rolled back during the incident to a previous state?

I am also wondering what happened to modifications/cancellations that had been delivered during the incident. We didn’t obsere any during that time though ...

So, for colleagues using D-EDGE: check your reservations delivery from Sunday? Do you have similar observations?

Maybe the MEWS connectivity team can look into what happened together with D-EDGE Connectivity Team ?

Kind regards,

Jean-Philipp.

Hello!

Since I have received a follow up on my support inquiry regarding some lost channel manager reservations I would like to add to this thread the information that was provided to me by mews support:

Regarding the reservations failed on 17/11 due to the incident, seems like during these kinds of incidents, some reservations may have to be manually pushed once more once the incident resolves.

The success message in D-Edge, is because from their side they have correctly sent the reservations to Mews, however, if it is Mews, who is at the time having technical difficulties, it may not be able to process the reservations that arrived from the channel manager in your case D-Edge. Thus, once it's confirmed from our side that the incident has been resolved, it is recommended to check your channel manager reservations and manually push any that did not make it.

We have also previously (not related to an outage) lost a reservation occasionally, and the information provided on those incidents basically sums up to “However, when this SQL errors are present it seems, the system does not retry to integrate the reservation again.” (reservations delivery failed silently, although the channel manager had it confirmed as delivered to the PMS)

So, I am wondering if the software engineers responsible for the channel manager integrations would be able to design the processes to be more robust/fail safe. Especially, issuing confirmations to sending channel managers only once reservations really have been successfully commited to the live database would be an important improvement.

Finally, after big incidents the communications should include a recommendation to check incomming channel manager reservations after outages are resolved.

Kind regards,

Jean-Philipp.

Hello Jean-Philip,

Thank you for highlighting these issues. The synchronizations from D-Edge were indeed impacted by the outage. We have been in contact with D-Edge support and identified some reservations that were missing in Mews during that timeframe. As a result, some customers did a manual action, but also D-Edge repushed missing reservations.

This outage was an exceptional event, and while we hope not to experience anything similar again, we want to be better prepared. There are ongoing discussions about implementing more robust solutions to improve resilience, even during significant outages of our core systems.

Once again, we apologize for the inconvenience caused, but I wanted to ensure you that these topics are firmly on our radar and we will also take a look to D-Edge connection specifically (as you’re mentioning that they are failing silently).

Thank you for understanding,

Vladimir

Engineering Manager, Distribution team