tag:status.ipfone.com,2005:/historyIPFone Status - Incident History2024-03-29T01:23:05-04:00IPFonetag:status.ipfone.com,2005:Incident/107109562022-07-28T21:51:06-04:002022-07-28T21:51:06-04:00WebEx Partial Outage<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>21:51</var> EDT</small><br><strong>Resolved</strong> - Services remain healthy and stable and this issue is now resolved. We appreciate your patience while we addressed this service incident.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>20:02</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented by Cisco and we are currently monitoring the results.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>16:47</var> EDT</small><br><strong>Update</strong> - The majority of services have been restored and are operational. Some users may still experience intermittent issues accessing their Webex board and devices. For users who are still unable to log into the Webex App, we recommend that you close and re-launch the Webex App.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>14:07</var> EDT</small><br><strong>Update</strong> - WebEx app list of affected services:<br /><br />Room creation and management - Major Impact<br />Send and read messages - Major Impact<br />File sharing and viewing - Major Impact<br />Client calling and meetings - Partial Impact<br />Sign in and sign up - Major Impact<br />Web and desktop apps - Major Impact<br />Mobile apps - Major Impact<br />Search - Major Impact<br />Integrations - Major Impact</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>13:26</var> EDT</small><br><strong>Investigating</strong> - Cisco engineering is working on mitigation for the affected services. Webex for BroadWorks users may also experience issues with signing in and instant messaging. Users who are utilizing Webex Developer API may also experience intermittent issues as well.<br /><br />All hands are on deck to restore services and we apologize for the inconvenience this may cause.</p>tag:status.ipfone.com,2005:Incident/105821792022-07-14T18:06:34-04:002022-07-14T18:06:34-04:00WebEx services - Partial Outage for the US region -<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>18:06</var> EDT</small><br><strong>Resolved</strong> - Services remain healthy and stable and this issue is now resolved.<br /><br />We appreciate your patience while we addressed this service incident.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>18:06</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>17:46</var> EDT</small><br><strong>Monitoring</strong> - Monitoring - <br /><br />Engineering has completed service remediation activities and services are now restored.<br /><br />We will continue to monitor to ensure service stability.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>15:57</var> EDT</small><br><strong>Identified</strong> - - Identified - Cisco engineering continues to address the issue and rollout fixes. <br /><br />Although some users may still experience intermittent failures, service availability is improving.<br /><br />All hands are currently on deck and are working to restore services. We apologize for the inconvenience this may cause.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>14:45</var> EDT</small><br><strong>Update</strong> - Cisco engineering continues to work on mitigation for the affected services. Webex for BroadWorks users may also experience issues with signing in and the provisioning of new accounts. Users who are utilizing Webex Developer API may also experience intermittent issues as well.<br /><br />All hands are on deck to restore services and we apologize for the inconvenience this may cause.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>14:18</var> EDT</small><br><strong>Investigating</strong> - Dear customer,<br /><br />Cisco engineering dept continues to work on restoring the affected services. In addition, some users may experience intermittent failures logging into Webex Control Hub.<br />We will provide updates as they become available.<br /><br />IPfone Support.</p>tag:status.ipfone.com,2005:Incident/96193832022-03-24T12:45:03-04:002022-03-24T12:45:03-04:00MiaRec - Call Recording service<p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>12:45</var> EDT</small><br><strong>Resolved</strong> - Our NOC has informed that call recording service issue has been resolved.</p><p><small>Mar <var data-var='date'>24</var>, <var data-var='time'>09:22</var> EDT</small><br><strong>Investigating</strong> - Our NOC and the software vendor team are investigating a few failures detected at the call recording service.<br />Updates will be provided through this channel as they are received.</p>tag:status.ipfone.com,2005:Incident/82566722021-10-21T09:25:59-04:002021-10-21T09:25:59-04:00Partial inbound and outbound calls, router issue.<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>09:25</var> EDT</small><br><strong>Resolved</strong> - Continuous monitoring did not reveal any new SIP disconnections. New version of the firmware is running stable.<br />Please reach out to our support team if you notice any issues.<br />We apologize of the inconvenience.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>12:53</var> EDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>12:47</var> EDT</small><br><strong>Monitoring</strong> - Our NOC has been applying a fix suggested by our vendor (Edgewater Networks/Ribbon) during after hours. We are currently monitoring all the customers affected, if you experience difficulties making and receiving calls let our support team know, and please reboot your local Edgemarc device.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>11:44</var> EDT</small><br><strong>Identified</strong> - We are aware that some customers are experiencing issues making and receiving calls, this gets resolved by restarting the Edgewater router we have provided you with.<br />OUR NOC has tracked down the root cause of this to be related to a bug on the existing firmware for our routers.<br />Our NOC is currently working on a permanent solution for this working alongside our vendor (Edgewater Networks/Ribbon), but if you experience difficulties making and receiving calls let our support team know and please reboot your local Edgemarc device.<br />In order to reboot the IPFone router, you can unplug it from power, wait a few seconds, and plug it back in. We will keep you posted on our progress and will let you know before we apply a permanent fix.</p>tag:status.ipfone.com,2005:Incident/81117962021-10-04T16:48:03-04:002021-10-04T16:48:03-04:00Partial inbound and outbound calls issues.<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>16:48</var> EDT</small><br><strong>Resolved</strong> - Bandwidth have indicated normal traffic patterns through the weekend and today. Our team will continued monitoring traffic to make sure all inbound and outbound call are working as expected.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>12:56</var> EDT</small><br><strong>Update</strong> - Bandwidth.com has reported that their services have been stable. We are asking customers to report any additional issues to our support team for further investigation. Monitoring of services will continue, and additional information will be posted as it becomes available.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>11:02</var> EDT</small><br><strong>Update</strong> - Bandwidth teams are observing expected network behavior with successful call completion. We will continue to monitor and provide updates accordingly.<br />We will continue to provide updates as they become available.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>09:27</var> EDT</small><br><strong>Update</strong> - Bandwidth teams are observing expected network behavior with successful call completion. We will continue to monitor and provide updates accordingly.<br />We will continue to provide updates as they become available.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>18:25</var> EDT</small><br><strong>Monitoring</strong> - Bandwidth teams are observing expected network behavior with successful call completion. We will continue to monitor and provide updates accordingly.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>16:06</var> EDT</small><br><strong>Update</strong> - We continue monitoring intermittent inbound calling issues. Customers may experience intermittent call failures or voice quality issues and/or one-way audio scenarios. All teams are actively engaged</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>14:45</var> EDT</small><br><strong>Investigating</strong> - We continue monitoring intermittent inbound calling issues. Customers may experience intermittent call failures or voice quality issues and/or one-way audio scenarios. All teams are actively engaged.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>12:02</var> EDT</small><br><strong>Monitoring</strong> - Bandwidth is currently investigating an incident impacting inbound calls from Verizon Wireless to the Bandwidth network. Inbound calls may experience intermittent failures.<br />Bandwidth’s engineers are responding in real-time and their management has committed to keeping us informed of the latest developments.<br />We will continue to provide updates as they become available.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>09:35</var> EDT</small><br><strong>Investigating</strong> - We continue to work with Bandwidth as we noticed intermittent inbound calling issues. Customers may experience drop calls or one-way audio issues. Bandwidth.com has acknowledged to IPFone they have been the victim of a Distributed Denial of Service (DDoS) attack and are currently working to mitigate the impact. Bandwidth’s engineers are responding in real-time and their management has committed to keeping us informed of the latest developments.</p>tag:status.ipfone.com,2005:Incident/81020242021-09-29T07:33:57-04:002021-09-29T07:33:57-04:00Partial inbound and outbound calls issues.<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>07:33</var> EDT</small><br><strong>Resolved</strong> - Our underlying carrier, bandwidth.com, is reporting network stability, we consider this incident resolved. Teams will continue to monitor our network. Should additional updates be needed, they will be posted here.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>17:49</var> EDT</small><br><strong>Update</strong> - Our underlying carrier, bandwidth.com, is reporting that they are continue to have some intermittent issues, but largely improved and resolved. They continue to fortify mitigation measures until services are fully restored and stable.<br />We will continue to provide updates as they become available.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>13:59</var> EDT</small><br><strong>Update</strong> - Our underlying carrier is reporting that they are continue to have some intermittent issues, but largely improved and resolved. They continue to fortify mitigation measures until services are fully restored and stable.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>13:33</var> EDT</small><br><strong>Identified</strong> - We continue to work with Bandwidth on the intermittent inbound calling issues. Customers may experience drop calls or one-way audio issues. Bandwidth.com has acknowledged to IPFone they have been the victim of a Distributed Denial of Service (DDoS) attack and are currently working to mitigate the impact. This attack has been published by various media outlets as Bandwidth is a significant provider of telecommunications services.<br />Bandwidth’s engineers are responding in real-time and their management has committed to keeping IPFone informed of the latest developments</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>12:33</var> EDT</small><br><strong>Update</strong> - A small portion of IPFone customers may continue experiencing inbound call connection delays and dead air. We are working with the underlying carrier to resolve this issue. Will provide updates as they become available.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>10:18</var> EDT</small><br><strong>Update</strong> - Multiple customers are still currently experiencing issues with incoming phone calls due to a nationwide outage impacting all major VoIP carriers. We continue to work with our PSTN provider affecting their service provider interconnects. We will continue to provide updates as they become available.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>10:03</var> EDT</small><br><strong>Investigating</strong> - As of 9:51 EDT we are observing degraded calling with Bandwidth, and have received customer reports that correlate with this. We will continue to pursue with the vendor for resolution.</p>tag:status.ipfone.com,2005:Incident/80953622021-09-27T19:43:44-04:002021-09-27T19:43:44-04:00Partial inbound and outbound calls issues.<p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>19:43</var> EDT</small><br><strong>Resolved</strong> - Dear customer,<br />At this moment, we observed inbound calling, outbound calling and Toll-Free services operating as designed. Full voice services are back and working. We will continue monitoring the services and the vendor services and we will provide any updates if necessary. In the next 24/48 hours we will provide you with a RFO (reason for outage) to explain what happened and what actions are taken.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>18:18</var> EDT</small><br><strong>Identified</strong> - We continue to work with our PSTN partner, Bandwidth.com, At this moment, we observed Toll-Free services and outbound calling operating as designed. They are working diligently and we will continue to provide updates as they become available.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>16:32</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>16:32</var> EDT</small><br><strong>Update</strong> - Dear Customer, Several customers are still currently experiencing issues with incoming phone calls due to a nationwide outage impacting all major VoIP carriers. We continue to work with our PSTN provider affecting their service provider interconnects. We will continue to provide updates as they become available. Sorry for the inconvenience.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>16:03</var> EDT</small><br><strong>Update</strong> - We continue to work with our PSTN partner, Bandwidth.com, who has acknowledged that the cause of degradation is a Distributed Denial of Service (DDoS) attack that is affecting all of their service provider interconnects. They are working diligently to contain the attack, but are unable to say more at this time. We will continue to provide updates as they become available.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>15:54</var> EDT</small><br><strong>Investigating</strong> - We continue to work with our PSTN partner, Bandwidth.com, who has acknowledged that the cause of degradation is a Distributed Denial of Service (DDoS) attack that is affecting all of their service provider interconnects. They are working diligently to contain the attack, but are unable to say more at this time. We will continue to provide updates as they become available</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>15:07</var> EDT</small><br><strong>Update</strong> - A portion of IPFone customers may be experiencing inbound call connection delays and dead air. We are working with the underlying carrier to resolve this issue ASAP. IPFone NOC is continuing to work with the underlying carrier to resolve this issue and will provide updates as they become available.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:47</var> EDT</small><br><strong>Update</strong> - A portion of IPFone customers may be experiencing intermittent inbound and outbound calling issues, including to emergency services due to an underlying carrier. IPFone NOC is continuing to work with the underlying carrier to resolve this issue and will provide updates as they become available.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:27</var> EDT</small><br><strong>Identified</strong> - We have identified the problem and currently working with one of our Carrier Partners to recover service.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:25</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:18</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>14:17</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>10:30</var> EDT</small><br><strong>Investigating</strong> - We are currently aware of ongoing issues affecting inbound and outbound calls to certain telephone number ranges. Our NOC team is investigating the issue with priority.</p>