État du réseau Nouveautés & Information

We are aware of an issue with CRM users who are receiving a license expiry when this is not the case

We have traced the issue to a DNS issue at our data centres and are working to resolve.

We've been made aware of an issue affecting some Virgin Media customers that are losing contact with our services.

Unfortunately this problem lies with Virgin media or a network node between them and us. 

We have informed our data centre engineers who are working on a workaround.

Update: 14/04/2021 10:58

We're receiving reports that services are starting to come back up. We're downgrading severity to medium.

Update: 14/04/2021 11:04

No further reports and all known affected customers now returned to service.

Closing issue.

A planned action on our network over the weekend resulted in dangerous unintended and unforeseeable load on our CRM network. The cluster rejected connections in order to preserve data continuity.

Customers affected 

• Tonic CRM customers with API access
• Tonic CRM customers with Automated reporting

Symptoms

• UI and API calls rejected

We will be reaching out to affected customers today to advise on the issue and to request re-submission of failed API calls. If you believe you could be one of the affected customers, please check your API log over the weekend and re-submit any failed requests.

Remedial Action Taken

• We have stabilised the network and all systems are now responding as requested, there was no working downtime. 
• API requests on Saturday 26th and Sunday the 27th September may have failed, please check your logs and resubmit if necessary.

We have experience an issue in the data centre. Some services are functioning at a reduced capacity while others are not functioning at all.

We investigate.

An outage has been reported on our telephony network, we are investigating

Update 12/03/2020 14:45

We have discovered the cause of the issue and are working on a resolution. 

ETA 5 minutes.

Update 2/03/2020 14:47

The instability has been corrected and the issue is now resolved.

We are currently dealing with an issue with our licensing server that is blocking access to tonic CRM user interface.

Scheduled actions, API and Introducer access are unaffected. 

We aim to have resolved this issue by 09.30AM on 12.07.2019

Dear Customer,

We're experiencing some issues with e-mail reception this morning. The cause is being investigated.

If you're attempting to contact us via any @tonicpower email address, please contact us via telephone instead on 03333353533 or open a support ticket at https://portal.tonicpower.co.uk

Hello,

Some customers are reporting issues with areas of the postcode checker not being able to connect.

We use multiple postcode lookup sources in order to ensure continuity of service so currently advice is to "keep trying" if you receive any errors.

We're currently looking into this, however, we believe the problem to be with one of our PAF providers leading to intermittent issues. 

We'll update via this ticket with updates.

Hello,

We're investigating an issue where some customers are experiencing some connectivity issues to the CRM system.

This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.

Hello,

We're investigating an issue where some customers are experiencing some connectivity issues to the telephone system.

This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.

From Adobe Sign:

Adobe Sign is currently experiencing an incident which impacts service on the NA1 shard.

 

The Adobe Operations team is working to resolve the incident as quickly as possible.

 

We apologize for any inconvenience this may cause.

 

Adobe Sign Support

We have resolved the incident causing instability on Adobe Sign. Full Service was restored at 06:17 AM Pacific Time, August 18, 2017.

We appreciate your business and apologize for any inconvenience you may have experienced.

We're aware of an issue affecting a diverse range of systems within the Tonic network. 

We've marked the issue as critical, however the symptoms are currently limited to low bandwidth which is causing some slowdown in some services.

The issue appears to be in one of the data centers and has been reported. A repair is underway.

Hi All,

We're experiencing some issues with the Tonic CRM network which is resulting in a general slowdown of tonic systems. 

We have reason to believe that the issue resides further down the network than our systems and engineers are currently investigating.

We'll keep you updated as the issue progresses.

We've suffered an internet outage on our main trunk at our main office which is having an impact on some of our call termination.

We're up and running now on our failover trunk but some customers are still experiencing issues reaching us by telephone. Please direct queries via-email to support@tonicpower.co.uk

We are aware of an issue relating to outbound e-mail delivery for customers using the admin.tonicpower.co.uk server. This has been disabled as a precaution and is due to a compromised customer website.

We are working with the customer in order to resolve the issue and resume normal operation.

We are aware of an issue within the Tonic CRM infrastructure that is resulting in some general performance issues.

Currently, only Tonic CRM installations are affected.

Engineers are investigating and we'll inform you of any updates via this thread.

-- Update 10-05-17 --

The issue was relating to a server that was communicating incorrectly. The server has now been repaired and normal service has been resumed.

Dear Customer,

Thursday afternoon, Cloudflare published a blog post describing a memory leak caused by a serious bug that impacted their systems. If you haven't yet, We encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

As Tonic Solutions has, in the past, recommended Cloudflare to many of our customers. We feel it is our responsibility to bring this information to your attention. Please note, however, this issue was not caused by Tonic Solutions and is not related to any of our services. We publish this issue purely as a courtesy.


Kind Regards,
Tonic Support.

The server admin.tonicpower.co.uk is suffering a sustained DDoS attack that is resulting in issues with e-mail delivery and slow page load times.

The issue appears to be caused by insecure Wordpress installations triggering our spam protection routine. Please can you ensure that if you install third party software to your webspace that it is completely up-to-date and that you have carried out the relevant security tests to ensure the integrity of your website.

ETA for a fix is 1:40pm 07/01/2016

-- Update - The issue is resolved but as a precaution we are performing more tests before re-enabling mail transport. Thank you for your patience.

-- Update - We mitigated the attack and have secured against future attacks.

We are aware of an issue affecting customers on the admin.tonicpower.co.uk shared server.

Due to a clients compromised system sending large amounts of unsolicited mail we have temporarily found our IP range on a blacklist. This is affecting email delivery to some Yahoo and other smaller mail providers.

Our engineers are currently liasing with the blacklist provider and hope to reach a resolution shortly. We will update via this notice as the issue progresses.

Scheduled maintenance tonight to restore normal operation from failover to main trunk.

Service may be disrupted.

We are currently investigating an issue with our shared hosting platform served from admin.tonicpower.co.uk.

The issues appears to be a failure with a drive array which will need to move to failover and have a backup restored. During this time some customers services may be affected.

We're experiencing issues with Echosign termination.

Agreements are being created but not sent to the recipient. The issue lies with Echosign and we'll update when we have more information.

Agreements can still be created safely but will not be received until the issue is resolved.

We were notified of a system issue that affected some clients using Tonic CRM. The system was inacessible for a period of about 2 hours.

Engineers responded to fault in a Command & Control (C&C) server in our datacentre and resolved an issue.