Email Issues (Resolved)
  • Priority - Low
  • Affecting Other - Email Receipt
  • Dear Customer,

    We're experiencing some issues with e-mail reception this morning. The cause is being investigated.

    If you're attempting to contact us via any @tonicpower email address, please contact us via telephone instead on 03333353533 or open a support ticket at https://portal.tonicpower.co.uk

  • Date - 09/08/2018 10:34 - 10/08/2018 09:05
  • Last Updated - 02/11/2018 14:29
Intermittent postcode lookup issues (Resolved)
  • Priority - Low
  • Affecting Other - PAF Lookup
  • Hello,

    Some customers are reporting issues with areas of the postcode checker not being able to connect.

    We use multiple postcode lookup sources in order to ensure continuity of service so currently advice is to "keep trying" if you receive any errors.

    We're currently looking into this, however, we believe the problem to be with one of our PAF providers leading to intermittent issues. 

    We'll update via this ticket with updates.


    Update: 16/05/18 10:41

    The issue definitely resides with one of our providers, we're currently working with them to investigate the issue.


    Update: 16/05/18 11:16

    Engineers are unable to locate any issue with infrastructure and all tests are coming back normal. We're closing this issue off as we are confident that the issue is most likely a temporary transport problem.

    We'll work with individual clients if they are still having issues, so please reach out to support@tonicpower.co.uk if you are still having any problems.

  • Date - 16/05/2018 10:34 - 16/05/2018 11:16
  • Last Updated - 16/05/2018 11:18
Intermittent Connectivity Issues (Resolved)
  • Priority - Medium
  • Affecting Server - Tonic CRM Network
  • Hello,

    We're investigating an issue where some customers are experiencing some connectivity issues to the CRM system.

    This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.



    Update:

    Engineers believe the problem has been resolved, please report any further occurrences to us through the proper channels.

  • Date - 04/04/2018 16:21 - 04/04/2018 20:29
  • Last Updated - 05/04/2018 08:50
Intermittent Connectivity Issues (Resolved)
  • Priority - High
  • Affecting Server - Tonic TC3S
  • Hello,

    We're investigating an issue where some customers are experiencing some connectivity issues to the telephone system.

    This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.



    Update:

    Engineers believe the problem has been resolved, please report any further occurrences to us through the proper channels.

  • Date - 04/04/2018 16:32 - 04/04/2018 20:29
  • Last Updated - 05/04/2018 08:49
Tonic Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Tonic CRM Network
  • Today one of our providers suffered a major outage across multiple datacentres in multiple locations. Tonic services were impacted for roughly 2 hours between 7 and 10am. Normal service has now been resumed.
     
    If you're still experiencing any issues, please contact us directly on: 03333353533
     
    Thank you for your patience.

  • Date - 09/11/2017 07:49 - 09/11/2017 09:53
  • Last Updated - 09/11/2017 10:11
Adobe Sign Issues (Resolved)
  • Priority - Critical
  • Affecting Other - Adobe Sign / Echosign
  • From Adobe Sign:

    Adobe Sign is currently experiencing an incident which impacts service on the NA1 shard.
     
    The Adobe Operations team is working to resolve the incident as quickly as possible.
     
    We apologize for any inconvenience this may cause.
     
    Adobe Sign Support

    Update 18/08/17 14:49

    We have resolved the incident causing instability on Adobe Sign. Full Service was restored at 06:17 AM Pacific Time, August 18, 2017.

    We appreciate your business and apologize for any inconvenience you may have experienced.

  • Date - 18/08/2017 14:01 - 18/08/2017 14:49
  • Last Updated - 18/08/2017 14:50
Network issue (Resolved)
  • Priority - Medium
  • Affecting Other - DataCenter
  • We're aware of an issue affecting a diverse range of systems within the Tonic network. 

    We've marked the issue as critical, however the symptoms are currently limited to low bandwidth which is causing some slowdown in some services.

    The issue appears to be in one of the data centers and has been reported. A repair is underway.


    Update 26/07/2017 12:09

    We're currently in contact with our supplier to understand and resolve the issue.


    Update 26/07/2017 12:16

    The majority of services are now performing normally, however, the issue is still present. Downgrading to Medium Priority.

    Update 26/07/2017 14:33

    The monitoring period for this ticket is over and the network appears stable so we're closing this issue. 

    If you're still having any latency issues or noticing a slowdown, please open a support ticket

  • Date - 26/07/2017 11:54 - 26/07/2017 14:33
  • Last Updated - 26/07/2017 14:36
Tonic Slow (Resolved)
  • Priority - High
  • Affecting Server - Tonic CRM Network
  • Hi All,

    We're experiencing some issues with the Tonic CRM network which is resulting in a general slowdown of tonic systems. 

    We have reason to believe that the issue resides further down the network than our systems and engineers are currently investigating.

    We'll keep you updated as the issue progresses.


    Update 19/07/2017 16:39

    We have located the source of the issue and are currently working on a fix.


    Update 19/07/2917 17:37

    We have now resolved the issue and are taking steps to prevent recurrence. Anyone still experiencing issues should open a support ticket.

  • Date - 19/07/2017 15:52 - 19/07/2917 17:37
  • Last Updated - 19/07/2017 17:38
Tonic HQ - Internet (Resolved)
  • Priority - Medium
  • Affecting Other - Tonic Support
  • We've suffered an internet outage on our main trunk at our main office which is having an impact on some of our call termination.

    We're up and running now on our failover trunk but some customers are still experiencing issues reaching us by telephone. Please direct queries via-email to support@tonicpower.co.uk



    Update 22/06/2017 10:32

    We're now back to full capacity and normal service has resumed. We apologise for any inconvenience caused.


  • Date - 22/06/2017 09:38 - 22/06/2017 10:32
  • Last Updated - 22/06/2017 10:34
admin.tonicpower.co.uk E-mail delivery issue (Resolved)
  • Priority - Low
  • Affecting Server - Tonic Hosting Platform
  • We are aware of an issue relating to outbound e-mail delivery for customers using the admin.tonicpower.co.uk server. This has been disabled as a precaution and is due to a compromised customer website.

    We are working with the customer in order to resolve the issue and resume normal operation.



    16/05/2017 11:11

    The issue is in a resolution stage, however service is still impacted due to appearing on one RBL. E-mail delivery may still have issues.


    29/05/2017 16:08

    We have been monitoring the email system without issue for over a week so are now closing this issue down as resolved. Please report any further issues to support@tonicpower.co.uk

  • Date - 15/05/2017 17:45 - 29/05/2017 00:00
  • Last Updated - 29/05/2017 16:10
Infrastructure Issue (Resolved)
  • Priority - High
  • Affecting System - Tonic CRM
  • We are aware of an issue within the Tonic CRM infrastructure that is resulting in some general performance issues.

    Currently, only Tonic CRM installations are affected.

    Engineers are investigating and we'll inform you of any updates via this thread.

    -- Update 10-05-17 --

    The issue was relating to a server that was communicating incorrectly. The server has now been repaired and normal service has been resumed.

  • Date - 09/05/2017 08:34
  • Last Updated - 10/05/2017 10:28
Cloudflare Memory Leakage (Resolved)
  • Priority - Critical
  • Affecting System - Cloudflare
  • Dear Customer,

    Thursday afternoon, Cloudflare published a blog post describing a memory leak caused by a serious bug that impacted their systems. If you haven't yet, We encourage you to read that post on the bug:

    https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

    As Tonic Solutions has, in the past, recommended Cloudflare to many of our customers. We feel it is our responsibility to bring this information to your attention. Please note, however, this issue was not caused by Tonic Solutions and is not related to any of our services. We publish this issue purely as a courtesy.


    Kind Regards,
    Tonic Support.

  • Date - 23/02/2017 17:14
  • Last Updated - 05/04/2017 08:24
admin DDos (Resolved)
  • Priority - High
  • Affecting Server - Tonic Hosting Platform
  • The server admin.tonicpower.co.uk is suffering a sustained DDoS attack that is resulting in issues with e-mail delivery and slow page load times.

    The issue appears to be caused by insecure Wordpress installations triggering our spam protection routine. Please can you ensure that if you install third party software to your webspace that it is completely up-to-date and that you have carried out the relevant security tests to ensure the integrity of your website.

    ETA for a fix is 1:40pm 07/01/2016

    -- Update - The issue is resolved but as a precaution we are performing more tests before re-enabling mail transport. Thank you for your patience.

    -- Update - We mitigated the attack and have secured against future attacks.

  • Date - 06/01/2017 13:51 - 09/01/2017 00:00
  • Last Updated - 11/01/2017 16:44
E-Mail Delivery (Resolved)
  • Priority - Medium
  • Affecting Server - Tonic Hosting Platform
  • We are aware of an issue affecting customers on the admin.tonicpower.co.uk shared server.

    Due to a clients compromised system sending large amounts of unsolicited mail we have temporarily found our IP range on a blacklist. This is affecting email delivery to some Yahoo and other smaller mail providers.

    Our engineers are currently liasing with the blacklist provider and hope to reach a resolution shortly. We will update via this notice as the issue progresses.



    Update 22/03/16: Blacklist checks are currently clear and have been holding steady for the past week. Please note that we will be amending our TOS for hosting customers in the coming weeks, we'll email details.

  • Date - 03/03/2016 09:50 - 22/03/2016 00:00
  • Last Updated - 22/03/2016 10:58
Scheduled Maintenance (Resolved)
  • Priority - High
  • Affecting Server - Tonic Hosting Platform
  • Scheduled maintenance tonight to restore normal operation from failover to main trunk.

    Service may be disrupted.

  • Date - 26/01/2016 23:59 - 27/01/2016 03:00
  • Last Updated - 03/03/2016 10:02
Failure of Shared Hosting (admin.tonicpower.c (Resolved)
  • Priority - Critical
  • Affecting Server - Tonic Hosting Platform
  • We are currently investigating an issue with our shared hosting platform served from admin.tonicpower.co.uk.

    The issues appears to be a failure with a drive array which will need to move to failover and have a backup restored. During this time some customers services may be affected.


    Update 26/01/16 14:39

    Service should be resumed on failover, scheduled for maintenance tonight to restore normal operation.

  • Date - 26/01/2016 14:06 - 26/01/2016 14:39
  • Last Updated - 26/01/2016 14:40
E-Sign Issues (Resolved)
  • Priority - Medium
  • Affecting Other - Tonic CRM (With Echosign)
  • We're experiencing issues with Echosign termination.

    Agreements are being created but not sent to the recipient. The issue lies with Echosign and we'll update when we have more information.

    Agreements can still be created safely but will not be received until the issue is resolved.


    Update 12:34 07/01/2016

    Engineers are working on the issue and expect to have a resolution shortly.

    Update 16:01 07/01/2016

    The issue leading to E-Sign transmission has been stable for the past 30 minutes.

    We are being advised to recommend clients re-send agreements created before 15:30 today to ensure delivery.

  • Date - 07/01/2016 11:34
  • Last Updated - 07/01/2016 16:03
Issue Affecting Tonic CRM Standard (Resolved)
  • Priority - High
  • Affecting System - Tonic CRM
  • We were notified of a system issue that affected some clients using Tonic CRM. The system was inacessible for a period of about 2 hours.

    Engineers responded to fault in a Command & Control (C&C) server in our datacentre and resolved an issue.

  • Date - 14/08/2015 16:24 - 14/08/2015 18:35
  • Last Updated - 17/08/2015 11:24

Server Status

Below is a real-time overview of our servers where you can check if there's any known issues.

Server Name HTTP FTP POP3 PHP Info Server Load Uptime
Tonic CRM Network PHP Info
Tonic Hosting Platform PHP Info
Tonic SMS Platform PHP Info
Tonic TC3S PHP Info