Issue affecting CRM Users (已解決)
  • 優先級 - 重大
  • 影響範圍 伺服器 - Tonic CRM Network
  • We are aware of an issue with CRM users who are receiving a license expiry when this is not the case

    We have traced the issue to a DNS issue at our data centres and are working to resolve.

  • 日期 - 17/06/2021 09:02 - 17/06/2022 10:00
  • 最後更新 - 01/02/2023 10:55
Networking issues (已解決)
  • 優先級 - 中
  • 影響範圍 其他 - Virgin Media
  • We've been made aware of an issue affecting some Virgin Media customers that are losing contact with our services.

    Unfortunately this problem lies with Virgin media or a network node between them and us. 

    We have informed our data centre engineers who are working on a workaround.


    Update: 14/04/2021 10:58

    We're receiving reports that services are starting to come back up. We're downgrading severity to medium.


    Update: 14/04/2021 11:04

    No further reports and all known affected customers now returned to service.

    Closing issue.

  • 日期 - 14/04/2021 10:50 - 14/04/2021 11:04
  • 最後更新 - 14/04/2021 11:04
API Loop issue (已解決)
  • 優先級 - 重大
  • 影響範圍 伺服器 - Tonic CRM Network
  • A planned action on our network over the weekend resulted in dangerous unintended and unforeseeable load on our CRM network. The cluster rejected connections in order to preserve data continuity.

    Customers affected 

    • Tonic CRM customers with API access
    • Tonic CRM customers with Automated reporting

    Symptoms

    • UI and API calls rejected

    We will be reaching out to affected customers today to advise on the issue and to request re-submission of failed API calls. If you believe you could be one of the affected customers, please check your API log over the weekend and re-submit any failed requests.

    Remedial Action Taken

    • We have stabilised the network and all systems are now responding as requested, there was no working downtime. 
    • API requests on Saturday 26th and Sunday the 27th September may have failed, please check your logs and resubmit if necessary.

  • 日期 - 26/09/2020 08:14 - 28/09/2020 08:00
  • 最後更新 - 28/09/2020 08:27
Widespread Server Outage (已解決)
  • 優先級 - 中
  • 影響範圍 伺服器 - Tonic CRM Network
  • We have experience an issue in the data centre. Some services are functioning at a reduced capacity while others are not functioning at all.

    We investigate.



    Update: 30/03/2020 16:34 - We expect to be operating back at full capacity within 30 minutes.



    Update: 30/03/2020 16:46 - The majority of services are now restored. We continue work on the rest.



    Update: 30/03/2020 17:15 - The issue is resolved, all systems go.

  • 日期 - 30/03/2020 15:15 - 30/03/2020 17:15
  • 最後更新 - 30/03/2020 16:18
Telephony Outage (已解決)
  • 優先級 - 重大
  • 影響範圍 伺服器 - Tonic TC3S
  • An outage has been reported on our telephony network, we are investigating


    Update 12/03/2020 14:45

    We have discovered the cause of the issue and are working on a resolution. 

    ETA 5 minutes.


    Update 2/03/2020 14:47

    The instability has been corrected and the issue is now resolved.

  • 日期 - 12/03/2020 14:41 - 12/03/2020 14:47
  • 最後更新 - 12/03/2020 14:47
Tonic Licensing Issue (已解決)
  • 優先級 - 重大
  • 影響範圍 伺服器 - Tonic CRM Network
  • We are currently dealing with an issue with our licensing server that is blocking access to tonic CRM user interface.

    Scheduled actions, API and Introducer access are unaffected. 

    We aim to have resolved this issue by 09.30AM on 12.07.2019

  • 日期 - 11/07/2019 22:42 - 12/07/2019 10:42
  • 最後更新 - 26/02/2020 10:14
Email Issues (已解決)
  • 優先級 - 低
  • 影響範圍 其他 - Email Receipt
  • Dear Customer,

    We're experiencing some issues with e-mail reception this morning. The cause is being investigated.

    If you're attempting to contact us via any @tonicpower email address, please contact us via telephone instead on 03333353533 or open a support ticket at https://portal.tonicpower.co.uk

  • 日期 - 09/08/2018 10:34 - 10/08/2018 09:05
  • 最後更新 - 02/11/2018 14:29
Intermittent postcode lookup issues (已解決)
  • 優先級 - 低
  • 影響範圍 其他 - PAF Lookup
  • Hello,

    Some customers are reporting issues with areas of the postcode checker not being able to connect.

    We use multiple postcode lookup sources in order to ensure continuity of service so currently advice is to "keep trying" if you receive any errors.

    We're currently looking into this, however, we believe the problem to be with one of our PAF providers leading to intermittent issues. 

    We'll update via this ticket with updates.


    Update: 16/05/18 10:41

    The issue definitely resides with one of our providers, we're currently working with them to investigate the issue.


    Update: 16/05/18 11:16

    Engineers are unable to locate any issue with infrastructure and all tests are coming back normal. We're closing this issue off as we are confident that the issue is most likely a temporary transport problem.

    We'll work with individual clients if they are still having issues, so please reach out to support@tonicpower.co.uk if you are still having any problems.

  • 日期 - 16/05/2018 10:34 - 16/05/2018 11:16
  • 最後更新 - 16/05/2018 11:18
Intermittent Connectivity Issues (已解決)
  • 優先級 - 中
  • 影響範圍 伺服器 - Tonic CRM Network
  • Hello,

    We're investigating an issue where some customers are experiencing some connectivity issues to the CRM system.

    This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.



    Update:

    Engineers believe the problem has been resolved, please report any further occurrences to us through the proper channels.

  • 日期 - 04/04/2018 16:21 - 04/04/2018 20:29
  • 最後更新 - 05/04/2018 08:50
Intermittent Connectivity Issues (已解決)
  • 優先級 - 高
  • 影響範圍 伺服器 - Tonic TC3S
  • Hello,

    We're investigating an issue where some customers are experiencing some connectivity issues to the telephone system.

    This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.



    Update:

    Engineers believe the problem has been resolved, please report any further occurrences to us through the proper channels.

  • 日期 - 04/04/2018 16:32 - 04/04/2018 20:29
  • 最後更新 - 05/04/2018 08:49
Tonic Outage (已解決)
  • 優先級 - 重大
  • 影響範圍 伺服器 - Tonic CRM Network
  • Today one of our providers suffered a major outage across multiple datacentres in multiple locations. Tonic services were impacted for roughly 2 hours between 7 and 10am. Normal service has now been resumed.
     
    If you're still experiencing any issues, please contact us directly on: 03333353533
     
    Thank you for your patience.

  • 日期 - 09/11/2017 07:49 - 09/11/2017 09:53
  • 最後更新 - 09/11/2017 10:11
Adobe Sign Issues (已解決)
  • 優先級 - 重大
  • 影響範圍 其他 - Adobe Sign / Echosign
  • From Adobe Sign:

    Adobe Sign is currently experiencing an incident which impacts service on the NA1 shard.
     
    The Adobe Operations team is working to resolve the incident as quickly as possible.
     
    We apologize for any inconvenience this may cause.
     
    Adobe Sign Support

    Update 18/08/17 14:49

    We have resolved the incident causing instability on Adobe Sign. Full Service was restored at 06:17 AM Pacific Time, August 18, 2017.

    We appreciate your business and apologize for any inconvenience you may have experienced.

  • 日期 - 18/08/2017 14:01 - 18/08/2017 14:49
  • 最後更新 - 18/08/2017 14:50
Network issue (已解決)
  • 優先級 - 中
  • 影響範圍 其他 - DataCenter
  • We're aware of an issue affecting a diverse range of systems within the Tonic network. 

    We've marked the issue as critical, however the symptoms are currently limited to low bandwidth which is causing some slowdown in some services.

    The issue appears to be in one of the data centers and has been reported. A repair is underway.


    Update 26/07/2017 12:09

    We're currently in contact with our supplier to understand and resolve the issue.


    Update 26/07/2017 12:16

    The majority of services are now performing normally, however, the issue is still present. Downgrading to Medium Priority.

    Update 26/07/2017 14:33

    The monitoring period for this ticket is over and the network appears stable so we're closing this issue. 

    If you're still having any latency issues or noticing a slowdown, please open a support ticket

  • 日期 - 26/07/2017 11:54 - 26/07/2017 14:33
  • 最後更新 - 26/07/2017 14:36
Tonic Slow (已解決)
  • 優先級 - 高
  • 影響範圍 伺服器 - Tonic CRM Network
  • Hi All,

    We're experiencing some issues with the Tonic CRM network which is resulting in a general slowdown of tonic systems. 

    We have reason to believe that the issue resides further down the network than our systems and engineers are currently investigating.

    We'll keep you updated as the issue progresses.


    Update 19/07/2017 16:39

    We have located the source of the issue and are currently working on a fix.


    Update 19/07/2917 17:37

    We have now resolved the issue and are taking steps to prevent recurrence. Anyone still experiencing issues should open a support ticket.

  • 日期 - 19/07/2017 15:52 - 19/07/2917 17:37
  • 最後更新 - 19/07/2017 17:38
Tonic HQ - Internet (已解決)
  • 優先級 - 中
  • 影響範圍 其他 - Tonic Support
  • We've suffered an internet outage on our main trunk at our main office which is having an impact on some of our call termination.

    We're up and running now on our failover trunk but some customers are still experiencing issues reaching us by telephone. Please direct queries via-email to support@tonicpower.co.uk



    Update 22/06/2017 10:32

    We're now back to full capacity and normal service has resumed. We apologise for any inconvenience caused.


  • 日期 - 22/06/2017 09:38 - 22/06/2017 10:32
  • 最後更新 - 22/06/2017 10:34
admin.tonicpower.co.uk E-mail delivery issue (已解決)
  • 優先級 - 低
  • 影響範圍 伺服器 - Tonic Hosting Platform
  • We are aware of an issue relating to outbound e-mail delivery for customers using the admin.tonicpower.co.uk server. This has been disabled as a precaution and is due to a compromised customer website.

    We are working with the customer in order to resolve the issue and resume normal operation.



    16/05/2017 11:11

    The issue is in a resolution stage, however service is still impacted due to appearing on one RBL. E-mail delivery may still have issues.


    29/05/2017 16:08

    We have been monitoring the email system without issue for over a week so are now closing this issue down as resolved. Please report any further issues to support@tonicpower.co.uk

  • 日期 - 15/05/2017 17:45 - 29/05/2017 00:00
  • 最後更新 - 29/05/2017 16:10
Infrastructure Issue (已解決)
  • 優先級 - 高
  • 影響範圍 系統 - Tonic CRM
  • We are aware of an issue within the Tonic CRM infrastructure that is resulting in some general performance issues.

    Currently, only Tonic CRM installations are affected.

    Engineers are investigating and we'll inform you of any updates via this thread.

    -- Update 10-05-17 --

    The issue was relating to a server that was communicating incorrectly. The server has now been repaired and normal service has been resumed.

  • 日期 - 09/05/2017 08:34
  • 最後更新 - 10/05/2017 10:28
Cloudflare Memory Leakage (已解決)
  • 優先級 - 重大
  • 影響範圍 系統 - Cloudflare
  • Dear Customer,

    Thursday afternoon, Cloudflare published a blog post describing a memory leak caused by a serious bug that impacted their systems. If you haven't yet, We encourage you to read that post on the bug:

    https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

    As Tonic Solutions has, in the past, recommended Cloudflare to many of our customers. We feel it is our responsibility to bring this information to your attention. Please note, however, this issue was not caused by Tonic Solutions and is not related to any of our services. We publish this issue purely as a courtesy.


    Kind Regards,
    Tonic Support.

  • 日期 - 23/02/2017 17:14
  • 最後更新 - 05/04/2017 08:24
admin DDos (已解決)
  • 優先級 - 高
  • 影響範圍 伺服器 - Tonic Hosting Platform
  • The server admin.tonicpower.co.uk is suffering a sustained DDoS attack that is resulting in issues with e-mail delivery and slow page load times.

    The issue appears to be caused by insecure Wordpress installations triggering our spam protection routine. Please can you ensure that if you install third party software to your webspace that it is completely up-to-date and that you have carried out the relevant security tests to ensure the integrity of your website.

    ETA for a fix is 1:40pm 07/01/2016

    -- Update - The issue is resolved but as a precaution we are performing more tests before re-enabling mail transport. Thank you for your patience.

    -- Update - We mitigated the attack and have secured against future attacks.

  • 日期 - 06/01/2017 13:51 - 09/01/2017 00:00
  • 最後更新 - 11/01/2017 16:44
E-Mail Delivery (已解決)
  • 優先級 - 中
  • 影響範圍 伺服器 - Tonic Hosting Platform
  • We are aware of an issue affecting customers on the admin.tonicpower.co.uk shared server.

    Due to a clients compromised system sending large amounts of unsolicited mail we have temporarily found our IP range on a blacklist. This is affecting email delivery to some Yahoo and other smaller mail providers.

    Our engineers are currently liasing with the blacklist provider and hope to reach a resolution shortly. We will update via this notice as the issue progresses.



    Update 22/03/16: Blacklist checks are currently clear and have been holding steady for the past week. Please note that we will be amending our TOS for hosting customers in the coming weeks, we'll email details.

  • 日期 - 03/03/2016 09:50 - 22/03/2016 00:00
  • 最後更新 - 22/03/2016 10:58
Scheduled Maintenance (已解決)
  • 優先級 - 高
  • 影響範圍 伺服器 - Tonic Hosting Platform
  • Scheduled maintenance tonight to restore normal operation from failover to main trunk.

    Service may be disrupted.

  • 日期 - 26/01/2016 23:59 - 27/01/2016 03:00
  • 最後更新 - 03/03/2016 10:02
Failure of Shared Hosting (admin.tonicpower.c (已解決)
  • 優先級 - 重大
  • 影響範圍 伺服器 - Tonic Hosting Platform
  • We are currently investigating an issue with our shared hosting platform served from admin.tonicpower.co.uk.

    The issues appears to be a failure with a drive array which will need to move to failover and have a backup restored. During this time some customers services may be affected.


    Update 26/01/16 14:39

    Service should be resumed on failover, scheduled for maintenance tonight to restore normal operation.

  • 日期 - 26/01/2016 14:06 - 26/01/2016 14:39
  • 最後更新 - 26/01/2016 14:40
E-Sign Issues (已解決)
  • 優先級 - 中
  • 影響範圍 其他 - Tonic CRM (With Echosign)
  • We're experiencing issues with Echosign termination.

    Agreements are being created but not sent to the recipient. The issue lies with Echosign and we'll update when we have more information.

    Agreements can still be created safely but will not be received until the issue is resolved.


    Update 12:34 07/01/2016

    Engineers are working on the issue and expect to have a resolution shortly.

    Update 16:01 07/01/2016

    The issue leading to E-Sign transmission has been stable for the past 30 minutes.

    We are being advised to recommend clients re-send agreements created before 15:30 today to ensure delivery.

  • 日期 - 07/01/2016 11:34
  • 最後更新 - 07/01/2016 16:03
Issue Affecting Tonic CRM Standard (已解決)
  • 優先級 - 高
  • 影響範圍 系統 - Tonic CRM
  • We were notified of a system issue that affected some clients using Tonic CRM. The system was inacessible for a period of about 2 hours.

    Engineers responded to fault in a Command & Control (C&C) server in our datacentre and resolved an issue.

  • 日期 - 14/08/2015 16:24 - 14/08/2015 18:35
  • 最後更新 - 17/08/2015 11:24

伺服器狀態

以下是我們所有伺服器的即時概況,您可以在此查看是否有任何已知問題發生。

伺服器名稱 HTTP FTP POP3 PHP 資訊 伺服器負載 可用時間
Tonic CRM Network PHP 資訊
Tonic Hosting Platform PHP 資訊
Tonic TC3S PHP 資訊