Issue affecting CRM Users (Решени)
  • Важност на пораката - Критична
  • Влијаат Сервер - Tonic CRM Network
  • We are aware of an issue with CRM users who are receiving a license expiry when this is not the case

    We have traced the issue to a DNS issue at our data centres and are working to resolve.

  • Дата - 17/06/2021 09:02 - 17/06/2022 10:00
  • Последна промена - 01/02/2023 10:55
Networking issues (Решени)
  • Важност на пораката - Средна
  • Влијаат Други - Virgin Media
  • We've been made aware of an issue affecting some Virgin Media customers that are losing contact with our services.

    Unfortunately this problem lies with Virgin media or a network node between them and us. 

    We have informed our data centre engineers who are working on a workaround.


    Update: 14/04/2021 10:58

    We're receiving reports that services are starting to come back up. We're downgrading severity to medium.


    Update: 14/04/2021 11:04

    No further reports and all known affected customers now returned to service.

    Closing issue.

  • Дата - 14/04/2021 10:50 - 14/04/2021 11:04
  • Последна промена - 14/04/2021 11:04
API Loop issue (Решени)
  • Важност на пораката - Критична
  • Влијаат Сервер - Tonic CRM Network
  • A planned action on our network over the weekend resulted in dangerous unintended and unforeseeable load on our CRM network. The cluster rejected connections in order to preserve data continuity.

    Customers affected 

    • Tonic CRM customers with API access
    • Tonic CRM customers with Automated reporting

    Symptoms

    • UI and API calls rejected

    We will be reaching out to affected customers today to advise on the issue and to request re-submission of failed API calls. If you believe you could be one of the affected customers, please check your API log over the weekend and re-submit any failed requests.

    Remedial Action Taken

    • We have stabilised the network and all systems are now responding as requested, there was no working downtime. 
    • API requests on Saturday 26th and Sunday the 27th September may have failed, please check your logs and resubmit if necessary.

  • Дата - 26/09/2020 08:14 - 28/09/2020 08:00
  • Последна промена - 28/09/2020 08:27
Widespread Server Outage (Решени)
  • Важност на пораката - Средна
  • Влијаат Сервер - Tonic CRM Network
  • We have experience an issue in the data centre. Some services are functioning at a reduced capacity while others are not functioning at all.

    We investigate.



    Update: 30/03/2020 16:34 - We expect to be operating back at full capacity within 30 minutes.



    Update: 30/03/2020 16:46 - The majority of services are now restored. We continue work on the rest.



    Update: 30/03/2020 17:15 - The issue is resolved, all systems go.

  • Дата - 30/03/2020 15:15 - 30/03/2020 17:15
  • Последна промена - 30/03/2020 16:18
Telephony Outage (Решени)
  • Важност на пораката - Критична
  • Влијаат Сервер - Tonic TC3S
  • An outage has been reported on our telephony network, we are investigating


    Update 12/03/2020 14:45

    We have discovered the cause of the issue and are working on a resolution. 

    ETA 5 minutes.


    Update 2/03/2020 14:47

    The instability has been corrected and the issue is now resolved.

  • Дата - 12/03/2020 14:41 - 12/03/2020 14:47
  • Последна промена - 12/03/2020 14:47
Tonic Licensing Issue (Решени)
  • Важност на пораката - Критична
  • Влијаат Сервер - Tonic CRM Network
  • We are currently dealing with an issue with our licensing server that is blocking access to tonic CRM user interface.

    Scheduled actions, API and Introducer access are unaffected. 

    We aim to have resolved this issue by 09.30AM on 12.07.2019

  • Дата - 11/07/2019 22:42 - 12/07/2019 10:42
  • Последна промена - 26/02/2020 10:14
Email Issues (Решени)
  • Важност на пораката - Ниска
  • Влијаат Други - Email Receipt
  • Dear Customer,

    We're experiencing some issues with e-mail reception this morning. The cause is being investigated.

    If you're attempting to contact us via any @tonicpower email address, please contact us via telephone instead on 03333353533 or open a support ticket at https://portal.tonicpower.co.uk

  • Дата - 09/08/2018 10:34 - 10/08/2018 09:05
  • Последна промена - 02/11/2018 14:29
Intermittent postcode lookup issues (Решени)
  • Важност на пораката - Ниска
  • Влијаат Други - PAF Lookup
  • Hello,

    Some customers are reporting issues with areas of the postcode checker not being able to connect.

    We use multiple postcode lookup sources in order to ensure continuity of service so currently advice is to "keep trying" if you receive any errors.

    We're currently looking into this, however, we believe the problem to be with one of our PAF providers leading to intermittent issues. 

    We'll update via this ticket with updates.


    Update: 16/05/18 10:41

    The issue definitely resides with one of our providers, we're currently working with them to investigate the issue.


    Update: 16/05/18 11:16

    Engineers are unable to locate any issue with infrastructure and all tests are coming back normal. We're closing this issue off as we are confident that the issue is most likely a temporary transport problem.

    We'll work with individual clients if they are still having issues, so please reach out to support@tonicpower.co.uk if you are still having any problems.

  • Дата - 16/05/2018 10:34 - 16/05/2018 11:16
  • Последна промена - 16/05/2018 11:18
Intermittent Connectivity Issues (Решени)
  • Важност на пораката - Средна
  • Влијаат Сервер - Tonic CRM Network
  • Hello,

    We're investigating an issue where some customers are experiencing some connectivity issues to the CRM system.

    This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.



    Update:

    Engineers believe the problem has been resolved, please report any further occurrences to us through the proper channels.

  • Дата - 04/04/2018 16:21 - 04/04/2018 20:29
  • Последна промена - 05/04/2018 08:50
Intermittent Connectivity Issues (Решени)
  • Важност на пораката - Висока
  • Влијаат Сервер - Tonic TC3S
  • Hello,

    We're investigating an issue where some customers are experiencing some connectivity issues to the telephone system.

    This is an issue in the datacenter and is under investigation. There is no risk to the server infrastructure.



    Update:

    Engineers believe the problem has been resolved, please report any further occurrences to us through the proper channels.

  • Дата - 04/04/2018 16:32 - 04/04/2018 20:29
  • Последна промена - 05/04/2018 08:49
Tonic Outage (Решени)
  • Важност на пораката - Критична
  • Влијаат Сервер - Tonic CRM Network
  • Today one of our providers suffered a major outage across multiple datacentres in multiple locations. Tonic services were impacted for roughly 2 hours between 7 and 10am. Normal service has now been resumed.
     
    If you're still experiencing any issues, please contact us directly on: 03333353533
     
    Thank you for your patience.

  • Дата - 09/11/2017 07:49 - 09/11/2017 09:53
  • Последна промена - 09/11/2017 10:11
Adobe Sign Issues (Решени)
  • Важност на пораката - Критична
  • Влијаат Други - Adobe Sign / Echosign
  • From Adobe Sign:

    Adobe Sign is currently experiencing an incident which impacts service on the NA1 shard.
     
    The Adobe Operations team is working to resolve the incident as quickly as possible.
     
    We apologize for any inconvenience this may cause.
     
    Adobe Sign Support

    Update 18/08/17 14:49

    We have resolved the incident causing instability on Adobe Sign. Full Service was restored at 06:17 AM Pacific Time, August 18, 2017.

    We appreciate your business and apologize for any inconvenience you may have experienced.

  • Дата - 18/08/2017 14:01 - 18/08/2017 14:49
  • Последна промена - 18/08/2017 14:50
Network issue (Решени)
  • Важност на пораката - Средна
  • Влијаат Други - DataCenter
  • We're aware of an issue affecting a diverse range of systems within the Tonic network. 

    We've marked the issue as critical, however the symptoms are currently limited to low bandwidth which is causing some slowdown in some services.

    The issue appears to be in one of the data centers and has been reported. A repair is underway.


    Update 26/07/2017 12:09

    We're currently in contact with our supplier to understand and resolve the issue.


    Update 26/07/2017 12:16

    The majority of services are now performing normally, however, the issue is still present. Downgrading to Medium Priority.

    Update 26/07/2017 14:33

    The monitoring period for this ticket is over and the network appears stable so we're closing this issue. 

    If you're still having any latency issues or noticing a slowdown, please open a support ticket

  • Дата - 26/07/2017 11:54 - 26/07/2017 14:33
  • Последна промена - 26/07/2017 14:36
Tonic Slow (Решени)
  • Важност на пораката - Висока
  • Влијаат Сервер - Tonic CRM Network
  • Hi All,

    We're experiencing some issues with the Tonic CRM network which is resulting in a general slowdown of tonic systems. 

    We have reason to believe that the issue resides further down the network than our systems and engineers are currently investigating.

    We'll keep you updated as the issue progresses.


    Update 19/07/2017 16:39

    We have located the source of the issue and are currently working on a fix.


    Update 19/07/2917 17:37

    We have now resolved the issue and are taking steps to prevent recurrence. Anyone still experiencing issues should open a support ticket.

  • Дата - 19/07/2017 15:52 - 19/07/2917 17:37
  • Последна промена - 19/07/2017 17:38
Tonic HQ - Internet (Решени)
  • Важност на пораката - Средна
  • Влијаат Други - Tonic Support
  • We've suffered an internet outage on our main trunk at our main office which is having an impact on some of our call termination.

    We're up and running now on our failover trunk but some customers are still experiencing issues reaching us by telephone. Please direct queries via-email to support@tonicpower.co.uk



    Update 22/06/2017 10:32

    We're now back to full capacity and normal service has resumed. We apologise for any inconvenience caused.


  • Дата - 22/06/2017 09:38 - 22/06/2017 10:32
  • Последна промена - 22/06/2017 10:34
admin.tonicpower.co.uk E-mail delivery issue (Решени)
  • Важност на пораката - Ниска
  • Влијаат Сервер - Tonic Hosting Platform
  • We are aware of an issue relating to outbound e-mail delivery for customers using the admin.tonicpower.co.uk server. This has been disabled as a precaution and is due to a compromised customer website.

    We are working with the customer in order to resolve the issue and resume normal operation.



    16/05/2017 11:11

    The issue is in a resolution stage, however service is still impacted due to appearing on one RBL. E-mail delivery may still have issues.


    29/05/2017 16:08

    We have been monitoring the email system without issue for over a week so are now closing this issue down as resolved. Please report any further issues to support@tonicpower.co.uk

  • Дата - 15/05/2017 17:45 - 29/05/2017 00:00
  • Последна промена - 29/05/2017 16:10
Infrastructure Issue (Решени)
  • Важност на пораката - Висока
  • Влијаат Систем - Tonic CRM
  • We are aware of an issue within the Tonic CRM infrastructure that is resulting in some general performance issues.

    Currently, only Tonic CRM installations are affected.

    Engineers are investigating and we'll inform you of any updates via this thread.

    -- Update 10-05-17 --

    The issue was relating to a server that was communicating incorrectly. The server has now been repaired and normal service has been resumed.

  • Дата - 09/05/2017 08:34
  • Последна промена - 10/05/2017 10:28
Cloudflare Memory Leakage (Решени)
  • Важност на пораката - Критична
  • Влијаат Систем - Cloudflare
  • Dear Customer,

    Thursday afternoon, Cloudflare published a blog post describing a memory leak caused by a serious bug that impacted their systems. If you haven't yet, We encourage you to read that post on the bug:

    https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

    As Tonic Solutions has, in the past, recommended Cloudflare to many of our customers. We feel it is our responsibility to bring this information to your attention. Please note, however, this issue was not caused by Tonic Solutions and is not related to any of our services. We publish this issue purely as a courtesy.


    Kind Regards,
    Tonic Support.

  • Дата - 23/02/2017 17:14
  • Последна промена - 05/04/2017 08:24
admin DDos (Решени)
  • Важност на пораката - Висока
  • Влијаат Сервер - Tonic Hosting Platform
  • The server admin.tonicpower.co.uk is suffering a sustained DDoS attack that is resulting in issues with e-mail delivery and slow page load times.

    The issue appears to be caused by insecure Wordpress installations triggering our spam protection routine. Please can you ensure that if you install third party software to your webspace that it is completely up-to-date and that you have carried out the relevant security tests to ensure the integrity of your website.

    ETA for a fix is 1:40pm 07/01/2016

    -- Update - The issue is resolved but as a precaution we are performing more tests before re-enabling mail transport. Thank you for your patience.

    -- Update - We mitigated the attack and have secured against future attacks.

  • Дата - 06/01/2017 13:51 - 09/01/2017 00:00
  • Последна промена - 11/01/2017 16:44
E-Mail Delivery (Решени)
  • Важност на пораката - Средна
  • Влијаат Сервер - Tonic Hosting Platform
  • We are aware of an issue affecting customers on the admin.tonicpower.co.uk shared server.

    Due to a clients compromised system sending large amounts of unsolicited mail we have temporarily found our IP range on a blacklist. This is affecting email delivery to some Yahoo and other smaller mail providers.

    Our engineers are currently liasing with the blacklist provider and hope to reach a resolution shortly. We will update via this notice as the issue progresses.



    Update 22/03/16: Blacklist checks are currently clear and have been holding steady for the past week. Please note that we will be amending our TOS for hosting customers in the coming weeks, we'll email details.

  • Дата - 03/03/2016 09:50 - 22/03/2016 00:00
  • Последна промена - 22/03/2016 10:58
Scheduled Maintenance (Решени)
  • Важност на пораката - Висока
  • Влијаат Сервер - Tonic Hosting Platform
  • Scheduled maintenance tonight to restore normal operation from failover to main trunk.

    Service may be disrupted.

  • Дата - 26/01/2016 23:59 - 27/01/2016 03:00
  • Последна промена - 03/03/2016 10:02
Failure of Shared Hosting (admin.tonicpower.c (Решени)
  • Важност на пораката - Критична
  • Влијаат Сервер - Tonic Hosting Platform
  • We are currently investigating an issue with our shared hosting platform served from admin.tonicpower.co.uk.

    The issues appears to be a failure with a drive array which will need to move to failover and have a backup restored. During this time some customers services may be affected.


    Update 26/01/16 14:39

    Service should be resumed on failover, scheduled for maintenance tonight to restore normal operation.

  • Дата - 26/01/2016 14:06 - 26/01/2016 14:39
  • Последна промена - 26/01/2016 14:40
E-Sign Issues (Решени)
  • Важност на пораката - Средна
  • Влијаат Други - Tonic CRM (With Echosign)
  • We're experiencing issues with Echosign termination.

    Agreements are being created but not sent to the recipient. The issue lies with Echosign and we'll update when we have more information.

    Agreements can still be created safely but will not be received until the issue is resolved.


    Update 12:34 07/01/2016

    Engineers are working on the issue and expect to have a resolution shortly.

    Update 16:01 07/01/2016

    The issue leading to E-Sign transmission has been stable for the past 30 minutes.

    We are being advised to recommend clients re-send agreements created before 15:30 today to ensure delivery.

  • Дата - 07/01/2016 11:34
  • Последна промена - 07/01/2016 16:03
Issue Affecting Tonic CRM Standard (Решени)
  • Важност на пораката - Висока
  • Влијаат Систем - Tonic CRM
  • We were notified of a system issue that affected some clients using Tonic CRM. The system was inacessible for a period of about 2 hours.

    Engineers responded to fault in a Command & Control (C&C) server in our datacentre and resolved an issue.

  • Дата - 14/08/2015 16:24 - 14/08/2015 18:35
  • Последна промена - 17/08/2015 11:24

Статус на сервери

Преглед на нашите сервери во реално време.

Сервер HTTP FTP POP3 PHP инфо % на искористеност Активно време
Tonic CRM Network PHP инфо
Tonic Hosting Platform PHP инфо
Tonic TC3S PHP инфо