What happened last night (14th Nov)

To our writers!

With our usual willingness to over-share, I thought I’d outline why you couldn’t log in last night in one post. The main reason is we needed to sleep so there was no one up to kickstart the server when it went down. We’re really sorry about that and we’re looking at ways of avoiding it happening again (see below).

Your words

We also want to say that although there were problems logging in – if somehow this happens again, don’t worry your words are extremely safe. Whether or not you can log in, your words are still there and we save in three separate places so be assured that once Novlr is back up they will be there.

The tech

So, one of the ‘node processes’ on our server got stuck. I know that ‘stuck’ doesn’t sound like a technical term but it’s the word used. What that meant is the other processes couldn’t cope with the load, so many of you couldn’t log in and write. What was needed was the server to be restarted.

The timing

Unfortunately, it coincided with the few hours in the day when none of us were around – almost exactly – between 2am and 6am UK time.

What we’re doing about it

We totally understand that the fact it happened when we were asleep isn’t any great consolation to the amazing number of you using Novlr from the US/Canada/Australia and lots of other places in different timezones. Our servers also shouldn’t go down at all. No one is more frustrated about it than us.

  1. We’ll be getting in touch with each of you who got in touch last night to make sure all is well
  2. We’re going to be adding node workers to the server to cope with load, so that if one gets ‘stuck’ the others can cope fine.
  3. We’re also going to look at server monitoring options, to automate a restart if we hit the same problem (this might take a little longer as we don’t want it restarting unnecessarily)
  4. We’re also going to look at the language we use because we don’t think it was that helpful. When you were trying to log in the site was telling you ‘an account with that email doesn’t exist’ – the account does exist, just the code couldn’t reach the server, so thought the account didn’t exist. We will change this because it is scary to think that your account might be gone. (It would never be gone!)
  5. We’re going to be encouraging our writers to sync their work automatically daily with Google Drive or Dropbox (switch it on in Settings > Export. You don’t need to, because we back everything up a lot, but it just gives you another layer of reassurance).
  6. Kim and Thomas will sleep less 🙂

We’re really sorry this happened. And we’re annoyed with ourselves for letting it happen. It’s the last thing you need during a NaNoWriMo push.

Thank you for being so patient with us – we are trying to build the best possible tool we can for you.

6 replies
    • Thomas says:

      Thanks Nicholas. You’re absolutely right it is – we’ll make sure we’re getting enough otherwise we’ll be no use to anyone 🙂

      Reply
  1. Akimika says:

    Hey, you guys are really great! Don’t let a hung up process make you too crazy. One could suggest writing a process monitor, but sometimes it creates false positives that do more harm than good. I think there might be a compromise to be reached that a regular restart of the services can usually help avoid. That’s just my suggestion, though. ^^; I have to say you guys have created a really great tool, and I admire that you do get back to everyone. Thanks a lot!

    Reply
    • Thomas says:

      Thanks Akimika. Good idea, we’ll look at regular restarts too as an option.

      It’s just the beginning, once we get through teething issues like this, we can get on to the really exciting stuff.

      Reply
  2. Nik says:

    Servers, any servers can do that.

    I reckon as long as people KNOw it’s just a ‘glitch’, they don’t mind. I know I don’t, as working in the computer industry I see these things occasionally happen to any company at any time. Keep up the great job!

    Reply

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *