PDA

View Full Version : Linux Lotus Server



m8internet
14-04-2011, 03:58 PM
Has there been an issue with this server, between 16:00 and 17:00 today?

rogerd
15-04-2011, 12:08 PM
Has there been an issue with this server, between 16:00 and 17:00 today?

We can assume yes as my customers reported sites down but it was back after 40 minutes.

m8internet
15-04-2011, 04:59 PM
See I don't get this, the Server Status page showed no issue (but a high server load)
I was able to visit websites without any issue, making sure it was dynamic content and not ISP cached

m8internet
23-07-2011, 09:59 AM
Has there been an issue with this server, since 10:45?
Websites are slow to respond and often unable to connect to DB / MySQL

The Server Status shows Critical Load, and has been like that since 10:00

Natasha
01-08-2011, 03:06 PM
We've checked with our operation team and they did investigate the servers crond process and daily tasks performed on the server.

With a conclusion they found that during backup process which generated tarball files which takes up CPU usage causing server load.

To to minimize the server load they have changed the backup cron time which will run after midnight and will make alternate days to run backup on the server.

Thanks

m8internet
24-09-2011, 11:59 AM
Has there been an issue with this server, since 13:00 yesterday?
The Server Status has showed various errors and noticed, but typically resulting in no connection to database and/or slow response

It has been unable to connect since 12:30 today

m8internet
02-10-2011, 09:03 AM
Again today, downtime from 09:13 to 09:44

Daniel Smith
03-10-2011, 11:38 AM
Thanks for your heads up on this regards and sorry for the delay in response.

Well, the server was not down but it was an firewall issue which I have corrected it as soon as I was informed by one of our client and notified by server monitoring system.

I hope everything is working fine for you now.

rogerd
03-10-2011, 03:12 PM
I'm becoming increasingly worried about the Lotus server as its instability and variable load issues are starting to have an impact on my customers.

The loads are regularly high with a number of occasions today above 10 and I am now regularly getting downtime alerts from monitoring programs but by the time it is queried support say server is fine.

Not so.

Luke Williams
04-10-2011, 07:21 AM
Hello,

We have changed the backup script time due to these ongoing issues, I'm sure server will be fine in peak hours now.

Thanks.

m8internet
04-10-2011, 03:58 PM
The loads are regularly high with a number of occasions today above 10 and I am now regularly getting downtime alerts from monitoring programs
I was previously using webhostingstuff but in the last few weeks their monitor seems to have disappeared
Before it disappeared I had noticed the uptime value was dropping steadily

Last week I installed uptimerobot as this monitors the server at 5 minute intervals rather than hourly (if the server returns offline then it increases to every minute until back online)
However, it has quite a lot of configurations and I believe it was giving me false positive reports, so simplified it
I believe this was due to the IP reporting being blocked by the firewall

If you notice irregular activity, then post the times as I have done above

rogerd
04-10-2011, 06:22 PM
If you notice irregular activity, then post the times as I have done above

There is a heavy load regularly about every five to 10 minutes past the hour:

02/10/2011-22:05:30 +0100 = 15.60, 8.64, 4.53
3/10/2011-00:06:55 +0100 = 12.57, 7.59, 4.00
03/10/2011-02:05:38 +0100 = 9.30, 4.55, 2.43
03/10/2011-04:06:42 +0100 = 14.02, 7.74, 3.74
03/10/2011-06:05:34 +0100 = 13.00, 7.38, 4.07
03/10/2011-08:10:02 +0100 = 10.25, 8.99, 4.67
etc, etc.......

and I seem to remember the backup scripts were already previously altered some time ago to take them out of peak hours ?????

m8internet
04-10-2011, 08:01 PM
That would suggest someone is running a cronjob at x:05 either every hour or every second (even) hour

I know many people will run them at x:00 or at midnight

When backups run they shouldn't affect the visitors as much as this

m8internet
04-10-2011, 08:09 PM
Scrutinised my customers and found one that meets this criteria
If Webhost UK want to discuss then please create a new ticket on my account and advise of account number 0311

m8internet
04-10-2011, 08:19 PM
Also account 0264, this also meets the same criteria
Looks like it is a recommendation to run cron jobs as x:05

rogerd
04-10-2011, 09:06 PM
That would suggest someone is running a cronjob at x:05 either every hour or every second (even) hour

I know many people will run them at x:00 or at midnight

When backups run they shouldn't affect the visitors as much as this

It is obviously cron jobs or something similar, due to the regularity.

I would add there's nothing wrong with running cron jobs - as long as they are tested to see they are not bad scripts, rather than just setup and left.

You can always guarantee I am trying to login to WHM or a cpanel or FTP when it happens and I have to wait a while before I can get access, but hey, that's life.

My concerns on the server were more pointed at the general "average" load rise over recent times especially yesterday. I'm happy to say the averages have settled since - so fingers crossed it stays that way.

m8internet
04-10-2011, 09:45 PM
I found several customers with cronjobs, including my own, and they all look to be acceptable
My attention was drawn to these two as they look highly unusual

The first appears to be calling files within cpanel and I have not yet been able to establish why a customer would do this

The second appears to be calling a flie within a folder that does not exist, so the customer is probably not aware of this but it should send a warning email but there is no email account setup on their account

As a reseller, any failed cronjobs should be notified to my account
Equally, it takes some time to look through every account to firstly see if there are any and secondly to determine how they work if there are any

m8internet
04-10-2011, 09:48 PM
FTP when it happens and I have to wait a while before I can get access
I reported this some time ago and have never had this issue resolved
The FTP server suddenly cuts off during connection
I was given various settings to change but it made no difference
I then connected to another provider and there was no issue
On reconnecting the issue persists

rogerd
04-10-2011, 10:01 PM
I found several customers with cronjobs, including my own, and they all look to be acceptable
My attention was drawn to these two as they look highly unusual

The first appears to be calling files within cpanel and I have not yet been able to establish why a customer would do this

The second appears to be calling a flie within a folder that does not exist, so the customer is probably not aware of this but it should send a warning email but there is no email account setup on their account

As a reseller, any failed cronjobs should be notified to my account
Equally, it takes some time to look through every account to firstly see if there are any and secondly to determine how they work if there are any

We should be doing this maybe on the reseller forum, rather than network status.

The cron calling the non-existant shouldn't be major, could be left over remnants of an old Softaculous app since removed, but the other one is intriguing isn't it?

I'll shift to the reseller thread.

Luke Williams
05-10-2011, 05:02 AM
Hello,

We're aware that each user on our reseller server is running cron jobs, we're trying to find out the cron jobs which are set to run in less that 10 minutes. We're thinking to describe this on our website that each cron job should have interval of 30 minutes.

As well as I mentioned in my previous post, we have changed the backup script time to off-peak hours which will help us to keep the reseller server stable and fine.

I personally now monitoring the server for any server load issues or any intermittent downtime you're facing.

We appreciate your patience during this period.

Thanks.

rogerd
05-10-2011, 08:55 AM
Hello,

We're aware that each user on our reseller server is running cron jobs, we're trying to find out the cron jobs which are set to run in less that 10 minutes. We're thinking to describe this on our website that each cron job should have interval of 30 minutes.

As well as I mentioned in my previous post, we have changed the backup script time to off-peak hours which will help us to keep the reseller server stable and fine.

I personally now monitoring the server for any server load issues or any intermittent downtime you're facing.

We appreciate your patience during this period.

Thanks.

Hi Luke,

It's not the frequency of cronjobs but the type of script running intense cpu processes which can cause the load increase. There is a danger of overlimiting resources for no valid reason which may then be detrimental to your customers operations.

Recently, average loads have settled and spikes seemed to be only every alternate hour for a short period. Even they have settled in the last 24 hours. As I said previously there is a great improvement so some work has obviously improved things.

Thank you.

Natasha
05-10-2011, 12:18 PM
From the cron logs I can see crond service was restarted today in early morning at 3:13 am

5 Oct 5 03:13 /var/run/crond.pid

;and it was auto restarted immediately wit hthe help of chkservd service.

Oct 5 03:13:07 lotus crond[2635]: (CRON) STARTUP (V5.0)
Oct 5 03:14:01 lotus crond[2635]: (root) RELOAD (cron/root)

So As far as I know there is no issue with cron service on Lotus server and most of the time cron service is running fine and if any case it is down/unavailable then chkservd service checks all the system services including crond service and restarts immediately if found in stopped state.

rogerd
05-10-2011, 12:33 PM
From the cron logs I can see crond service was restarted today in early morning at 3:13 am

5 Oct 5 03:13 /var/run/crond.pid

;and it was auto restarted immediately wit hthe help of chkservd service.

Oct 5 03:13:07 lotus crond[2635]: (CRON) STARTUP (V5.0)
Oct 5 03:14:01 lotus crond[2635]: (root) RELOAD (cron/root)

So As far as I know there is no issue with cron service on Lotus server and most of the time cron service is running fine and if any case it is down/unavailable then chkservd service checks all the system services including crond service and restarts immediately if found in stopped state.

I totally agree with you, cron runs fine as far as I can see.

Regarding whether there has been an issue with cron - not especially. The comments and discussion started with possible reasons for high load issues which were having a performance impact on the server and cron participation was thrown into the conversation.

This has gone on to Luke suggesting limiting cron frequency to a minimum of 10 minutes which prompted my comment of "not unless you have a valid reason" which I really can't see.

As long as abuses on the server are caught and actioned which seems to be happening, there doesn't seem to be a need for any other changes.

Just my 2 pennyworth.....

Natasha
05-10-2011, 01:03 PM
You're correct this thread was for high load issues and I've checked with the operation team that with some user account those running cronjobs with the time stamp less than 5 minutes and some of them were to update the RSS feeds and clients more often get it done through running cronjobs. If the feed data is big enough it scans the entire data and sometime it happens to get the cron service to get killed due to amount of memory and cpu usage it eats up.

We indeed receives alerts for max number of emails sent from the server, anu user overloading the server by utilizing max memory or cpu usage, and on that basis we get the culprit account suspended to make sure the shared server performs well.

Honestly it''s been quite a while that we'd received any complaints or such reports especially for this server and hope it delivers with stable performance.

m8internet
06-10-2011, 10:00 PM
Honestly it''s been quite a while that we'd received any complaints or such reports especially for this server and hope it delivers with stable performance
April, July, and October; once every three months

On the plus side I have not received any warning emails about any of my customers, so assume it is not one of mine...

george
07-10-2011, 03:11 AM
I've confirmed with the operation team and it was not your customers. We are still closely monitoring the server.

m8internet
20-10-2011, 10:56 PM
I've had eighteen customers report downtime since 22:50 and nearly an hour later it is no different

Albert Kent
21-10-2011, 03:25 AM
Hello,

We have checked with our support team and there is no any server or services related issue reported at yesterday.

rogerd
21-10-2011, 08:42 AM
Hello,

We have checked with our support team and there is no any server or services related issue reported at yesterday.

Strange as I spent 30 minutes on live chat last night complaining about loads on the server being high all day followed by me receiving two seperate downtime alerts from third party monitoring sites at the times mentioned by m8internet

Maybe your support team are looking at a different server.

Today we have started with loads of 5 - pretty steady, which does not bode well for the rest of the day

rogerd
21-10-2011, 09:04 AM
Hello,

As well as I mentioned in my previous post, we have changed the backup script time to off-peak hours which will help us to keep the reseller server stable and fine.

Thanks.

Message for Luke: We had already been told backups were changed to out of hours months ago - so now you are just changing it last week ??

Well guess what - on live chat last night at 20:22 I was told the high loads were due AT THAT TIME to backups running, it was killed and the loads reduced then the server went down.

This morning we are running at 5 average to start the day.

Please note: WE are monitoring the loads as well

PS The dates of recent posts in this thread have been screwed up as the quote here was only from a post last week yet the post is shown as dated May this year.

rogerd
21-10-2011, 09:08 AM
PS The dates of recent posts in this thread have been screwed up as the quote here was only from a post last week yet the post is shown as dated May this year.


My mistake - stupid US dates again on a UK forum

Why do you do it admin?

m8internet
21-10-2011, 11:24 AM
My mistake - stupid US dates again on a UK forum
The dates are in UK format for me
Perhaps the setup in your User CP has changed back to default

Anyway, my apologies for digging up this old fossil in error, I searched Lotus and this thread appeared first so replied but there was actually a newer reply else where!

I have five customers operating much busier websites as Christmas approaches
These are usually the first to report issues as they receive lots of visits every day, plus complaints from their customers about orders suddenly stopping (website unavailable)
However when other customers contact me with much quieter websites I start to worry

Strangely I did not receive any warning emails from my external status provider, uptimerobot.com, which up to now has been very reliable
However, I notice the uptime they list no longer matches the server status that appears in my WHMCS installation

m8internet
21-10-2011, 11:25 AM
The dates are in UK format for me
Perhaps the setup in your User CP has changed back to default
Apologies, you are right month-day-year
This has changed since the update

Abbie
22-10-2011, 08:50 AM
The date format issue has now been set to dd-mm-yy.

m8internet
22-10-2011, 11:25 AM
The date format issue has now been set to dd-mm-yy.
That's better, makes it much easier to read
Now why couldn't you also do this with the server time?
http://www.webhost.uk.net/forums/showthread.php?t=371
However there is little point now as the clocks go back next week and so will be correct!
I will revive that thread nearer the time in 2012

m8internet
22-10-2011, 11:29 AM
Today we have started with loads of 5 - pretty steady, which does not bode well for the rest of the day
I have sat with the Server Status page for the last few hours, and it is stil showing at least 5, so it is no surprise the average is at least 5

I also had a response advising there was an issue with the backups and they will now be performed on non-business days
However, for many of my customers every day is a business day
The quietest time in the UK is likely to be :
Saturday 01:00 to 04:00
Sunday 01:00 to 07:00
You might want to reconsider these windows for running backups
These are also the windows I use for website maintenance so that customers and visitors don't notice any downtime

m8internet
22-10-2011, 11:35 AM
I have sat with the Server Status page for the last few hours, and it is stil showing at least 5, so it is no surprise the average is at least 5
Replied too soon!

12:33 (server time 11:30) and load hits :
WARNING - load average: 6.80, 6.78, 6.20

This seems to come in short bursts

m8internet
06-11-2011, 05:08 PM
High server loads almost all day, slow websites

Daniel Smith
06-11-2011, 10:48 PM
Yes, we were aware of the issue...
There was backup scheduled at that time and the server was under maintenance the load was recorded by you.

We have managed to lower down the server load and now everything is working fine now.

m8internet
07-11-2011, 02:48 PM
Yes, we were aware of the issue...
There was backup scheduled at that time and the server was under maintenance the load was recorded by you
A backup scheduled to place all day?

Luke Williams
08-11-2011, 04:10 AM
Hello,

Backup schedule is placed in off-peak hours. But it is possible that the backup process is running in working hours depends on number of accounts need to backup.

m8internet
18-11-2011, 06:22 PM
The monitor Lotus Server Secure (92.48.78.224) is currently down 18/11/2011 18:14:46

m8internet
18-11-2011, 06:23 PM
The monitor Lotus Server A (213.229.113.14) is currently down 18/11/2011 18:21:39

Natasha
24-11-2011, 03:55 PM
We've corrected the network status page, there was some issue with nagios settings which is now corrected.

m8internet
11-12-2011, 11:32 PM
22 minutes of downtime
22:47 to 23:09

m8internet
07-01-2012, 02:34 PM
A backup scheduled to place all day?
Looks like the backup has been running since since this morning and is still running
I am sorry but this is outrageous and unacceptable
A solution to this needs to be found as it is affecting everyone on this server every weekend now

m8internet
26-01-2012, 05:38 PM
There has been no access to FTP since 11:30 today
6 hours later and no change