Monthly Archives: September 2011

Blackboard Down Again

Update: Sep-14-2011 7:00 am

Roger has successfully rebooted the BB servers and the system is functioning normally again.


Update: 4:00 pm

Tulio, Tom and Roger have investigated the problem with the Blackboard servers and are confident that this is the same issue we experienced several months ago. The immediate fix is to restart the BB file server, followed by the app servers. Due to the critical nature of BB, we will be performing that restart beginning at 5:00 am tomorrow rather than waiting till the regularly scheduled maintenance window on Thursday. System restart time is estimated to be about 30 minutes, but could be as long as 2 hours. I will be sending out an e-mail to the campus shortly.


Tulio has shutdown all but one of the apps servers and restarted the services on that app server to get BB logins working again. BB is up again but only running on one server. More later.


Blackboard is currently not functioning. More information to follow.

NGDC Status Update – September 13th

Dell Consulting has completed the configuration of the blade server system, the SAN storage system and the VSphere virtual environment. Except for the delivery of as-built documentation, their work is done.

Last week, we performed some test virtual-to-virtual (aka V2V) migrations of servers that were stationed on the VMware ESX 3.5 hosts. We also performed some limited testing of physical-to-virtual (aka P2V) migrations. Everything is working as expected.

A relationship map view of our VSphere environment with one production server.

A fair amount of time is now being spent performing a deeper inventory of our existing servers, including what each server does, which department it belongs to, and whether or not it is used for live access (aka ‘production’) or is only for development and testing. This information will allow us to target the migration of production servers to the NGDC system first, and to coordinate those migrations with the affected communities to have the least negative impact. We are also configuring the VSphere environment security levels, building virtual server deployment templates, developing virtual server access and management strategies and tools, and working on ensuring that all production servers residing on the new system are protected with backup processes consistent with our disaster recovery plans.

Our goal for the NGDC project this next 12 months is to convert approximately one-third of our existing physical servers to virtual servers, about 30 machines. The left over physical boxes will then be powered off and removed from the datacenter, reducing power and cooling requirements, lowering operating costs, and providing additional maneuvering room in our datacenter racks. All production-level servers will also gain additional high-availability functionality. In addition, we will be migrating all of the production servers that now live on the VMware ESX 2.x/3.x environments to the new VSphere platform. Finally, any data housed on the existing MPC/Lefthand iSCSI SAN will be migrated to the Dell Equallogic SANs. The MPC units will then be removed from service, which will complete Phase 1 of this 3 phase project (for a review of the different phases and goals, see the NGDC wiki, which is linked from this previous post:  http://blogs.uis.edu/its/2010/12/09/reminder-next-generation-datacenter-discussion-meeting/)

Email Issues Update

It was determined that a failed hard drive on one of the exchange servers was preventing correct startup.  A manual reboot with manipulations around the failed drive has restored exchange services.

A replacement drive will be requested via dell service today but we expect somewhat slower exchange system preformance until the drive is replaced and the server is brought completely online.

Estimate full restoration of mail services by end of day Tuesday, September 13th, 2011

Email Server Issues

We are currently experiencing issues with the exchange servers.  Problem is unknown but seems to be affecting outlook connections, campus webmail servers appear to still be functional at this time.

We have restarted servers to no effect, technicians are going over to UHB server room to investigate further.

Blackboard Security Issue

We received the following email from Chris Mueller.

I wanted to let everyone know that after several complaints from students of not being able to login to Blackboard, we discovered that if you type in ‘UIS Blackboard’ into Google search, you get taken to a site that looks just like the UIS Blackboard login page but the address is wrong. 

Here is the link Google takes you too: http://www.bubz2020.appspot.com/bb.uis.edu  

Sending out an e-mail of some kind might be necessary. I’ve told the student workers here tonight that if they receive any more phone calls, to have the student reset their password.

I have checked the link and it looks exactly like our blackboard site. I am sure they are probably logging passwords when someone logs in. This is a severe security risk. Someone should probably send out a message to the campus warning them of the situation.

Pharos

The Pharos print server is down. The server service had stopped earlier today and in an attempt to get it to restart, I restarted the Pharos server. However, the server did not restart and now won’t restart. I contacted Curtisand he is going to look at it when he gets off the road. Pam R.