Skip to main content

Hardware Troubleshooting

Alex Kirst
Updated

Table of contents

If you are having trouble understanding the language in this document, please reference the Glossary.

Common issues and solutions

Graphics card issues

Symptoms

  • Projector display issues
  • Missing or blank displays
  • System overheating
  • Non-functioning video ports
  • Graphics card fan failure

Confirm port functionality

  1. Verify all video ports are functional.
    • Each port on your graphics card drives one projector output. If you notice that one projector is failing to output, try swapping a known working projector to the port that is having the issue. If the issue persists on the new projector, the graphics card may be faulty and need to be replaced.
  2. When using HDMI, ensure adapters are "Active" rather than "Passive".
    • If there are any identifying markers (model number, serial number) on your cable adapters, you can search for them online to determine if they are "passive" or "active".
    • For Display Port to HDMI conversions, use active adapters only, and confirm that the cables are not "unidirectional".

Replace failed graphics cards

Hard drive failures

Symptoms

  • System won't boot
  • Kernel errors
  • Slow performance
  • Dataset having trouble loading
  • System freezes

Important: Confirm backups of all of your site-custom datasets and playlists exist

  1. These are located at /shared/sos/site-backup.sos1/. You'll want to confirm that your important data is safe, namely files under /home/sos/sosrc, /home/sosdemo/sosrc/, and /shared/sos/media/site-custom/.
  2. If your backups are out of date, please make manual backups of your site-custom datasets and playlists.

Check repair and recovery options

  1. Check system logs for errors
    • If familiar with system logs, Ubuntu comes preinstalled with an application called Logs. If you look through the Hardware section of this application, you may find errors related to hard drive operation.
  2. Run disk health diagnostics
    • Open the Disks application on your SOS machine, and navigate to each of your hard drives on the left hand side. Click the vertical "..." icon in the top right, and choose the "SMART data and self-tests" option. If errors occur here, there may be failing hard drive sectors and require hard drive replacement.
  3. Recovery options
    • Consider upgrading to upgrading to Solid State Drives (SSD) for better reliability.
    • For complete failure:
      • Install a fresh Ubuntu OS with the help of the SOS Support Team.
      • Reinstall SOS software with the help of the SOS Support Team.
      • Restore backed-up datasets and playlists.
  4. Preventive measures
    • Maintain regular backups.
      • These should be automatic, but confirming they exist under the folder /shared/sos/site-backup.sos1 should be done regularly.
    • Monitor disk health.
      • See "Run disk health diagnostics" detailed above.
    • Consider dual-drive setup (500GB boot drive + 1TB shared drive).
      • This is our standard setup configuration. If the main boot drive fails, your custom data and settings are left untouched in case of failure.

Boot-related issues

Symptoms

  • System stuck at boot screen
  • Kernel panic errors during startup
  • Black/blank screen after booting
  • Server termination errors

Check hardware connections

  1. Verify RAM seating.
    • Disconnecting and reconnecting your RAM chips from the motherboard may help.
  2. Confirm hard drive cables are connected properly within the SOS computer.

Remove the Nvidia driver

  • Boot into recovery mode if possible, or connect through SSH from another machine.
    • If able to boot into recovery mode, try uninstalling your Nvidia drivers. The best way to do this is to open b. Terminal window and run the command sudo apt-get purge *nvidia* && sudo apt-get autoremove. Then try to boot as normal.
      • SOS requires an Nvidia driver to be installed, so you'll want to reinstall the driver once you've booted up successfully.

Power-related issues

Symptoms

  • Sudden shutdowns
  • System instability
  • Failed boot attempts
  • Component failures after power events

After power events

  1. Check all power connections.
    • If you're using a power strip for your SOS machine, try to bypass that and plug your machine directly into the wall outlet.
  2. Verify UPS functionality if present.
    • There should be status indicators on the front screen of this device. If any are red or blinking, you may have a UPS failure.
  3. Test system stability after restart by running the SOS software for some time.

Preventive measures

  1. Install UPS systems.
    • These systems help to maintain system power in the event of a power loss event at your location. These are essentially large backup batteries that help to keep your infrastructure running.
  2. Document proper shutdown procedures.
    • Disconnecting the power cable while the machine is running could lead to software issues that prevent the system from booting. Only do this if absolutely necessary, and the machine will not shut down using the standard Shut Down operation.

Performance issues

Symptoms

  • Slow dataset loading
  • System lag
  • Long boot times
  • Delayed response to controls

Performance optimization

  1. Verify disk space availability on your OS hard drive.
    • You can see your available Hard Drive space using the Disks application on your SOS computer.
  2. Check system resources usage.
    • Open the application System Monitor on your SOS machine and verify that no resources are exceeding 80%.
  3. Monitor temperature levels in the System Monitor application or by running the nvidia-settings -c :2 command in a terminal.

System maintenance

  1. Regular system updates.
    • These can be done through the Software Updater application, built into Ubuntu. These are all safe to do on a regular basis. Please refrain from updating to a new version of the Ubuntu operating system (i.e Ubuntu 24.04) without first contacting SOS Support, as this will most likely prevent your SOS application from running.
  2. Remove unnecessary applications.
  3. Schedule regular reboots.
    • We do recommend keeping your SOS machine on most of the time, but after any updates it is a good idea to reboot the system so that they take effect.

Replace failing components

  1. Most commonly, performance issues are a symptom of failing hardware.
  2. Replace failing hard drives, including both the boot drive and "SHARED" drive.
  3. Replace failing graphics cards.
    • Removing the graphics cards from your system will help determine if they are the cause. Sometimes failing graphics cards will slow the system to a crawl.

 

Emergency procedures

  1. System failure
    • Switch to backup system if available.
      • Most sites have an identical Hot Swap computer, SOS2. In the event of a system failure, please swap all connections to SOS2 and confirm it is working as expected. If the backup system crashes as well, the issue may lie in your network configuration.
    • Document error messages.
      • Take screenshots of errors to provide to Support.
    • Contact support with detailed information.
  2. Data recovery
    • Maintain offline backups.
      • Keeping a copy of your /shared/sos/site-backup.sos1/ folder somewhere other than the SOS computer itself will help to recover data in case of a hard drive failure.
    • Keep spare hardware available.
      • The most important items that are at risk of failure are the graphics cards and the hard drives.
    • Test recovery processes regularly and confirm your backups are current.

 

Best practices

Backup system management

  1. Maintain functioning backup system.
    • This is your Hot Swap computer, usually named SOS2.
  2. Regular testing of backup hardware.
    • Confirming that your backup SOS2 computer is functional on a regular basis will help in the event of a system failure, as connections can quickly be swapped to migrate to the new system.
  3. Keep backup system updated with security patches and SOS software updates.

Documentation

  1. Keep hardware specifications of your SOS computer recorded.
    • Graphics card models
    • Dell computer model (ex. Precision Tower 5860)
  2. Document all system changes.
    • Software updates, Hardware replacements.
  3. Maintain software update history.
  4. Record custom configurations.
    • These can include custom Kiosk configurations located at /shared/sos/kiosk/.

 

Hardware tips and recommendations

Graphics cards

  1. Please see our current recommendations here:

Storage

  1. Consider SSD upgrades.
  2. Maintain separate boot/data drives, as is standard in SOS installations.
  3. Practice regular backup solutions outside of the automatic backup.
  4. Monitor drive health using the Disks tool.

Contacting support

If issues persist after trying these solutions, contact SOS support at sos.support@noaa.gov with the following information:

  • SOS version number
  • iPad app version number
  • iOS version
  • Detailed description of the issue
  • Screenshots if applicable
  • Steps already attempted to resolve the issue

For urgent hardware issues:

  • Document specific error messages
  • Provide system specifications
  • Include recent system changes
  • Prepare TeamViewer access capability (if available)

Related articles

Comments

0 comments

Please sign in to leave a comment.

Powered by Zendesk