Dataset Troubleshooting

Table of contents

• Common issues and solutions
  • Real-Time Dataset update issues
  • Static Dataset update issues
• Dataset management
• Special Dataset requests
• Prevention tips
• Contacting support

If you are having trouble understanding the language in this document, please reference the Glossary.

---

Common issues and solutions

Real-Time Dataset update issues

Symptoms

• Datasets not updating since a specific date
• Missing recent data
• Real-time visualizations showing outdated information
• Black sections on visualizations

Check firewall settings

1. Ensure the following IPs are allowed through your network firewall:
   • ftp.sos.noaa.gov: 137.75.88.43
   • public.sos.noaa.gov: 137.75.88.60
2. See our Firewall Configuration & Troubleshooting Guide for more information.

Verify disk space

1. Check available space in /shared directory using the command df -h in a terminal or in the Disks application.
2. Remove old backups if necessary.
3. Ensure lftpNomads.sh has sufficient space to run.
   • The SOS dataset database needs about 500GB-600GB of storage for the entire catalog, and is stored on your secondary "SHARED" hard drive under the folder location /shared

Script issues

1. Check if lftpNomads.sh process is hanging:
   • Check for existing lock files using the command ps -ef | grep lftpNomads.
     • If you see any entries here, the process is running or may be stuck.
2. Restart the SOS system.
3. Check the following day to confirm that your machine is now getting up-to-date data frames.
   • Make sure to choose a Real-Time Dataset, for example Real-Time Clouds.
4. Contact support if issues persist.

Static Dataset update issues

Symptoms

• FTP connection issues
• Downloads hanging or timing out
• Missing datasets
• Incomplete downloads

Missing Datasets

1. Datasets should automatically update each Monday overnight.
   • Ensure you have stable network connection to the internet.
   • Ensure you have sufficient space to download all datasets as detailed above.
2. In the SOS Stream GUI application window, choose Download All Missing Datasets from the top File menu.
   • This process can take anywhere from 6-12 hours.
   • This should not interrupt the running process of your sphere, but it may be best to schedule it overnight.

FTP access problems

1. Verify network connectivity to the internet by opening any website.
2. Check for institutional firewall restrictions that are blocking SOS computers as detailed above.

Script issues

1. Check if lftp_playlist.sh process is hanging:
   • Check for existing lock files using the command ps -ef | grep lftp_playlist.
     • If you see any entries here, the process is running or may be stuck.
2. Restart the SOS system.
3. In the SOS Stream GUI application window, choose Download All Missing Datasets from the top File menu.
4. Contact support if issues persist.

---

Dataset management

Regular updates:

1. Schedule routine checks to confirm static datasets are updating.
   • Staying current with the SOS Network's email updates will let you know of new static dataset releases.
     • You can also check our website for new datasets, and then confirm they are on your computers.
     • Reach out to our support team to be added to this network if you are not a part of it currently.
2. Monitor the real-time dataset schedule, and confirm you have up to date images regularly.
   • Keep track of last successful update dates by checking the final frame on any Real-Time Dataset on your sphere.

Storage management:

1. Maintain adequate free space on your secondary "SHARED" hard drive.
   • You can check how much space you have available using the command df -h.
2. Regularly clean up old or unused custom datasets.
3. Custom Datasets should only be placed in the location /shared/sos/media/site-custom/ on your SOS computer.

System maintenance:

1. Restart SOS system periodically.
2. Keep software up to date.

---

Special Dataset requests

Under certain scenarios, the SOS Team does field specific dataset requests for new dataset ideas, as well as help investigate the process for adding your own custom datasets to our global catalog for all sites to use. If providing a Real-Time Dataset, please provide the associated code / scripts. Please reach out to us if any of these interest you.

Process for suggesting or providing custom Datasets

1. Test your new dataset thoroughly.
2. Contact the SOS support team.
3. Provide detailed requirements to load or run the dataset.
4. Allow sufficient time for processing.

The SOS Team archives all Datasets and can make specific time frame Datasets available upon request (if available). If you'd like one of the current Real-Time Datasets to be time specific (ex. a hurricane event that affected your location), please reach out to us.

Available Dataset types

1. Real-time weather and climate data
2. Historical earth science data
3. Custom visualization datasets
4. Specialized research data

---

Issue prevention tips

Regular maintenance

1. Monitor disk space usage using the command df -h.
2. Verify real-time dataset functionality by loading a dataset and confirming the time stamps are current.
3. Test visualization quality and consistency regularly by visual inspection.

Network configuration

1. Maintain updated firewall rules.
2. Document network changes.
3. Keep IP allowlist current if you have one.

---

Contacting support

If issues persist after trying these solutions, contact SOS support at [e-mail] with the following information:

• SOS version number
• iPad app version number
• iOS version
• Detailed description of the issue
• Screenshots if applicable
• Steps already attempted to resolve the issue

For urgent hardware issues:

• Document specific error messages
• Provide system specifications
• Include recent system changes
• Prepare TeamViewer access capability (if available)