How To Monitor And Access Logs On ORS

How To Monitor Your ORS
  • Last Update:2024-06-17
  • Version:003
  • Language:en

Requirements

This tutorial will show you how to monitor your ORS and access radio / core networks logs (and more) on the Rapid.Space panel.

Monitor your ORS - Monitor Setup URL

To investigate potential errors with the services or to get statistics you can use our monitoring app. Please go to your service page (by clicking on the service in the service section) and find the "monitor-setup-url" connection parameter, as shown on the screenshot. If the parameter is not present, please wait for 20 minutes after starting the service.

 

Monitor your ORS - Synchronization

Monitor your ORS - Synchronization

You will notice that when you enter the monitor App, the information of your ORS has been filled in automatically and started synchronisation. Once synchronisation finishes, the Monitor will load and display the list of Promises for your hosting subscriptions. 

Before we get into the promise list, let's have a look at the App. The Monitor has a multiple-functions side panel:

  • Promises: all promises of all services on all computers.
  • Software Instances: all instances (hosting subscriptions) of all services on all computers.
  • Instance Trees: all services from all computers (similar to Rapid.Space Panel).
  • Monitor Configurations: all data sources (instances), loaded from master.
  • Syncronize: update the monitor data by querying all services and crawling data.
  • OPML Import / Export: used to import and export the current configuration.

For instance you can click on "Software Instances" and then on the enb instance to get statistics regarding eNB.

Monitor your ORS - Promise List

Here you can see the monitor page for the enb instance. On this page you can see the list of promises for this instance. Each promises checks a specific part of the instance is working correctly. Promises will appear orange or red if they are failing, which can help you investigate why your ORS is not working properly. Promises will also give you useful data.

Promises in Rapid.Space are executables doing arbitrary tasks and exiting with exit code 0 ("it works") or greater ("it doesn't work"). Everything in Rapid.Space is based on such Promises in order to automate the management of a Rapid.Space network. In case a Promise fails (eg. a computer not responding), a ticket will be created on the Rapid.Space Master in order for a user to follow up with this failing promise (more info in Understanding SlapOS Promises).

There are 3 statuses for the promise status: error, warning and OK. An error indicates that the promise has failed, a warning indicates that the Monitor App is having trouble retrieving the promise status (e.g. the server is offline), and OK indicates that the promise is functioning properly.

Here is a thorough explanation about some promises in Error, especially for the case of RUs: 

RU*_config_log

Promise source code: check_lopcomm_config_log.py

Common causes: Netconf connection lost; cu_config.xml is improperly configured (e.g., setting out-of-range frequencies).

Solution: Refer to the related RU*-config.log (see "Access ORS log" section) in the private log to debug the causes. If the logs indicate a connection loss, check for CPRI locks and ensure the RU gets an IPv6 address from the BBU. If the edit-config RPC request is unsuccessful, adjust the user input on the panel accordingly. If you need further assistance, contact the Rapid.Space Team.

RU*_cpri_lock

Promise source code: check_cpri_lock.py

Common causes: Hardware issues (e.g., disconnected hardware, unplugged cables, RU powered off); software issues (e.g., failed frame synchronization).

Solution: If "HW Lock is missing", check the physical connection between the RU and BBU. If "SW Lock is missing", which rarely happens, it indicates frame loss. Contact Rapid.Space for assistance.

RU*_firmware

Promise source code: Unavailable

Common causes: The RU is running unverified firmware.

Solution: Provide the correct SSH key to the RU to enable firmware download and upgrade from the BBU.

RU*_lof

Promise source code: check_lopcomm_lof.py

Common causes: Loss of frame.

Solution: Follow the same steps as for RU*_cpri_lock when "SW Lock is missing".

RU*_netconf_connection

Promise source code: Unavailable

Common causes: Netconf connection lost.

Solution: Check for CPRI locks and ensure the RU gets an IPv6 address from the BBU.

RU*_netconf_socket

Promise source code: Unavailable

Common causes: Netconf connection lost; the RU is not listening for Netconf.

Solution: Check for CPRI locks and ensure the RU gets an IPv6 address from the BBU.

RU*_pa_current

Promise source code: check_lopcomm_pa_current.py

Common causes: RU's PA over current.

Solution: Contact Rapid.Space support.

RU*_pa_output_power

Promise source code: check_lopcomm_pa_output_power.py

Common causes: RU's PA Over Output Power.

Solution: Contact Rapid.Space support.

RU*_rssi

Promise source code: test_check_lopcomm_rssi.py

Common causes: RU's RSSI imbalance; RX diversity lost.

Solution: Contact Rapid.Space support.

RU*_rx_saturated

Promise source code: check_rx_saturated.py

Common causes: RU's RX antennas saturated.

Solution: Contact Rapid.Space support.

RU*_sdr_busy

Promise source code: check_sdr_busy.py

Common causes: ENB doesn't properly use the CPRI card.

Solution: Refer to the related enb-output.log (see "Access ORS log" section) in the private log to debug the causes. Possible issues include uninitialized trx_sdr kernel, CPRI card in use by another process, incorrect SPF port, or missing license to launch LTEENB. Contact Rapid.Space for assistance if needed.

RU*_stats_log

Promise source code: check_lopcomm_stats_log.py

Common causes: Netconf connection lost; subscription for notification from RU failed.

Solution: Refer to the related RU*-stats.log (see "Access ORS log" section) in the private log to debug the causes. Check for CPRI locks and ensure the RU gets an IPv6 address from the BBU. Contact Rapid.Space for further assistance if needed.

RU*_sync

Promise source code: check_lopcomm_sync.py

Common causes: Similar to RU*_cpri_lock.

Solution: Similar to RU*_cpri_lock.

RU*_vswr

Promise source code: check_lopcomm_sync.py

Common causes: RU's VSWR alarm.

Solution: Ensure antennas are connected. Try rebooting the RU to clear the alarm. Contact Lopcomm for help if necessary.

amarisoft_stats_log

Promise source code: check_sdr_busy.py

Common causes: ENB doesn't properly use the CPRI card.

Solution: Refer to the related enb-output.log (see "Access ORS log" section) in the private log to debug the causes. Potential issues include uninitialized trx_sdr kernel, CPRI card in use by another process, incorrect SPF port, or missing license to launch LTEENB. Contact Rapid.Space for assistance if needed.

buildout_slappart*_status

Promise source code: Unavailable

Common causes: Fault in the software's buildout.

Solution: Contact Rapid.Space for a patch.

check_baseband_latency

Promise source code: check_baseband_latency.py

Common causes: Insufficient processing time for LTEENB due to other processes consuming too much CPU on the server (ORS/BBU).

Solution: If you have access to the server, identify and resolve the disturbing process. Otherwise, contact Rapid.Space for help.

check_monitor_frontend_password

Promise source code: Unavailable

Common causes: monitor-setup-url with username and password cannot be accessed.

Solution: Ensure the server is online. Contact Rapid.Space for assistance.

monitor_bootstrap_status

Promise source code: monitor_bootstrap_status.py

Common causes: Fault in the software or request parameters.

Solution: Contact Rapid.Space for help.

sshd

Promise source code: Unavailable

Common causes: sshd on BBU for RU to download the firmware is unavailable.

Solution: Contact Rapid.Space for help.

monitor_httpd_listening_on_tcp

Promise source code: Unavailable

Common causes: Server's IPv6 is not accessible.

Solution: Check the server's connection.

monitor_http_frontend

Promise source code: Unavailable

Common causes: Monitor frontend URL is not ready.

Solution: Check the frontend server.

Health instance checks the computer health (CPU, memory, internet connection, etc.)

check_cpu_temperature

Promise source code: check_cpu_temperature.py

Common causes: CPU temperature too high.

Solution: Check the device's environment.

check_cpu_load

Promise source code: check_server_cpu_load.py

Common causes: CPU overload.

Solution: Check running processes on the server.

check_free_disk_space

Promise source code: check_free_disk_space.py

Common causes: Insufficient disk space on the server.

Solution: If you have access to the server, free up some space. Otherwise, contact Rapid.Space for help.

check_network_errors

Promise source code: check_network_errors_packets.py

Common causes: Network packet loss.

Solution: Check the server's connection.

check_partition_space

Promise source code: Unavailable

Common causes: Server's IPv6 is not accessible.

Solution: Check the server's partition Usage. Contact Rapid.Space for help.

check_ram_usage

Promise source code: check_ram_usage.py

Common causes: High RAM usage.

Solution: Check the server's RAM usage. Contact Rapid.Space for help.

check_re6stnet_certificate

Promise source code: Unavailable

Common causes: Re6stnet certificate expired.

Solution: Contact Rapid.Space for help.

check_network_transit

Promise source code: check_network_transit.py

Common causes: Network congestion.

Solution: Check the server's connection. Contact Rapid.Space for help if needed.

check_disk_space

Promise source code: Unavailable

Common causes: Same as check_free_disk_space.

Solution: Same as check_free_disk_space.

Monitor your ORS - Access ORS log

Please click on the "access private files" link to access the ORS logs.

Monitor your ORS - Access ORS logs

Next you can go the log folder.

Monitor your ORS - Access ORS logs

Here are some useful entries from the log folder:

  • amarisoft-stats.json.log: JSON dump of all radio statistics, such as RF power levels, saturation events etc...
  • enb-output.log: Standard output from Amarisoft lteenb software
  • enb.log: Log from Amarisoft lteenb software
  • monitor: Logs from the promises, can contain useful data

To learn how to interpret the logs, please refer to How To Interpret ORS Logs.

Monitor your ORS - Access ORS logs

After navigating to the monitor folder and then the promise folder, you will find logs from the promises. For instance, the "check-rx-saturated" JSON log will contain data about the RF levels from the RX antennas, which can be used to adjust rx_gain.

Monitor your ORS - Access ORS logs

The sample values should ideally be below -10, if they are too high then the reception will be saturated which can prevent the ORS from correctly receiving radio.