Portal:Cloud VPS/Admin/Haproxy
This page contains information about our HAProxy configuration in the CloudVPS environment.
OpenStack API Endpoints
Backend HA Status
Backend services that are full active / active will automatically be pooled and depooled by HAProxy. As long as there is one instance available there will be no interruption in service.
EQIAD1
| Service | active backends | standby backends | notes |
|---|---|---|---|
| Designate | cloudservices1003 cloudservices1004 | NA | |
| Glance API | cloudcontrol1003 | cloudcontrol1004 | Waiting on shared storage (CEPH) for full active/active |
| Glance Registry | cloudcontrol1003 | cloudcontrol1004 | Waiting on shared storage (CEPH) for full active/active |
| Keystone | cloudcontrol1003 cloudcontrol1004 | NA | |
| Neutron | cloudcontrol1003 cloudcontrol1004 | NA | |
| Nova | cloudcontrol1003 cloudcontrol1004 | NA | |
Port Mappings
| Service | Backend Port | Frontend Port |
|---|---|---|
| Designate API | 9001 | 9001 |
| Glance API | 19292 | 9292 |
| Glance Registry | 19191 | 9191 |
| Keystone Admin | 15000 | 5000 |
| Keystone Public | 45357 | 35357 |
| Neutron | 19696 | 9696 |
| Nova API | 18774 | 8774 |
| Nova Metadata | 18775 | 8775 |
Load Balancing and Session Handling
Backend connections for each service are dynamically directed to the server with the least amount of connections using the `leastconn` load balancing algorithm.
Session persistence is maintained with HAProxy stick tables using the IP address of the client. This will ensure new client connections get forwarded to the same server every time.
Monitoring
Icinga
Icinga is configured with the following HAProxy process checks:
- nrpe_check!check_haproxy_alive!10
- nrpe_check!check_haproxy!10
Prometheus
The prometheus haproxy exporter is collecting haproxy backend and frontend metrics directly from haproxy through the URL http://localhost:9900/stats;csv. This URL is read-only and only provides statistic information through the loopback interface.
Grafana dashboard: https://grafana.wikimedia.org/d/tanisM2Zz/wmcs-openstack-eqiad1-api-stats
Troubleshooting
Monitoring
HAProxy statistics and metrics can be queried through the "stats socket" at `/run/haproxy/haproxy.sock`.
Process information:
echo "show info" | sudo socat /run/haproxy/haproxy.sock stdio
Statistics:
echo "show stat" | sudo socat /run/haproxy/haproxy.sock stdio
Log files
- /var/log/haproxy/haproxy.log
Primary Failover
Design
Each controller is running HAProxy with no cross dependencies on other controllers or HAProxy instances. Each load balancer is operating on its own with no shared information like client sessions between them.
DNS
Each region has a dedicated FQDN (openstack.<region>.wikimediacloud.org) which maps to one of the OpenStack controllers running HAProxy. In the event of failure or maintenance this DNS entry will need to be remapped to the other controller in the operations/dns git repository at https://gerrit.wikimedia.org/r/admin/projects/operations/dns
