Miscweb

miscweb is a new service on kubernetes.

Since 2022-01-20 it serves production traffic for static-bugzilla. Other miscweb releases in the same namespace contain different services, like annual.wikimedia.org and transparency.wikimedia.org.

It was requested in task T281538 to replace the legacy service "miscweb" running on Ganeti VMs in production.

Also see: miscweb1002, miscweb2002 for the legacy machines still serving other microsites.

Sites running on miscweb k8s

The first of the sites hosted on miscweb-k8s is static-bugzilla.

Since 2022-01-20 static-bugzilla.wikimedia.org is served from k8s. Other microsites are running on miscweb as well. All services in miscweb:

• annual.wikimedia.org and 15.wikipedia.org
• bienvenida.wikimedia.org
• transparency.wikimedia.org
• static-bugzilla.wikimedia.org
• tendril.wikimedia.org and dbtree.wikimedia.org
• research.wikimedia.org
• wikiworkshop.org
• static-codereview.wikimedia.org/
• design.wikimedia.org

Where does the code live?

Most code for the html files and blubber container images is hosted under https://gitlab.wikimedia.org/repos/sre/miscweb. Each service has a project there. The projects contain 4 important files and folders:

• .pipeline folder contains blubber configuration for the image build
• html folder contains the static html which is served by the container image. Note: this folder is also called html-compressed for big sites which use lfs and compressed storage of html files.
• production folder contains the apache configuration for the apache2 process in the container
• .gitlab-ci.yml contains the CI configuration for building and publishing the image to our image registry.

The helm charts for kubernetes can be found in operations/deployment-charts.

How to deploy changes

Most service changes contain of updates of the used container image and updates to the Kubernetes helm deployment.

#13 pushing layers
#13 pushing layers 4.9s done
13# pushing manifest for docker-registry.discovery.wmnet/repos/sre/miscweb/annualreport:2023-08-14-083827
77bf6661c95d2ee40d007b0e9 0.6s done
#13 DONE 10.2s

Service names

miscweb.svc.eqiad.wmnet has address 10.2.2.58  (eqiad)
miscweb.svc.codfw.wmnet has address 10.2.1.58  (codfw)

miscweb.discovery.wmnet has address 10.2.2.58  (DNS/Discovery)

LVS / discovery

https://config-master.wikimedia.org/pybal/eqiad/miscweb

https://config-master.wikimedia.org/pybal/codfw/miscweb

https://config-master.wikimedia.org/discovery/

Metrics and dashboards

https://grafana.wikimedia.org/d/exdTE7kSk/miscweb?orgId=1

How this service was made

Here I am trying to compile a table / list of all the changes made to get this service from scratch into WMF production, in chronological order of how they were merged.

How to switch microsites hosted on legacy VMs from one datacenter to another

This section is not about the kubernetes service but about the ganeti VMs hosting microsites that have not moved yet.

This is how to switch those from one DC to another:

1. Check the google doc "sre collab service matrix" to see which services are hosted on miscweb* and pick one to switch
2. go to the puppet repo and find your service in the file common/profile/trafficserver/backend.yaml
3. confirm that it points to the name "https://webserver-misc-sites.discovery.wmnet"
4. look in the DNS repository in the file ./templates/wmnet and find this section:

;webserver-misc-sites  300 IN CNAME miscweb1003.eqiad.wmnet.
webserver-misc-sites 300 IN CNAME miscweb2003.codfw.wmnet.

1. Now decide to either switch ALL sites at once and switch around which server is commented out and which is not
   1. or introduce a different name, like webserver-misc-sites but separately, that points to miscweb1003 instead of miscweb2003 (or vice versa)
   2. if you are doing the second, go back to the ATS config (backend.yaml) mentioned above and switch just one of the sites to your new DNS discovery name
2. run puppet on all cp* hosts via cumin, or just on the hosts for your data center (ex. "cp4*" for ulsfo) or wait half an hour
3. on old and new miscweb machine do a tail -f /var/log/apache2/*.log and watch the access logs, each site has its own access and error log file
4. open the site in your browser and confirm the request hits your new backend and the site looks like before
5. run a httpbb test against your new (or old and new) server and confirm there are no errors or differences (from a deployment server, example:)

   1. [deploy1002:~] $ httpbb --hosts miscweb1003.eqiad.wmnet /srv/deployment/httpbb-tests/miscweb/test_miscweb.yaml

6. mark the site as switched in doc/ticket. !log in SAL
7. double check sites that have external deployers (deploy method column in the service matrix doc)