On-premise solution for monitoring isolated clusters been established. The various metrics from every node is collected with the collectd package, which sending the data to the Grafana monitoring server, where the data is being stored in the influxdb to be displayed with Grafana.

The firewall rules were enabled to allow static routing on UPD 25826 port to make possible for isolated clusters to communicate with the monitoring server.

Also Grafana was configured to send alerting messages to the dedicated Slack channel in the case of potential node health issues in real time. The messages could be configured to be sent to different devices – smartphone or desktop.

I have been involved in development and support of both types of Web Servers – dynamic and static.

The former involves from big projects with Django as CMS with University HPC cluster as the backend, and small web-apps powered by Flask.

1. webserver for Protein Simulation GPCR-MODSIM
2. internal Flask apps

The latter involves the Jekyll-generated websites:

1. Combient AB website
2. personal webpage

The ReadTheDocs is online service which allows to host docs for free. But what if the docs are for internal use or describing the commercial project?

Here comes the localized or forked version of ReadTheDocs! It gives all the advantages but keep your code and docs in private, allowing access only from the inside the company’s network.

Sometimes public cloud is not a decent solution for your needs. When instead a bunch of on-premises clusters could be an option.

The distinguished features are:

1. the clusters, being on different VLANs, are isolated from each other by default. However, that could be tuned immediately by changing the rules on the firewall. This allows to scale up and down the cluster, but in a blob manner, which is not an option for most of public clouds

2. the local DNS service is enabled for all the clusters globally. The cron job performs scheduled scans of clusters, creating a global DNS table, served by DNSmasq. So when allowed all the hosts on all clusters can talk to each other by names, not IPs.

3. the provision of the nodes is done through PXE-booting with Cobbler using the local mirror for Linux distributions and packages

4. the remote access is established via secure VPN connection

The clusters are being established and provisioned in a largely automatic manner, which leads to reproducibility and fast deployment time.

The Open Nebula is a solid approach to orchestrate a bunch of virtual machines (VMs). Cloud helps to create and provision VMs carrying the needed services, which are, perhaps, not very suitable for running in Docker containers.

The setup is built on Intel NUCs and configured to operate in Highly Available mode with 3 front-ends, meaning that the service functionality will remain, even:

1. if one of the front-ends degrades
2. the worker NUC or the VM running inside will fail

The storage is NFS-attached from the Synology NAS solution (RAID5).

Hadoop/Spark cluster (50 CPU cores/200 GB RAM/ 40 TB disk space) is suitable for solving medium size tasks in Data Science, Bioinformatics and similar data intensive domains.

Neglecting a separate fat (128 GB RAM, 20 cores) node, the cluster setup is pretty compact, and can reside on the desk, still demanding less then 1 kW of energy and being reasonably quite even under the high load.

The cluster expansion is easy due to user-friendly Apache Ambari. The NUCs are supplied with 1 Gbit NIC, so USB3-connected HDDs are not the performance bottleneck. The portable HDDs, even not providing the top performance, are truly easy to change and add new ones, reaching 80-100 TB.

Spark is being utilized by the Machine Learning tasks. The Jupyter web-based Notebook is running on the fat node and used for coding. The jobs can be:

1. submitted to run on the cluster
2. running locally on the fat node

The Jupyter kernel is configured so that the Spark driver resides on the fat node with 128 GB RAM, helping Data Scientists make interactive analysis in the Jupyter notebook.