Update How to set up MLflow authored by David Benjamin Goll's avatar David Benjamin Goll
...@@ -2,6 +2,7 @@ ...@@ -2,6 +2,7 @@
title: How to set up MLflow title: How to set up MLflow
--- ---
# How to set up MLFlow on the cluster (with VSCode) # How to set up MLFlow on the cluster (with VSCode)
*Work in progress* *Work in progress*
...@@ -20,6 +21,8 @@ module load anaconda # to load conda ...@@ -20,6 +21,8 @@ module load anaconda # to load conda
conda create -n mlflow-env python=3.10 # name your environment as you like. I chose "mlflow-env" conda create -n mlflow-env python=3.10 # name your environment as you like. I chose "mlflow-env"
source activate mlflow-env # activate your environment source activate mlflow-env # activate your environment
``` ```
> **Note:** Instead of anaconda and pip, you could also use uv, to set up and manage your environment. Click [here](https://gitlab.pik-potsdam.de/sdmc/cluster-wiki/-/wikis/Tutorials/UV-for-Python), to find out more.
- Install MLFlow, ipykernel, and other Python packages you need: <br> - Install MLFlow, ipykernel, and other Python packages you need: <br>
* `pip install mlflow` Note: This will also install `numpy, pandas, scikit-learn` and others <br> * `pip install mlflow` Note: This will also install `numpy, pandas, scikit-learn` and others <br>
* `pip install ipykernel` To run jupyter notebooks in VSCode <br> * `pip install ipykernel` To run jupyter notebooks in VSCode <br>
...@@ -56,9 +59,23 @@ If this is not set within your notebook or runtime environment, the runs will be ...@@ -56,9 +59,23 @@ If this is not set within your notebook or runtime environment, the runs will be
3. Run your ML run (e.g. the quickstart notebook) and look at your logged data in the mlflow UI: <br> 3. Run your ML run (e.g. the quickstart notebook) and look at your logged data in the mlflow UI: <br>
## Forward a port from the cluster to your local browser to view the MLFlow UI ## Open the MLflow UI
You should be able to start the mlflow ui via the command
```bash
mlflow ui
```
or `uv run mlflow ui` (if you use uv). By default this will run the ui on the local web server [http://127.0.0.1:5000](http://127.0.0.1:5000). If the connection is already in use, you need to manually set a different port, e.g. 5001 instead of 5000:
To view the UI, you must create a secure tunnel from your local localhost:YourPortNumber (on your laptop) to the remote 127.0.0.1:YourPortNumber (on the cluster). In VScode: ```bash
mlflow ui --host 127.0.0.1 --port 5001
```
If you did not set a server, new runs or file changes in mlruns/ may not appear immediately. To refresh, you may need to restart the mlflow ui process.
If there are any other problems with `mlflow ui` on the cluster, you can alternatively create a secure tunnel from your local localhost:YourPortNumber (on your laptop) to the remote 127.0.0.1:YourPortNumber (on the cluster). In VScode:
* Open the Command Palette: Ctrl + Shift + P (or Cmd + Shift + P on Mac) * Open the Command Palette: Ctrl + Shift + P (or Cmd + Shift + P on Mac)
* "Forward a Port": When prompted, enter: YourPortNumber e.g. 8081 * "Forward a Port": When prompted, enter: YourPortNumber e.g. 8081
* VS Code will now show a forwarded port in the “Ports” panel at the bottom. Click on the globe symbol to see the UI in your local browser. * VS Code will now show a forwarded port in the “Ports” panel at the bottom. Click on the globe symbol to see the UI in your local browser.
...@@ -68,13 +85,6 @@ To view the UI, you must create a secure tunnel from your local localhost:YourPo ...@@ -68,13 +85,6 @@ To view the UI, you must create a secure tunnel from your local localhost:YourPo
![image](uploads/da29e2267a664335a0a57ab9a2847ba0/image.png){width=1438 height=523} ![image](uploads/da29e2267a664335a0a57ab9a2847ba0/image.png){width=1438 height=523}
If you did not set up a tracking server, you can access the MLFlow UI simply via this command:
`mlflow ui --port 8081` (this is equivalent to `mlflow ui --host 127.0.0.1 --port 8081`)
Then forward the prompt to your local computer (see the steps above).
Note: The UI in this mode is read-only, and new runs or file changes in mlruns/ may not appear immediately. To refresh, you may need to restart the mlflow ui process.