From a755758dcf05e6fc0705b2908b75ecbebe7b0e69 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Mon, 1 Sep 2025 20:06:52 +0200 Subject: [PATCH 01/16] start updating production deployment section (WIP) --- docs/deployment/production-deployment.rst | 64 ++++++++++++++++++++--- 1 file changed, 58 insertions(+), 6 deletions(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 75dcf97..1b2a4a3 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -4,14 +4,66 @@ Production Deployment ********************* -If you want to run the FAIR Data Point in production it is recommended to use HTTPS protocol with valid certificates. -You can easily configure FDP to run behind a reverse proxy which takes care of the certificates. +Disclaimer +========== -In this example, we will configure FDP to run on ``https://fdp.example.com``. -We will see how to configure the reverse proxy in the same Docker Compose file. -However, it is not necessary, and the proxy can be configured elsewhere. +Running a FAIR Data Point in production is a bit more involved than running one offline on your development machine. +The configuration details of a production deployments depend on many factors, such as available resources and security requirements. +Whether you're setting up your own bare metal server or using a cloud provider that offers all kinds of managed services, you'll need to think about many of the same topics. -First of all, we need to generate the certificates on the server where we want to run the FDP. +Here's just a few topics that come to mind, in no particular order: + +- network security +- secrets storage +- identity and access management (IAM) +- data management (security, privacy, replication, backups) +- service availability (e.g. container orchestration) +- performance +- audit logging +- infrastructure as code +- and so on and so forth... + +Obviously this list is far from exhaustive. + +Due to this complexity we cannot provide a generic solution for a production deployment. +However, we can provide some pointers and suggestions to help you get started. +Assuming basic infrastructure hardening is already in place (see e.g. `OWASP cheat sheets`_), we'll look at a few things: + +- HTTPS (encrypted communication based on TLS) +- database authentication + + +Encrypted communication +======================= + +One of the first requirements for a production deployment is to set up Transport Layer Security (TLS) to provide encrypted communication, better known as HTTPS (HTTP over TLS). + +Database authentication +======================= + +`database security cheat sheet`_ + + + +.. _OWASP cheat sheets: https://cheatsheetseries.owasp.org +.. _: https://cheatsheetseries.owasp.org/cheatsheets/Web_Service_Security_Cheat_Sheet.html +.. _database security cheat sheet: https://cheatsheetseries.owasp.org/cheatsheets/Database_Security_Cheat_Sheet.html +.. _: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html +.. _: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html +.. _: https://ubuntu.com/blog/what-is-system-hardening-definition-and-best-practices + + + +TODO: update the text below + + +If you want to run the FAIR Data Point (FDP) in production we recommend setting up TLS so you can serve all traffic over HTTPS. +It is convenient to configure the FDP to run behind a reverse proxy which takes care of the TLS certificates. + +In this example, we will configure the FDP to run on the domain ``fdp.example.com``. +In this example, we configure the reverse proxy in the same Docker Compose file, but this is not required. + +First of all, we need to generate the TLS certificates on the server that will run the FDP. You can use `Let's Encrypt `__ and create the certificates with `certbot `__. The certificates are generated in a standard location, e.g., ``/etc/letsencrypt/live/fdp.example.com`` for ``fdp.example.com`` domain. We will mount the whole ``letsencrypt`` folder to the reverse proxy container later so that it can use the certificates. From 54ee4f206212da1c2698dec369ea91a120be9288 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Mon, 1 Sep 2025 20:15:28 +0200 Subject: [PATCH 02/16] mention list of secrets etc. --- docs/deployment/production-deployment.rst | 34 ++++++++++++++++------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 1b2a4a3..15b53f9 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -30,28 +30,42 @@ However, we can provide some pointers and suggestions to help you get started. Assuming basic infrastructure hardening is already in place (see e.g. `OWASP cheat sheets`_), we'll look at a few things: - HTTPS (encrypted communication based on TLS) -- database authentication - +- database security +- database backups Encrypted communication ======================= One of the first requirements for a production deployment is to set up Transport Layer Security (TLS) to provide encrypted communication, better known as HTTPS (HTTP over TLS). -Database authentication -======================= +Database security +================= + +- `database security cheat sheet`_ +- `mongodb security cecklist`_ -`database security cheat sheet`_ +Secrets +======= +List of secrets: +- jwt token secret key +- default user accounts +- mongodb credentials +- triple store credentials + +Backups +======= + +(...) .. _OWASP cheat sheets: https://cheatsheetseries.owasp.org -.. _: https://cheatsheetseries.owasp.org/cheatsheets/Web_Service_Security_Cheat_Sheet.html +.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Web_Service_Security_Cheat_Sheet.html .. _database security cheat sheet: https://cheatsheetseries.owasp.org/cheatsheets/Database_Security_Cheat_Sheet.html -.. _: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html -.. _: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html -.. _: https://ubuntu.com/blog/what-is-system-hardening-definition-and-best-practices - +.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html +.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html +.. .. _: https://ubuntu.com/blog/what-is-system-hardening-definition-and-best-practices +.. _mongodb security cecklist: https://www.mongodb.com/docs/manual/administration/security-checklist/ TODO: update the text below From bc2a4e7dd16836d0c6143cd1c4d31c191bd52959 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 16:12:33 +0200 Subject: [PATCH 03/16] add nginx/server.conf and update nginx/compose.yml --- docs/deployment/nginx/compose.yml | 9 +++++++ docs/deployment/nginx/server.conf | 45 +++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 docs/deployment/nginx/compose.yml create mode 100644 docs/deployment/nginx/server.conf diff --git a/docs/deployment/nginx/compose.yml b/docs/deployment/nginx/compose.yml new file mode 100644 index 0000000..23826d8 --- /dev/null +++ b/docs/deployment/nginx/compose.yml @@ -0,0 +1,9 @@ +services: + nginx: + image: nginx + ports: + - "80:80" + - "443:443" + volumes: + - "/etc/letsencrypt:/etc/letsencrypt:ro" + - "server.conf:/etc/nginx/conf.d/server.conf:ro" diff --git a/docs/deployment/nginx/server.conf b/docs/deployment/nginx/server.conf new file mode 100644 index 0000000..72a7c11 --- /dev/null +++ b/docs/deployment/nginx/server.conf @@ -0,0 +1,45 @@ +# allow http traffic from localhost +# (required because the fdp-client service acts as a reverse-proxy for the fdp service) +server { + listen 80; + server_name localhost; + include conf.d/includes/proxy.conf; +} + +# redirect all other traffic from port 80 to port 443 +server { + listen 80; + listen [::]:80; + server_name fdp.example.com; + return 301 https://$host$request_uri; +} + +# listen for tls (ssl) traffic +server { + listen 443 ssl; + listen [::]:443 ssl; + server_name fdp.example.com; + ssl_certificate /etc/letsencrypt/live/fdp.example.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/fdp.example.com/privkey.pem; + + location / { + # pass request to the upstream fdp-client service (on the docker network) + proxy_pass http://fdp-client:80; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} + +# catch all +server { + listen 80 default_server; + listen [::]:80 default_server; + listen 443 ssl default_server; + listen [::]:443 ssl default_server; + ssl_reject_handshake on; + server_name _; # match none + return 444; +} From f2691e1e8e5b92737125eb6c1bb2b1fda7c3b565 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 16:13:52 +0200 Subject: [PATCH 04/16] rewrite nginx sections (WIP) --- docs/deployment/production-deployment.rst | 220 ++++++++-------------- 1 file changed, 82 insertions(+), 138 deletions(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 15b53f9..01b5c04 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -9,9 +9,9 @@ Disclaimer Running a FAIR Data Point in production is a bit more involved than running one offline on your development machine. The configuration details of a production deployments depend on many factors, such as available resources and security requirements. -Whether you're setting up your own bare metal server or using a cloud provider that offers all kinds of managed services, you'll need to think about many of the same topics. -Here's just a few topics that come to mind, in no particular order: +Whether you're setting up your own bare metal server or using a cloud provider that offers all kinds of managed services, you'll need to think about many of the same topics. +Here's just a few that come to mind, in no particular order: - network security - secrets storage @@ -33,174 +33,121 @@ Assuming basic infrastructure hardening is already in place (see e.g. `OWASP che - database security - database backups +These topics are covered by extending the :ref:`local-deployment` examples with some additional configuration. + Encrypted communication ======================= One of the first requirements for a production deployment is to set up Transport Layer Security (TLS) to provide encrypted communication, better known as HTTPS (HTTP over TLS). -Database security -================= - -- `database security cheat sheet`_ -- `mongodb security cecklist`_ - -Secrets -======= - -List of secrets: - -- jwt token secret key -- default user accounts -- mongodb credentials -- triple store credentials +Https connections can be handled, for example, by configuring a load balancer or a reverse proxy. -Backups -======= +As a minimal example, we'll describe how to use Docker compose to configure an ``nginx`` container as reverse proxy that terminates https, on the same host that runs the FDP containers. +In this case, communication between the ``nginx`` container and the upstream web server (either ``fdp`` or ``fdp-client``) still uses plain http, but in our case that occurs on the private Docker network. +It is also possible to configure the ``fdp``'s `embedded web server`_ and the ``fdp-client``'s embedded nginx instance to handle https connections, but that is outside the scope of this document. -(...) +TLS certificates +---------------- -.. _OWASP cheat sheets: https://cheatsheetseries.owasp.org -.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Web_Service_Security_Cheat_Sheet.html -.. _database security cheat sheet: https://cheatsheetseries.owasp.org/cheatsheets/Database_Security_Cheat_Sheet.html -.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html -.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html -.. .. _: https://ubuntu.com/blog/what-is-system-hardening-definition-and-best-practices -.. _mongodb security cecklist: https://www.mongodb.com/docs/manual/administration/security-checklist/ +In order to set up HTTPS, a valid TLS certificate is required (a.k.a. SSL certificate). +For this example, we assume that a TLS certificate is already available, *on the Docker host*, for our domain ``fdp.example.com``. +Certificate files can be obtained from various sources. +Our example assumes that the `certbot`_ tool was used to obtain a certificate from `Let's Encrypt`_. +The certificate file and corresponding key file can then be found in `certbot's default location`_ ``/etc/letsencrypt/live/fdp.example.com`` on the host. -TODO: update the text below +Nginx compose service +--------------------- +A minimal compose service definition for nginx is described below. +Bind mounts are used to make the nginx configuration files and certificates from the Docker host available in the ``nginx`` container. -If you want to run the FAIR Data Point (FDP) in production we recommend setting up TLS so you can serve all traffic over HTTPS. -It is convenient to configure the FDP to run behind a reverse proxy which takes care of the TLS certificates. +.. literalinclude:: nginx/compose.yml + :name: nginx compose config + :caption: minimal nginx service + :language: yaml + :lines: 2- -In this example, we will configure the FDP to run on the domain ``fdp.example.com``. -In this example, we configure the reverse proxy in the same Docker Compose file, but this is not required. +This is just a minimal example, so you may want to specify an image version, adjust the paths where necessary, and/or add some addional config. -First of all, we need to generate the TLS certificates on the server that will run the FDP. -You can use `Let's Encrypt `__ and create the certificates with `certbot `__. -The certificates are generated in a standard location, e.g., ``/etc/letsencrypt/live/fdp.example.com`` for ``fdp.example.com`` domain. -We will mount the whole ``letsencrypt`` folder to the reverse proxy container later so that it can use the certificates. +.. note:: -As a reverse proxy, we will use `nginx `__. -We need to prepare some configuration, so create a new folder called ``nginx`` with the following structure and files: + Here we assume that the ``nginx`` container shares a Docker network with the ``fdp-client`` container. + Only the ``nginx`` container ports (``80`` and ``443``) should be exposed to the public internet. + Make sure to remove any lines exposing ports from other components, such as the following for ``fdp-client``: -:: + .. literalinclude:: compose/fdp/components/v1/fdp-client.yml + :name: remove exposed ports + :language: yaml + :lines: 2,5-6 - nginx/ - ├ nginx.conf - ├ sites-available - │ └ fdp.conf - └ sites-enabled - └ fdp.conf -> ../sites-available/fdp.conf +.. note:: -The file ``nginx.conf`` is the configuration of the whole nginx, and it includes all the files from ``sites-enabled`` which contains configuration for individual servers (we can use one nginx, for example, to handle multiple servers on different domains). -All available configurations for different servers are in the ``sites-available``, but only those linked to ``sites-enabled`` are used. + The default ``/etc/nginx/nginx.conf`` in the `official nginx image`_ automatically includes ``*.conf`` files from ``/etc/nginx/conf.d`` in the ``http`` block. + That's why we mount our config files to ``/etc/nginx/conf.d``. -Let's see what should be the content of the configuration files. + .. code-block:: + :caption: default nginx.conf http block includes files from conf.d -.. code-block:: nginx + ... + http { + ... + include /etc/nginx/conf.d/*.conf; + } - # nginx/nginx.conf - - # Main nginx config - user www-data www-data; - worker_processes 5; +Nginx virtual server configuration +---------------------------------- - events { - worker_connections 4096; - } +.. literalinclude:: nginx/server.conf + :name: nginx server config + :caption: minimal nginx virtual server configuration + :language: none - http { - # Docker DNS resolver - # We can then use docker container names as hostnames in other configurations - resolver 127.0.0.11 valid=10s; - # Include all the configurations files from sites-enabled - include /etc/nginx/sites-enabled/*.conf; - } +Database security +================= -Then, we need to configure the FDP server. +The FDP uses two types of database: -.. code-block:: nginx +- MongoDB, for application data +- A triple store, such as GraphDB, for the actual metadata - # nginx/sites-available/fdp.conf +Both need to be secured. - server { - listen 443 ssl; +- `database security cheat sheet`_ +- `mongodb security cecklist`_ - # Generated certificates using certbot, we will mount these in compose.yml - ssl_certificate /etc/letsencrypt/live/fdp.example.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/fdp.example.com/privkey.pem; +Secrets +======= - server_name fdp.example.com; +List of secrets: - # We pass all the request to the fdp-client container, we can use HTTP in the internal network - # fdp-client_1 is the name of the client container in our configuration, we can use it as host - location / { - proxy_set_header Host $host; - proxy_set_header X-Forwarded-Proto $scheme; - proxy_pass_request_headers on; - proxy_pass http://fdp-client_1; - } - } +- jwt token secret key +- default user accounts +- mongodb credentials +- triple store credentials - # We redirect all request from HTTP to HTTPS - server { - listen 80; - server_name fdp.example.com; - return 301 https://$host$request_uri; - } +Backups +======= -Finally, we need to create a soft link from sites-enabled to sites-available for the FDP configuration. +(...) -:: +.. _OWASP cheat sheets: https://cheatsheetseries.owasp.org +.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Web_Service_Security_Cheat_Sheet.html +.. _database security cheat sheet: https://cheatsheetseries.owasp.org/cheatsheets/Database_Security_Cheat_Sheet.html +.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html +.. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html +.. .. _: https://ubuntu.com/blog/what-is-system-hardening-definition-and-best-practices +.. _mongodb security cecklist: https://www.mongodb.com/docs/manual/administration/security-checklist/ +.. _embedded web server: https://docs.spring.io/spring-boot/how-to/webserver.html#howto.webserver.configure-ssl +.. _certbot: https://certbot.eff.org/instructions +.. _Let's Encrypt: https://letsencrypt.org +.. _certbot's default location: https://eff-certbot.readthedocs.io/en/stable/using.html#where-are-my-certificates +.. _official nginx image: https://hub.docker.com/_/nginx - $ cd nginx/sites-enabled && ln -s ../sites-available/fdp.conf +TODO: update the text below -We have certificates generated and configuration for proxy ready. -Now we need to add the proxy to our ``compose.yml`` file so we can run the whole FDP behind the proxy. -.. code-block:: yaml - :substitutions: - - # compose.yml - - services: - proxy: - image: nginx:1.17.3 - ports: - - 80:80 - - 443:443 - volumes: - # Mount the nginx folder with the configuration - - ./nginx:/etc/nginx:ro - # Mount the letsencrypt certificates - - /etc/letsencrypt:/etc/letsencrypt:ro - - fdp: - image: fairdata/fairdatapoint:|compose_ver| - volumes: - - ./application.yml:/fdp/application.yml:ro - - fdp-client: - image: fairdata/fairdatapoint-client:|compose_ver| - environment: - - FDP_HOST=fdp - - mongo: - image: mongo:4.0.12 - ports: - - "127.0.0.1:27017:27017" - volumes: - - ./mongo/data:/data/db - - graphdb: - image: ontotext/graphdb:10.7.6 - volumes: - - ./graphdb:/opt/graphdb/home - -Don't forget to create the GraphDB repository as described in the :ref:`Persistent Repository ` section. The last thing to do is to update our ``application.yml`` file. We need to add ``clientUrl`` so that FDP knows the actual URL even if hidden behind the reverse proxy. @@ -242,17 +189,14 @@ Of course, the domain you want to access the FDP on must be configured to the se Don't forget to change the default user accounts as soon as your FAIR Data Point becomes publicly available. -.. DANGER:: - - Do not expose mongo port unless you secured the database with username and password. -.. WARNING:: +.. warning:: In order to improve findability of itself and its content, the FAIR Data Point has a built-in feature that registers its URL into our server and pings it once a week. This feature facilitates the indexing of the metadata of each registered and active FAIR Data Point. If you do not want your FAIR Data Point to be included in this registry, add these lines to your application configuration: - .. code-block:: yaml + .. code-block:: yaml # application.yml From 52526f85667c5e60a05cf1fd56b6e0970f066fe6 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 16:17:45 +0200 Subject: [PATCH 05/16] update nginx config comments --- docs/deployment/nginx/server.conf | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/deployment/nginx/server.conf b/docs/deployment/nginx/server.conf index 72a7c11..250d69b 100644 --- a/docs/deployment/nginx/server.conf +++ b/docs/deployment/nginx/server.conf @@ -6,7 +6,7 @@ server { include conf.d/includes/proxy.conf; } -# redirect all other traffic from port 80 to port 443 +# redirect all other traffic from port 80 (http) to port 443 (https) server { listen 80; listen [::]:80; @@ -14,7 +14,7 @@ server { return 301 https://$host$request_uri; } -# listen for tls (ssl) traffic +# pass traffic from port 443 to the upstream web server, on the docker network server { listen 443 ssl; listen [::]:443 ssl; @@ -23,7 +23,6 @@ server { ssl_certificate_key /etc/letsencrypt/live/fdp.example.com/privkey.pem; location / { - # pass request to the upstream fdp-client service (on the docker network) proxy_pass http://fdp-client:80; proxy_http_version 1.1; proxy_set_header Host $host; @@ -33,7 +32,7 @@ server { } } -# catch all +# catch all remaining traffic server { listen 80 default_server; listen [::]:80 default_server; From 6aac649721b3c1086676c5f9f7e055042548ca6a Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 16:30:27 +0200 Subject: [PATCH 06/16] remove localhost server config --- docs/deployment/nginx/server.conf | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/deployment/nginx/server.conf b/docs/deployment/nginx/server.conf index 250d69b..bcbd9e0 100644 --- a/docs/deployment/nginx/server.conf +++ b/docs/deployment/nginx/server.conf @@ -1,11 +1,3 @@ -# allow http traffic from localhost -# (required because the fdp-client service acts as a reverse-proxy for the fdp service) -server { - listen 80; - server_name localhost; - include conf.d/includes/proxy.conf; -} - # redirect all other traffic from port 80 (http) to port 443 (https) server { listen 80; From 911875ab3071e66eacaf8df2e7f383c6c665f90f Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 17:14:44 +0200 Subject: [PATCH 07/16] fix bind mount source path --- docs/deployment/nginx/compose.yml | 2 +- docs/deployment/production-deployment.rst | 22 ++++++++++++++-------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/deployment/nginx/compose.yml b/docs/deployment/nginx/compose.yml index 23826d8..8a04873 100644 --- a/docs/deployment/nginx/compose.yml +++ b/docs/deployment/nginx/compose.yml @@ -6,4 +6,4 @@ services: - "443:443" volumes: - "/etc/letsencrypt:/etc/letsencrypt:ro" - - "server.conf:/etc/nginx/conf.d/server.conf:ro" + - "./server.conf:/etc/nginx/conf.d/server.conf:ro" diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 01b5c04..57ebb74 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -81,6 +81,20 @@ This is just a minimal example, so you may want to specify an image version, ad :language: yaml :lines: 2,5-6 +Nginx server configuration +-------------------------- + +Here's a minimal example of an nginx server configuration that does the following: + +- redirect ``http`` to ``https`` +- pass ``https`` requests for ``fdp.example.com`` on to the upstream ``fdp-client`` container (over ``http``) +- catch any other requests + +.. literalinclude:: nginx/server.conf + :name: nginx server config + :caption: server.conf + :language: none + .. note:: The default ``/etc/nginx/nginx.conf`` in the `official nginx image`_ automatically includes ``*.conf`` files from ``/etc/nginx/conf.d`` in the ``http`` block. @@ -95,14 +109,6 @@ This is just a minimal example, so you may want to specify an image version, ad include /etc/nginx/conf.d/*.conf; } -Nginx virtual server configuration ----------------------------------- - -.. literalinclude:: nginx/server.conf - :name: nginx server config - :caption: minimal nginx virtual server configuration - :language: none - Database security ================= From 09774b0990ebfb6c920a07481031bef031fc32b7 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 17:23:10 +0200 Subject: [PATCH 08/16] update server conf comments --- docs/deployment/nginx/server.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deployment/nginx/server.conf b/docs/deployment/nginx/server.conf index bcbd9e0..d7ba808 100644 --- a/docs/deployment/nginx/server.conf +++ b/docs/deployment/nginx/server.conf @@ -1,4 +1,4 @@ -# redirect all other traffic from port 80 (http) to port 443 (https) +# redirect all traffic for fdp.example.com port 80 (http) to port 443 (https) server { listen 80; listen [::]:80; @@ -6,7 +6,7 @@ server { return 301 https://$host$request_uri; } -# pass traffic from port 443 to the upstream web server, on the docker network +# pass traffic for fdp.example.com port 443 (https) to the upstream web server, on the docker network server { listen 443 ssl; listen [::]:443 ssl; From 466c1ff940543a77dd6ce62a2e2ba8337ff79110 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 17:35:33 +0200 Subject: [PATCH 09/16] DRY server_name in nginx config --- docs/deployment/nginx/compose.yml | 2 ++ docs/deployment/nginx/server.conf | 11 ++++++----- docs/deployment/production-deployment.rst | 1 + 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/deployment/nginx/compose.yml b/docs/deployment/nginx/compose.yml index 8a04873..3523886 100644 --- a/docs/deployment/nginx/compose.yml +++ b/docs/deployment/nginx/compose.yml @@ -1,5 +1,7 @@ services: nginx: + environment: + FDP_HOST: fdp.example.com image: nginx ports: - "80:80" diff --git a/docs/deployment/nginx/server.conf b/docs/deployment/nginx/server.conf index d7ba808..c5104d9 100644 --- a/docs/deployment/nginx/server.conf +++ b/docs/deployment/nginx/server.conf @@ -2,17 +2,18 @@ server { listen 80; listen [::]:80; - server_name fdp.example.com; + server_name ${FDP_HOST}; return 301 https://$host$request_uri; } -# pass traffic for fdp.example.com port 443 (https) to the upstream web server, on the docker network +# pass traffic for fdp.example.com port 443 (https) to the upstream web server, +# on the docker network server { listen 443 ssl; listen [::]:443 ssl; - server_name fdp.example.com; - ssl_certificate /etc/letsencrypt/live/fdp.example.com/fullchain.pem; - ssl_certificate_key /etc/letsencrypt/live/fdp.example.com/privkey.pem; + server_name ${FDP_HOST}; + ssl_certificate /etc/letsencrypt/live/${FDP_HOST}/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/${FDP_HOST}/privkey.pem; location / { proxy_pass http://fdp-client:80; diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 57ebb74..2d8a0c3 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -61,6 +61,7 @@ Nginx compose service A minimal compose service definition for nginx is described below. Bind mounts are used to make the nginx configuration files and certificates from the Docker host available in the ``nginx`` container. +The ``FDP_HOST`` environment variable is used in the ``server.conf`` config file described in the next section. .. literalinclude:: nginx/compose.yml :name: nginx compose config From 1ca7e24e7ebfccb1210ca993ceda6c1f7e606d18 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 18:12:54 +0200 Subject: [PATCH 10/16] use the correct templates dir for nginx --- docs/deployment/nginx/compose.yml | 2 +- docs/deployment/nginx/{server.conf => server.conf.template} | 0 docs/deployment/production-deployment.rst | 5 +++-- 3 files changed, 4 insertions(+), 3 deletions(-) rename docs/deployment/nginx/{server.conf => server.conf.template} (100%) diff --git a/docs/deployment/nginx/compose.yml b/docs/deployment/nginx/compose.yml index 3523886..0e37c82 100644 --- a/docs/deployment/nginx/compose.yml +++ b/docs/deployment/nginx/compose.yml @@ -8,4 +8,4 @@ services: - "443:443" volumes: - "/etc/letsencrypt:/etc/letsencrypt:ro" - - "./server.conf:/etc/nginx/conf.d/server.conf:ro" + - "./server.conf.template:/etc/nginx/templates/server.conf.template:ro" diff --git a/docs/deployment/nginx/server.conf b/docs/deployment/nginx/server.conf.template similarity index 100% rename from docs/deployment/nginx/server.conf rename to docs/deployment/nginx/server.conf.template diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 2d8a0c3..ebdbc6a 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -98,8 +98,9 @@ Here's a minimal example of an nginx server configuration that does the followin .. note:: - The default ``/etc/nginx/nginx.conf`` in the `official nginx image`_ automatically includes ``*.conf`` files from ``/etc/nginx/conf.d`` in the ``http`` block. - That's why we mount our config files to ``/etc/nginx/conf.d``. + The `official nginx image`_ has the ability to render templated configuration files with environment variables. + Any ``*.template`` files from the ``/etc/nginx/templates`` directory are rendered into ``/etc/nginx/conf.d``. + The image's default ``/etc/nginx/nginx.conf`` then automatically includes ``*.conf`` files from ``/etc/nginx/conf.d`` in the ``http`` block. .. code-block:: :caption: default nginx.conf http block includes files from conf.d From 1daed6e7912635e776130d1d835a987d58fe1da8 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 18:15:28 +0200 Subject: [PATCH 11/16] fix nginx template filename --- docs/deployment/production-deployment.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index ebdbc6a..b5debda 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -91,14 +91,14 @@ Here's a minimal example of an nginx server configuration that does the followin - pass ``https`` requests for ``fdp.example.com`` on to the upstream ``fdp-client`` container (over ``http``) - catch any other requests -.. literalinclude:: nginx/server.conf +.. literalinclude:: nginx/server.conf.template :name: nginx server config - :caption: server.conf + :caption: server.conf template :language: none .. note:: - The `official nginx image`_ has the ability to render templated configuration files with environment variables. + The `official nginx image`_ has the ability to render configuration file templates with environment variables. Any ``*.template`` files from the ``/etc/nginx/templates`` directory are rendered into ``/etc/nginx/conf.d``. The image's default ``/etc/nginx/nginx.conf`` then automatically includes ``*.conf`` files from ``/etc/nginx/conf.d`` in the ``http`` block. From c4f09ecd83a49a285bf80eff7836a2eee00ef26e Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Wed, 3 Sep 2025 18:17:55 +0200 Subject: [PATCH 12/16] mention nginx unprivileged --- docs/deployment/production-deployment.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index b5debda..a92ec8e 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -69,7 +69,8 @@ The ``FDP_HOST`` environment variable is used in the ``server.conf`` config file :language: yaml :lines: 2- -This is just a minimal example, so you may want to specify an image version, adjust the paths where necessary, and/or add some addional config. +This is just a minimal example, so you may want to specify an image version, adjust the paths, where necessary, and/or add some addional config. +There is also an ``nginxinc/nginx-unprivileged`` image, but that will require a bit more configuration. .. note:: From 736d2b5c9d22c3fcd2bb59bbd699c87ca603442b Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Mon, 27 Oct 2025 13:46:53 +0100 Subject: [PATCH 13/16] clarify production some deployment lines --- docs/deployment/production-deployment.rst | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index a92ec8e..8527423 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -10,17 +10,17 @@ Disclaimer Running a FAIR Data Point in production is a bit more involved than running one offline on your development machine. The configuration details of a production deployments depend on many factors, such as available resources and security requirements. -Whether you're setting up your own bare metal server or using a cloud provider that offers all kinds of managed services, you'll need to think about many of the same topics. +Whether you're setting up your own bare metal server or using a cloud provider with lots of managed services, many of the same topics will need attention. Here's just a few that come to mind, in no particular order: - network security - secrets storage - identity and access management (IAM) - data management (security, privacy, replication, backups) -- service availability (e.g. container orchestration) +- service availability (container orchestration, monitoring) - performance - audit logging -- infrastructure as code +- deployment automation (infrastructure as code, CI/CD) - and so on and so forth... Obviously this list is far from exhaustive. @@ -35,8 +35,8 @@ Assuming basic infrastructure hardening is already in place (see e.g. `OWASP che These topics are covered by extending the :ref:`local-deployment` examples with some additional configuration. -Encrypted communication -======================= +HTTPS setup +=========== One of the first requirements for a production deployment is to set up Transport Layer Security (TLS) to provide encrypted communication, better known as HTTPS (HTTP over TLS). @@ -50,7 +50,7 @@ TLS certificates ---------------- In order to set up HTTPS, a valid TLS certificate is required (a.k.a. SSL certificate). -For this example, we assume that a TLS certificate is already available, *on the Docker host*, for our domain ``fdp.example.com``. +For this example, we assume a TLS certificate is already available, *on the Docker host*, for our domain ``fdp.example.com``. Certificate files can be obtained from various sources. Our example assumes that the `certbot`_ tool was used to obtain a certificate from `Let's Encrypt`_. @@ -129,10 +129,13 @@ Both need to be secured. Secrets ======= +The best way to handle application secrets strongly depends on your use-case. +In our minimal example we take one of the simplest approaches, viz. using environment variables. + List of secrets: - jwt token secret key -- default user accounts +- default fdp user accounts - mongodb credentials - triple store credentials From cc5d724a120aa7c133afcf5a02980a8090dc0d7f Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Mon, 3 Nov 2025 17:30:54 +0100 Subject: [PATCH 14/16] reason for https --- docs/deployment/production-deployment.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 8527423..a89f0ae 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -26,7 +26,7 @@ Here's just a few that come to mind, in no particular order: Obviously this list is far from exhaustive. Due to this complexity we cannot provide a generic solution for a production deployment. -However, we can provide some pointers and suggestions to help you get started. +However, we *can* provide some pointers and suggestions to help you get started. Assuming basic infrastructure hardening is already in place (see e.g. `OWASP cheat sheets`_), we'll look at a few things: - HTTPS (encrypted communication based on TLS) @@ -39,6 +39,7 @@ HTTPS setup =========== One of the first requirements for a production deployment is to set up Transport Layer Security (TLS) to provide encrypted communication, better known as HTTPS (HTTP over TLS). +This is very important, because, among many other things, it prevents unauthorized parties from intercepting and reading your login credentials. Https connections can be handled, for example, by configuring a load balancer or a reverse proxy. @@ -130,7 +131,7 @@ Secrets ======= The best way to handle application secrets strongly depends on your use-case. -In our minimal example we take one of the simplest approaches, viz. using environment variables. +In our minimal example we take one of the simplest approaches, which is using environment variables. List of secrets: From bbd9bb8276d9134a39bbb944149a2032b821d8d4 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Fri, 7 Nov 2025 15:57:01 +0100 Subject: [PATCH 15/16] add link to graphdb security setup docs --- docs/deployment/production-deployment.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index a89f0ae..00f1235 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -126,6 +126,7 @@ Both need to be secured. - `database security cheat sheet`_ - `mongodb security cecklist`_ +- `graphdb security`_ Secrets ======= @@ -152,6 +153,7 @@ Backups .. .. _: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html .. .. _: https://ubuntu.com/blog/what-is-system-hardening-definition-and-best-practices .. _mongodb security cecklist: https://www.mongodb.com/docs/manual/administration/security-checklist/ +.. _graphdb security: https://graphdb.ontotext.com/documentation/11.1/enabling-security.html .. _embedded web server: https://docs.spring.io/spring-boot/how-to/webserver.html#howto.webserver.configure-ssl .. _certbot: https://certbot.eff.org/instructions .. _Let's Encrypt: https://letsencrypt.org From d69c62be1233b092e8f802463c9106d60fe53db3 Mon Sep 17 00:00:00 2001 From: dennisvang <29799340+dennisvang@users.noreply.github.com> Date: Fri, 7 Nov 2025 15:58:30 +0100 Subject: [PATCH 16/16] mention env security qualifier --- docs/deployment/production-deployment.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployment/production-deployment.rst b/docs/deployment/production-deployment.rst index 00f1235..c9f0207 100644 --- a/docs/deployment/production-deployment.rst +++ b/docs/deployment/production-deployment.rst @@ -132,7 +132,7 @@ Secrets ======= The best way to handle application secrets strongly depends on your use-case. -In our minimal example we take one of the simplest approaches, which is using environment variables. +In our minimal example we take one of the simplest (and least secure) approaches, which is using environment variables. List of secrets: