From 1ff9e8737e533a9522baa6e77720b4704a4dd971 Mon Sep 17 00:00:00 2001 From: zachary painter <60552605+ZPain8464@users.noreply.github.com> Date: Tue, 30 Jan 2024 17:10:27 -0500 Subject: [PATCH] Docs: adds robots.txt notice (#1208) * adds robots.txt notice * runs prettier * Update content/docs/reference/routes/public-access.mdx Co-authored-by: Kenneth Jenkins <51246568+kenjenkins@users.noreply.github.com> * Update content/docs/reference/routes/public-access.mdx Co-authored-by: Kenneth Jenkins <51246568+kenjenkins@users.noreply.github.com> * Update content/docs/reference/routes/public-access.mdx Co-authored-by: Kenneth Jenkins <51246568+kenjenkins@users.noreply.github.com> * Update content/docs/reference/routes/public-access.mdx Co-authored-by: Kenneth Jenkins <51246568+kenjenkins@users.noreply.github.com> * rewrites text * fixes cspell typo --------- Co-authored-by: Kenneth Jenkins <51246568+kenjenkins@users.noreply.github.com> --- .../docs/reference/routes/public-access.mdx | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/content/docs/reference/routes/public-access.mdx b/content/docs/reference/routes/public-access.mdx index c7ccccab1..e7c68a90f 100644 --- a/content/docs/reference/routes/public-access.mdx +++ b/content/docs/reference/routes/public-access.mdx @@ -16,9 +16,24 @@ import TabItem from '@theme/TabItem'; ## Summary -**Use with caution:** **Public Access** allows all requests for a given route, bypassing authentication and authorization. Suitable for publicly exposed web services. +:::caution -If this setting is enabled, no whitelists (for example, Allowed Users) should be provided in this route. +Because the **Public Access** setting bypasses authentication and authorization checks, you should enable it only for publicly exposed web services. + +::: + +The **Public Access** setting instructs Pomerium to grant unauthorized and unauthenticated access to all requests to the upstream service. If you enable this setting, no other policy should be provided for the route. + +### Robots.txt behavior + +By default, Pomerium serves a **robots.txt** response directly, instructing search engines _not_ to crawl the route domain: + +```txt +User-agent: * +Disallow: / +``` + +For routes with `allow_public_unauthenticated_access` enabled, Pomerium will not serve **robots.txt** directly. Instead, requests for `/robots.txt` will be proxied to the upstream service. ## How to configure