Website : customer workflow #10
Labels
No Label
User Research
backend
billing
frontend
gitpad
operations
post-mvp
priority-1
priority-2
priority-3
shared hosting
No Milestone
No project
No Assignees
2 Participants
Notifications
Total Time Spent: 4 days 10 hours
Due Date
realaravinth
4 days 2 hours
dachary
8 hours
No due date set.
Dependencies
No dependencies set.
Reference: Hostea/july-mvp#10
Loading…
Reference in New Issue
There is no content yet.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?
Priority levels:
Resources:
Hostea/dashboard(not accounted for in time logged; mostly copied Hostea/payments) is bootstrapped for this purpose.
Tasks accomplished
i. django-odic-provider
- Lacks maintenance
- Supports very old Django releases(2.0 while latest Django release and the one used by Hostea/dashboard is 4.x)
ii. Authlib
- Lacks documentation
i. Deployed a local instance via Docker
ii. Configured an application for OIDC login and integrated with keycloak-provided demo app to evaluate user experience
iii. Tried to integrate with Gitea but wasn't successful. Task is unimportant, but will come back to it if Gitea/any Hostea software integration with Hostea/dashboard
I made the decision to roll out a custom OIDC provider implementation so spent time reading the OIDC Core specification. I'm half-way through and worst-case scenario, I should be done with it by Sunday.
s/django-odic-provider/django-oidc-provider/
OIDC Core Specification was heavily referring to validation mechanisms detailed in the OAuth 2.0 specification(RFC 6749) so I read it to familiarize myself with the OAuth 2.0.
Summary
OAuth 2.0 offers four mechanisms to receive authorization:
1. Authorization Code Grant:
i. client server requests the user for consent via the authorization server(OAuth server/Hostea dashboard). ii. If user consents, authorization server will redirect the user with the authorization code encoded in URI parameters
iii. The client server uses this code to get the access token.
The access token is the secret that gives access to the user's resources on the OAuth server.
Conclusions
i. Secure since access token isn't present in URI parameter
ii. Suitable for Hostea since Hostea has server components.
2. Implicit Grant:
Conclusions
3. Resource Owner Password Credentials Grant
Conclusions
4. Client credentials grant
Decisions
For the MVP, I only plan to support Authorization code grant flow.
Interestingly, the authorization flows have parallels in OpenID connect where there are three authentication flows:
Yesterday, I made the decision to only implement support for Authorization Code Flow for the MVP and conveniently, implementing OAuth's Authorization code grant flow will allow us to do just that! :)
I read two OAuth-related RFCs today. OAuth and to some extent, even OIDC core is incomplete for our use cases, so I plan to work my way through OAuth-related and OIDC-related RFCs listed here over the weekend to gain a reasonably complete picture of how things work.
RFC 6750: OAuth 2.0 Bearer Token Usage
Bearer token is used as an opaque secret to authenticate a
client/relying party to a resource server. Using it requires no
additional proof-of-possession.
Summary
RFC 6750 lists three methods of transmitting bearer tokens in requests
to resource servers
Authorizationheader of therequest.
Transmit in
application/x-www-form-urlencodedrequest body with theaccess_tokenparameter. There are some conditions associated with it, which are mentioned in the linked section but not going into detail as they are irrelevant for the MVP.Support: MAY
access_tokenparameterThreat Mitigation
Use digital signature or Message Authentication Code(MAC) or opaque tokens containing references to authorization information rather than encoding information directly.
Bearer tokens MUST NOT be stored in cookies as they are typically transmitted in the clear
replay attack mitigation: limit lifetime(1 hour or less is recommended by the RFC).
RFC 7636: Proof Key for Code Exchange by OAuth Public Clients
An attacker can intercept authorization code issued to public clients and exchange it to gain access tokens.
Attack scenario
authorization codevia browser and register redirection URI with custom schemes for call back purposes.authorization codecall backs.authorization codereturned post user consent may be intercepted by malicious app and exchanged foraccess tokenThe RFC lists a long set of pre-conditions that will have to be met for the attack to be successful. Though long and specific, the RFC says this type of attack has been observed in the wild.
Mitigation
This extension uses a one-time cryptographic key sent over TLS with the authorization request for code proof-of-procession verification.
Client generates and stores secret called
code_verifierand derivest(code_verifier) = code_challengefrom it and sends it with authorization requestAuthorization server returns
authorization codeand storescode_challengeand transformation method(example:SHA256)Client sends
authorization codeinaccess token requestand as usual but also includescode_verifiersecret generated at step 1.Authorization server performs transformation on
code_verifierwith the recorded method in step 2 and compares it withcode_challenge. If they match, then proof-of-procession is verified.Attacker that intercepts callbacks is unaware of
code_verifiersecret so they won't be able to send accesstoken requestConclusions
As stated in the entry remark, PKCE is not in the scope of July MVP but implementing support post-MVP is strongly recommended.
I couldn't log my time during the weekend(went on a holiday) but here are the RFCs that I read during that time:
RFC 7009: OAuth 2.0 Token Revocation
Relying parties(all services dependent on the Hostea/dashboard SSO) should revoke unused tokens(refresh, access, etc.) for security reasons. Scenarios that warrant token revocation:
This RFC describes the request and response and schemas for token revocation but doesn't say anything about the location of the revocation endpoint(recommends documenting it in the service documentation)
Example request
token_type_hintin the revocation request payload for optimizing database lookups. The RFC also warns that the endpoint can be a source of DOS attacks so recommends the usage of this parameter. Whentoken_type_hintis absent in request, the RFC recommends searching for a match against all issued tokens are revoking successful matches. If all token types(in Hostea's context: refresh and access tokens) share the same namespace, then only one match will be yielded when thetoken_type_hintis not present and so all token types must be searched and matches revoked.Security recommendations
Conclusions
There are two ways to implement tokens: digital signature or Message Authentication Code(MAC) and opaque tokens. Revoking MAC tokens would require the authorization server to either store a list of blacklisted/revoked tokens and check token-authenticated request against it or rotate the key that signed produced MAC. The blacklist approach does away with the advantage of using MAC --- avoiding database lookups for authentication. And the key rotation is impractical at scale. Therefore, the opaque approach, though costly, is the feasible on.
RFC 7662: Auth 2.0 Token Introspection
Token introspection standards to query token metadata by authenticated parties(token audience, etc.).
Example introspection request:
Example response
Client ID and secret are used for authentication
activestatus of the recommended metadata is mandatory. For full list kindly see here.Other RFCs/documents
I also had a cursory glance of OAuth 2.0 Security Best Current Practice (Security BCP) and Threat Model and Security Considerations (RFC 6819). They document implicit suggestions by RFC6749(OAuth 2.0 core) and other extensions or assumptions that one would make to secure their OAuth implementation, so there's nothing unique to report. I will, however, use those documents as a checklist to audit our implementation every step of the way.
Game plan
I'm currently reading JWT (RFC7519) and Dynamic Client Registration (RFC 7591) and I have the following in my reading list:
And if time permits, I will read the OpenID Connect optional specifications listed here. But this will be async and not blocking progress as it is the situation right now.
Log
Bootstrapped OIDC Client registration:
9c5f4d,d3eb1,152603Docs: README(
761e3b) and HACKING instructions(43d60e)Unfortunately(or maybe fortunately?) all this code is going drown the drain:
I was looking up information on implementing basic and bearer token authentication, since the default Django authentication uses cookie based sessions which is not ergonomic for programmatic access. I landed on the Django REST framework page for authentication, which contained a link to Django OAuth toolkit(BSD 3-Clause). Django OAuth toolkit implements OIDC and looks alive(last commit was from two days ago).
I will investigate and evaluate
django-oauth-toolkitfeasibility for use in Hostea software.Tasks
OAuth2 setup
Setup dummy OAuth2 application with a bunch of scripts and manual browser link clicking
Configured
django-oauth-toolkitwithin a django project to OAuth2 provider capabilities(Manually)Verified OAuth2 workflow implemented within
django-oauth-toolkitConfigured OAuth2 authenticated access to protected resources in the django project
OIDC provider
Modified
django-oauth-toolkitconfiguration to enable OIDC capabilitiesdjango-oauth-toolkitby default enforces PKCE(RFC 7636, notes) but Gitea in client/Relying Party mode doesn't do PKCE since it is a confidential client and so not required. So disabled PKCE ondjango-oauth-toolkitglobally. Ideally, I wish this were a per-application basis PKCE configuration. The library supports multiple client types, so not enforcing PKCE on confidential clients would be nice.Configured Gitea to use OIDC via the web interface:
i) OIDC configuration was a little weird:
In Site Administration> Add Authentication Source select "OAuth2" for "Authentication Type" and under "OAuth2 Provider" select "OpenID Connect".
By default, selcting "OAuth2" for "Authentication Type" shows popular third-party SSO providers like GitHub, Google, Discord, etc and OIDC is hidden deep in the menu. Unintuitive. Wasted ~20 minutes.
ii) Gitea's redirection URI(required when configuring app on
django-oauth-toolkit) is http://gitea.example.org/user/oauth2//callback where authentication name is a field in "Add Authentication Source" formiii)
django-oauth-toolkitrequires scope to be set toopenid. If scope unspecified, request will be treated as regular OAuth2 request.Gitea OIDC integration was partially successful, the following things worked out of the box:
i) OIDC service auto-discovery
ii) Authorization and subsequent authorization code grant generation
But access token generation fails with:
json { "error": "invalid_client" }I tried identifying the cause fist on Gitea's side, which took through multiple interfaces and then to gothic, this SSO library that Gitea uses. Everything checked out fine. And then I tried doing the same on
django-oauth-toolkit's side and the same process ensued: multiple interfaces and then to the OAuth dependency it uses.I faced similar issues with plain OAuth2 trials(
{"error": "invalid_grant"}due to no support for PKCE on client side) so I believedjango-oauth-toolkitis expecting something that Gitea isn't providing.Tomorrow I will intercept Gitea's request payload and try to compare it with
django-oauth-toolkit's expected payload.If I'm able to figure out how to get the library working with Gitea, we'll have zero-code SSO via Hostea/dashbaord 🙃
OIDC SSO is ready and working 🎉
Long story
django-oauth-toolkitoffers OIDC with with two cryptographic options:The difference between the options is available in the docs.
I couldn't get
RSA + SHA2 256working yesterday, so as mentioned I wrote a simple HTTP server to intercept all Gitea communications with the OIDC provider to figure out why OIDC integration with Gitea was failing. Everything from Gitea's side checked out so I switched to thesymmetric key HMAC + SHA2 256to see if it was working. And it did! :)In Hostea's context, we could get by with
symmetric key HMAC + SHA2 256since all our clients/relying parties will be running in trusted environments.Currently, to add an application to
django-oauth-toolkit, one has to interact with the forms it exposes, which is not ergonomic for automation. I will create APIs to enable programmatic interaction withdjango-oauth-toolking, which, I think, will be the only customisation that Hostea will be making.Good sleuth work 👍
Tasks accomplished
django-oauth-toolkitin subsequent commitsl10n can be done post MVP, this is sensible. Enough has a weblate playbook and that can be integrated with the CI so the gettext binary files are updated on a regular basis.
Tasks
Tasks
python manage.pyto clear unverified users that exceed a configurable tolerance periodTasks:
Good call on using "t-shirt" sizes rather than allowing the user to independently choose the size of each resource 👍 Here are the available sizes for the MVP:
As for the price, although it will need adjusting, I suggest making it roughly double what it actually costs to be realistic:
Update:
The VM configurations can be set at runtime. I feel this options will allow each Hostea admin to choose a price and offering that is optimal for their operation.
Tomorrow, I will create a management command(
manage.pycommands) to create VM configurations.Do you want me to set VM configurations defaults per the list you've mentioned above?
It would make sense since it is what the backend implements. It has a lot more but those three are the cheapest.
Tasks accomplished
Default VM configuration creation is causing problems. My local DB instance's migrations were current when I ran the test but spinning up a fresh DB instance and then trying to apply migrations is causing issues(failed CI run.
The error says that the target table is unavailable but everything works when I comment out the stuff that creates default configurations.
My guess is that Python->SQL translation is entering into some circular loop, because of the way I'm creating default configurations.
Will fix it tomorrow.
Tasks accomplished
get_user_idmanagement command, which could have been used withdjango-oauth-toolkit'screateapplicationmanagement command. Removed becuase it is now integrated withcreate_oidc(see below).create_oidcto create OIDC applications, that is opinionated in a manner that is suitable for Hostea(sets default for algorithm, authorisation grant type and client type) and more ergonomic thandjango-oauth-toolkit'screateapplication.create_oidcUsage:
Example:
Please note that redirection URI(third positional parameter and only URI passed in the above command) is an actual Gitea redirection URL.
Gitea's redirection URI is in the form:
client_idandclient_secretwill be auto-generated bycreate_oidcand should be parsed and supplied to Gitea.Configuring SSO on Gitea
To configure Gitea with the SSO implemented in
Hostea/dashboard:create_oidc. Parse and saveclient_idandclient_secret, as mentioned above.The goal here is that the dashboard provides a SSO so that the user can:
Is this correct?
The best way to script this seems to be gitea admin auth add-oauth.
Yes. But, at the moment the support application in Hostea/Dashboard only redirects to the issue tracker support repository.
Additionally, the SSO can be configured to work with https://forum.hostea.org, too. But that's for another time.
This is out of scope of the SSO.
If a user's Hostea instance is configured to work with the SSO, then all Hostea users will be able to log in to the user's private Hostea instance.
Also, Gitea shows SSO options on the login webpage, so users on the private Gitea might necessarily try to create accounts on the Dashboard.
The Gitea screenshot is for illustrative purposes only. You are welcome to use any method that you prefer to configure the Gitea side of things. :)
Thanks for clarifying. In that case the easiest way to achive this
So the workflow would be:
Or is it something else?
With the Gitea configuration, we can offer a much more seamless experience.
Please see this screen capture(ephemeral link, valid for 1 week from now).
Thanks for clarifying. I now understand how it fits. Nothing related to OpenID really. Perfect.
So I'll make a note to modify the playbook so that it:
python manage.py create_oidc gitea atm http://localhost:8080/user/oauth2/hostea-dash-local/callbackI'll leave the writing of the test to check that it works to you, if that's ok?
Copy/pasting from hoteasetup.py may help since it has web scraping logic for woodpecker already.
Just to make sure there is no misunderstanding, I'll wait on you to give me the greenlight before I do that. Preferably when everything else is ready so I can do it at once.
Actually, we can do better and skip the whole Gitea login screen:
Instead of redirecting to https://gitea.hostea.org/Hostea/support/issues/new, I can redirect the user to http://gitea.hostea.org/user/oauth2/hostea-sso?redirect_to=/hostea/support/issues/new(assuming the SSO is named "hostea-sso" on the Hostea Gitea). This way, the user will be automatically authenticated on the Hostea Gitea via the SSO and directly visit new issues webpage!
edit: accidentally closed the issue and also this method doesn't work. Gitea doesn't send
redirect_toalong with the authorisation request. There is a parallel redirection mechanism via cookies that Gitea uses in such cases. We could run a custom JavaScript script at http://gitea.hostea.org/user/oauth2/hostea-sso?redirect_to=/hostea/support/issues/new to set the cookie but doesn't seem right.Yes, perfect!
Gladly!
@dachary: These prices include tax, right? The billing system requires me to specify taxes. Tax rates might be different for each Hostea operator.
For the MVP I'd like to set the price with taxes included and set the
tax=0. Tax computation based on configurable rates will be implemented post-MVP.Yes, lets' make it tax included.
Tasks
Copy/pasting from commit message
The tests are working locally, because I run all service containers in
network_mode=host, which is apparently a privileged operation in Woodpecker CI. I could enable higher privileges in the CI dashboard, but I think that would be unsafe: as the containers might bleed into the underlying host's networking.It is important that the Gitea container is able to talk to the dashboard instance that the integration test script spins up. Without that, I won't be able to test SSO functionality.
Tomorrow, I will try packaging the Dashboard into a Docker container, and run it instead of spinning up an instance using
python manage.py.I tried to spin up a Gitea instance outside of docker, in an attempt to simplify the networking issues I faced earler but wasn't successful.
There is a step in the integration test script which checks if all needed services are up and running. For some reason, both Gitea and the Dashboard instances are not running and so the script went into a deadlock.
I ran the build locally, using woodpecker-cli
And I faced the same issue.
I then logged into the woodpecker build container with
docker execand found issues with the Gitea instance: Gitea'sRUN_USERwas set to my local user, where as woodpecker was running as root.I don't know why the dashboard was failing.
It's very tricky. You're essentially retracing the steps that brought me to write hosteasetup.py and it took me a day or two to get it right.
Tasks
Please see here for the WIP pull reqeust
Tasks(Hostea/Dashboard)
git pullbefore write andgit pushpost writeTasks
Implemented add/delete VM from command line and related tests
wrote the corresponding test and reported an issue on the dashboard, prepared the coworking session
Poll and send email notification to user when their Gitea instance is online and reachable. Please see associated PR
Tasks accomplished:
debug tests
(logging yesterday's work)
help with debug
Enough details were fixed to satisfy the MVP. Still rough around the edges but MVP good.