Authenticating 3rd Party Integrations in Go

While building Swish, I came across an interesting problem: How do I create an Interface for authenticating integrations?.

When you want to enable a user integrate with a specific service, that can be easy to model. But it is trickier to model an authentication flow that works for almost any service.

In the article, I'll walk you through the process I went through, several versions I tried, and my current solution.

Why is this necessary?

In Swish, the plan is to support a multitude of integrations, allowing a user to connect their blog to the many tools they are also using.

Now, when authenticating to external services, there are an almost infinite amount of ways the authentication flow will work. But because I needed to create a common interface for integration authentication, I needed to figure out how this will be modeled.

Version 1: Using Forms

Describing our Form

To display our form to a user, we need a few parts.

1. The name of the form as a whole. Used to make sure we are reading the correct imput
2. Instructions to show the user to understand what the options mean
3. The options which represent a part of the form

package integrations

import (
	"fmt"
	"net/url"
	"template/html"

	"gitlab.com/swishink/swishink/internal/types"
)

// An authForm represents a form to be shown to the user
// The name of the form is combined with the ID of the option to
// get the name in the HTML field
// e.g. <input name="form_name[option_id]">
// This is so that we can differentiate the options when submitted
// Instructions are any information to show the user to understand what the options mean
type AuthForm struct {
	Name         string
	Instructions template.HTML
	Options      []types.Option
}

// Used to get the values for the form when given url.Values
func (a AuthForm) GetVals(source url.Values) map[string]string {
	vals := map[string]string{}
	if a.Name != "" {
		for _, key := range a.Options {
			val := source.Get(fmt.Sprintf("%s[%s]", a.Name, key.ID))
			if val != "" {
				vals[key.ID] = val
			}
		}
	}

	return vals
}

About types.Option

types.Option is a struct I have to represent a HTML form input. I will write about it in a separate article, I don't want to clutter this one.

Modeling the authentication

To model these, my Integration authentication interface would look like this:

package integrations

import "gitlab.com/swishink/swishink/internal/types"

type IntegrationAuth interface {
	// Returns the authetication form
	Form() AuthForm
	// When the auth form is submitted, the values are passed to this function
	// to validate and get the credentials
	FormValidate(vals map[string]string) types.IntegrationCreds
}

Problems with v1

The method assumes that there is only one type of authentication. We will display a form to the user, and they will submit it.

With this method we cannot support integrations that use OAuth1 (like Twitter), or OAuth2(like LinkedIn).

Version 2: Supporting More Authentication Types

Trying to add more integrations, we then need to extend our authentication to support other authentication types.

1. Forms: This is for integrations that require the user to enter information such as an API key, or username/password. E.g. DEV, Medium, Hashnode.
2. Oauth1: For integrations that are authenticated using the OAuth1 protocol. E.g. Twitter.
3. OAuth2: For integrations that use the OAuth2 protocol: E.g. LinkedIn

Modeling the authentication

To model these, my Integration struct would look like this:

package integrations

import (
	"context"

	"github.com/dghubble/oauth1"
	"gitlab.com/swishink/swishink/internal/types"
	"golang.org/x/oauth2"
)

type authType string

var (
	authTypeForm   authType = "form"
	authTypeOauth1 authType = "oauth1"
	authTypeOauth2 authType = "oauth2"
)

type IntegrationAuth interface {
	// Returns the authentication type to use
	Type() authType

	// Returns the authetication form
	// Used when the authentication type is "form"
	Form() AuthForm
	// When the auth form is submitted, the values are passed to this function
	// to validate and get the credentials
	FormValidate(vals map[string]string) types.IntegrationCreds

	// Gets the Oauth1 config in the oauth1 flow
	Oauth1Config() oauth1.Config
	// In the Oauth1 flow, this function is used to retrieve the oauth1 credentials
	GetOauth1Creds(ctx context.Context, config *oauth1.Config, token *oauth1.Token) (types.IntegrationCreds, error)

	// Gets the Oauth2 config in the oauth2 flow
	Oauth2Config() *oauth2.Config
	// In the Oauth2 flow, this function is used to retrieve the oauth1 credentials
	GetOauth2Creds(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error)
}

This would work quite well, however, the immediate worst part of it is that every implementation will have to implement ALL the methods to be a valid Integration even if they don't use it.

We can immediately fix that by breaking up the interface.

package integrations

import (
	"context"

	"github.com/dghubble/oauth1"
	"gitlab.com/swishink/swishink/internal/types"
	"golang.org/x/oauth2"
)

type IntegrationFormAuth interface {
	// Returns the authetication form
	// Used when the authentication type is "form"
	Form() AuthForm
	// When the auth form is submitted, the values are passed to this function
	// to validate and get the credentials
	FormValidate(vals map[string]string) types.IntegrationCreds
}

type IntegrationOauth1Auth interface {
	// Gets the Oauth1 config in the oauth1 flow
	Oauth1Config() oauth1.Config
	// In the Oauth1 flow, this function is used to retrieve the oauth1 credentials
	GetOauth1Creds(ctx context.Context, config *oauth1.Config, token *oauth1.Token) (types.IntegrationCreds, error)
}

type IntegrationOauth2Auth interface {
	// Gets the Oauth2 config in the oauth2 flow
	Oauth2Config() *oauth2.Config
	// In the Oauth2 flow, this function is used to retrieve the oauth1 credentials
	GetOauth2Creds(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error)
}

In the process, we also got rid of the AuthType() method. Since they are now different interfaces, we can check the AuthType by seeing what interface was implemented. For example:

package integrations

import "errors"

type authType string

var (
	authTypeForm   authType = "form"
	authTypeOauth1 authType = "oauth1"
	authTypeOauth2 authType = "oauth2"
)

func GetAuthType(integration interface{}) (authType, error) {
	switch integration.(type) {
	case IntegrationFormAuth:
		return authTypeForm, nil
	case IntegrationOauth1Auth:
		return authTypeOauth1, nil
	case IntegrationOauth2Auth:
		return authTypeOauth2, nil
	default:
		return "", errors.New("No valid authentication type")
	}
}

Alright, everything seems clean, modular. This should be great right? NO

Problems with v2

This solution starts to fall apart when we encounter a more custom authentication flow. Here are some examples.

• Self hosted software: We may need to get the URL of the server, along with generated clientID/Secrets. Then maybe switch to Oauth1/Oauth2 authentication.
• Multiple supported authentication types. We may need to ask the user what method they want to use to authenticate, and then use that method. This would not only require multiple steps, it would require multiple steps based on what the user chose on a previous step.

Version 3: A recursive flow

Requirements

1. A flexible number of steps

We do not know how many steps an integration authentication will take. So our flow must be able to take an arbitrary amount of steps.

First thought could be, each integration will return a list of the steps and then we will take the users through the flow.

However, this will not work because we could have situations where the number of steps changes because of something in a previous step. And this leads to our next requirement.

If the authentication flow can change depending on the values from a previous step, then each step must receive the values from the previous step.

2. Ability to display a form at the end of any step

Some authentication flows will likely require that we show a form to the user. This could be options on the authentication method, getting an API key, getting the self-hosted URL, e.t.c.

3. Ability to redirect the user to a URL at the end of any step

Some authentication flows may require a redirect, this is common on OAuth flows.

4. A way to know the flow has finished

If we will keep calling this authentication method recursively, we need to be able to signal that the authentication has been completed.

Inputs

1. The authentication URL. For something like OAuth, we need to know the URL to set as the Callback URL.
2. The values gotten from the previous step.

package integrations

import  "net/url"

type AuthReq struct {
	// The URL is the URL that the Flow should redirect to for extra steps
	// For example, it will be used to set the callbackURL in Oauth flows
	URL string

	// FormValues are the submitted values from the request
	// We use `url.Values` because that is also the type of http.Request.Form
	// Makes it easy to retrieve these values
	FormValues url.Values
}

Output

1. What to do next?
   • A form to display to the user. OR
   • A URL to redirect the user to. OR
   • The final credentials of the Integration.
2. An error if we encountered any

package integrations

import 	"gitlab.com/swishink/swishink/internal/types"

type AuthResp struct {
	// Set when we need to display a form
	Form AuthForm

	// RedirectTo is a URL we want to redirect the user to
	// Useful in OAuth authentication flows
	RedirectTo string

	// Credentials are the credentials to save
	// When this value is not nil, the authentication flow ends
	Credentials types.IntegrationCreds
}

The Flow

With these, we can create an AuthFlow interface like this:

package integrations

import "context"

type IntegrationAuth interface {
	Do(ctx context.Context, req AuthReq) (resp *AuthResp, err error)
}

Quite elegant if I do say so myself 😎.
Fitting that like many of the interfaces in the standard library, we end up with a single method interface.

Implementations

Let's go over some implementations of this IntegrationAuth.

No Authentication

Some integrations don't require credentials. Just a one-click activation.

This is what powers the Blogstreak and Disqus integrations on Siwsh.ink, so you can check it out in the demo to see how it works.

package integrations

import (
	"context"

	"gitlab.com/swishink/swishink/internal/types"
)

type AuthFlowNone struct{}

func (a AuthFlowNone) Do(ctx context.Context, req AuthReq) (resp *AuthResp, err error) {
	return &AuthResp{Credentials: types.IntegrationCreds{}}, nil
}

Single Form

A common type of integration we will see is a single form authentication. We display one form, and we validate the values submitted.
The effect is identical to v1.

This is what powers the DEV, Medium and Hashnode integrations on Siwsh.ink, so you can check it out in the demo to see how it works.

package integrations

import (
	"context"
	"fmt"

	"gitlab.com/swishink/swishink/internal/types"
)

type AuthFlowForm struct {
	Form      AuthForm
	Validator func(ctx context.Context, val map[string]string) (types.IntegrationCreds, error)
}

func (a AuthFlowForm) Do(ctx context.Context, req AuthReq) (*AuthResp, error) {
	ourVals := a.Form.GetVals(req.FormValues)

	// if we have no form values, display the form
	if len(ourVals) == 0 {
		return &AuthResp{Form: a.Form}, nil
	}

	// If the form is submitted, validate
	creds, err := a.Validator(ctx, a.Form.GetVals(req.FormValues))
	if err != nil {
		return nil, fmt.Errorf("Could not validate submitted form: %w", err)
	}

	return &AuthResp{Credentials: creds}, nil
}

Oauth1

This is an implementation for an Oauth1 flow. An integration such as Twitter will be able to use this.

This is what powers the Twitter integration on Siwsh.ink, so you can check it out in the demo to see how it works.

package integrations

import (
	"context"
	"errors"
	"fmt"

	"github.com/dghubble/oauth1"
	"gitlab.com/swishink/swishink/internal/types"
)

type AuthFlowOauth1 struct {
	// The Oauth1 config
	Config *oauth1.Config

	// Used to save a request token/secret pair
	SaveTokenPair func(token, secret string)

	// Uset to retrieve a request secret given the token
	GetSecretByToken func(token string) (secret string)

	// Set by the user, use the config/token to get and marshal the credentials to save
	GetCreds func(ctx context.Context, config *oauth1.Config, token *oauth1.Token) (types.IntegrationCreds, error)
}

func (a AuthFlowOauth1) Do(ctx context.Context, req AuthReq) (*AuthResp, error) {
	if req.URL == "" {
		return nil, fmt.Errorf("No authURL in request context")
	}

	if a.Config == nil {
		return nil, fmt.Errorf("oauth2 config is nil")
	}
	a.Config.CallbackURL = req.URL

	// To know if we are in the Oauth flow, we check if the oauth verifier parameter is present
	// If it is not present, redirect to authorizationURL
	if req.FormValues.Get("oauth_verifier") == "" {
		return a.redirectToAuthorizationURL()
	}

	// If it is present, it means we got a callback
	// Validate the callback
	return a.validateCallback(ctx, req)
}

func (a AuthFlowOauth1) redirectToAuthorizationURL() (*AuthResp, error) {
	requestToken, requestSecret, err := a.Config.RequestToken()
	if err != nil {
		return nil, fmt.Errorf("could not get request token: %w", err)
	}
	a.SaveTokenPair(requestToken, requestSecret)

	authorizationURL, err := a.Config.AuthorizationURL(requestToken)
	if err != nil {
		return nil, fmt.Errorf("could not get authorizationURL: %w", err)
	}

	return &AuthResp{RedirectTo: authorizationURL.String()}, nil
}

func (a AuthFlowOauth1) validateCallback(ctx context.Context, req AuthReq) (*AuthResp, error) {
	requestToken := req.FormValues.Get("oauth_token")
	verifier := req.FormValues.Get("oauth_verifier")
	if requestToken == "" || verifier == "" {
		return nil, errors.New("Request missing oauth_token or oauth_verifier")
	}

	// Get request secret
	requestSecret := a.GetSecretByToken(requestToken)
	accessToken, accessSecret, err := a.Config.AccessToken(requestToken, requestSecret, verifier)
	if err != nil {
		return nil, fmt.Errorf("could not get access token and secret: %w", err)
	}

	creds, err := a.GetCreds(ctx, a.Config, oauth1.NewToken(accessToken, accessSecret))
	if err != nil {
		return nil, fmt.Errorf("Could not get oauth1 credentials: %w", err)
	}

	return &AuthResp{Credentials: creds}, nil
}

OAuth2

OAuth2 authentication is pretty much the industry standard these days. Facebook, Google, MailChimp, GitHub, LinkedIn and many more use OAuth2 authentication.

This is what powers the LinkedIn integration on Siwsh.ink. So you can check it out in the demo to see how it works.

package integrations

import (
	"context"
	"fmt"

	"gitlab.com/swishink/swishink/internal"
	"gitlab.com/swishink/swishink/internal/types"
	"golang.org/x/oauth2"
)

type AuthFlowOauth2 struct {
	Config *oauth2.Config

	// Set by the user, will be called in the Finish method
	GetCreds func(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error)
}

func (a AuthFlowOauth2) Do(ctx context.Context, req AuthReq) (*AuthResp, error) {
	if req.URL == "" {
		return nil, fmt.Errorf("No authURL in auth request")
	}

	if a.Config == nil {
		return nil, fmt.Errorf("oauth2 config is nil")
	}
	a.Config.RedirectURL = req.URL

	// To know if we are in the Oauth flow, we check if the code parameter is present
	// If it is not present, redirect to the authorization URL
	if req.FormValues.Get("code") == "" {
		return a.redirectToAuthorizationURL()
	}

	// If it is present, validate the callback
	return a.validateCallback(ctx, req)
}

func (a AuthFlowOauth2) redirectToAuthorizationURL() (*AuthResp, error) {
	// This is a dashboard page, we already validate that is is a legitimate user/request
	// So we can just use a random code
	authorizationURL := a.Config.AuthCodeURL(internal.GenRandomToken(12))
	return &AuthResp{RedirectTo: authorizationURL}, nil
}

func (a AuthFlowOauth2) validateCallback(ctx context.Context, req AuthReq) (*AuthResp, error) {
	code := req.FormValues.Get("code")
	token, err := a.Config.Exchange(ctx, code)
	if err != nil {
		return nil, fmt.Errorf("code exchange failed: %s", err.Error())
	}

	creds, err := a.GetCreds(ctx, a.Config, token)
	if err != nil {
		return nil, fmt.Errorf("Could not get oauth2 credentials: %w", err)
	}

	return &AuthResp{Credentials: creds}, nil
}

WordPress: A more complex flow

To integrate with WordPress, we needed an interesting flow. Here are the things to consider.

1. There's 2 flavors of WordPress. Hosted version(.com) and the Self hosted version(.org).
2. The hosted version (.com) uses OAuth2, while the self-hosted version needs a plugin to generate an application password which we would then use to validate.

Here's how the authentication flow for WordPress goes:

• Ask if it is a Hosted or Self-hosted WordPress install
• If it is Hosted, revert to an OAuth2 flow
• If it is self-hosted, ask for the website URL, the username and the application password. Then validate after it is submitted.

This is what powers the WordPress integration on Siwsh.ink. So you can check it out in the demo to see how it all works.

The full implementation is shown below.

package wordpress

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"html/template"
	"io/ioutil"
	"net/http"
	"net/url"
	"path/filepath"
	"time"

	"gitlab.com/swishink/swishink/internal/types"
	"gitlab.com/swishink/swishink/services/integrations"
	"golang.org/x/oauth2"
)

const (
	// Keys for our credentials
	wpAuthKey = "wp_auth"

	// Keys for self-hosted wordpress credentials
	wpURLKey         = "wp_url"
	wpUsernameKey    = "wp_user_login"
	wpAppPasswordKey = "wp_app_password"

	// Keys for wordpress.com credentials
	wpUserIDKey = "wp_user_id"
	wpBlogIDKey = "wp_blog_id"
	wpTokenKey  = "wp_token"

	wpTypeKey = "wp_type"
	wpTypeCom = "wp_type_com"
	wpTypeOrg = "wp_type_org"
)

var wordpressOauth2Endpoint = oauth2.Endpoint{
	AuthURL:  "https://public-api.wordpress.com/oauth2/authorize",
	TokenURL: "https://public-api.wordpress.com/oauth2/token",
}

type wordpress struct {
	ClientID     string
	ClientSecret string
}

func (w *wordpress) Auth() integrations.AuthFlow {
	return wpAuthFlow{
		Config: w.getOauth2Config(),
	}
}

type wpAuthFlow struct {
	Config *oauth2.Config
}

func (w wordpress) getOauth2Config() *oauth2.Config {
	return &oauth2.Config{
		ClientID:     w.ClientID,
		ClientSecret: w.ClientSecret,
		// Needs a "global" scope to access the v2 API
		// Scopes:       []string{"auth", "posts", "media"},
		Endpoint: wordpressOauth2Endpoint,
	}
}

func (w wpAuthFlow) oauth2() integrations.AuthFlowOauth2 {
	return integrations.AuthFlowOauth2{
		Config:   w.Config,
		GetCreds: w.validateComCreds,
	}
}

func (w wpAuthFlow) Do(ctx context.Context, req integrations.AuthReq) (*integrations.AuthResp, error) {
	if req.URL == "" {
		return nil, fmt.Errorf("No authURL in request context")
	}

	comOrOrgVals := comORorg.GetVals(req.FormValues)
	if len(comOrOrgVals) > 0 {
		wpType := comOrOrgVals["wp_type"]
		switch wpType {
		case wpTypeOrg:
			return &integrations.AuthResp{Form: wpOrgDetails}, nil
		default:
			return w.oauth2().Do(ctx, req)
		}
	}

	wpOrgVals := wpOrgDetails.GetVals(req.FormValues)
	if len(wpOrgVals) > 0 {
		creds, err := wpValidateOrgCreds(wpOrgVals)
		if err != nil {
			return nil, fmt.Errorf("Could not validate wp org credentials: %w", err)
		}

		return &integrations.AuthResp{Credentials: creds}, nil
	}

	// If "code" exists in the query parameters, continue the Oauth2 authentication
	if req.FormValues.Get("code") != "" {
		return w.oauth2().Do(ctx, req)
	}

	// If nothing else, sent the first form
	return &integrations.AuthResp{Form: comORorg}, nil
}

func (w wpAuthFlow) validateComCreds(ctx context.Context, config *oauth2.Config, token *oauth2.Token) (types.IntegrationCreds, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet,
		fmt.Sprintf("https://public-api.wordpress.com/oauth2/token-info?client_id=%s&token=%s",
			w.Config.ClientID, token.AccessToken), nil)
	if err != nil {
		return nil, fmt.Errorf("could not create WordPress token verify request: %w", err)
	}

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, fmt.Errorf("could not do WordPress token verify request: %w", err)
	}
	defer resp.Body.Close()

	bodyBytes, _ := ioutil.ReadAll(resp.Body)

	if resp.StatusCode != 200 {
		return nil, fmt.Errorf("Got status code %d in WordPress token verify response: %s", resp.StatusCode, string(bodyBytes))
	}

	var response = struct {
		ClientID string `json:"client_id"`
		UserID   string `json:"user_id"`
		BlogID   string `json:"blog_id"`
		Scope    string `json:"scope"`
	}{}
	if err = json.NewDecoder(bytes.NewBuffer(bodyBytes)).Decode(&response); err != nil {
		return nil, fmt.Errorf("could not decode WordPress token verify response: %w", err)
	}

	return types.IntegrationCreds{
		wpTypeKey:   wpTypeCom,
		wpUserIDKey: response.UserID,
		wpBlogIDKey: response.BlogID,
		wpTokenKey:  token.AccessToken,
	}, nil
}

func wpValidateOrgCreds(vals map[string]string) (types.IntegrationCreds, error) {

	wpURL, err := url.Parse(vals[wpURLKey])
	if err != nil {
		return nil, fmt.Errorf("could not parse wordpress url")
	}

	wpURL.Path = filepath.ToSlash(filepath.Join(wpURL.Path, "wp-json", "wp", "v2", "users", "me"))
	ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
	defer cancel()

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, wpURL.String(), nil)
	if err != nil {
		return nil, fmt.Errorf("could not create org /me request")
	}

	req.SetBasicAuth(vals[wpUsernameKey], vals[wpAppPasswordKey])

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, fmt.Errorf("Could not do WordPress /me request: %w", err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("Got status code %d from WordPress /me request", resp.StatusCode)
	}

	creds := types.IntegrationCreds{}
	creds[wpTypeKey] = wpTypeOrg
	creds[wpURLKey] = vals[wpURLKey]
	creds[wpUsernameKey] = vals[wpUsernameKey]
	creds[wpAppPasswordKey] = vals[wpAppPasswordKey]

	return creds, nil
}

var comORorg = integrations.AuthForm{
	Name: wpTypeKey,
	// Instructions: "Is your WordPress blog hosted on WordPress.com or do you have a self-hosted instance?",
	Options: []types.Option{
		{
			ID:       wpTypeKey,
			Required: true,
			Label:    "Type of website",
			Kind:     types.MustKind("select"),
			HelpText: "Is your WordPress blog hosted on WordPress.com or do you have a self-hosted instance?",
			Options: []types.SelectOption{
				{
					Label: "WordPress.com",
					Value: wpTypeCom,
				},
				{
					Label: "Self Hosted (WordPress.org)",
					Value: wpTypeOrg,
				},
			},
		},
	},
}

// Form to get the URL of the self-hosted wordpress instance
var wpOrgDetails = integrations.AuthForm{
	Name:         "wp_api_details",
	Instructions: template.HTML(`You should have <a href="https://wordpress.org/plugins/application-passwords/" target="_blank" rel="noopener">this plugin</a> installed on your site so we can authenticate to the API.<br>You should then generate a new application password from your profile settings page.`),
	Options: []types.Option{
		{
			ID:       wpURLKey,
			Label:    "Website URL",
			HelpText: "The URL of your WordPress website.",
			Required: true,
			Kind:     types.MustKind("url"),
		},
		{
			ID:       wpUsernameKey,
			Label:    "Username",
			HelpText: "Your username on the WordPress website.",
			Required: true,
			Kind:     types.MustKind("text"),
		},
		{
			ID:       wpAppPasswordKey,
			Label:    "Application Password",
			HelpText: "The generated application password. <b>DO NOT</b> use your login password!",
			Required: true,
			Kind:     types.MustKind("text"),
		},
	},
}

Conclusion

We went through a couple iterations and ended up with an authentication flow that can be used for almost any possible flow.

Works somewhat like recursion. We'll keep calling the method with the values from the previous step until we get the final credentials or an error.

We also looked a few implementations of this authentication flow to show how it looks in practice.

If you have any questions/comments, let me know on Twitter.