REA

Stephen M. Reaves

2022-05-25

Reverse Engineering Apis

REA

rea is a tool used to reverse-engineer apis. Specifically, it is meant to take JSON API response and build Golang structs in a way that makes them easy to digest in existing go code.

Examples

Basic

The simplest example is to give it a json file and to look at the result. For example:

curl 'https://www.boredapi.com/api/activity' | rea && cat out.go

and you should see something like this:

// Auto-generated by rea
package main

// Object was auto-generated by rea
type Object struct {
	Accessibility float64 `json:"accessibility"`
	Activity      string  `json:"activity"`
	Key           string  `json:"key"`
	Link          string  `json:"link"`
	Participants  float64 `json:"participants"`
	Price         float64 `json:"price"`
	Type          string  `json:"type"`
}

boom. You got a struct. But you have to use an external command? That’s no bueno. You can instead store the json in a file, then specify the file with the -f flag, like so:

rea -f joo.json && cat out.go

But what if you don’t want to store something on your local machine either? Probably the best way to use rea is to let it make the call for you with the -w flag:

rea -w 'https://www.boredapi.com/api/activity' && cat out.go

The last command will handle everything internally and produce the same output as above. Actually all three of these methods will produce the same output, but you may want to use different options depending on your situation.

Advanced

There are quite a few more flags that can be used to change the output. Running

rea -h

will produce the following result:

Usage of rea:
  -a    Append?
  -f string
        File to read from
  -o string
        File to output to (default "out.go")
  -p string
        Package name of generated file (default "main")
  -s string
        Name of struct to be generated (default "Object")
  -v    Verbose output?
  -w curl
        Provide a url instead of a filename to have rea curl it for you

In fact, we can use multiple flags in conjuction with each other.

Let’s say you have the following Golang code:

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"os"
)

func main() {
	if err := run(); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
}

func run() error {
	var a Account
	a.First_name = "Stephen"
	fmt.Println(a)

	resp, err := http.Get("https://www.boredapi.com/api/activity")
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return err
	}

	var activity Activity
	Jerr := json.Unmarshal(body, &activity)
	if Jerr != nil {
		return err
	}

	fmt.Println(activity.Activity)
	fmt.Println(activity.Type)

	return nil
}

This code will not work, precisely because the structs Account and Activity do not exist. Luckily, we can let rea fill in the blanks.

Simply running ocm whoami | rea -s account -o main.go -a and rea -w 'https://www.boredapi.com/api/activity' -o main.go -a -s activity will give you the following code:

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
	"os"
)

func main() {
	if err := run(); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
}

func run() error {
	var a Account
	a.First_name = "Stephen"
	fmt.Println(a)

	resp, err := http.Get("https://www.boredapi.com/api/activity")
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return err
	}

	var activity Activity
	Jerr := json.Unmarshal(body, &activity)
	if Jerr != nil {
		return err
	}

	fmt.Println(activity.Activity)
	fmt.Println(activity.Type)

	return nil
}

// Account was auto-generated by rea
type Account struct {
	Created_at   string       `json:"created_at"`
	Email        string       `json:"email"`
	First_name   string       `json:"first_name"`
	Href         string       `json:"href"`
	Id           string       `json:"id"`
	Kind         string       `json:"kind"`
	Last_name    string       `json:"last_name"`
	Organization Organization `json:"organization"`
	Updated_at   string       `json:"updated_at"`
	Username     string       `json:"username"`
}

// Organization was auto-generated by rea
type Organization struct {
	Created_at     string `json:"created_at"`
	Ebs_account_id string `json:"ebs_account_id"`
	External_id    string `json:"external_id"`
	Href           string `json:"href"`
	Id             string `json:"id"`
	Kind           string `json:"kind"`
	Name           string `json:"name"`
	Updated_at     string `json:"updated_at"`
}

// Activity was auto-generated by rea
type Activity struct {
	Accessibility float64 `json:"accessibility"`
	Activity      string  `json:"activity"`
	Key           string  `json:"key"`
	Link          string  `json:"link"`
	Participants  float64 `json:"participants"`
	Price         float64 `json:"price"`
	Type          string  `json:"type"`
}

Now the Account and Activity structs are available and the code is working! Running the code will produce something similar to:

{  Stephen     {       }  }
Look at pictures and videos of cute animals
relaxation

We could also put the code in a different file and/or package by using the -o and -p options.

Code Deep Dive

So how does this magic work? Let’s take a look.

cmd/rea/main.go

func run() error {
	var b []byte
	var e error

	if config.FromStdIn {
		b, e = ioutil.ReadAll(os.Stdin)
	} else if config.WebsiteName != "" {
		b, e = get(config.WebsiteName)
	} else {
		b, e = os.ReadFile(config.Input)
	}
	if e != nil {
		return e
	}

	if config.Verbose {
		fmt.Println("Input:")
		fmt.Println(string(b))
	}

	if err := parseAndWrite(b); err != nil {
		return err
	}

	if err := formatOutput(config.Output); err != nil {
		return err
	}

	if config.Verbose {
		fmt.Println("Output written to: ", config.Output)
	}

	return nil
}

This is the high-level overview of what the code does. You can see we handle some input, then we parse the JSON and format the output. Most of that is sort of boilerplate, generic go-code. The parsing of the JSON is the real meat and potatoes here, so let’s dive into the parseAndWrite function.

func parseAndWrite(j []byte) error {
	var b map[string]interface{}
	if err := json.Unmarshal(j, &b); err != nil {
		return err
	}

	data := generators.TplTuple{
		Name:   config.StructName,
		Fields: b,
	}

	var out *os.File
	var err error

	if config.Append {
		out, err = os.OpenFile(config.Output, os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)
	} else {
		out, err = os.Create(config.Output)
	}
	if err != nil {
		return err
	}
	defer out.Close()

	t := generators.New()
	t.NestedParse(data)

	_, err = t.Parse()
	if err != nil {
		return err
	}

	err = t.Execute(out, data)
	if err != nil {
		return err
	}

	return nil
}

So we start by taking in j []byte, which holds the raw json request, then we shove that into a generic map. There a numerous examples online of how to do this, so I won’t spend a lot of time on it, but if you are confused as to how this works, it would behoove you to read up on that subject then come back.

Next, we move on to creating a simple struct to hold the jsonMap and the name of the top level struct, and we create or append a file.

The exciting part is the

t := generators.New()
t.NestedParse(data)

The New function returns a generic template that looks like this:

func DefaultTemplate() string {
	var s string

	if !config.Append {
		s += `
// Auto-generated by rea
package ` + config.PkgName
	}

	s += `

// {{(Title .Name)}} was auto-generated by rea
type {{(Title .Name)}} struct {
{{- range $jsonName, $val := .Fields -}}
	{{if (IsNested $jsonName $val) -}}
		{{(Title $jsonName)}} {{(Title $jsonName)}} ` + "`" + `json:"{{$jsonName}}"` + "`" + `
	{{else -}}
		{{(Title $jsonName)}} {{(TypeOf $val)}} ` + "`" + `json:"{{$jsonName}}"` + "`" + `
	{{end -}}
{{- end -}}
}

This is using the text/template package which is extremely powerful. Basically what this template is saying is that we need to loop over the fields, extract their name and type, then place that information where golang expects to find it.