Edward's Tech Site

this site made with Next.js 13, see the code

FORAY: Jul 12, 2024 - Go
Get Go API running at Hetzner with read/write routes using JWT authentication

DOING: taking a few courses on Go to get up to speed, then will connect ArangoDB...

  • background
    • before I start working through more Go courses
      • I want to get a base, realistic Go project online
      • so that I can constantly add to it while learning Go
    • in order for it to be as realistic as possible
      • it should be an online read/write API
      • that has a e.g. local React frontend which allows a user to log in and stay logged in with JWT
    • at this point, the database is not important
      • we can just use file access to a JSON file
      • here is some code that reads an online JSON file and produces an HTML file from it
    • after this is running, the Go courses will be more useful
      • since any interesting/userful concept I learn
      • I can build it into a live project that is online and
      • can be used as a base for creating useful, public-facing projects e.g.
        • Book Notes site
        • Tech Vibe site
        • Vim Skills site
        • Project/Time Management site (local)
    • steps after this will be adding
      • database connectivity including ArangoDB
      • GraphQL capability
      • more robust frontends for this API with React/Next.js
      • these frontends can run at Hetzner for practice
        • but more easily and more stable: at Vercel
  • infos
  • overview
  • >>> 1. local read-only Go API
    • main.go
      • package main
         
        import (
        "encoding/json"
        "fmt"
        "net/http"
        "strconv"
        )
         
        func main() {
         
        port := 7788
         
        http.HandleFunc("/languages", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Access-Control-Allow-Origin", "*")
        w.Header().Set("Access-Control-Allow-Methods", "GET, OPTIONS")
        w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
         
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode([]string{"C#", "Java", "Ruby", "Python", "JavaScript", "Go"})
        })
        fmt.Printf("listening at http://localhost:%v/languages\n", port)
        http.ListenAndServe(":"+strconv.Itoa(port), nil)
        }
    • start with go run main.go, works
    • looking how to do hot-reloading
      • you could use reflex, air, or fresh
        • air is recommended
    • making frontend to consume this
      • creating site showcase-go-use-api
      • using:
      • PageWelcome.tsx
        • import { useEffect, useState } from "react";
           
          const url = "http://localhost:7788/languages";
           
          export const PageWelcome = () => {
          const [languages, setLanguages] = useState<string[]>([]);
           
          useEffect(() => {
          (async () => {
          const response = await fetch(url);
          const _languages: string[] = await response.json();
          setLanguages(_languages);
          })();
          }, []);
           
          return (
          <>
          <p>There are there {languages.length} languages.</p>
          </>
          );
          };
    • works locally
  • >>> 2. create online read-only Go API at Hetzner
    • creating directory at Hetzner and cloning, ok
    • for some reason the CPU shot up:
    • it seems every time I start go run main.go, it shoots up to over 150% CPU
    • it finally ran though
    • trying a simpler API:
      • main2.go
        • package main
           
          import (
          "fmt"
          "net/http"
          )
           
          func main() {
          http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
          fmt.Fprintln(w, "Hello, World!")
          })
          fmt.Println("Listening on :8080")
          http.ListenAndServe(":8080", nil)
          }
      • this works online:
    • by the way, you have to rename any other file that has a main() function it it
    • ok, looking for differences that would cause the first to hang
      • trying again as it was before
      • and it runs just fine, no changes:
    • setting up subdomain for Go API
    • adding main page with link
      • works
    • adding scripts to deploy with PM2
      • I'm sure there is a better way, but using npm scripts for now since I know they work
      • I'm also sure that the better way is to have a compile process and run the compiled version, but we are going to run it as in dev for now when we serve, so it works
      • package.json
        • {
          "name": "showcase-go-api",
          "version": "1.0.0",
          "description": "",
          "main": "index.js",
          "scripts": {
          "start": "go run main.go",
          "setup": "pm2 start --name showcase-go-api npm -- start && pm2 save",
          "deploy": "git pull --no-rebase && pm2 restart showcase-go-api --update-env --time && pm2 save"
          },
          "keywords": [],
          "author": "",
          "license": "ISC"
          }
    • works:
  • >>> 3. set up frontend site at Vercel that reads data from this API
  • >>> 4. connect an SQLite database
    • ultimately I want to get this API running with ArangoDB
      • but since I am new to Go, want to at least get most standard and simple database connection set up: SQLite
    • set up environment variable so
      • development uses: ../localhost:7788http://localhost:7788
      • production uses: ../showcase-go-api.tanguay.euhttps://showcase-go-api.tanguay.eu
      • .env
        • TEST = 12345
      • this is empty
        • test := os.Getenv("TEST")
          fmt.Printf("Test is: [%v]\n", test)
      • it doesn't work, I get for go get github.com/joho/godotenv
        • go get github.com/joho/godotenv
          go: go.mod file not found in current directory or any parent directory.
          'go get' is no longer supported outside a module.
          To build and install a command, use 'go install' with a version,
          like 'go install example.com/cmd@latest'
          For more information, see https://golang.org/doc/go-get-install-deprecation
          or run 'go help get' or 'go help install'.
    • ok that was just a test
    • I need to install the environment variables on the frontend anyway
      • works, loading languages from tools
    • set up to read from SQLite database
      • install SQLite
        • go get github.com/mattn/go-sqlite3
          • got: go: go.mod file not found in current directory or any parent directory.
      • have to set up new Go Module
        • go mod init showcase-go-api
      • setting up SQLite database
      • added this to tools.go
        • func getHowtos() {
           
          // Open the database
          db, err := sql.Open("sqlite3", "./data/main.sqlite")
          if err != nil {
          fmt.Println("Error opening database:", err)
          return
          }
          defer db.Close()
           
          // Query the database
          rows, err := db.Query("SELECT id, category, title FROM howtos")
          if err != nil {
          fmt.Println("Error querying database:", err)
          return
          }
          defer rows.Close()
           
          // Iterate over the rows
          for rows.Next() {
          var id int
          var category string
          var title string
          err = rows.Scan(&id, &category, &title)
          if err != nil {
          fmt.Println("Error scanning row:", err)
          return
          }
          fmt.Printf("id: %d, category: %s, title: %s\n", id, category, title)
          }
           
          // Check for errors from iterating over rows
          err = rows.Err()
          if err != nil {
          fmt.Println("Error iterating over rows:", err)
          return
          }
          }
      • but when I run it, I get the error:
        • Error querying database: Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub
        • a solution is to build as a Dockerfile
      • so, aborting SQLite for now until I can set it up with Docker
  • >>> 5. connect an MongoDB Atlas database
    • local database
      • checking local Mongo database connection, it works
      • connect with Node to make sure connection string is correct
      • getting code to read from the following collection
        • MONGO_DATABASE = template-api-read-write-mongo
          MONGO_COLLECTION = todos
      • install driver
        • go get go.mongodb.org/mongo-driver/mongo
          go get go.mongodb.org/mongo-driver/mongo/options
      • it gets data from local Mongo
    • MongoDB Atlas
      • get environment variables working, since we don't want to included Atlas
        • reading ARTICLE: Different ways to use environment variables in Golang
          • install driver
            • go get github.com/joho/godotenv
          • tools.go
            • import (
              "os"
              "github.com/joho/godotenv"
              )
               
              func testEnvironmentVariable() {
              // Load the .env file
              err := godotenv.Load()
              if err != nil {
              fmt.Println("Error loading .env file")
              return
              }
              test := os.Getenv("TEST")
              fmt.Printf("Test = [%s]", test)
              }
          • works
          • .env
            • TEST = 123
          • adding .gitignore
            • .env
      • environment variables with local connection works
        • func getTodosWithMongo() {
          err := godotenv.Load()
          if err != nil {
          fmt.Println("Error loading .env file")
          return
          }
          mongo_conn := os.Getenv("MONGO_CONNECTION")
          mongo_database := os.Getenv("MONGO_DATABASE")
          mongo_collection := os.Getenv("MONGO_COLLECTION")
           
          clientOptions := options.Client().ApplyURI(mongo_conn)
           
          client, err := mongo.Connect(context.TODO(), clientOptions)
          if err != nil {
          log.Fatal(err)
          }
           
          err = client.Ping(context.TODO(), nil)
          if err != nil {
          log.Fatal(err)
          }
           
          fmt.Println("Connected to MongoDB!")
           
          collection := client.Database(mongo_database).Collection(mongo_collection)
           
          filter := bson.D{}
           
          var results []bson.M
          ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
          defer cancel()
          cursor, err := collection.Find(ctx, filter)
          if err != nil {
          log.Fatal(err)
          }
          if err = cursor.All(ctx, &results); err != nil {
          log.Fatal(err)
          }
           
          for _, result := range results {
          fmt.Printf("Found document: %v\n", result)
          }
           
          err = client.Disconnect(context.TODO())
          if err != nil {
          log.Fatal(err)
          }
          fmt.Println("Connection to MongoDB closed.")
          }
      • also works with Atlas connection data
      • getting useful data
      • at MongoDB Atlas, set up mock-data user and database for skills
      • outputting data from MongoDB:
        • type Skill struct {
          IDCode string
          Name string
          }
           
          func getSkillsFromMongo() []Skill {
          err := godotenv.Load()
          if err != nil {
          fmt.Println("Error loading .env file")
          }
          mongo_conn := os.Getenv("MONGO_CONNECTION")
          mongo_database := os.Getenv("MONGO_DATABASE")
          mongo_collection := os.Getenv("MONGO_COLLECTION")
           
          clientOptions := options.Client().ApplyURI(mongo_conn)
           
          client, err := mongo.Connect(context.TODO(), clientOptions)
          if err != nil {
          log.Fatal(err)
          }
           
          err = client.Ping(context.TODO(), nil)
          if err != nil {
          log.Fatal(err)
          }
           
          fmt.Println("Connected to MongoDB!")
           
          collection := client.Database(mongo_database).Collection(mongo_collection)
           
          filter := bson.D{}
           
          var results []bson.M
          ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
          defer cancel()
          cursor, err := collection.Find(ctx, filter)
          if err != nil {
          log.Fatal(err)
          }
          if err = cursor.All(ctx, &results); err != nil {
          log.Fatal(err)
          }
           
          err = client.Disconnect(context.TODO())
          if err != nil {
          log.Fatal(err)
          }
          fmt.Println("Connection to MongoDB closed.")
           
          var skills []Skill
          for _, result := range results {
          var idCode, name string
          if result["idCode"] != nil {
          idCode = result["idCode"].(string)
          }
          if result["name"] != nil {
          name = result["name"].(string)
          }
          skill := Skill{
          IDCode: idCode,
          Name: name,
          }
          skills = append(skills, skill)
          }
           
          return skills
           
          }
           
          http.HandleFunc("/skills", func(w http.ResponseWriter, r *http.Request) {
          w.Header().Set("Access-Control-Allow-Origin", "*")
          w.Header().Set("Access-Control-Allow-Methods", "GET, OPTIONS")
          w.Header().Set("Access-Control-Allow-Headers", "Content-Type")
          w.Header().Set("Content-Type", "application/json")
          json.NewEncoder(w).Encode(getSkillsFromMongo())
          })
      • so works locally
      • now getting MongoDB to work on Hetzner
        • add .env file and deploy
        • Hetzner machine went to 200% CPU again
        • restarted and is ok
        • works now with skills coming from MongoDB atlas
    • updating frontpage for Hetzner MongoDB example
      • set up mainsite as open project and push link to this project
      • works well, online at frontend:
    1. connect a ArangoDB database
    • .. taking a few courses on Go to get up to speed, then will connect ArangoDB