[Updated] Reflection

README.md

@@ -36,3 +36,14 @@ sudo reboot
 docker compose up
 ```
 - Visit localhost:4200 to play the quiz!
+
+
+## Benchmarking
+- Go to the /scripts folder and run the LoadTestMultiple.sh script
+- Inside you can change how many sessions should be created (set to 6 by default)
+- In the /scripts/benchmarks folder you can find some benchmarks I've done
+
+## Notes
+- If the data is not present, it will try to query it from the external API, which takes significantly longer (only 1 request per 5 seconds)
+- When running it locally, give the backend a few minutes to gather data in bulk (should be completely done in ~7 minutes)
+- It refreshes this data every night in the background, so it stays up to date

Reflection.md

@@ -1,20 +1,20 @@
-# Issues
+# Reflection
 
-## Issues during testing
+## Issues during testing (October 30th)
 ### Scheduler
 I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed.
 I don't know how to solve this yet
 
 
-## Caching solution
+## Caching solution (October 30th)
 ### Lack of data
-Not being able to gather blocks of 50 is a huge issue. The idea of reusing blocks becomes less viable if I can just get a big chunk every time for others to use
+Not being able to gather blocks of 50 is a big issue. The idea of reusing blocks becomes less viable if I can't just get a big chunk every time for others to use.
 I can't force a big chunk, because it might just return a status 1, and then the user gets nothing.
 
 Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain.
 
 
-## Possible solution
+## Possible solution (October 31st)
 Cache all data, 50 at a time
 Store it in a database
 When its done, let the db make blocks for quick cache access
@@ -28,3 +28,36 @@ Then start replacing the cache items one at a time.
 
 Users shouldn't feel this at all, because the active sessions are stored in their own cache items
 
+# Progress so far (November 3rd)
+There only seem to be ~4100 items, making the collecting phase only last around 7 minutes.
+
+Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up
+
+TODO: Add a caching state, and start filling up each unique combination with the items from the db
+This should lead to performance gains
+
+## Possible future alternative
+In the current state (when the cache is done), the backend might hold data 3 times (Cache, active DB table, and background DB table during the night)
+
+### Purely Cached solution
+- Every night
+- Check for each category+difficulty combination how many questions there are (API has an endpoint for this)
+- Track which one you are on, and how many you have
+- Take blocks of 50, until the remaining items are less than 50, then get that exact amount
+- When they are gathered, either 
+    1. Divide them between 3 cache items: cat:dif:any/multiple/boolean 
+    - (Manually filter on the type, since the API sends questions with either option together), or 
+    2. In one like cat:dif:any and then filter it by hand when a user asks for it
+    - (This will replace the collection put there the night before)
+- Then move to the next one
+- Stop when the last combination has been collected
+
+#### When a user requests data: 
+- Get the items from the cache
+- Less items than requested: Inform the user to try a different combination
+- More items than requested: Randomize and select a subset
+
+### Tradeoffs
+- Advantage: Less total data held on the server
+
+- Downside: Slower gathering, since there are 25+3+4=300 unique combinations (each of which will end with a request of less than 50 items), it will require a decent number requests more than the bulk collecting that the DB does.

backend/Dockerfile

@@ -22,11 +22,8 @@ CMD ["tail","-f","/dev/null"]
 #---Building
 FROM base AS build
 COPY . .
-RUN mvn clean package -DskipTests
+RUN mvn clean package 
-
+# Add -DskipTests for slightly faster build, but no testing
-#--Testing
-FROM build AS testing
-RUN mvn clean test
 
 #---Production
 # Pulled again to get rid of maven, to decrease the size

backend/pom.xml

@@ -32,10 +32,10 @@
     	<caffeine.version>3.1.8</caffeine.version>
 	</properties>
 	<dependencies>
-		<!-- <dependency>
+		<dependency>
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-starter-data-jpa</artifactId>
-		</dependency> -->
+		</dependency>
 		<dependency>
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-starter-web</artifactId>
@@ -47,11 +47,11 @@
 			<scope>runtime</scope>
 			<optional>true</optional>
 		</dependency>
-		<!-- <dependency>
+		<dependency>
 			<groupId>com.h2database</groupId>
 			<artifactId>h2</artifactId>
 			<scope>runtime</scope>
-		</dependency> -->
+		</dependency>
 		<dependency>
 			<groupId>org.springframework.boot</groupId>
 			<artifactId>spring-boot-starter-test</artifactId>
@@ -72,6 +72,11 @@
     		<groupId>org.springframework.boot</groupId> 
     		<artifactId>spring-boot-starter-validation</artifactId> 
 		</dependency>
+		<dependency>
+    		<groupId>org.apache.commons</groupId>
+    		<artifactId>commons-text</artifactId>
+    		<version>1.10.0</version>
+		</dependency>
 	</dependencies>
 
 	<build>

backend/src/main/java/com/ninjaws/quiz/components/StartupRunner.java

@@ -0,0 +1,40 @@
+package com.ninjaws.quiz.components;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.CommandLineRunner;
+import org.springframework.stereotype.Component;
+
+import com.ninjaws.quiz.services.ApiService;
+import com.ninjaws.quiz.services.StateManagementService;
+
+
+@Component
+public class StartupRunner implements CommandLineRunner {
+
+    @Autowired
+    private StateManagementService stateManagementService;
+    @Autowired
+    private ApiService apiService;
+
+    @Override
+    public void run(String... args) {
+
+        /**
+         * On startup:
+         * - Set ProcessingState to Startup
+         * - Store the categories
+         * - Get a sessiontoken
+         * 
+         * * Inside the SchedulerService
+         * - Start filling the question table
+         * 
+         * * Inside the Response Handling Service
+         * - No more questions? Stop
+         * - Set ProcessingState to Normal
+         */
+
+        stateManagementService.setCurrentProcessingState(StateManagementService.ProcessingState.STARTUP);
+        apiService.getCategories();
+        apiService.getSessionToken();
+    }
+}

backend/src/main/java/com/ninjaws/quiz/controllers/QuizController.java

@@ -1,5 +1,6 @@
 package com.ninjaws.quiz.controllers;
 
+import java.util.List;
 import java.util.Optional;
 
 import org.springframework.beans.factory.annotation.Autowired;
@@ -11,9 +12,11 @@ import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.RequestParam;
 import org.springframework.web.bind.annotation.RestController;
 
+import com.ninjaws.quiz.entities.Category;
 import com.ninjaws.quiz.models.Answers;
 import com.ninjaws.quiz.models.QuizSettings;
 import com.ninjaws.quiz.models.Session;
+import com.ninjaws.quiz.services.DataService;
 import com.ninjaws.quiz.services.QuizService;
 
 @RestController
@@ -22,6 +25,9 @@ public class QuizController {
 
     @Autowired
     private QuizService quizService;
+    @Autowired
+    private DataService dataService;
+
 
     /**
      * Start a session
@@ -68,4 +74,9 @@ public class QuizController {
         String jsonString = String.format("{\"score\":\"%s\"}", quizService.getScore(answers));
         return jsonString;
     }
+
+    @GetMapping("/categories")
+    public List<Category> getCategories(){
+        return dataService.getCategories();
+    }
 }

backend/src/main/java/com/ninjaws/quiz/entities/BaseQuestionEntity.java

@@ -0,0 +1,34 @@
+package com.ninjaws.quiz.entities;
+
+import java.util.List;
+import java.util.UUID;
+
+import jakarta.persistence.GeneratedValue;
+import jakarta.persistence.GenerationType;
+import jakarta.persistence.Id;
+import jakarta.persistence.JoinColumn;
+import jakarta.persistence.ManyToOne;
+import jakarta.persistence.MappedSuperclass;
+import lombok.Getter;
+import lombok.Setter;
+
+@MappedSuperclass
+@Getter
+@Setter
+public class BaseQuestionEntity {
+    @Id
+    @GeneratedValue(strategy = GenerationType.UUID)
+    private UUID id;
+
+    @ManyToOne
+    @JoinColumn(name = "category", nullable = false)
+    private Category category;
+
+    private String difficulty;
+    private String type;
+    private String question;
+    
+    private String correctAnswer;    
+    // @ElementCollection // This stores it in a separate table, it's just a set of strings so not required right now
+    private List<String> incorrectAnswers;
+}

backend/src/main/java/com/ninjaws/quiz/entities/Category.java

@@ -0,0 +1,25 @@
+package com.ninjaws.quiz.entities;
+
+import jakarta.persistence.Entity;
+import jakarta.persistence.Id;
+import jakarta.persistence.Table;
+import lombok.Getter;
+import lombok.Setter;
+
+@Entity
+@Table(name = "category")
+@Getter
+@Setter
+public class Category {
+    @Id
+    private Integer id;
+    private String name;
+
+    public Category() {}
+
+    public Category(int id, String name) {
+        this.id = id;
+        this.name = name;
+    }
+    
+}

backend/src/main/java/com/ninjaws/quiz/entities/QuestionCollectingEntity.java

@@ -0,0 +1,10 @@
+package com.ninjaws.quiz.entities;
+
+import jakarta.persistence.Entity;
+import jakarta.persistence.Table;
+
+@Entity
+@Table(name = "question_collecting")
+public class QuestionCollectingEntity extends BaseQuestionEntity{
+    
+}

backend/src/main/java/com/ninjaws/quiz/entities/QuestionEntity.java

@@ -0,0 +1,10 @@
+package com.ninjaws.quiz.entities;
+
+import jakarta.persistence.Entity;
+import jakarta.persistence.Table;
+
+@Entity
+@Table(name = "question")
+public class QuestionEntity extends BaseQuestionEntity {    
+
+}

backend/src/main/java/com/ninjaws/quiz/models/ApiResponse.java

@@ -17,5 +17,5 @@ public class ApiResponse {
     @JsonProperty("response_code")
     private int responseCode;
     @JsonProperty("results")
-    private List<Question> results;
+    private List<QuestionDTO> results;
 }

backend/src/main/java/com/ninjaws/quiz/models/CategoryResponse.java

@@ -0,0 +1,14 @@
+package com.ninjaws.quiz.models;
+
+import java.util.List;
+
+import com.ninjaws.quiz.entities.Category;
+
+import lombok.Getter;
+import lombok.Setter;
+
+@Getter
+@Setter
+public class CategoryResponse {
+    List<Category> trivia_categories;
+}

backend/src/main/java/com/ninjaws/quiz/models/QuestionDTO.java

@@ -8,14 +8,16 @@ import com.fasterxml.jackson.annotation.JsonIgnore;
 import com.fasterxml.jackson.annotation.JsonProperty;
 
 import lombok.Getter;
+import lombok.Setter;
 
 @Getter
-public class Question {
+@Setter
-    private String type;
+public class QuestionDTO {
-    private String difficulty;
-    private String category;
-    private String question;
 
+    private String category;
+    private String difficulty;
+    private String type;
+    private String question;
     
     //* Hidden from the user */
     private String correctAnswer;    

backend/src/main/java/com/ninjaws/quiz/models/QuizSettings.java

@@ -1,8 +1,5 @@
 package com.ninjaws.quiz.models;
 
-import com.ninjaws.quiz.models.QuizSettings.Category;
-import com.ninjaws.quiz.models.QuizSettings.Difficulty;
-
 import jakarta.validation.Valid;
 import jakarta.validation.constraints.Max;
 import jakarta.validation.constraints.Min;
@@ -13,56 +10,24 @@ import lombok.Setter;
 @Setter
 public class QuizSettings {
 
-    public enum Type {
-        ANY, BOOLEAN, MULTIPLE;
-
-        public String getApiValue() {
-            return this.name().toLowerCase();
-        }
-    }
-
-    public enum Difficulty {
-        ANY, EASY, MEDIUM, HARD;
-
-        public String getApiValue() {
-            return this.name().toLowerCase();
-        }
-    }
-
-    public enum Category {
-        ANY, GEN_KNOWLEDGE, ENT_BOOKS, ENT_FILM, ENT_MUSIC, ENT_MUSICAL_THEATRE, ENT_TELEVISION, 
-        ENT_VIDEOGAMES, ENT_BOARDGAMES, SCIENCE_NATURE, SCIENCE_COMPUTERS, SCIENCE_MATH, MYTHOLOGY, SPORTS, GEOGRAPHY, HISTORY, POLITICS, ART, CELEBRITIES, 
-        ANIMALS, VEHICLES, ENT_COMICS, SCIENCE_GADGETS, ENT_ANIME_MANGA, ENT_CARTOON_ANIMATIONS;
-        
-        /**
-         * For some reason the external API starts with a value of 9 for the General Knowledge category, so we have to adjust
-         * ANY also gets adjusted, but it doesn't matter since that one doesn't get used as parameter anyway
-         * @return The corrected value
-         */
-        public int getApiValue() {
-            return 8 + this.ordinal();
-        }
-    }
-
     @Min(1)
     @Max(50)
     private int amount = 10;
-
+    private Integer category;
-    private Type type = Type.ANY;
+    private String difficulty;
-
+    private String type;
-    private Difficulty difficulty = Difficulty.ANY;
-
-    private Category category = Category.ANY;
-
 
     public QuizSettings(){}
 
     /** Inverse of toApiQuery */
     public QuizSettings(int amount, Integer category, String difficulty, String type) {    
         this.amount = amount;
-        this.category = category == null ? Category.ANY : Category.values()[category];
+        this.category = category;
-        this.difficulty = difficulty == null ? Difficulty.ANY : Difficulty.valueOf(difficulty.toUpperCase());
+        this.difficulty = difficulty;
-        this.type = type == null ? Type.ANY : type.equals("boolean") ? Type.BOOLEAN : Type.MULTIPLE;
+        this.type = type;
+        // this.category = category == null ? Category.ANY : Category.values()[category];
+        // this.difficulty = difficulty == null ? Difficulty.ANY : Difficulty.valueOf(difficulty.toUpperCase());
+        // this.type = type == null ? Type.ANY : type.equals("boolean") ? Type.BOOLEAN : Type.MULTIPLE;
     }
 
     /**
@@ -80,16 +45,16 @@ public class QuizSettings {
     public String toApiQuery() {
         StringBuilder query = new StringBuilder("?amount=" + amount);
         
-        if (category != null && category != Category.ANY) {
+        if (category != null) {
-            query.append("&category=").append(category.getApiValue());
+            query.append("&category=").append(category);
         }
 
-        if (difficulty != null && difficulty != Difficulty.ANY) {
+        if (difficulty != null ) {
-            query.append("&difficulty=").append(difficulty.getApiValue());
+            query.append("&difficulty=").append(difficulty);
         }
 
-        if (type != null && type != Type.ANY) {
+        if (type != null ) {
-            query.append("&type=").append(type.getApiValue());
+            query.append("&type=").append(type);
         }
 
         return query.toString();

backend/src/main/java/com/ninjaws/quiz/models/Session.java

@@ -11,7 +11,7 @@ import lombok.Setter;
 public class Session {
     private final String id;
     private int statusCode = 0;
-    private List<Question> questions;
+    private List<QuestionDTO> questions;
 
     public Session() {
         this.id = UUID.randomUUID().toString();

backend/src/main/java/com/ninjaws/quiz/models/SessionTokenDTO.java

@@ -0,0 +1,15 @@
+package com.ninjaws.quiz.models;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import lombok.Getter;
+import lombok.Setter;
+
+@Getter
+@Setter
+public class SessionTokenDTO {
+    @JsonProperty("response_code")
+    private int responseCode;
+    @JsonProperty("token")
+    private String token;    
+}

backend/src/main/java/com/ninjaws/quiz/repositories/CategoryRepository.java

@@ -0,0 +1,11 @@
+package com.ninjaws.quiz.repositories;
+
+import java.util.Optional;
+
+import org.springframework.data.jpa.repository.JpaRepository;
+
+import com.ninjaws.quiz.entities.Category;
+
+public interface CategoryRepository extends JpaRepository<Category, Long>{
+    Optional<Category> findByName(String name);
+}

backend/src/main/java/com/ninjaws/quiz/repositories/QuestionCollectingRepository.java

@@ -0,0 +1,14 @@
+package com.ninjaws.quiz.repositories;
+
+import org.springframework.data.jpa.repository.JpaRepository;
+
+import com.ninjaws.quiz.entities.QuestionCollectingEntity;
+
+/**
+ * This table is used for collecting questions from the external API
+ * When it finishes, the data from this repository will replace the data in the QuestionRepository, to ensure it stays fresh
+ * This should be scheduled once every day
+ */
+public interface QuestionCollectingRepository extends JpaRepository<QuestionCollectingEntity, Long>{
+    
+}

backend/src/main/java/com/ninjaws/quiz/repositories/QuestionRepository.java

@@ -0,0 +1,39 @@
+package com.ninjaws.quiz.repositories;
+
+import java.util.List;
+
+import org.springframework.data.jpa.repository.JpaRepository;
+import org.springframework.data.jpa.repository.Modifying;
+import org.springframework.data.jpa.repository.Query;
+import org.springframework.data.repository.query.Param;
+
+import com.ninjaws.quiz.entities.QuestionEntity;
+
+import jakarta.transaction.Transactional;
+
+public interface QuestionRepository extends JpaRepository<QuestionEntity, Long>{//, JpaSpecificationExecutor<Question> {
+
+    // @Query(value = "SELECT * FROM question AS q " +
+    // "WHERE (?1 IS NULL OR q.category = ?1) " + 
+    // "AND (?2 IS NULL OR q.difficulty = ?2) " +
+    // "AND (?3 IS NULL OR q.type = ?3) " +
+    // "ORDER BY RAND() LIMIT ?4", nativeQuery = true)
+    @Query(value = "SELECT * FROM question AS q " +
+        "WHERE (:categoryId IS NULL OR q.category = :categoryId) " + 
+        "AND (:difficulty IS NULL OR q.difficulty = :difficulty) " +
+        "AND (:type IS NULL OR q.type = :type) " +
+        "ORDER BY RAND() LIMIT :limit;", nativeQuery = true)
+    List<QuestionEntity> lookupQuestions(
+        @Param("categoryId") Integer categoryId, 
+        @Param("difficulty") String difficulty, 
+        @Param("type") String type, 
+        @Param("limit") int limit);
+
+    /**
+     * Moves all data from one table to another
+     */
+    @Modifying
+    @Transactional
+    @Query(value="INSERT INTO question SELECT * FROM question_collecting;", nativeQuery=true)
+    void moveCollection();
+}

backend/src/main/java/com/ninjaws/quiz/services/ApiService.java

@@ -9,8 +9,10 @@ import org.springframework.web.client.RestClientException;
 import org.springframework.web.client.RestTemplate;
 
 import com.ninjaws.quiz.models.ApiResponse;
+import com.ninjaws.quiz.models.CategoryResponse;
 import com.ninjaws.quiz.models.QuizSettings;
 import com.ninjaws.quiz.models.Request;
+import com.ninjaws.quiz.models.SessionTokenDTO;
 
 import jakarta.validation.Valid;
 
@@ -26,13 +28,14 @@ public class ApiService {
 
     private static final System.Logger logger = System.getLogger(ApiService.class.getName());
 
-    private final String apiUrl = "https://opentdb.com/api.php";
+    private final String apiUrl = "https://opentdb.com";
+    private String sessionToken;
 
     public ApiService() {
     }
 
     private String buildRequestString(QuizSettings quizSettings) {
-        return apiUrl + quizSettings.toApiQuery();
+        return apiUrl + "/api.php" + quizSettings.toApiQuery();
     }
 
     public void handleRequest(@Valid Request request) {
@@ -54,15 +57,54 @@ public class ApiService {
      * Store them in the DataService
      */
     public void collectBulk() {
-        String url = apiUrl + "?amount=50";
+        String tokenStr = sessionToken == null ? "" : "&token="+sessionToken;
+        String url = apiUrl + "/api.php?amount=50" + tokenStr;
         try {
             ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
             String json = response.getBody();
             ApiResponse apiResponse = cleanerService.extractApiResponse(json);
             responseHandler.handleResponse(apiResponse);
             
+            /** 
+             * A bit dirty, but otherwise it causes circular dependencies 
+             * Makes sure the categories are refreshed after finishing to replace the questions, to make sure the ids match
+             * And resets the sessionToken, it will be remade during the next REFRESH
+            */
+            if(apiResponse.getResponseCode() == 4) {
+                getCategories();
+                sessionToken = null;
+            }
+            
+        } catch (RestClientException | IOException e) {
+            logger.log(System.Logger.Level.ERROR, "Critical failure", e);
+        }
+
+    }
+
+    public void getSessionToken() {
+        String url = apiUrl + "/api_token.php?command=request";
+        try {
+            ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
+            String json = response.getBody();
+            SessionTokenDTO tokenDto = cleanerService.extractSessionToken(json);
+            sessionToken = tokenDto.getToken();
+            
         } catch (RestClientException | IOException e) {
             logger.log(System.Logger.Level.ERROR, "Critical failure", e);
         }
     }
+
+    public void getCategories() {
+        String url = apiUrl + "/api_category.php";
+        try {
+            ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
+            String json = response.getBody();
+            CategoryResponse categoryResponse = cleanerService.extractCategoryResponse(json);
+            responseHandler.handleCategoryResponse(categoryResponse);
+            
+        } catch (RestClientException | IOException e) {
+            logger.log(System.Logger.Level.ERROR, "Critical failure", e);
+        }
+
+    }
 }

backend/src/main/java/com/ninjaws/quiz/services/CleanerService.java

@@ -1,22 +1,123 @@
 package com.ninjaws.quiz.services;
 
 import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
 
+import org.apache.commons.lang3.StringEscapeUtils;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
 import com.fasterxml.jackson.databind.ObjectMapper;
+import com.ninjaws.quiz.entities.Category;
+import com.ninjaws.quiz.entities.QuestionCollectingEntity;
+import com.ninjaws.quiz.entities.QuestionEntity;
 import com.ninjaws.quiz.models.ApiResponse;
+import com.ninjaws.quiz.models.CategoryResponse;
+import com.ninjaws.quiz.models.QuestionDTO;
+import com.ninjaws.quiz.models.SessionTokenDTO;
+import com.ninjaws.quiz.repositories.CategoryRepository;
 
+/**
+ * Deals with conversions from one form of data to another
+ * Json to Object
+ * Entity to DTO
+ * etc
+ */
 @Service
 public class CleanerService {
 
     @Autowired
     private ObjectMapper objectMapper;
+    @Autowired
+    private CategoryRepository categoryRepository;
 
-    public CleanerService() {}
+    public CleanerService() {
+    }
 
     public ApiResponse extractApiResponse(String json) throws IOException {
         return objectMapper.readValue(json, ApiResponse.class);
     }
+
+    public CategoryResponse extractCategoryResponse(String json) throws IOException {
+        return objectMapper.readValue(json, CategoryResponse.class);
+    }
+
+    public List<QuestionDTO> questionListEntitytoDTO(List<QuestionEntity> entities) {
+        List<QuestionDTO> dtos = new ArrayList<>();
+        for (int i = 0; i < entities.size(); i++) {
+            QuestionEntity entity = entities.get(i);
+            dtos.add(questionEntityToDTO(entity));
+        }
+        return dtos;
+    }
+
+    public SessionTokenDTO extractSessionToken(String json) throws IOException {
+        return objectMapper.readValue(json, SessionTokenDTO.class);
+    }
+
+    public QuestionDTO questionEntityToDTO(QuestionEntity entity) {
+        QuestionDTO dto = new QuestionDTO();
+        dto.setQuestion(entity.getQuestion());
+        dto.setCorrectAnswer(entity.getCorrectAnswer());
+        dto.setIncorrectAnswer(entity.getIncorrectAnswers());
+        dto.setCategory(entity.getCategory().getName());
+        dto.setType(entity.getType());
+        dto.setDifficulty(entity.getDifficulty());
+        return dto;
+    }
+
+    public List<QuestionEntity> questionListDTOtoEntity(List<QuestionDTO> dtos) {
+        List<QuestionEntity> entities = new ArrayList<>();
+        for (int i = 0; i < dtos.size(); i++) {
+            QuestionDTO dto = dtos.get(i);
+            entities.add(questionDTOtoEntity(dto));
+        }
+        return entities;
+    }
+   
+    public QuestionEntity questionDTOtoEntity(QuestionDTO dto) {
+        QuestionEntity entity = new QuestionEntity();
+        Category category = categoryRepository.findByName(decodeString(dto.getCategory())).orElseThrow(() -> new IllegalStateException("Category not in DB!"));
+        entity.setCategory(category);
+        entity.setDifficulty(dto.getDifficulty());
+        entity.setType(dto.getType());
+        entity.setQuestion(dto.getQuestion());
+        entity.setCorrectAnswer(dto.getCorrectAnswer());
+        entity.setIncorrectAnswers(dto.getIncorrectAnswers());
+        return entity;
+    }
+
+
+    // TODO: Find a simpler way to convert from one table to another if the data is identical so this nonsense isn't necessary
+    public List<QuestionCollectingEntity> questionListDTOtoCollectingEntity(List<QuestionDTO> dtos) {
+        List<QuestionCollectingEntity> entities = new ArrayList<>();
+        for (int i = 0; i < dtos.size(); i++) {
+            QuestionDTO dto = dtos.get(i);
+            entities.add(questionDTOtoCollectingEntity(dto));
+        }
+        return entities;
+    }
+   
+    public QuestionCollectingEntity questionDTOtoCollectingEntity(QuestionDTO dto) {
+        QuestionCollectingEntity entity = new QuestionCollectingEntity();
+        Category category = categoryRepository.findByName(decodeString(dto.getCategory())).orElseThrow(() -> new IllegalStateException("Category not in DB!"));
+        entity.setCategory(category);
+        entity.setDifficulty(dto.getDifficulty());
+        entity.setType(dto.getType());
+        entity.setQuestion(dto.getQuestion());
+        entity.setCorrectAnswer(dto.getCorrectAnswer());
+        entity.setIncorrectAnswers(dto.getIncorrectAnswers());
+        return entity;
+    }
+
+
+    /** 
+     * For some reason, the categories come with special characters,
+     * But the categories from the questions dont
+     * So this has to be manually fixed if they are to be stored in the db (otherwise namematching doesnt work)
+     */
+    private String decodeString(String text){
+        return StringEscapeUtils.unescapeHtml4(text);
+    }
 }

backend/src/main/java/com/ninjaws/quiz/services/DataService.java

@@ -0,0 +1,70 @@
+package com.ninjaws.quiz.services;
+
+import java.util.List;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import com.ninjaws.quiz.entities.Category;
+import com.ninjaws.quiz.entities.QuestionCollectingEntity;
+import com.ninjaws.quiz.entities.QuestionEntity;
+import com.ninjaws.quiz.models.QuizSettings;
+import com.ninjaws.quiz.repositories.CategoryRepository;
+import com.ninjaws.quiz.repositories.QuestionCollectingRepository;
+import com.ninjaws.quiz.repositories.QuestionRepository;
+
+import jakarta.transaction.Transactional;
+
+@Service
+public class DataService {
+
+    @Autowired
+    private QuestionRepository questionRepository;
+    @Autowired
+    private QuestionCollectingRepository collectingRepository;
+    @Autowired
+    private CategoryRepository categoryRepository;
+
+    public List<QuestionEntity> getAllQuestions() {
+        return questionRepository.findAll();
+    }
+
+    public QuestionEntity saveQuestion(QuestionEntity question) {
+        return questionRepository.save(question);
+    }
+
+    public List<QuestionEntity> saveQuestions(List<QuestionEntity> questions) {
+        return questionRepository.saveAll(questions);
+    }
+
+    /**
+     * Tries to get questions that match the specified QuizSettings
+     *
+     * @return The questions if the query was possible, and an empty array if not
+     */
+    public List<QuestionEntity> lookupQuestions(QuizSettings quizSettings) {
+        return questionRepository.lookupQuestions(quizSettings.getCategory(), quizSettings.getDifficulty(), quizSettings.getType(), quizSettings.getAmount());
+    }
+
+    public List<Category> saveCategories(List<Category> categories) {
+        return categoryRepository.saveAll(categories);
+    }
+
+    public List<Category> getCategories(){
+        return categoryRepository.findAll();
+    }
+    /**
+     * Moving the collected questions to the questions list
+     *
+     */
+    @Transactional
+    public void migrateQuestions() {
+        questionRepository.deleteAll();
+        questionRepository.moveCollection();
+        collectingRepository.deleteAll();
+    }
+
+    public List<QuestionCollectingEntity> collectQuestions(List<QuestionCollectingEntity> questions) {
+        return collectingRepository.saveAll(questions);
+    }
+}

backend/src/main/java/com/ninjaws/quiz/services/QuizService.java

@@ -1,10 +1,12 @@
 package com.ninjaws.quiz.services;
 
+import java.util.List;
 import java.util.Optional;
 
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
+import com.ninjaws.quiz.entities.QuestionEntity;
 import com.ninjaws.quiz.models.Answers;
 import com.ninjaws.quiz.models.QuizSettings;
 import com.ninjaws.quiz.models.Request;
@@ -18,6 +20,12 @@ public class QuizService {
     private CacheService cacheService;
     @Autowired
     private QueueService queueService;
+    @Autowired
+    private DataService dataService;
+    @Autowired
+    private CleanerService cleanerService;
+    @Autowired
+    private StateManagementService stateManager;
 
     /**
      * Actions:
@@ -29,6 +37,22 @@ public class QuizService {
      */
     public Optional<String> createSession(QuizSettings quizSettings) {
         Session session = new Session();
+
+        /** First, we check if there are already enough items in our db */
+        List<QuestionEntity> questionsRetrieved = dataService.lookupQuestions(quizSettings);
+        /** If the items are not matching the required amount, and the db is not currently being filled up, then they simply don't exist */
+        if (questionsRetrieved.size() < quizSettings.getAmount() && stateManager.getCurrentProcessingState() != StateManagementService.ProcessingState.STARTUP) {
+            session.setStatusCode(1);
+            cacheService.addToCache(session.getId(), session);
+            return Optional.of(session.getId());
+        }
+        /** All items are collected in from the database, it doesn't need to be added to the queue */
+        if(!questionsRetrieved.isEmpty() && questionsRetrieved.size() == quizSettings.getAmount()) {
+            session.setQuestions(cleanerService.questionListEntitytoDTO(questionsRetrieved));
+            cacheService.addToCache(session.getId(), session);
+            return Optional.of(session.getId());
+        }
+
         cacheService.addToCache(session.getId(), session);
         boolean addedToQueue = queueService.addToQueue(new Request(session.getId(), quizSettings));
         /** In this order, because doing the queue before the cache could lead to race conditions (request completion immediately calls cache, but then the item doesn't exist) */
@@ -54,15 +78,17 @@ public class QuizService {
             throw new IllegalArgumentException("Invalid session ID: " + sessionId);
         }
 
+        /** Anything other than statuscode 0 doesn't result in active sessions, so remove them from the cache after reporting to the frontend */
+        if(session.getStatusCode() != 0) {
+            cacheService.removeFromCache(session.getId());
+            return Optional.of(session);
+        }
+
         /** Still waiting for a response */
         if (session.getQuestions() == null) {
             return Optional.empty();
         }
 
-        /** Anything other than statuscode 0 doesn't result in active sessions, so remove them from the cache after reporting to the frontend */
-        if(session.getStatusCode() != 0) {
-            cacheService.removeFromCache(session.getId());
-        }
 
         return Optional.of(session);
     }
@@ -71,7 +97,7 @@ public class QuizService {
      * Receives: sessionId and answers
      * 
      * Actions: 
-     * - retrieves session from cache
+     * - Retrieves session from cache
      * - Deletes session from cache
      * - Matches answers answers to correctAnswers
      * - Calculates score

backend/src/main/java/com/ninjaws/quiz/services/ResponseHandlingService.java

@@ -4,13 +4,20 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
 import com.ninjaws.quiz.models.ApiResponse;
+import com.ninjaws.quiz.models.CategoryResponse;
 import com.ninjaws.quiz.models.Session;
 
 @Service
 public class ResponseHandlingService {
 
+    @Autowired
+    private StateManagementService stateManager;
     @Autowired
     private CacheService cacheService;
+    @Autowired
+    private DataService dataService;
+    @Autowired
+    private CleanerService cleanerService;
 
     /**
      * If there is no sessionId, then this is a request from the scheduler
@@ -28,6 +35,8 @@ public class ResponseHandlingService {
             case 0:
                 // Success
                 updateSessionInCache(session, response);
+                /** Managing the token gets difficult since there is a timelimit, so simply dont add these, let the bulk collector do it */
+                // dataService.saveQuestions(cleanerService.questionListDTOtoEntity(response.getResults()));
                 break;
             case 1:
                 // Not enough questions to satisfy the request. 
@@ -61,11 +70,26 @@ public class ResponseHandlingService {
     public void handleResponse(ApiResponse response) {
         switch (response.getResponseCode()) {
             case 0:
-                // TODO: Store in DB
+                if(stateManager.getCurrentProcessingState() == StateManagementService.ProcessingState.STARTUP) {
+                    /** Initial saving during the startup */
+                    dataService.saveQuestions(cleanerService.questionListDTOtoEntity(response.getResults()));
+                } else if (stateManager.getCurrentProcessingState() == StateManagementService.ProcessingState.REFRESH) {
+                    /** Background collecting during the night */
+                    dataService.collectQuestions(cleanerService.questionListDTOtoCollectingEntity(response.getResults()));
+                }
                 break;
             case 4:
-                // Signal DataService that we have extracted all the data. Stop gathering, switch to caching-mode
+                // Signal DataService that we have extracted all the data. Stop gathering, 
-                // TODO: Signal DB that it's full, start db-swapping process
+                // TODO: Switch to caching-mode
+                
+                /** 
+                 * If it was in the REFRESH state, move the data from collecting to the used questions
+                 * Otherwise simply switch to the NORMAL state
+                 */
+                if (stateManager.getCurrentProcessingState() == StateManagementService.ProcessingState.REFRESH) {
+                    dataService.migrateQuestions();
+                }
+                stateManager.setCurrentProcessingState(StateManagementService.ProcessingState.NORMAL);
                 break;
             case 5:
                 // Should be prevented by the scheduler
@@ -80,4 +104,7 @@ public class ResponseHandlingService {
         cacheService.addToCache(session.getId(), session);
     }
 
+    public void handleCategoryResponse(CategoryResponse categoryResponse) {
+        dataService.saveCategories(categoryResponse.getTrivia_categories());
+    }
 }

backend/src/main/java/com/ninjaws/quiz/services/SchedulerService.java

@@ -13,6 +13,8 @@ public class SchedulerService {
     private QueueService queueService;
     @Autowired
     private ApiService apiService;
+    @Autowired
+    private StateManagementService stateService;
 
     public SchedulerService() {}
 
@@ -25,9 +27,23 @@ public class SchedulerService {
     @Scheduled(fixedDelay = 5000)
     protected void processQueue() {
         Request request = queueService.popFromQueue();
+
         if (request != null) {
             apiService.handleRequest(request);
+        } else {
+            if (stateService.getCurrentProcessingState() == StateManagementService.ProcessingState.STARTUP || stateService.getCurrentProcessingState() == StateManagementService.ProcessingState.REFRESH) {
+                apiService.collectBulk();
+            }
         }
     }   
 
+    /**
+     * Starts the renewing of the questions, to ensure they stay fresh
+     */
+    @Scheduled(cron = "0 0 1 * * ?")
+    protected void StartDbRefresh() {
+        apiService.getSessionToken();
+        stateService.setCurrentProcessingState(StateManagementService.ProcessingState.REFRESH);
+    }
+    
 }

backend/src/main/java/com/ninjaws/quiz/services/StateManagementService.java

@@ -0,0 +1,29 @@
+package com.ninjaws.quiz.services;
+
+import org.springframework.stereotype.Service;
+
+@Service
+public class StateManagementService {
+
+    /**
+     * STARTUP: When starting the app
+     * REFRESH: Scheduled at night, so the database stays fresh
+     * NORMAL: No special activities
+     */
+    public enum ProcessingState {
+        STARTUP,
+        REFRESH,
+        NORMAL
+    }
+    
+    private ProcessingState currentProcessingState;
+    
+    public ProcessingState getCurrentProcessingState() {
+        return currentProcessingState;
+    }
+
+    public void setCurrentProcessingState(ProcessingState currentProcessingState) {
+        this.currentProcessingState = currentProcessingState;
+    }
+    
+}

backend/src/main/java/com/ninjaws/quiz/services/StorageService.java

@@ -1,5 +0,0 @@
-package com.ninjaws.quiz.services;
-
-public class StorageService {
-    
-}

backend/src/main/resources/application.properties

@@ -1,2 +1,34 @@
 spring.application.name=quiz
 server.port=9090
+
+# H2 Database configuration
+spring.h2.console.enabled=true
+# spring.h2.console.path=/console
+
+
+spring.datasource.url=jdbc:h2:mem:testdb
+spring.datasource.driverClassName=org.h2.Driver
+spring.datasource.username=sa
+spring.datasource.password=
+
+spring.jpa.hibernate.ddl-auto=create-drop
+spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
+
+
+# # Primary database configuration
+# spring.datasource.primary.url=jdbc:h2:mem:testdb
+# # ;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
+# spring.datasource.primary.driver-class-name=org.h2.Driver
+# spring.datasource.primary.username=sa
+# spring.datasource.primary.password=
+# spring.jpa.primary.hibernate.ddl-auto=create-drop
+# spring.jpa.primary.database-platform=org.hibernate.dialect.H2Dialect
+
+# # Secondary database configuration
+# spring.datasource.secondary.url=jdbc:h2:mem:secondarydb
+# # ;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
+# spring.datasource.secondary.driver-class-name=org.h2.Driver
+# spring.datasource.secondary.username=sa
+# spring.datasource.secondary.password=
+# spring.jpa.secondary.hibernate.ddl-auto=create-drop
+# spring.jpa.secondary.database-platform=org.hibernate.dialect.H2Dialect

backend/src/test/java/com/ninjaws/quiz/scala/QuestionStatusSimulation.scala

@@ -3,6 +3,7 @@ import io.gatling.core.Predef._
 import io.gatling.http.Predef._
 import scala.concurrent.duration._
 
+/** Does not work, use the LoadTestMultiple.sh in the /scripts folder in the root instead */
 class QuestionStatusSimulation extends Simulation {
     val httpProtocol = http.baseUrl("http://localhost:9090").acceptHeader("application/json")
 

backend/src/test/java/com/ninjaws/quiz/services/CacheServiceTest.java

@@ -0,0 +1,59 @@
+package com.ninjaws.quiz.services;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import com.ninjaws.quiz.models.Session;
+
+public class CacheServiceTest {
+
+    private CacheService cacheService;
+
+    @BeforeEach
+    public void setUp() {
+        cacheService = new CacheService();
+    }
+
+    @Test
+    public void testAddToCache() {
+        String key = "session1";
+        Session session = new Session();
+
+        cacheService.addToCache(key, session);
+
+        assertNotNull(cacheService.getFromCache(key));
+        assertEquals(session, cacheService.getFromCache(key));
+    }
+
+    @Test
+    public void testGetFromCache() {
+        String key = "session1";
+        Session session = new Session();
+
+        cacheService.addToCache(key, session);
+
+        Session retrievedSession = cacheService.getFromCache(key);
+        assertNotNull(retrievedSession);
+        assertEquals(session, retrievedSession);
+    }
+
+    @Test
+    public void testRemoveFromCache() {
+        String key = "session1";
+        Session session = new Session();
+
+        cacheService.addToCache(key, session);
+        cacheService.removeFromCache(key);
+
+        assertNull(cacheService.getFromCache(key));
+    }
+
+    @Test
+    public void testGetFromCacheWithNonExistentKey() {
+        assertNull(cacheService.getFromCache("nonExistentKey"));
+    }
+}

backend/src/test/java/com/ninjaws/quiz/services/QueueServiceTest.java

@@ -0,0 +1,63 @@
+package com.ninjaws.quiz.services;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import com.ninjaws.quiz.models.QuizSettings;
+import com.ninjaws.quiz.models.Request;
+
+public class QueueServiceTest {
+
+    private QueueService queueService;
+
+    @BeforeEach
+    public void setUp() {
+        queueService = new QueueService();
+    }
+
+    @Test
+    public void testAddToQueue() {
+        QuizSettings quizSettings = new QuizSettings();
+        Request request = new Request("session1", quizSettings);
+
+        boolean added = queueService.addToQueue(request);
+        assertTrue(added);
+
+        for (int i = 0; i < 4; i++) {
+            added = queueService.addToQueue(new Request("session" + (i + 2), new QuizSettings()));
+            assertTrue(added);
+        }
+
+        added = queueService.addToQueue(new Request("session6", new QuizSettings()));
+        assertFalse(added);
+    }
+
+    @Test
+    public void testPopFromQueue() {
+        QuizSettings quizSettings1 = new QuizSettings();
+        QuizSettings quizSettings2 = new QuizSettings();
+
+        Request request1 = new Request("session1", quizSettings1);
+        Request request2 = new Request("session2", quizSettings2);
+
+        queueService.addToQueue(request1);
+        queueService.addToQueue(request2);
+
+        Request poppedRequest = queueService.popFromQueue();
+        assertEquals(request1, poppedRequest);
+
+        poppedRequest = queueService.popFromQueue();
+        assertEquals(request2, poppedRequest);
+
+        assertNull(queueService.popFromQueue());
+    }
+
+    @Test
+    public void testPopFromEmptyQueue() {
+        assertNull(queueService.popFromQueue());
+    }
+}

backend/src/test/java/com/ninjaws/quiz/services/backupApiServiceTest.txt

@@ -1,114 +0,0 @@
-package com.ninjaws.quiz.services;
-
-import static org.assertj.core.api.Assertions.assertThat;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.Test;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.test.context.SpringBootTest;
-
-import com.fasterxml.jackson.databind.JsonNode;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.ninjaws.quiz.models.QuizSettings;
-
-@SpringBootTest
-public class ApiServiceTest {
-
-    @Autowired
-    private ApiService apiService;
-
-    @Autowired
-    private ObjectMapper objectMapper;
-
-    /**
-     * The external API only allows one request every 5 seconds
-     */
-    @AfterEach
-    public void cooldown() throws InterruptedException {
-        Thread.sleep(5000);
-    }
-
-    @Test
-    public void testRequestQuestions() throws Exception {
-        QuizSettings settings = new QuizSettings();
-        String jsonResponse = apiService.handleRequest(settings);
-        JsonNode jsonNode = objectMapper.readTree(jsonResponse);
-
-        // Check if the response code is as expected
-        assertThat(jsonNode.has("response_code")).isTrue();
-        assertThat(jsonNode.get("response_code").asInt()).isEqualTo(0);
-
-        // Check the results array
-        assertThat(jsonNode.has("results")).isTrue();
-        assertThat(jsonNode.get("results").isArray()).isTrue();
-        assertThat(jsonNode.get("results").size()).isEqualTo(10);
-
-        // Verify the first result has the expected fields
-        JsonNode firstResult = jsonNode.get("results").get(0);
-        assertThat(firstResult.has("type")).isTrue();
-        assertThat(firstResult.has("difficulty")).isTrue();
-        assertThat(firstResult.has("category")).isTrue();
-        assertThat(firstResult.has("question")).isTrue();
-        assertThat(firstResult.has("correct_answer")).isTrue();
-        assertThat(firstResult.has("incorrect_answers")).isTrue();
-        assertThat(firstResult.get("incorrect_answers").size()).isGreaterThan(0);
-    }
-
-    @Test
-    public void testAllParams() throws Exception {
-        int amount = 10;
-
-        QuizSettings settings = new QuizSettings();
-        settings.setAmount(amount);
-        settings.setCategory(QuizSettings.Category.HISTORY);
-        settings.setType(QuizSettings.Type.MULTIPLE);
-        settings.setDifficulty(QuizSettings.Difficulty.MEDIUM);
-
-        String jsonResponse = apiService.requestQuestions(settings);
-        JsonNode jsonNode = objectMapper.readTree(jsonResponse);
-
-        // Check if the response code is as expected
-        assertThat(jsonNode.has("response_code")).isTrue();
-        assertThat(jsonNode.get("response_code").asInt()).isEqualTo(0);
-
-        // Check the results array
-        assertThat(jsonNode.has("results")).isTrue();
-        assertThat(jsonNode.get("results").isArray()).isTrue();
-        assertThat(jsonNode.get("results").size()).isEqualTo(amount);
-
-        // Verify the first result has the expected fields
-        JsonNode firstResult = jsonNode.get("results").get(0);
-        String retrievedType = firstResult.get("type").toString();
-        String retrievedDifficulty = firstResult.get("difficulty").toString();
-        String retrievedCategory = firstResult.get("category").toString();
-        assertThat(retrievedType.equals(settings.getType().getApiValue()));
-        assertThat(retrievedDifficulty.equals(settings.getDifficulty().getApiValue()));
-        assertThat(retrievedCategory.equals("Animals"));
-        assertThat(firstResult.get("incorrect_answers").size()).isEqualTo(3);
-    }
-
-    /**
-     * If the API doesn't have the requested amount of questions, it will return an empty list and respond with 1
-     */
-    @Test
-    public void testCombinationThatResultsInZeroQuestions() throws Exception {
-        int amount = 30;
-
-        QuizSettings settings = new QuizSettings();
-        settings.setAmount(amount);
-        settings.setCategory(QuizSettings.Category.ANIMALS);
-        settings.setType(QuizSettings.Type.MULTIPLE);
-        settings.setDifficulty(QuizSettings.Difficulty.MEDIUM);
-
-        String jsonResponse = apiService.requestQuestions(settings);
-        JsonNode jsonNode = objectMapper.readTree(jsonResponse);
-
-        // Check if the response code is as expected
-        assertThat(jsonNode.has("response_code")).isTrue();
-        assertThat(jsonNode.get("response_code").asInt()).isEqualTo(1);
-
-        // Check the results array
-        assertThat(jsonNode.has("results")).isTrue();
-        assertThat(jsonNode.get("results").isArray()).isTrue();
-        assertThat(jsonNode.get("results").size()).isEqualTo(0);
-    }
-}

backend/src/test/java/com/ninjaws/quiz/services/backupQuizServiceTest.txt

@@ -1,72 +0,0 @@
-package com.ninjaws.quiz.services;
-
-import org.springframework.boot.test.context.SpringBootTest;
-
-@SpringBootTest
-// @EnableScheduling
-// @TestInstance(TestInstance.Lifecycle.PER_CLASS)
-public class QuizServiceTest {
-    
-    // @InjectMocks
-    // private QuizService quizService;
-
-    // @Mock
-    // private ApiService apiService;
-
-    // @BeforeAll
-    // public void setUp() {
-    //     MockitoAnnotations.openMocks(this);
-    //     // Cache<String, Session> cache = Caffeine.newBuilder()
-    //     //     .expireAfterWrite(1, TimeUnit.HOURS)
-    //     //     .build(); // Create a mock cache or a real cache for tests
-    //     // quizService = new QuizService(cache); 
-    // }
-
-    // @Test
-    // public void testCreateSession() {
-    //     QuizSettings settings = new QuizSettings();
-    //     String response = quizService.createSession(settings);
-    //     assertTrue(response instanceof String);
-    // }
-
-    // /**
-    //  * TODO: Fix the test
-    //  * Seems to clear the cache and queue when the createSession is finished 
-    //  */
-    // @Test
-    // public void testCheckSession() throws InterruptedException {
-        
-    //     QuizSettings settings = new QuizSettings();
-    //     when(apiService.requestQuestions(settings)).thenReturn("{\"response_code\":0,\"results\":[{\"type\":\"multiple\",\"difficulty\":\"hard\",\"category\":\"Entertainment: Video Games\",\"question\":\"The map featured in Arma 3 named &quot;Altis&quot; is based off of what Greek island?\",\"correct_answer\":\"Lemnos\",\"incorrect_answers\":[\"Ithaca\",\"Naxos\",\"Anafi\"]},{\"type\":\"multiple\",\"difficulty\":\"medium\",\"category\":\"Entertainment: Music\",\"question\":\"What is the last song on the first Panic! At the Disco album?\",\"correct_answer\":\"Build God, Then We&#039;ll Talk\",\"incorrect_answers\":[\"Nails for Breakfast, Tacks for Snacks\",\"I Write Sins Not Tragedies\",\"Lying Is The Most Fun A Girl Can Have Without Taking Her Clothes Off\"]},{\"type\":\"multiple\",\"difficulty\":\"easy\",\"category\":\"Entertainment: Video Games\",\"question\":\"Which of these is NOT a playable character in the 2016 video game Overwatch?\",\"correct_answer\":\"Invoker\",\"incorrect_answers\":[\"Mercy\",\"Winston\",\"Zenyatta\"]},{\"type\":\"multiple\",\"difficulty\":\"medium\",\"category\":\"Science: Computers\",\"question\":\"What does AD stand for in relation to Windows Operating Systems? \",\"correct_answer\":\"Active Directory\",\"incorrect_answers\":[\"Alternative Drive\",\"Automated Database\",\"Active Department\"]},{\"type\":\"multiple\",\"difficulty\":\"medium\",\"category\":\"Entertainment: Japanese Anime &amp; Manga\",\"question\":\"In &quot;Highschool DxD&quot;, Koneko Toujou is from what race?\",\"correct_answer\":\"Nekomata\",\"incorrect_answers\":[\"Kitsune\",\"Human\",\"Kappa\"]}]}");
-    //     String sessionId = quizService.createSession(settings);
-
-
-    //     int interval = 1000;
-    //     int maxTime = 5000;
-    //     long startTime = System.currentTimeMillis();
-    //     Session session = null;
-
-    //     while((System.currentTimeMillis() - maxTime) < startTime) {
-    //         Optional<Session> retrieved = quizService.checkSession(sessionId);
-    //         System.out.println(retrieved);
-    //         if (retrieved.isPresent()) {
-    //             System.out.print("Present!");
-    //             session = retrieved.get();
-    //             break;
-    //         } else {
-    //             try {
-    //                 Thread.sleep(interval);
-    //             } catch (InterruptedException e) {
-    //                 Thread.currentThread().interrupt(); // Restore interrupted status
-    //                 fail("Thread was interrupted during sleep");
-    //             }
-    //         }
-    //     }
-
-    //     assertNotEquals(session, null);
-    // }
-
-    // public void testCalculateScore(){
-
-    // }
-}

frontend/src/app/Quiz/models/category.ts

@@ -0,0 +1,4 @@
+export interface Category {
+    id: number;
+    name: string;
+}

frontend/src/app/Quiz/models/quiz-params.ts

@@ -1,5 +1,7 @@
+import { Category } from "./category";
+
 export interface QuizParams {
-    category: string;
+    category: number;
     type: string;
     amount: number;
     difficulty: string;

frontend/src/app/Quiz/services/quiz.service.ts

@@ -19,6 +19,7 @@ import { StorageService } from './storage.service';
 import { CleanerService } from './cleaner.service';
 import { ResponseHandlerService } from './response-handler.service';
 import { environment } from '../../../environments/environment';
+import { Category } from '../models/category';
 
 @Injectable({
   providedIn: 'root',
@@ -40,8 +41,7 @@ export class QuizService {
     }
     this.storageService.setSession(sessionIdObj.sessionId);
     const sessionObj = await lastValueFrom(this.pollQuestions(sessionIdObj.sessionId));
-
+    return this.reponseHandler.handleResponse(sessionObj.statusCode, sessionObj);
-    return this.reponseHandler.handleResponse(sessionIdObj.statusCode, sessionObj);
   }
 
   public pollQuestions(sessionId: string) {
@@ -67,11 +67,11 @@ export class QuizService {
 
   private convertParamsToUrl(quizParams: QuizParams) {
     let url = `?amount=${quizParams.amount}`;
-    const catIndex = this.getCategories().indexOf(quizParams.category);
+    const catIndex = quizParams.category;
     url += catIndex !== 0 ? `&category=${catIndex}` : '';
 
     const diff = quizParams.difficulty.toLowerCase();
-    url += diff !== 'any difficulty' ? `&category=${diff}` : '';
+    url += diff !== 'any difficulty' ? `&difficulty=${diff}` : '';
 
     const typeIndex = this.getTypes().indexOf(quizParams.type);
     url +=
@@ -83,34 +83,11 @@ export class QuizService {
     return url;
   }
 
-  public getCategories(): string[] {
+  public async getCategories(): Promise<Category[]> {
-    return [
+    let categories: Category[] = await firstValueFrom(this.http.get<any>(this.apiUrl + '/categories'));
-      'Any Category',
+    categories.push({id:0,name:"Any Category"});
-      'General Knowledge',
+    categories.sort((a,b) => a.id - b.id);
-      'Entertainment: Books',
+    return categories;
-      'Entertainment: Film',
-      'Entertainment: Music',
-      'Entertainment: Musicals & Theatres',
-      'Entertainment: Television',
-      'Entertainment: Video Games',
-      'Entertainment: Board Games',
-      'Science & Nature',
-      'Science: Computers',
-      'Science: Mathematics',
-      'Mythology',
-      'Sports',
-      'Geography',
-      'History',
-      'Politics',
-      'Art',
-      'Celebrities',
-      'Animals',
-      'Vehicles',
-      'Entertainment: Comics',
-      'Science: Gadgets',
-      'Entertainment: Japanese Anime & Manga',
-      'Entertainment: Cartoon & Animations',
-    ];
   }
 
   public getDifficulties(): string[] {

frontend/src/app/Quiz/setup/setup.component.html

@@ -5,7 +5,7 @@
         <div class="mb-4">
             <label for="category" class="block mb-1">Category</label>
             <select id="category" class="w-full p-2 border rounded" [(ngModel)]="category">
-                <option *ngFor="let category of quizService.getCategories()" [value]="category">{{ category }}</option>
+                <option *ngFor="let category of categories" [value]="category.id">{{ category.name }}</option>
             </select>
         </div>
 

frontend/src/app/Quiz/setup/setup.component.ts

@@ -7,6 +7,7 @@ import { QuizParams } from '../models/quiz-params';
 import { SpinnerService } from '../../Spinner/services/spinner.service';
 import { CommonModule } from '@angular/common';
 import { Question } from '../models/question';
+import { Category } from '../models/category';
 
 @Component({
   selector: 'app-setup',
@@ -17,8 +18,8 @@ import { Question } from '../models/question';
 })
 export class SetupComponent {
 
-
+  categories!: Category[];
-  category: string = "";
+  category!: number;
   difficulty: string = "";
   type: string = "";
   amount: number = 10;
@@ -28,10 +29,13 @@ export class SetupComponent {
     public quizService: QuizService,
     private spinnerService: SpinnerService
   ) {
+    quizService.getCategories().then((categories: Category[]) => {
       /** Setting default values */
-    this.category = this.quizService.getCategories()[0];
+      this.categories = categories;
+      this.category = this.categories.find(item => item.id == 0)!.id;
       this.difficulty = this.quizService.getDifficulties()[0];
       this.type = this.quizService.getTypes()[0];      
+    });
   }
 
   startQuiz() {

scripts/LoadTestMultiple.sh

@@ -3,15 +3,14 @@
 start_time=$(date +%s)
 
 for i in {1..6}; do
-  ./LoadTestOne.sh &
+  ./LoadTestOne-Silent.sh &
-  sleep 0.1
+#   sleep 0.1
 done
 
-wait  # Wait for all background jobs to finish
+wait
 
-end_time=$(date +%s)  # Capture the end time
+end_time=$(date +%s)
 
-# Calculate and print the elapsed time
 elapsed_time=$((end_time - start_time))
 echo "Done!"
 echo "Total time taken: $elapsed_time seconds"

scripts/LoadTestOne-Silent.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+# Step 1: Get the sessionId from the questions endpoint
+sessionIdJson=$(curl -s "http://localhost:9090/quiz/questions?amount=10&category=23&difficulty=medium&type=multiple")
+
+# Extract the status code
+statusCode=$(echo "$sessionIdJson" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+if [ "$statusCode" -ne 0 ]; then
+    exit 1
+fi
+
+# Extract the session ID
+sessionId=$(echo "$sessionIdJson" | grep -o '"sessionId":"[^"]*"' | sed 's/"sessionId":"//;s/"//')
+
+if [ -z "$sessionId" ]; then
+    exit 1
+fi
+
+# Step 2: Poll the status endpoint every 2 seconds
+while true; do
+    status_response=$(curl -s "http://localhost:9090/quiz/status/$sessionId")
+
+    if [ "$status_response" != "null" ]; then
+        # Extract the status code from the response
+        statusCode=$(echo "$status_response" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+        if [ "$statusCode" -ne 0 ]; then
+            exit 1
+        fi
+
+        # Check if the response is JSON (starts with '{')
+        if [[ $status_response == \{* ]]; then
+            break
+        fi
+    fi
+
+    sleep 2
+done

scripts/LoadTestOne-WithParams.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Step 1: Get the sessionId from the questions endpoint
+sessionIdJson=$(curl -s "http://localhost:9090/quiz/questions?amount=20&category=27&difficulty=medium&type=multiple")
+
+statusCode=$(echo "$sessionIdJson" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+if [ "$statusCode" -ne 0 ]; then
+    echo "Exiting: statusCode is not 0."
+    exit 1
+fi
+
+sessionId=$(echo "$sessionIdJson" | grep -o '"sessionId":"[^"]*"' | sed 's/"sessionId":"//;s/"//')
+
+if [ -z "$sessionId" ]; then
+    echo "Failed to get sessionId"
+    exit 1
+fi
+
+echo "Received sessionId: $sessionId"
+
+# Step 2: Poll the status endpoint every 2 seconds
+while true; do
+    status_response=$(curl -s http://localhost:9090/quiz/status/"$sessionId")
+    if [ "$status_response" != "null" ] && [ $? -eq 0 ]; then
+
+        statusCode=$(echo "$status_response" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+        if [ "$statusCode" -ne 0 ]; then
+            echo "Exiting: statusCode is $statusCode"
+            exit 1
+        fi
+
+        # Check if the response is JSON (starts with '{')
+        if [[ $status_response == \{* ]]; then
+            echo "$sessionId: done! status: $statusCode"
+            break
+        fi
+    else
+        echo "Status check failed, retrying..."
+    fi
+
+    sleep 2
+done

scripts/LoadTestOne.sh

@@ -1,5 +1,4 @@
 #!/bin/bash
-
 # Step 1: Get the sessionId from the questions endpoint
 sessionIdJson=$(curl -s http://localhost:9090/quiz/questions?amount=5)
 
@@ -22,11 +21,18 @@ echo "Received sessionId: $sessionId"
 # Step 2: Poll the status endpoint every 2 seconds
 while true; do
     status_response=$(curl -s http://localhost:9090/quiz/status/"$sessionId")
+    if [ "$status_response" != "null" ] && [ $? -eq 0 ]; then
+
+        statusCode=$(echo "$status_response" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+        if [ "$statusCode" -ne 0 ]; then
+            echo "Exiting: statusCode is $statusCode"
+            exit 1
+        fi
 
-    if [ $? -eq 0 ]; then
         # Check if the response is JSON (starts with '{')
         if [[ $status_response == \{* ]]; then
-            echo "$sessionId: done!"
+            echo "$sessionId: done! status: $statusCode"
             break
         fi
     else