Merge pull request #3 from Ninjaws/develop

[Updated] Reflection

README.md

@@ -36,3 +36,14 @@ sudo reboot
 docker compose up
 ```
 - Visit localhost:4200 to play the quiz!
+
+
+## Benchmarking
+- Go to the /scripts folder and run the LoadTestMultiple.sh script
+- Inside you can change how many sessions should be created (set to 6 by default)
+- In the /scripts/benchmarks folder you can find some benchmarks I've done
+
+## Notes
+- If the data is not present, it will try to query it from the external API, which takes significantly longer (only 1 request per 5 seconds)
+- When running it locally, give the backend a few minutes to gather data in bulk (should be completely done in ~7 minutes)
+- It refreshes this data every night in the background, so it stays up to date

Reflection.md

@@ -1,20 +1,20 @@
-# Issues
+# Reflection
 
-## Issues during testing
+## Issues during testing (October 30th)
 ### Scheduler
 I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed.
 I don't know how to solve this yet
 
 
-## Caching solution
+## Caching solution (October 30th)
 ### Lack of data
-Not being able to gather blocks of 50 is a huge issue. The idea of reusing blocks becomes less viable if I can just get a big chunk every time for others to use
+Not being able to gather blocks of 50 is a big issue. The idea of reusing blocks becomes less viable if I can't just get a big chunk every time for others to use.
 I can't force a big chunk, because it might just return a status 1, and then the user gets nothing.
 
 Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain.
 
 
-## Possible solution
+## Possible solution (October 31st)
 Cache all data, 50 at a time
 Store it in a database
 When its done, let the db make blocks for quick cache access
@@ -28,3 +28,36 @@ Then start replacing the cache items one at a time.
 
 Users shouldn't feel this at all, because the active sessions are stored in their own cache items
 
+# Progress so far (November 3rd)
+There only seem to be ~4100 items, making the collecting phase only last around 7 minutes.
+
+Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up
+
+TODO: Add a caching state, and start filling up each unique combination with the items from the db
+This should lead to performance gains
+
+## Possible future alternative
+In the current state (when the cache is done), the backend might hold data 3 times (Cache, active DB table, and background DB table during the night)
+
+### Purely Cached solution
+- Every night
+- Check for each category+difficulty combination how many questions there are (API has an endpoint for this)
+- Track which one you are on, and how many you have
+- Take blocks of 50, until the remaining items are less than 50, then get that exact amount
+- When they are gathered, either 
+    1. Divide them between 3 cache items: cat:dif:any/multiple/boolean 
+    - (Manually filter on the type, since the API sends questions with either option together), or 
+    2. In one like cat:dif:any and then filter it by hand when a user asks for it
+    - (This will replace the collection put there the night before)
+- Then move to the next one
+- Stop when the last combination has been collected
+
+#### When a user requests data: 
+- Get the items from the cache
+- Less items than requested: Inform the user to try a different combination
+- More items than requested: Randomize and select a subset
+
+### Tradeoffs
+- Advantage: Less total data held on the server
+
+- Downside: Slower gathering, since there are 25+3+4=300 unique combinations (each of which will end with a request of less than 50 items), it will require a decent number requests more than the bulk collecting that the DB does.

backend/src/main/java/com/ninjaws/quiz/components/StartupRunner.java

@@ -1,27 +1,40 @@
 package com.ninjaws.quiz.components;
 
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.boot.CommandLineRunner;
-import org.springframework.jdbc.core.JdbcTemplate;
+import org.springframework.stereotype.Component;
 
-/**
+import com.ninjaws.quiz.services.ApiService;
- * Sets up the database, and adds the categories
+import com.ninjaws.quiz.services.StateManagementService;
- * 
- * Fills up the primary database
- * 
- * Then, at night, creates a new database, adds categories, and then starts filling it up
- * When full, swaps the databases, and wipes the old one
- */
-public class StartupRunner  implements CommandLineRunner {
-    private final JdbcTemplate primaryJdbcTemplate;
 
-    public StartupRunner(JdbcTemplate primaryJdbcTemplate) {
+
-        this.primaryJdbcTemplate = primaryJdbcTemplate;
+@Component
-    }
+public class StartupRunner implements CommandLineRunner {
+
+    @Autowired
+    private StateManagementService stateManagementService;
+    @Autowired
+    private ApiService apiService;
 
     @Override
     public void run(String... args) {
-        // Initialization logic here
+
-        primaryJdbcTemplate.execute("CREATE TABLE IF NOT EXISTS test_table (id INT PRIMARY KEY, name VARCHAR(255))");
+        /**
-        primaryJdbcTemplate.update("INSERT INTO test_table (id, name) VALUES (?, ?)", 1, "Sample Data");
+         * On startup:
+         * - Set ProcessingState to Startup
+         * - Store the categories
+         * - Get a sessiontoken
+         * 
+         * * Inside the SchedulerService
+         * - Start filling the question table
+         * 
+         * * Inside the Response Handling Service
+         * - No more questions? Stop
+         * - Set ProcessingState to Normal
+         */
+
+        stateManagementService.setCurrentProcessingState(StateManagementService.ProcessingState.STARTUP);
+        apiService.getCategories();
+        apiService.getSessionToken();
     }
 }

backend/src/main/java/com/ninjaws/quiz/config/DataSourceConfig.java

@@ -1,46 +0,0 @@
-package com.ninjaws.quiz.config;
-
-import javax.sql.DataSource;
-
-import org.springframework.beans.factory.annotation.Qualifier;
-import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
-import org.springframework.boot.context.properties.ConfigurationProperties;
-import org.springframework.context.annotation.Bean;
-import org.springframework.context.annotation.Primary;
-import org.springframework.jdbc.core.JdbcTemplate;
-
-public class DataSourceConfig {
-    @Primary
-    @Bean(name = "primaryDataSourceProperties")
-    @ConfigurationProperties("spring.datasource.primary")
-    public DataSourceProperties primaryDataSourceProperties() {
-        return new DataSourceProperties();
-    }
-
-    @Primary
-    @Bean(name = "primaryDataSource")
-    public DataSource primaryDataSource() {
-        return primaryDataSourceProperties().initializeDataSourceBuilder().build();
-    }
-
-    @Bean(name = "secondaryDataSourceProperties")
-    @ConfigurationProperties("spring.datasource.secondary")
-    public DataSourceProperties secondaryDataSourceProperties() {
-        return new DataSourceProperties();
-    }
-
-    @Bean(name = "secondaryDataSource")
-    public DataSource secondaryDataSource() {
-        return secondaryDataSourceProperties().initializeDataSourceBuilder().build();
-    }
-
-    @Bean
-    public JdbcTemplate primaryJdbcTemplate(@Qualifier("primaryDataSource") DataSource ds) {
-        return new JdbcTemplate(ds);
-    }
-
-    @Bean
-    public JdbcTemplate secondaryJdbcTemplate(@Qualifier("secondaryDataSource") DataSource ds) {
-        return new JdbcTemplate(ds);
-    }
-}

backend/src/main/java/com/ninjaws/quiz/controllers/QuizController.java

@@ -1,5 +1,6 @@
 package com.ninjaws.quiz.controllers;
 
+import java.util.List;
 import java.util.Optional;
 
 import org.springframework.beans.factory.annotation.Autowired;
@@ -11,9 +12,11 @@ import org.springframework.web.bind.annotation.RequestMapping;
 import org.springframework.web.bind.annotation.RequestParam;
 import org.springframework.web.bind.annotation.RestController;
 
+import com.ninjaws.quiz.entities.Category;
 import com.ninjaws.quiz.models.Answers;
 import com.ninjaws.quiz.models.QuizSettings;
 import com.ninjaws.quiz.models.Session;
+import com.ninjaws.quiz.services.DataService;
 import com.ninjaws.quiz.services.QuizService;
 
 @RestController
@@ -22,6 +25,9 @@ public class QuizController {
 
     @Autowired
     private QuizService quizService;
+    @Autowired
+    private DataService dataService;
+
 
     /**
      * Start a session
@@ -68,4 +74,9 @@ public class QuizController {
         String jsonString = String.format("{\"score\":\"%s\"}", quizService.getScore(answers));
         return jsonString;
     }
+
+    @GetMapping("/categories")
+    public List<Category> getCategories(){
+        return dataService.getCategories();
+    }
 }

backend/src/main/java/com/ninjaws/quiz/entities/BaseQuestionEntity.java

@@ -0,0 +1,34 @@
+package com.ninjaws.quiz.entities;
+
+import java.util.List;
+import java.util.UUID;
+
+import jakarta.persistence.GeneratedValue;
+import jakarta.persistence.GenerationType;
+import jakarta.persistence.Id;
+import jakarta.persistence.JoinColumn;
+import jakarta.persistence.ManyToOne;
+import jakarta.persistence.MappedSuperclass;
+import lombok.Getter;
+import lombok.Setter;
+
+@MappedSuperclass
+@Getter
+@Setter
+public class BaseQuestionEntity {
+    @Id
+    @GeneratedValue(strategy = GenerationType.UUID)
+    private UUID id;
+
+    @ManyToOne
+    @JoinColumn(name = "category", nullable = false)
+    private Category category;
+
+    private String difficulty;
+    private String type;
+    private String question;
+    
+    private String correctAnswer;    
+    // @ElementCollection // This stores it in a separate table, it's just a set of strings so not required right now
+    private List<String> incorrectAnswers;
+}

backend/src/main/java/com/ninjaws/quiz/entities/QuestionCollectingEntity.java

@@ -0,0 +1,10 @@
+package com.ninjaws.quiz.entities;
+
+import jakarta.persistence.Entity;
+import jakarta.persistence.Table;
+
+@Entity
+@Table(name = "question_collecting")
+public class QuestionCollectingEntity extends BaseQuestionEntity{
+    
+}

backend/src/main/java/com/ninjaws/quiz/entities/QuestionEntity.java

@@ -1,36 +1,10 @@
 package com.ninjaws.quiz.entities;
 
-import java.util.List;
-
 import jakarta.persistence.Entity;
-import jakarta.persistence.GeneratedValue;
-import jakarta.persistence.GenerationType;
-import jakarta.persistence.Id;
-import jakarta.persistence.JoinColumn;
-import jakarta.persistence.ManyToOne;
 import jakarta.persistence.Table;
-import lombok.Getter;
-import lombok.Setter;
 
 @Entity
 @Table(name = "question")
-@Getter
+public class QuestionEntity extends BaseQuestionEntity {    
-@Setter
-public class QuestionEntity {    
 
-    @Id
-    @GeneratedValue(strategy = GenerationType.IDENTITY)
-    private Long id;
-
-    @ManyToOne
-    @JoinColumn(name = "category", nullable = false)
-    private Category category;
-
-    private String difficulty;
-    private String type;
-    private String question;
-    
-    //* Hidden from the user */
-    private String correctAnswer;    
-    private List<String> incorrectAnswers;
 }

backend/src/main/java/com/ninjaws/quiz/models/QuizSettings.java

@@ -1,8 +1,5 @@
 package com.ninjaws.quiz.models;
 
-import com.ninjaws.quiz.models.QuizSettings.Category;
-import com.ninjaws.quiz.models.QuizSettings.Difficulty;
-
 import jakarta.validation.Valid;
 import jakarta.validation.constraints.Max;
 import jakarta.validation.constraints.Min;
@@ -13,45 +10,9 @@ import lombok.Setter;
 @Setter
 public class QuizSettings {
 
-    public enum Type {
-        ANY, BOOLEAN, MULTIPLE;
-
-        public String getApiValue() {
-            return this.name().toLowerCase();
-        }
-    }
-
-    public enum Difficulty {
-        ANY, EASY, MEDIUM, HARD;
-
-        public String getApiValue() {
-            return this.name().toLowerCase();
-        }
-    }
-
-    public enum Category {
-        ANY, GEN_KNOWLEDGE, ENT_BOOKS, ENT_FILM, ENT_MUSIC, ENT_MUSICAL_THEATRE, ENT_TELEVISION, 
-        ENT_VIDEOGAMES, ENT_BOARDGAMES, SCIENCE_NATURE, SCIENCE_COMPUTERS, SCIENCE_MATH, MYTHOLOGY, SPORTS, GEOGRAPHY, HISTORY, POLITICS, ART, CELEBRITIES, 
-        ANIMALS, VEHICLES, ENT_COMICS, SCIENCE_GADGETS, ENT_ANIME_MANGA, ENT_CARTOON_ANIMATIONS;
-        
-        /**
-         * For some reason the external API starts with a value of 9 for the General Knowledge category, so we have to adjust
-         * ANY also gets adjusted, but it doesn't matter since that one doesn't get used as parameter anyway
-         * @return The corrected value
-         */
-        public int getApiValue() {
-            return 8 + this.ordinal();
-        }
-    }
-
     @Min(1)
     @Max(50)
     private int amount = 10;
-
-    // private Type type = Type.ANY;
-
-    // private Difficulty difficulty = Difficulty.ANY;
-
     private Integer category;
     private String difficulty;
     private String type;
@@ -81,24 +42,6 @@ public class QuizSettings {
      * Converts the class to a format that can be provided as arguments for the api request
      * @return The resulting request arguments as String
      */
-    // public String toApiQuery() {
-    //     StringBuilder query = new StringBuilder("?amount=" + amount);
-        
-    //     if (category != null && category != Category.ANY) {
-    //         query.append("&category=").append(category.getApiValue());
-    //     }
-
-    //     if (difficulty != null && difficulty != Difficulty.ANY) {
-    //         query.append("&difficulty=").append(difficulty.getApiValue());
-    //     }
-
-    //     if (type != null && type != Type.ANY) {
-    //         query.append("&type=").append(type.getApiValue());
-    //     }
-
-    //     return query.toString();
-    // }
-
     public String toApiQuery() {
         StringBuilder query = new StringBuilder("?amount=" + amount);
         

backend/src/main/java/com/ninjaws/quiz/models/SessionTokenDTO.java

@@ -0,0 +1,15 @@
+package com.ninjaws.quiz.models;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import lombok.Getter;
+import lombok.Setter;
+
+@Getter
+@Setter
+public class SessionTokenDTO {
+    @JsonProperty("response_code")
+    private int responseCode;
+    @JsonProperty("token")
+    private String token;    
+}

backend/src/main/java/com/ninjaws/quiz/repositories/QuestionCollectingRepository.java

@@ -0,0 +1,14 @@
+package com.ninjaws.quiz.repositories;
+
+import org.springframework.data.jpa.repository.JpaRepository;
+
+import com.ninjaws.quiz.entities.QuestionCollectingEntity;
+
+/**
+ * This table is used for collecting questions from the external API
+ * When it finishes, the data from this repository will replace the data in the QuestionRepository, to ensure it stays fresh
+ * This should be scheduled once every day
+ */
+public interface QuestionCollectingRepository extends JpaRepository<QuestionCollectingEntity, Long>{
+    
+}

backend/src/main/java/com/ninjaws/quiz/repositories/QuestionRepository.java

@@ -3,24 +3,37 @@ package com.ninjaws.quiz.repositories;
 import java.util.List;
 
 import org.springframework.data.jpa.repository.JpaRepository;
+import org.springframework.data.jpa.repository.Modifying;
 import org.springframework.data.jpa.repository.Query;
+import org.springframework.data.repository.query.Param;
 
 import com.ninjaws.quiz.entities.QuestionEntity;
 
+import jakarta.transaction.Transactional;
+
 public interface QuestionRepository extends JpaRepository<QuestionEntity, Long>{//, JpaSpecificationExecutor<Question> {
 
+    // @Query(value = "SELECT * FROM question AS q " +
+    // "WHERE (?1 IS NULL OR q.category = ?1) " + 
+    // "AND (?2 IS NULL OR q.difficulty = ?2) " +
+    // "AND (?3 IS NULL OR q.type = ?3) " +
+    // "ORDER BY RAND() LIMIT ?4", nativeQuery = true)
     @Query(value = "SELECT * FROM question AS q " +
-    "WHERE (?1 IS NULL OR q.category = ?1) " + 
+        "WHERE (:categoryId IS NULL OR q.category = :categoryId) " + 
-    "AND (?2 IS NULL OR q.difficulty = ?2) " +
+        "AND (:difficulty IS NULL OR q.difficulty = :difficulty) " +
-    "AND (?3 IS NULL OR q.type = ?3) " +
+        "AND (:type IS NULL OR q.type = :type) " +
-    "ORDER BY RAND() LIMIT ?4", nativeQuery = true)
+        "ORDER BY RAND() LIMIT :limit;", nativeQuery = true)
-    List<QuestionEntity> lookupQuestions(Integer categoryId, String difficulty, String type, int limit);
+    List<QuestionEntity> lookupQuestions(
+        @Param("categoryId") Integer categoryId, 
+        @Param("difficulty") String difficulty, 
+        @Param("type") String type, 
+        @Param("limit") int limit);
 
-    // public List<Question> getQuestionsBy(String type, String category, String difficulty, int limit) {
+    /**
-    //     Specification<Product> spec = Specification.where(QuestionSpecification.hasType(type))
+     * Moves all data from one table to another
-    //     .and(QuestionSpecification.hasCategory(category))
+     */
-    //     .and(QuestionSpecification.hasDifficulty(difficulty));
+    @Modifying
-
+    @Transactional
-    //     return productRepository.findAll(spec);
+    @Query(value="INSERT INTO question SELECT * FROM question_collecting;", nativeQuery=true)
-    // }
+    void moveCollection();
 }

backend/src/main/java/com/ninjaws/quiz/services/ApiService.java

@@ -12,6 +12,7 @@ import com.ninjaws.quiz.models.ApiResponse;
 import com.ninjaws.quiz.models.CategoryResponse;
 import com.ninjaws.quiz.models.QuizSettings;
 import com.ninjaws.quiz.models.Request;
+import com.ninjaws.quiz.models.SessionTokenDTO;
 
 import jakarta.validation.Valid;
 
@@ -28,6 +29,7 @@ public class ApiService {
     private static final System.Logger logger = System.getLogger(ApiService.class.getName());
 
     private final String apiUrl = "https://opentdb.com";
+    private String sessionToken;
 
     public ApiService() {
     }
@@ -55,21 +57,42 @@ public class ApiService {
      * Store them in the DataService
      */
     public void collectBulk() {
-        // TODO: Make this happen only once, when a new database is generated
+        String tokenStr = sessionToken == null ? "" : "&token="+sessionToken;
-        getCategories();
+        String url = apiUrl + "/api.php?amount=50" + tokenStr;
-        String url = apiUrl + "/api.php?amount=50";
         try {
             ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
             String json = response.getBody();
             ApiResponse apiResponse = cleanerService.extractApiResponse(json);
             responseHandler.handleResponse(apiResponse);
             
+            /** 
+             * A bit dirty, but otherwise it causes circular dependencies 
+             * Makes sure the categories are refreshed after finishing to replace the questions, to make sure the ids match
+             * And resets the sessionToken, it will be remade during the next REFRESH
+            */
+            if(apiResponse.getResponseCode() == 4) {
+                getCategories();
+                sessionToken = null;
+            }
+            
         } catch (RestClientException | IOException e) {
             logger.log(System.Logger.Level.ERROR, "Critical failure", e);
         }
 
     }
 
+    public void getSessionToken() {
+        String url = apiUrl + "/api_token.php?command=request";
+        try {
+            ResponseEntity<String> response = restTemplate.getForEntity(url, String.class);
+            String json = response.getBody();
+            SessionTokenDTO tokenDto = cleanerService.extractSessionToken(json);
+            sessionToken = tokenDto.getToken();
+            
+        } catch (RestClientException | IOException e) {
+            logger.log(System.Logger.Level.ERROR, "Critical failure", e);
+        }
+    }
 
     public void getCategories() {
         String url = apiUrl + "/api_category.php";

backend/src/main/java/com/ninjaws/quiz/services/CleanerService.java

@@ -10,12 +10,20 @@ import org.springframework.stereotype.Service;
 
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.ninjaws.quiz.entities.Category;
+import com.ninjaws.quiz.entities.QuestionCollectingEntity;
 import com.ninjaws.quiz.entities.QuestionEntity;
 import com.ninjaws.quiz.models.ApiResponse;
 import com.ninjaws.quiz.models.CategoryResponse;
 import com.ninjaws.quiz.models.QuestionDTO;
+import com.ninjaws.quiz.models.SessionTokenDTO;
 import com.ninjaws.quiz.repositories.CategoryRepository;
 
+/**
+ * Deals with conversions from one form of data to another
+ * Json to Object
+ * Entity to DTO
+ * etc
+ */
 @Service
 public class CleanerService {
 
@@ -44,6 +52,10 @@ public class CleanerService {
         return dtos;
     }
 
+    public SessionTokenDTO extractSessionToken(String json) throws IOException {
+        return objectMapper.readValue(json, SessionTokenDTO.class);
+    }
+
     public QuestionDTO questionEntityToDTO(QuestionEntity entity) {
         QuestionDTO dto = new QuestionDTO();
         dto.setQuestion(entity.getQuestion());
@@ -76,6 +88,30 @@ public class CleanerService {
         return entity;
     }
 
+
+    // TODO: Find a simpler way to convert from one table to another if the data is identical so this nonsense isn't necessary
+    public List<QuestionCollectingEntity> questionListDTOtoCollectingEntity(List<QuestionDTO> dtos) {
+        List<QuestionCollectingEntity> entities = new ArrayList<>();
+        for (int i = 0; i < dtos.size(); i++) {
+            QuestionDTO dto = dtos.get(i);
+            entities.add(questionDTOtoCollectingEntity(dto));
+        }
+        return entities;
+    }
+   
+    public QuestionCollectingEntity questionDTOtoCollectingEntity(QuestionDTO dto) {
+        QuestionCollectingEntity entity = new QuestionCollectingEntity();
+        Category category = categoryRepository.findByName(decodeString(dto.getCategory())).orElseThrow(() -> new IllegalStateException("Category not in DB!"));
+        entity.setCategory(category);
+        entity.setDifficulty(dto.getDifficulty());
+        entity.setType(dto.getType());
+        entity.setQuestion(dto.getQuestion());
+        entity.setCorrectAnswer(dto.getCorrectAnswer());
+        entity.setIncorrectAnswers(dto.getIncorrectAnswers());
+        return entity;
+    }
+
+
     /** 
      * For some reason, the categories come with special characters,
      * But the categories from the questions dont

backend/src/main/java/com/ninjaws/quiz/services/DataService.java

@@ -6,17 +6,22 @@ import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 
 import com.ninjaws.quiz.entities.Category;
+import com.ninjaws.quiz.entities.QuestionCollectingEntity;
 import com.ninjaws.quiz.entities.QuestionEntity;
 import com.ninjaws.quiz.models.QuizSettings;
 import com.ninjaws.quiz.repositories.CategoryRepository;
+import com.ninjaws.quiz.repositories.QuestionCollectingRepository;
 import com.ninjaws.quiz.repositories.QuestionRepository;
 
+import jakarta.transaction.Transactional;
+
 @Service
 public class DataService {
 
     @Autowired
     private QuestionRepository questionRepository;
-
+    @Autowired
+    private QuestionCollectingRepository collectingRepository;
     @Autowired
     private CategoryRepository categoryRepository;
 
@@ -32,34 +37,34 @@ public class DataService {
         return questionRepository.saveAll(questions);
     }
 
-    // public List<Question> getRandomQuestions(String type, String category, String difficulty, int limit) {
-    //     return questionRepository.findRandomQuestions(type, limit);
-    // }
-
-    public List<QuestionEntity> lookupQuestions(QuizSettings quizSettings) {
-        // Long category = Long.valueOf(quizSettings.getCategory().getApiValue() == 1 ? null : quizSettings.getCategory().getApiValue());
-        // String difficulty = quizSettings.getDifficulty() == QuizSettings.Difficulty.ANY ? null : quizSettings.getDifficulty().getApiValue();
-        // String type = quizSettings.getType() == QuizSettings.Type.ANY ? null : quizSettings.getType().getApiValue();
-        // System.out.println(category);
-        // System.out.println(difficulty);
-        // System.out.println(type);
-        
-        return questionRepository.lookupQuestions(quizSettings.getCategory(), quizSettings.getDifficulty(), quizSettings.getType(), quizSettings.getAmount());
-    }
-
     /**
      * Tries to get questions that match the specified QuizSettings
+     *
      * @return The questions if the query was possible, and an empty array if not
      */
-    // public List<Question> getByQuizSettings(QuizSettings quizSettings) {
+    public List<QuestionEntity> lookupQuestions(QuizSettings quizSettings) {
-    //     Specification<Product> spec = Specification.where(QuestionSpecification.hasType(type))
+        return questionRepository.lookupQuestions(quizSettings.getCategory(), quizSettings.getDifficulty(), quizSettings.getType(), quizSettings.getAmount());
-    //     .and(QuestionSpecification.hasCategory(category))
+    }
-    //     .and(QuestionSpecification.hasDifficulty(difficulty));
-
-    //     return productRepository.findAll(spec);
-    // }
 
     public List<Category> saveCategories(List<Category> categories) {
         return categoryRepository.saveAll(categories);
     }
+
+    public List<Category> getCategories(){
+        return categoryRepository.findAll();
+    }
+    /**
+     * Moving the collected questions to the questions list
+     *
+     */
+    @Transactional
+    public void migrateQuestions() {
+        questionRepository.deleteAll();
+        questionRepository.moveCollection();
+        collectingRepository.deleteAll();
+    }
+
+    public List<QuestionCollectingEntity> collectQuestions(List<QuestionCollectingEntity> questions) {
+        return collectingRepository.saveAll(questions);
+    }
 }

backend/src/main/java/com/ninjaws/quiz/services/DatabaseScheduler.java

@@ -1,20 +0,0 @@
-package com.ninjaws.quiz.services;
-
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.jdbc.core.JdbcTemplate;
-import org.springframework.scheduling.annotation.Scheduled;
-import org.springframework.stereotype.Service;
-
-@Service
-public class DatabaseScheduler {
-    @Autowired
-    private JdbcTemplate secondaryJdbcTemplate;
-
-    @Scheduled(fixedRate = 60000) // Runs every 60 seconds
-    public void createAndFillSecondaryDatabase() {
-        // Create tables and insert data into the secondary database
-        secondaryJdbcTemplate.
-        .execute("CREATE TABLE IF NOT EXISTS sample_table (id INT PRIMARY KEY, name VARCHAR(255))");
-        secondaryJdbcTemplate.update("INSERT INTO sample_table (id, name) VALUES (?, ?)", 1, "Sample Data");
-    }
-}

backend/src/main/java/com/ninjaws/quiz/services/QuizService.java

@@ -24,6 +24,8 @@ public class QuizService {
     private DataService dataService;
     @Autowired
     private CleanerService cleanerService;
+    @Autowired
+    private StateManagementService stateManager;
 
     /**
      * Actions:
@@ -38,19 +40,20 @@ public class QuizService {
 
         /** First, we check if there are already enough items in our db */
         List<QuestionEntity> questionsRetrieved = dataService.lookupQuestions(quizSettings);
-        if(!questionsRetrieved.isEmpty()) {
+        /** If the items are not matching the required amount, and the db is not currently being filled up, then they simply don't exist */
-            session.setQuestions(cleanerService.questionListEntitytoDTO(questionsRetrieved));
+        if (questionsRetrieved.size() < quizSettings.getAmount() && stateManager.getCurrentProcessingState() != StateManagementService.ProcessingState.STARTUP) {
+            session.setStatusCode(1);
+            cacheService.addToCache(session.getId(), session);
+            return Optional.of(session.getId());
         }
-
+        /** All items are collected in from the database, it doesn't need to be added to the queue */
-        cacheService.addToCache(session.getId(), session);
+        if(!questionsRetrieved.isEmpty() && questionsRetrieved.size() == quizSettings.getAmount()) {
-
+            session.setQuestions(cleanerService.questionListEntitytoDTO(questionsRetrieved));
-        /**
+            cacheService.addToCache(session.getId(), session);
-         * If the questions were retrieved from the DB, then it doesn't need to be added to the queue
-         */
-        if(!questionsRetrieved.isEmpty()) {
             return Optional.of(session.getId());
         }
 
+        cacheService.addToCache(session.getId(), session);
         boolean addedToQueue = queueService.addToQueue(new Request(session.getId(), quizSettings));
         /** In this order, because doing the queue before the cache could lead to race conditions (request completion immediately calls cache, but then the item doesn't exist) */
         if(!addedToQueue) {
@@ -75,15 +78,17 @@ public class QuizService {
             throw new IllegalArgumentException("Invalid session ID: " + sessionId);
         }
 
+        /** Anything other than statuscode 0 doesn't result in active sessions, so remove them from the cache after reporting to the frontend */
+        if(session.getStatusCode() != 0) {
+            cacheService.removeFromCache(session.getId());
+            return Optional.of(session);
+        }
+
         /** Still waiting for a response */
         if (session.getQuestions() == null) {
             return Optional.empty();
         }
 
-        /** Anything other than statuscode 0 doesn't result in active sessions, so remove them from the cache after reporting to the frontend */
-        if(session.getStatusCode() != 0) {
-            cacheService.removeFromCache(session.getId());
-        }
 
         return Optional.of(session);
     }
@@ -92,7 +97,7 @@ public class QuizService {
      * Receives: sessionId and answers
      * 
      * Actions: 
-     * - retrieves session from cache
+     * - Retrieves session from cache
      * - Deletes session from cache
      * - Matches answers answers to correctAnswers
      * - Calculates score

backend/src/main/java/com/ninjaws/quiz/services/ResponseHandlingService.java

@@ -10,6 +10,8 @@ import com.ninjaws.quiz.models.Session;
 @Service
 public class ResponseHandlingService {
 
+    @Autowired
+    private StateManagementService stateManager;
     @Autowired
     private CacheService cacheService;
     @Autowired
@@ -33,7 +35,8 @@ public class ResponseHandlingService {
             case 0:
                 // Success
                 updateSessionInCache(session, response);
-                dataService.saveQuestions(cleanerService.questionListDTOtoEntity(response.getResults()));
+                /** Managing the token gets difficult since there is a timelimit, so simply dont add these, let the bulk collector do it */
+                // dataService.saveQuestions(cleanerService.questionListDTOtoEntity(response.getResults()));
                 break;
             case 1:
                 // Not enough questions to satisfy the request. 
@@ -67,11 +70,26 @@ public class ResponseHandlingService {
     public void handleResponse(ApiResponse response) {
         switch (response.getResponseCode()) {
             case 0:
-                dataService.saveQuestions(cleanerService.questionListDTOtoEntity(response.getResults()));
+                if(stateManager.getCurrentProcessingState() == StateManagementService.ProcessingState.STARTUP) {
+                    /** Initial saving during the startup */
+                    dataService.saveQuestions(cleanerService.questionListDTOtoEntity(response.getResults()));
+                } else if (stateManager.getCurrentProcessingState() == StateManagementService.ProcessingState.REFRESH) {
+                    /** Background collecting during the night */
+                    dataService.collectQuestions(cleanerService.questionListDTOtoCollectingEntity(response.getResults()));
+                }
                 break;
             case 4:
-                // Signal DataService that we have extracted all the data. Stop gathering, switch to caching-mode
+                // Signal DataService that we have extracted all the data. Stop gathering, 
-                // TODO: Signal DB that it's full, start db-swapping process
+                // TODO: Switch to caching-mode
+                
+                /** 
+                 * If it was in the REFRESH state, move the data from collecting to the used questions
+                 * Otherwise simply switch to the NORMAL state
+                 */
+                if (stateManager.getCurrentProcessingState() == StateManagementService.ProcessingState.REFRESH) {
+                    dataService.migrateQuestions();
+                }
+                stateManager.setCurrentProcessingState(StateManagementService.ProcessingState.NORMAL);
                 break;
             case 5:
                 // Should be prevented by the scheduler

backend/src/main/java/com/ninjaws/quiz/services/SchedulerService.java

@@ -13,6 +13,8 @@ public class SchedulerService {
     private QueueService queueService;
     @Autowired
     private ApiService apiService;
+    @Autowired
+    private StateManagementService stateService;
 
     public SchedulerService() {}
 
@@ -25,11 +27,23 @@ public class SchedulerService {
     @Scheduled(fixedDelay = 5000)
     protected void processQueue() {
         Request request = queueService.popFromQueue();
+
         if (request != null) {
             apiService.handleRequest(request);
         } else {
-            apiService.collectBulk();
+            if (stateService.getCurrentProcessingState() == StateManagementService.ProcessingState.STARTUP || stateService.getCurrentProcessingState() == StateManagementService.ProcessingState.REFRESH) {
+                apiService.collectBulk();
+            }
         }
     }   
 
+    /**
+     * Starts the renewing of the questions, to ensure they stay fresh
+     */
+    @Scheduled(cron = "0 0 1 * * ?")
+    protected void StartDbRefresh() {
+        apiService.getSessionToken();
+        stateService.setCurrentProcessingState(StateManagementService.ProcessingState.REFRESH);
+    }
+    
 }

backend/src/main/java/com/ninjaws/quiz/services/StateManagementService.java

@@ -0,0 +1,29 @@
+package com.ninjaws.quiz.services;
+
+import org.springframework.stereotype.Service;
+
+@Service
+public class StateManagementService {
+
+    /**
+     * STARTUP: When starting the app
+     * REFRESH: Scheduled at night, so the database stays fresh
+     * NORMAL: No special activities
+     */
+    public enum ProcessingState {
+        STARTUP,
+        REFRESH,
+        NORMAL
+    }
+    
+    private ProcessingState currentProcessingState;
+    
+    public ProcessingState getCurrentProcessingState() {
+        return currentProcessingState;
+    }
+
+    public void setCurrentProcessingState(ProcessingState currentProcessingState) {
+        this.currentProcessingState = currentProcessingState;
+    }
+    
+}

backend/src/main/resources/application.properties

@@ -6,29 +6,29 @@ spring.h2.console.enabled=true
 # spring.h2.console.path=/console
 
 
-# spring.datasource.url=jdbc:h2:mem:testdb
+spring.datasource.url=jdbc:h2:mem:testdb
-# spring.datasource.driverClassName=org.h2.Driver
+spring.datasource.driverClassName=org.h2.Driver
-# spring.datasource.username=sa
+spring.datasource.username=sa
-# spring.datasource.password=
+spring.datasource.password=
 
-# spring.jpa.hibernate.ddl-auto=create-drop
+spring.jpa.hibernate.ddl-auto=create-drop
-# spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
+spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
 
 
-# Primary database configuration
+# # Primary database configuration
-spring.datasource.primary.url=jdbc:h2:mem:testdb
+# spring.datasource.primary.url=jdbc:h2:mem:testdb
-# ;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
+# # ;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
-spring.datasource.primary.driver-class-name=org.h2.Driver
+# spring.datasource.primary.driver-class-name=org.h2.Driver
-spring.datasource.primary.username=sa
+# spring.datasource.primary.username=sa
-spring.datasource.primary.password=
+# spring.datasource.primary.password=
-spring.jpa.primary.hibernate.ddl-auto=create-drop
+# spring.jpa.primary.hibernate.ddl-auto=create-drop
-spring.jpa.primary.database-platform=org.hibernate.dialect.H2Dialect
+# spring.jpa.primary.database-platform=org.hibernate.dialect.H2Dialect
 
-# Secondary database configuration
+# # Secondary database configuration
-spring.datasource.secondary.url=jdbc:h2:mem:secondarydb
+# spring.datasource.secondary.url=jdbc:h2:mem:secondarydb
-# ;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
+# # ;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
-spring.datasource.secondary.driver-class-name=org.h2.Driver
+# spring.datasource.secondary.driver-class-name=org.h2.Driver
-spring.datasource.secondary.username=sa
+# spring.datasource.secondary.username=sa
-spring.datasource.secondary.password=
+# spring.datasource.secondary.password=
-spring.jpa.secondary.hibernate.ddl-auto=create-drop
+# spring.jpa.secondary.hibernate.ddl-auto=create-drop
-spring.jpa.secondary.database-platform=org.hibernate.dialect.H2Dialect
+# spring.jpa.secondary.database-platform=org.hibernate.dialect.H2Dialect

backend/src/test/java/com/ninjaws/quiz/scala/QuestionStatusSimulation.scala

@@ -3,6 +3,7 @@ import io.gatling.core.Predef._
 import io.gatling.http.Predef._
 import scala.concurrent.duration._
 
+/** Does not work, use the LoadTestMultiple.sh in the /scripts folder in the root instead */
 class QuestionStatusSimulation extends Simulation {
     val httpProtocol = http.baseUrl("http://localhost:9090").acceptHeader("application/json")
 

backend/src/test/java/com/ninjaws/quiz/services/CleanerServiceTest.txt

@@ -1,107 +0,0 @@
-package com.ninjaws.quiz.services;
-
-import java.io.IOException;
-import java.util.Arrays;
-
-import static org.assertj.core.api.Assertions.assertThat;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import org.junit.jupiter.api.extension.ExtendWith;
-import org.mockito.InjectMocks;
-import org.mockito.Mock;
-import static org.mockito.Mockito.times;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.when;
-import org.mockito.MockitoAnnotations;
-import org.mockito.junit.jupiter.MockitoExtension;
-
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.ninjaws.quiz.entities.Category;
-import com.ninjaws.quiz.models.ApiResponse;
-import com.ninjaws.quiz.models.QuestionDTO;
-
-// public class CleanerServiceTest {
-
-//     @InjectMocks
-//     private CleanerService cleanerService;
-
-//     @Mock
-//     private ObjectMapper objectMapper;
-
-//     @BeforeEach
-//     public void setUp() {
-//         MockitoAnnotations.openMocks(this);
-//     }
-
-//     @Test
-//     public void testExtractApiResponse() throws IOException {
-//         String json = "{\"response_code\":200,\"results\":[{\"type\":\"multiple choice\",\"difficulty\":\"easy\",\"category\":\"general knowledge\",\"question\":\"What is the capital of France?\",\"correct_answer\":\"Paris\",\"incorrect_answers\":[\"Berlin\",\"Madrid\",\"Rome\"]}]}"; // Example JSON input
-       
-//         Question question = new Question();
-//         question.setType("multiple choice");
-//         question.setDifficulty("easy");
-//         question.setCategory("general knowledge");
-//         question.setQuestion("What is the capital of France?");
-//         question.setCorrectAnswer("Paris");
-//         question.setIncorrectAnswers(Arrays.asList("Berlin", "Madrid", "Rome"));
-
-//         ApiResponse expectedResponse = new ApiResponse();
-//         expectedResponse.setResponseCode(200);
-//         expectedResponse.setResults(Arrays.asList(question));
-//         when(objectMapper.readValue(json, ApiResponse.class)).thenReturn(expectedResponse);
-        
-//         ApiResponse actualResponse = cleanerService.extractApiResponse(json);
-//         assertThat(actualResponse).isNotNull();
-//         assertThat(actualResponse.getResponseCode()).isEqualTo(expectedResponse.getResponseCode());
-//         assertThat(actualResponse.getResults()).isNotEmpty();
-//         assertThat(actualResponse.getResults().get(0).getQuestion()).isEqualTo(question.getQuestion());
-//         verify(objectMapper, times(1)).readValue(json, ApiResponse.class);
-//     }
-// }
-
-@ExtendWith(MockitoExtension.class)
-public class CleanerServiceTest {
-
-    @InjectMocks
-    private CleanerService cleanerService;
-
-    @Mock
-    private ObjectMapper objectMapper;
-
-    @BeforeEach
-    public void setUp() {
-        MockitoAnnotations.openMocks(this);
-    }
-
-    @Test
-    public void testExtractApiResponse() throws IOException {
-        String json = "{\"response_code\":200,\"results\":[{\"type\":\"multiple choice\",\"difficulty\":\"easy\",\"category\":\"general knowledge\",\"question\":\"What is the capital of France?\",\"correct_answer\":\"Paris\",\"incorrect_answers\":[\"Berlin\",\"Madrid\",\"Rome\"]}]}"; // Example JSON input
-        
-        // Create a Category object for the question
-        Category category = new Category();
-        category.setId(1L);
-        category.setName("General Knowledge");
-
-        // Create the Question object with the category
-        QuestionDTO question = new QuestionDTO();
-        question.setType("multiple choice");
-        question.setDifficulty("easy");
-        question.setCategory(category); // Set the Category object here
-        question.setQuestion("What is the capital of France?");
-        question.setCorrectAnswer("Paris");
-        question.setIncorrectAnswers(Arrays.asList("Berlin", "Madrid", "Rome"));
-
-        ApiResponse expectedResponse = new ApiResponse();
-        expectedResponse.setResponseCode(200);
-        expectedResponse.setResults(Arrays.asList(question));
-        when(objectMapper.readValue(json, ApiResponse.class)).thenReturn(expectedResponse);
-        
-        ApiResponse actualResponse = cleanerService.extractApiResponse(json);
-        assertThat(actualResponse).isNotNull();
-        assertThat(actualResponse.getResponseCode()).isEqualTo(expectedResponse.getResponseCode());
-        assertThat(actualResponse.getResults()).isNotEmpty();
-        assertThat(actualResponse.getResults().get(0).getQuestion()).isEqualTo(question.getQuestion());
-        assertThat(actualResponse.getResults().get(0).getCategory().getName()).isEqualTo(category.getName()); // Check category name
-        verify(objectMapper, times(1)).readValue(json, ApiResponse.class);
-    }
-}

backend/src/test/java/com/ninjaws/quiz/services/DataServiceTest.txt

@@ -1,202 +0,0 @@
-package com.ninjaws.quiz.services;
-
-import java.util.Arrays;
-import java.util.List;
-import java.util.Optional;
-
-import static org.assertj.core.api.Assertions.assertThat;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import org.junit.jupiter.api.extension.ExtendWith;
-import static org.mockito.ArgumentMatchers.any;
-import static org.mockito.ArgumentMatchers.anyList;
-import org.mockito.InjectMocks;
-import org.mockito.Mock;
-import static org.mockito.Mockito.times;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.when;
-import org.mockito.MockitoAnnotations;
-import org.mockito.junit.jupiter.MockitoExtension;
-
-import com.ninjaws.quiz.entities.Category;
-import com.ninjaws.quiz.models.QuestionDTO;
-import com.ninjaws.quiz.repositories.QuestionRepository;
-
-// public class DataServiceTest {
-    
-//     @InjectMocks
-//     private DataService dataService;
-
-//     @Mock
-//     private QuestionRepository questionRepository;
-
-//     private Question question;
-
-
-//     @BeforeEach
-//     public void setUp() {
-//         MockitoAnnotations.openMocks(this);
-        
-//         question = new Question();
-//         question.setType("multiple choice");
-//         question.setDifficulty("easy");
-//         question.setCategory("general knowledge");
-//         question.setQuestion("What is the capital of France?");
-//         question.setCorrectAnswer("Paris");
-//         question.setIncorrectAnswers(Arrays.asList("Berlin", "Madrid", "Rome"));
-//     }
-
-//     @Test
-//     public void testSaveQuestion() {
-//         when(questionRepository.save(any(Question.class))).thenReturn(question);
-
-//         Question savedQuestion = dataService.saveQuestion(question);
-//         assertThat(savedQuestion).isNotNull();
-//         assertThat(savedQuestion.getQuestion()).isEqualTo("What is the capital of France?");
-//         verify(questionRepository, times(1)).save(question);
-//     }
-
-//     @Test
-//     public void testGetAllQuestions() {
-//         when(questionRepository.findAll()).thenReturn(Arrays.asList(question));
-
-//         List<Question> questions = dataService.getAllQuestions();
-//         assertThat(questions).isNotEmpty();
-//         assertThat(questions.size()).isEqualTo(1);
-//         assertThat(questions.get(0).getQuestion()).isEqualTo("What is the capital of France?");
-//         verify(questionRepository, times(1)).findAll();
-//     }
-
-//     @Test
-//     public void testFindQuestionById() {
-//         when(questionRepository.findById(any(Long.class))).thenReturn(Optional.of(question));
-
-//         Optional<Question> foundQuestion = questionRepository.findById(1L);
-//         assertThat(foundQuestion).isPresent();
-//         assertThat(foundQuestion.get().getQuestion()).isEqualTo("What is the capital of France?");
-//         verify(questionRepository, times(1)).findById(1L);
-//     }
-
-//     @Test
-//     public void testSaveQuestions() {
-//         Question question1 = new Question();
-//         question1.setType("multiple choice");
-//         question1.setDifficulty("easy");
-//         question1.setCategory("general knowledge");
-//         question1.setQuestion("What is the capital of France?");
-//         question1.setCorrectAnswer("Paris");
-//         question1.setIncorrectAnswers(Arrays.asList("Berlin", "Madrid", "Rome"));
-
-//         Question question2 = new Question();
-//         question2.setType("multiple choice");
-//         question2.setDifficulty("medium");
-//         question2.setCategory("science");
-//         question2.setQuestion("What is the chemical symbol for water?");
-//         question2.setCorrectAnswer("H2O");
-//         question2.setIncorrectAnswers(Arrays.asList("CO2", "O2", "NaCl"));
-
-//         when(questionRepository.saveAll(anyList())).thenReturn(Arrays.asList(question1, question2));
-
-//         List<Question> questions = dataService.saveQuestions(Arrays.asList(question1, question2));
-
-//         assertThat(questions).hasSize(2);
-//         assertThat(questions.get(0).getQuestion()).isEqualTo("What is the capital of France?");
-//         assertThat(questions.get(1).getQuestion()).isEqualTo("What is the chemical symbol for water?");
-//         verify(questionRepository, times(1)).saveAll(anyList());
-//     }
-// }
-
-@ExtendWith(MockitoExtension.class)
-public class DataServiceTest {
-
-    @InjectMocks
-    private DataService dataService;
-
-    @Mock
-    private QuestionRepository questionRepository;
-
-    private QuestionDTO question;
-    private Category category;
-
-    @BeforeEach
-    public void setUp() {
-        MockitoAnnotations.openMocks(this);
-
-        category = new Category();
-        category.setId(1L);
-        category.setName("General Knowledge");
-
-        question = new QuestionDTO();
-        question.setType("multiple choice");
-        question.setDifficulty("easy");
-        question.setCategory(category); // Set the Category object here
-        question.setQuestion("What is the capital of France?");
-        question.setCorrectAnswer("Paris");
-        question.setIncorrectAnswers(Arrays.asList("Berlin", "Madrid", "Rome"));
-    }
-
-    @Test
-    public void testSaveQuestion() {
-        when(questionRepository.save(any(QuestionDTO.class))).thenReturn(question);
-
-        QuestionDTO savedQuestion = dataService.saveQuestion(question);
-        assertThat(savedQuestion).isNotNull();
-        assertThat(savedQuestion.getQuestion()).isEqualTo("What is the capital of France?");
-        assertThat(savedQuestion.getCategory().getName()).isEqualTo("General Knowledge"); // Check category name
-        verify(questionRepository, times(1)).save(question);
-    }
-
-    @Test
-    public void testGetAllQuestions() {
-        when(questionRepository.findAll()).thenReturn(Arrays.asList(question));
-
-        List<QuestionDTO> questions = dataService.getAllQuestions();
-        assertThat(questions).isNotEmpty();
-        assertThat(questions.size()).isEqualTo(1);
-        assertThat(questions.get(0).getQuestion()).isEqualTo("What is the capital of France?");
-        assertThat(questions.get(0).getCategory().getName()).isEqualTo("General Knowledge"); // Check category name
-        verify(questionRepository, times(1)).findAll();
-    }
-
-    @Test
-    public void testFindQuestionById() {
-        when(questionRepository.findById(any(Long.class))).thenReturn(Optional.of(question));
-
-        Optional<QuestionDTO> foundQuestion = questionRepository.findById(1L);
-        assertThat(foundQuestion).isPresent();
-        assertThat(foundQuestion.get().getQuestion()).isEqualTo("What is the capital of France?");
-        assertThat(foundQuestion.get().getCategory().getName()).isEqualTo("General Knowledge"); // Check category name
-        verify(questionRepository, times(1)).findById(1L);
-    }
-
-    @Test
-    public void testSaveQuestions() {
-        QuestionDTO question1 = new QuestionDTO();
-        question1.setType("multiple choice");
-        question1.setDifficulty("easy");
-        question1.setCategory(category); // Set the Category object here
-        question1.setQuestion("What is the capital of France?");
-        question1.setCorrectAnswer("Paris");
-        question1.setIncorrectAnswers(Arrays.asList("Berlin", "Madrid", "Rome"));
-
-        QuestionDTO question2 = new QuestionDTO();
-        Category category2 = new Category();
-        category2.setId(2L);
-        category2.setName("Science");
-        question2.setType("multiple choice");
-        question2.setDifficulty("medium");
-        question2.setCategory(category2); // Set the Category object here
-        question2.setQuestion("What is the chemical symbol for water?");
-        question2.setCorrectAnswer("H2O");
-        question2.setIncorrectAnswers(Arrays.asList("CO2", "O2", "NaCl"));
-
-        when(questionRepository.saveAll(anyList())).thenReturn(Arrays.asList(question1, question2));
-
-        List<QuestionDTO> questions = dataService.saveQuestions(Arrays.asList(question1, question2));
-
-        assertThat(questions).hasSize(2);
-        assertThat(questions.get(0).getQuestion()).isEqualTo("What is the capital of France?");
-        assertThat(questions.get(1).getQuestion()).isEqualTo("What is the chemical symbol for water?");
-        verify(questionRepository, times(1)).saveAll(anyList());
-    }
-}

backend/src/test/java/com/ninjaws/quiz/services/backupApiServiceTest.txt

@@ -1,114 +0,0 @@
-package com.ninjaws.quiz.services;
-
-import static org.assertj.core.api.Assertions.assertThat;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.Test;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.test.context.SpringBootTest;
-
-import com.fasterxml.jackson.databind.JsonNode;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.ninjaws.quiz.models.QuizSettings;
-
-@SpringBootTest
-public class ApiServiceTest {
-
-    @Autowired
-    private ApiService apiService;
-
-    @Autowired
-    private ObjectMapper objectMapper;
-
-    /**
-     * The external API only allows one request every 5 seconds
-     */
-    @AfterEach
-    public void cooldown() throws InterruptedException {
-        Thread.sleep(5000);
-    }
-
-    @Test
-    public void testRequestQuestions() throws Exception {
-        QuizSettings settings = new QuizSettings();
-        String jsonResponse = apiService.handleRequest(settings);
-        JsonNode jsonNode = objectMapper.readTree(jsonResponse);
-
-        // Check if the response code is as expected
-        assertThat(jsonNode.has("response_code")).isTrue();
-        assertThat(jsonNode.get("response_code").asInt()).isEqualTo(0);
-
-        // Check the results array
-        assertThat(jsonNode.has("results")).isTrue();
-        assertThat(jsonNode.get("results").isArray()).isTrue();
-        assertThat(jsonNode.get("results").size()).isEqualTo(10);
-
-        // Verify the first result has the expected fields
-        JsonNode firstResult = jsonNode.get("results").get(0);
-        assertThat(firstResult.has("type")).isTrue();
-        assertThat(firstResult.has("difficulty")).isTrue();
-        assertThat(firstResult.has("category")).isTrue();
-        assertThat(firstResult.has("question")).isTrue();
-        assertThat(firstResult.has("correct_answer")).isTrue();
-        assertThat(firstResult.has("incorrect_answers")).isTrue();
-        assertThat(firstResult.get("incorrect_answers").size()).isGreaterThan(0);
-    }
-
-    @Test
-    public void testAllParams() throws Exception {
-        int amount = 10;
-
-        QuizSettings settings = new QuizSettings();
-        settings.setAmount(amount);
-        settings.setCategory(QuizSettings.Category.HISTORY);
-        settings.setType(QuizSettings.Type.MULTIPLE);
-        settings.setDifficulty(QuizSettings.Difficulty.MEDIUM);
-
-        String jsonResponse = apiService.requestQuestions(settings);
-        JsonNode jsonNode = objectMapper.readTree(jsonResponse);
-
-        // Check if the response code is as expected
-        assertThat(jsonNode.has("response_code")).isTrue();
-        assertThat(jsonNode.get("response_code").asInt()).isEqualTo(0);
-
-        // Check the results array
-        assertThat(jsonNode.has("results")).isTrue();
-        assertThat(jsonNode.get("results").isArray()).isTrue();
-        assertThat(jsonNode.get("results").size()).isEqualTo(amount);
-
-        // Verify the first result has the expected fields
-        JsonNode firstResult = jsonNode.get("results").get(0);
-        String retrievedType = firstResult.get("type").toString();
-        String retrievedDifficulty = firstResult.get("difficulty").toString();
-        String retrievedCategory = firstResult.get("category").toString();
-        assertThat(retrievedType.equals(settings.getType().getApiValue()));
-        assertThat(retrievedDifficulty.equals(settings.getDifficulty().getApiValue()));
-        assertThat(retrievedCategory.equals("Animals"));
-        assertThat(firstResult.get("incorrect_answers").size()).isEqualTo(3);
-    }
-
-    /**
-     * If the API doesn't have the requested amount of questions, it will return an empty list and respond with 1
-     */
-    @Test
-    public void testCombinationThatResultsInZeroQuestions() throws Exception {
-        int amount = 30;
-
-        QuizSettings settings = new QuizSettings();
-        settings.setAmount(amount);
-        settings.setCategory(QuizSettings.Category.ANIMALS);
-        settings.setType(QuizSettings.Type.MULTIPLE);
-        settings.setDifficulty(QuizSettings.Difficulty.MEDIUM);
-
-        String jsonResponse = apiService.requestQuestions(settings);
-        JsonNode jsonNode = objectMapper.readTree(jsonResponse);
-
-        // Check if the response code is as expected
-        assertThat(jsonNode.has("response_code")).isTrue();
-        assertThat(jsonNode.get("response_code").asInt()).isEqualTo(1);
-
-        // Check the results array
-        assertThat(jsonNode.has("results")).isTrue();
-        assertThat(jsonNode.get("results").isArray()).isTrue();
-        assertThat(jsonNode.get("results").size()).isEqualTo(0);
-    }
-}

backend/src/test/java/com/ninjaws/quiz/services/backupQuizServiceTest.txt

@@ -1,72 +0,0 @@
-package com.ninjaws.quiz.services;
-
-import org.springframework.boot.test.context.SpringBootTest;
-
-@SpringBootTest
-// @EnableScheduling
-// @TestInstance(TestInstance.Lifecycle.PER_CLASS)
-public class QuizServiceTest {
-    
-    // @InjectMocks
-    // private QuizService quizService;
-
-    // @Mock
-    // private ApiService apiService;
-
-    // @BeforeAll
-    // public void setUp() {
-    //     MockitoAnnotations.openMocks(this);
-    //     // Cache<String, Session> cache = Caffeine.newBuilder()
-    //     //     .expireAfterWrite(1, TimeUnit.HOURS)
-    //     //     .build(); // Create a mock cache or a real cache for tests
-    //     // quizService = new QuizService(cache); 
-    // }
-
-    // @Test
-    // public void testCreateSession() {
-    //     QuizSettings settings = new QuizSettings();
-    //     String response = quizService.createSession(settings);
-    //     assertTrue(response instanceof String);
-    // }
-
-    // /**
-    //  * TODO: Fix the test
-    //  * Seems to clear the cache and queue when the createSession is finished 
-    //  */
-    // @Test
-    // public void testCheckSession() throws InterruptedException {
-        
-    //     QuizSettings settings = new QuizSettings();
-    //     when(apiService.requestQuestions(settings)).thenReturn("{\"response_code\":0,\"results\":[{\"type\":\"multiple\",\"difficulty\":\"hard\",\"category\":\"Entertainment: Video Games\",\"question\":\"The map featured in Arma 3 named &quot;Altis&quot; is based off of what Greek island?\",\"correct_answer\":\"Lemnos\",\"incorrect_answers\":[\"Ithaca\",\"Naxos\",\"Anafi\"]},{\"type\":\"multiple\",\"difficulty\":\"medium\",\"category\":\"Entertainment: Music\",\"question\":\"What is the last song on the first Panic! At the Disco album?\",\"correct_answer\":\"Build God, Then We&#039;ll Talk\",\"incorrect_answers\":[\"Nails for Breakfast, Tacks for Snacks\",\"I Write Sins Not Tragedies\",\"Lying Is The Most Fun A Girl Can Have Without Taking Her Clothes Off\"]},{\"type\":\"multiple\",\"difficulty\":\"easy\",\"category\":\"Entertainment: Video Games\",\"question\":\"Which of these is NOT a playable character in the 2016 video game Overwatch?\",\"correct_answer\":\"Invoker\",\"incorrect_answers\":[\"Mercy\",\"Winston\",\"Zenyatta\"]},{\"type\":\"multiple\",\"difficulty\":\"medium\",\"category\":\"Science: Computers\",\"question\":\"What does AD stand for in relation to Windows Operating Systems? \",\"correct_answer\":\"Active Directory\",\"incorrect_answers\":[\"Alternative Drive\",\"Automated Database\",\"Active Department\"]},{\"type\":\"multiple\",\"difficulty\":\"medium\",\"category\":\"Entertainment: Japanese Anime &amp; Manga\",\"question\":\"In &quot;Highschool DxD&quot;, Koneko Toujou is from what race?\",\"correct_answer\":\"Nekomata\",\"incorrect_answers\":[\"Kitsune\",\"Human\",\"Kappa\"]}]}");
-    //     String sessionId = quizService.createSession(settings);
-
-
-    //     int interval = 1000;
-    //     int maxTime = 5000;
-    //     long startTime = System.currentTimeMillis();
-    //     Session session = null;
-
-    //     while((System.currentTimeMillis() - maxTime) < startTime) {
-    //         Optional<Session> retrieved = quizService.checkSession(sessionId);
-    //         System.out.println(retrieved);
-    //         if (retrieved.isPresent()) {
-    //             System.out.print("Present!");
-    //             session = retrieved.get();
-    //             break;
-    //         } else {
-    //             try {
-    //                 Thread.sleep(interval);
-    //             } catch (InterruptedException e) {
-    //                 Thread.currentThread().interrupt(); // Restore interrupted status
-    //                 fail("Thread was interrupted during sleep");
-    //             }
-    //         }
-    //     }
-
-    //     assertNotEquals(session, null);
-    // }
-
-    // public void testCalculateScore(){
-
-    // }
-}

frontend/src/app/Quiz/models/category.ts

@@ -0,0 +1,4 @@
+export interface Category {
+    id: number;
+    name: string;
+}

frontend/src/app/Quiz/models/quiz-params.ts

@@ -1,5 +1,7 @@
+import { Category } from "./category";
+
 export interface QuizParams {
-    category: string;
+    category: number;
     type: string;
     amount: number;
     difficulty: string;

frontend/src/app/Quiz/services/quiz.service.ts

@@ -19,6 +19,7 @@ import { StorageService } from './storage.service';
 import { CleanerService } from './cleaner.service';
 import { ResponseHandlerService } from './response-handler.service';
 import { environment } from '../../../environments/environment';
+import { Category } from '../models/category';
 
 @Injectable({
   providedIn: 'root',
@@ -40,8 +41,7 @@ export class QuizService {
     }
     this.storageService.setSession(sessionIdObj.sessionId);
     const sessionObj = await lastValueFrom(this.pollQuestions(sessionIdObj.sessionId));
-
+    return this.reponseHandler.handleResponse(sessionObj.statusCode, sessionObj);
-    return this.reponseHandler.handleResponse(sessionIdObj.statusCode, sessionObj);
   }
 
   public pollQuestions(sessionId: string) {
@@ -67,7 +67,7 @@ export class QuizService {
 
   private convertParamsToUrl(quizParams: QuizParams) {
     let url = `?amount=${quizParams.amount}`;
-    const catIndex = this.getCategories().indexOf(quizParams.category);
+    const catIndex = quizParams.category;
     url += catIndex !== 0 ? `&category=${catIndex}` : '';
 
     const diff = quizParams.difficulty.toLowerCase();
@@ -83,34 +83,11 @@ export class QuizService {
     return url;
   }
 
-  public getCategories(): string[] {
+  public async getCategories(): Promise<Category[]> {
-    return [
+    let categories: Category[] = await firstValueFrom(this.http.get<any>(this.apiUrl + '/categories'));
-      'Any Category',
+    categories.push({id:0,name:"Any Category"});
-      'General Knowledge',
+    categories.sort((a,b) => a.id - b.id);
-      'Entertainment: Books',
+    return categories;
-      'Entertainment: Film',
-      'Entertainment: Music',
-      'Entertainment: Musicals & Theatres',
-      'Entertainment: Television',
-      'Entertainment: Video Games',
-      'Entertainment: Board Games',
-      'Science & Nature',
-      'Science: Computers',
-      'Science: Mathematics',
-      'Mythology',
-      'Sports',
-      'Geography',
-      'History',
-      'Politics',
-      'Art',
-      'Celebrities',
-      'Animals',
-      'Vehicles',
-      'Entertainment: Comics',
-      'Science: Gadgets',
-      'Entertainment: Japanese Anime & Manga',
-      'Entertainment: Cartoon & Animations',
-    ];
   }
 
   public getDifficulties(): string[] {

frontend/src/app/Quiz/setup/setup.component.html

@@ -5,7 +5,7 @@
         <div class="mb-4">
             <label for="category" class="block mb-1">Category</label>
             <select id="category" class="w-full p-2 border rounded" [(ngModel)]="category">
-                <option *ngFor="let category of quizService.getCategories()" [value]="category">{{ category }}</option>
+                <option *ngFor="let category of categories" [value]="category.id">{{ category.name }}</option>
             </select>
         </div>
 

frontend/src/app/Quiz/setup/setup.component.ts

@@ -7,6 +7,7 @@ import { QuizParams } from '../models/quiz-params';
 import { SpinnerService } from '../../Spinner/services/spinner.service';
 import { CommonModule } from '@angular/common';
 import { Question } from '../models/question';
+import { Category } from '../models/category';
 
 @Component({
   selector: 'app-setup',
@@ -17,8 +18,8 @@ import { Question } from '../models/question';
 })
 export class SetupComponent {
 
-
+  categories!: Category[];
-  category: string = "";
+  category!: number;
   difficulty: string = "";
   type: string = "";
   amount: number = 10;
@@ -28,10 +29,13 @@ export class SetupComponent {
     public quizService: QuizService,
     private spinnerService: SpinnerService
   ) {
-    /** Setting default values */
+    quizService.getCategories().then((categories: Category[]) => {
-    this.category = this.quizService.getCategories()[0];
+      /** Setting default values */
-    this.difficulty = this.quizService.getDifficulties()[0];
+      this.categories = categories;
-    this.type = this.quizService.getTypes()[0];
+      this.category = this.categories.find(item => item.id == 0)!.id;
+      this.difficulty = this.quizService.getDifficulties()[0];
+      this.type = this.quizService.getTypes()[0];      
+    });
   }
 
   startQuiz() {

scripts/LoadTestMultiple.sh

@@ -3,15 +3,14 @@
 start_time=$(date +%s)
 
 for i in {1..6}; do
-  ./LoadTestOne-WithParams.sh &
+  ./LoadTestOne-Silent.sh &
-  sleep 0.1
+#   sleep 0.1
 done
 
-wait  # Wait for all background jobs to finish
+wait
 
-end_time=$(date +%s)  # Capture the end time
+end_time=$(date +%s)
 
-# Calculate and print the elapsed time
 elapsed_time=$((end_time - start_time))
 echo "Done!"
 echo "Total time taken: $elapsed_time seconds"

scripts/LoadTestOne-Silent.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+# Step 1: Get the sessionId from the questions endpoint
+sessionIdJson=$(curl -s "http://localhost:9090/quiz/questions?amount=10&category=23&difficulty=medium&type=multiple")
+
+# Extract the status code
+statusCode=$(echo "$sessionIdJson" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+if [ "$statusCode" -ne 0 ]; then
+    exit 1
+fi
+
+# Extract the session ID
+sessionId=$(echo "$sessionIdJson" | grep -o '"sessionId":"[^"]*"' | sed 's/"sessionId":"//;s/"//')
+
+if [ -z "$sessionId" ]; then
+    exit 1
+fi
+
+# Step 2: Poll the status endpoint every 2 seconds
+while true; do
+    status_response=$(curl -s "http://localhost:9090/quiz/status/$sessionId")
+
+    if [ "$status_response" != "null" ]; then
+        # Extract the status code from the response
+        statusCode=$(echo "$status_response" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+        if [ "$statusCode" -ne 0 ]; then
+            exit 1
+        fi
+
+        # Check if the response is JSON (starts with '{')
+        if [[ $status_response == \{* ]]; then
+            break
+        fi
+    fi
+
+    sleep 2
+done

scripts/LoadTestOne-WithParams.sh

@@ -1,7 +1,6 @@
 #!/bin/bash
-
 # Step 1: Get the sessionId from the questions endpoint
-sessionIdJson=$(curl -s "http://localhost:9090/quiz/questions?amount=25&category=27&difficulty=medium&type=boolean")
+sessionIdJson=$(curl -s "http://localhost:9090/quiz/questions?amount=20&category=27&difficulty=medium&type=multiple")
 
 statusCode=$(echo "$sessionIdJson" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
 
@@ -22,8 +21,7 @@ echo "Received sessionId: $sessionId"
 # Step 2: Poll the status endpoint every 2 seconds
 while true; do
     status_response=$(curl -s http://localhost:9090/quiz/status/"$sessionId")
-
+    if [ "$status_response" != "null" ] && [ $? -eq 0 ]; then
-    if [ $? -eq 0 ]; then
 
         statusCode=$(echo "$status_response" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
 
@@ -34,7 +32,6 @@ while true; do
 
         # Check if the response is JSON (starts with '{')
         if [[ $status_response == \{* ]]; then
-#             echo "$statusCode"
             echo "$sessionId: done! status: $statusCode"
             break
         fi

scripts/LoadTestOne.sh

@@ -1,5 +1,4 @@
 #!/bin/bash
-
 # Step 1: Get the sessionId from the questions endpoint
 sessionIdJson=$(curl -s http://localhost:9090/quiz/questions?amount=5)
 
@@ -22,16 +21,22 @@ echo "Received sessionId: $sessionId"
 # Step 2: Poll the status endpoint every 2 seconds
 while true; do
     status_response=$(curl -s http://localhost:9090/quiz/status/"$sessionId")
+    if [ "$status_response" != "null" ] && [ $? -eq 0 ]; then
+
+        statusCode=$(echo "$status_response" | grep -o '"statusCode":[0-9]*' | sed 's/"statusCode"://')
+
+        if [ "$statusCode" -ne 0 ]; then
+            echo "Exiting: statusCode is $statusCode"
+            exit 1
+        fi
 
-    if [ $? -eq 0 ]; then
         # Check if the response is JSON (starts with '{')
         if [[ $status_response == \{* ]]; then
-            echo "$sessionId: done!"
+            echo "$sessionId: done! status: $statusCode"
             break
         fi
     else
-        echo "Status check failed, exiting..."
+        echo "Status check failed, retrying..."
-        exit 1
     fi
 
     sleep 2