14 Commits
Author SHA1 Message Date
Ninjaws e1b04745b2 [Updated] Readme 2024-11-03 13:39:50 +01:00
Ninjaws b82dfdbe17 [Updated] Frontend now uses categories retrieved from the backend 2024-11-03 13:29:52 +01:00
Ninjaws c26c298a41 [Removed] Unused file 2024-11-03 01:44:47 +01:00
Ninjaws ec5a1de584 [Added] Nightly filling up of fresh data (and swapping), Startup filling of DB, sessionToken for bulk gathering 2024-11-03 01:43:47 +01:00
Ninjaws 121dff910d [Added] Functionality for getting data from the database [NOTE: swapping seems to difficult for now, might have to go for a cache approach] 2024-11-02 20:02:36 +01:00
Ninjaws ddc421b4b3 [Added] Start of the DataService (and data collecting system) 2024-11-01 22:22:27 +01:00
Ninjaws 595b83b364 [Updated] Dockerfiles so everything runs using docker compose up, added the benchmark and speed tests 2024-11-01 19:52:53 +01:00
Ninjaws 675bdc7f8e [Updated] Finished Frontend, Added statushandling to backend and frontend [Todo: session token handling at lvl 2] 2024-11-01 17:02:24 +01:00
Ninjaws 3188f6ce20 [Added] Most of the frontend, split up backend services 2024-10-31 23:05:22 +01:00
Ninjaws 2505fce920 [Added] Tests and some more functionality to the Services 2024-10-30 21:47:45 +01:00
Ninjaws bf66869ebf [Added] Basis for Backend 2024-10-29 21:21:14 +01:00
Ninjaws 1a992a871e [Added] Setup for running production in docker-compose 2024-10-28 22:58:56 +01:00
Ninjaws e62369533e [Added] Basis for the project 2024-10-28 21:33:39 +01:00
Ninjaws 13b1ad0c2e Initial commit 2024-10-28 11:49:02 +01:00
10 changed files with 10 additions and 139 deletions
-14
View File
@@ -1,14 +0,0 @@
name: Quiz App Deployment
run-name: ${{ gitea.actor }} is deploying Quiz App
on:
push:
branches:
- main
jobs:
Quiz Runner:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v4
- run: docker compose up -d --build
-13
View File
@@ -1,13 +0,0 @@
name: Quiz App Testing
run-name: ${{ gitea.actor }} is testing Quiz App
on: [push]
jobs:
Quiz Runner:
runs-on: ubuntu-latest
steps:
- run: docker --version
- run: docker compose version
- name: Check out repository code
uses: actions/checkout@v4
# Run tests
+6 -31
View File
@@ -1,20 +1,20 @@
# Reflection # Issues
## Issues during testing (October 30th) ## Issues during testing
### Scheduler ### Scheduler
I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed. I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed.
I don't know how to solve this yet I don't know how to solve this yet
## Caching solution (October 30th) ## Caching solution
### Lack of data ### Lack of data
Not being able to gather blocks of 50 is a big issue. The idea of reusing blocks becomes less viable if I can't just get a big chunk every time for others to use. Not being able to gather blocks of 50 is a huge issue. The idea of reusing blocks becomes less viable if I can just get a big chunk every time for others to use
I can't force a big chunk, because it might just return a status 1, and then the user gets nothing. I can't force a big chunk, because it might just return a status 1, and then the user gets nothing.
Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain. Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain.
## Possible solution (October 31st) ## Possible solution
Cache all data, 50 at a time Cache all data, 50 at a time
Store it in a database Store it in a database
When its done, let the db make blocks for quick cache access When its done, let the db make blocks for quick cache access
@@ -28,7 +28,7 @@ Then start replacing the cache items one at a time.
Users shouldn't feel this at all, because the active sessions are stored in their own cache items Users shouldn't feel this at all, because the active sessions are stored in their own cache items
# Progress so far (November 3rd) # Progress so far
There only seem to be ~4100 items, making the collecting phase only last around 7 minutes. There only seem to be ~4100 items, making the collecting phase only last around 7 minutes.
Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up
@@ -36,28 +36,3 @@ Swapping dbs is replaced with swapping tables, seems to work just as fine, yet m
TODO: Add a caching state, and start filling up each unique combination with the items from the db TODO: Add a caching state, and start filling up each unique combination with the items from the db
This should lead to performance gains This should lead to performance gains
## Possible future alternative
In the current state (when the cache is done), the backend might hold data 3 times (Cache, active DB table, and background DB table during the night)
### Purely Cached solution
- Every night
- Check for each category+difficulty combination how many questions there are (API has an endpoint for this)
- Track which one you are on, and how many you have
- Take blocks of 50, until the remaining items are less than 50, then get that exact amount
- When they are gathered, either
1. Divide them between 3 cache items: cat:dif:any/multiple/boolean
- (Manually filter on the type, since the API sends questions with either option together), or
2. In one like cat:dif:any and then filter it by hand when a user asks for it
- (This will replace the collection put there the night before)
- Then move to the next one
- Stop when the last combination has been collected
#### When a user requests data:
- Get the items from the cache
- Less items than requested: Inform the user to try a different combination
- More items than requested: Randomize and select a subset
### Tradeoffs
- Advantage: Less total data held on the server
- Downside: Slower gathering, since there are 25+3+4=300 unique combinations (each of which will end with a request of less than 50 items), it will require a decent number requests more than the bulk collecting that the DB does.
@@ -12,7 +12,7 @@ public class WebConfig implements WebMvcConfigurer {
@Override @Override
public void addCorsMappings(CorsRegistry registry) { public void addCorsMappings(CorsRegistry registry) {
registry.addMapping("/**") // Allow all endpoints registry.addMapping("/**") // Allow all endpoints
.allowedOrigins("http://localhost:4200", "https://quiz.ianvink.nl") // Allow specific origin .allowedOrigins("http://localhost:4200") // Allow specific origin
.allowedMethods("GET", "POST", "PUT", "DELETE", "OPTIONS") // Allow specific methods .allowedMethods("GET", "POST", "PUT", "DELETE", "OPTIONS") // Allow specific methods
.allowedHeaders("*") // Allow all headers .allowedHeaders("*") // Allow all headers
.allowCredentials(true); // Allow credentials .allowCredentials(true); // Allow credentials
@@ -1,5 +1,4 @@
spring.application.name=quiz spring.application.name=quiz
server.address=0.0.0.0
server.port=9090 server.port=9090
# H2 Database configuration # H2 Database configuration
-13
View File
@@ -8,11 +8,6 @@ services:
container_name: quiz-frontend-prod-cont container_name: quiz-frontend-prod-cont
ports: ports:
- "4200:80" - "4200:80"
networks:
- quiz-network
restart: always
cpu_period: 100000 # Default is 100,000 microseconds (100ms)
cpu_quota: 5000 # 5% of the total CPU time
backend: backend:
build: build:
context: ./backend context: ./backend
@@ -22,11 +17,3 @@ services:
container_name: quiz-backend-prod-cont container_name: quiz-backend-prod-cont
ports: ports:
- "9090:9090" - "9090:9090"
networks:
- quiz-network
restart: always
cpu_period: 100000 # Default is 100,000 microseconds (100ms)
cpu_quota: 5000 # 5% of the total CPU time
networks:
quiz-network: # Define the network
-1
View File
@@ -27,6 +27,5 @@ RUN ng build --configuration production
#--- Production #--- Production
FROM nginx:alpine AS production FROM nginx:alpine AS production
COPY --from=build /app/dist/quiz/browser /usr/share/nginx/html COPY --from=build /app/dist/quiz/browser /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80 EXPOSE 80
CMD ["nginx", "-g", "daemon off;"] CMD ["nginx", "-g", "daemon off;"]
-61
View File
@@ -1,61 +0,0 @@
server {
listen 80;
# server_name quiz.ianvink.nl;
location / {
root /usr/share/nginx/html;
index index.html;
try_files $uri $uri/ /index.html;
}
location /api/ {
proxy_set_header Connection '';
proxy_http_version 1.1;
proxy_read_timeout 360s;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# add_header 'Access-Control-Allow-Origin' '*'; # Adjust this to your frontend domain
# add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; # Include POST
# add_header 'Access-Control-Allow-Headers' 'Content-Type, Authorization'; # Include any other headers
proxy_pass http://backend:9090/;
}
}
# server {
# listen 80;
# server_name quiz.ianvink.nl;
# location /.well-known/acme-challenge/ {
# root /usr/share/nginx/html; # Adjust this if needed
# }
# location / {
# return 301 https://$host$request_uri;
# }
# }
# server {
# listen 443 ssl;
# server_name quiz.example.com;
# ssl_certificate /etc/letsencrypt/live/quiz.example.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/quiz.example.com/privkey.pem;
# location / {
# root /usr/share/nginx/html;
# index index.html;
# try_files $uri $uri/ /index.html;
# }
# location /api/ {
# proxy_pass http://backend:9090/; # Adjust this if needed
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# proxy_set_header X-Forwarded-Proto $scheme;
# }
# }
@@ -2,6 +2,5 @@ export const environment = {
production: true, production: true,
/** Connecting through the docker compose network is not working for now, so I'll have to use the localhost */ /** Connecting through the docker compose network is not working for now, so I'll have to use the localhost */
// apiUrl: 'http://backend:9090', // apiUrl: 'http://backend:9090',
apiUrl: '/api' //'https://api.nosotros.ianvink.nl', apiUrl: 'http://localhost:9090',
// apiUrl: 'http://localhost:9090',
}; };