Ninjaws/Quiz
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #3 from Ninjaws/develop

Browse Source 
[Updated] Reflection
This commit is contained in:
Ninjaws
2024-11-03 14:29:37 +01:00
committed by GitHub
parent 1a52b7337c 2c2c561182
commit 2525dc8e76
1 changed files with 31 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Reflection.md
+31 -6
View File
@@ -1,20 +1,20 @@
# Issues # Reflection
## Issues during testing ## Issues during testing (October 30th)
### Scheduler ### Scheduler
I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed. I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed.
I don't know how to solve this yet I don't know how to solve this yet
## Caching solution ## Caching solution (October 30th)
### Lack of data ### Lack of data
Not being able to gather blocks of 50 is a huge issue. The idea of reusing blocks becomes less viable if I can just get a big chunk every time for others to use Not being able to gather blocks of 50 is a big issue. The idea of reusing blocks becomes less viable if I can't just get a big chunk every time for others to use.
I can't force a big chunk, because it might just return a status 1, and then the user gets nothing. I can't force a big chunk, because it might just return a status 1, and then the user gets nothing.
Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain. Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain.
## Possible solution ## Possible solution (October 31st)
Cache all data, 50 at a time Cache all data, 50 at a time
Store it in a database Store it in a database
When its done, let the db make blocks for quick cache access When its done, let the db make blocks for quick cache access
@@ -28,7 +28,7 @@ Then start replacing the cache items one at a time.
Users shouldn't feel this at all, because the active sessions are stored in their own cache items Users shouldn't feel this at all, because the active sessions are stored in their own cache items
# Progress so far # Progress so far (November 3rd)
There only seem to be ~4100 items, making the collecting phase only last around 7 minutes. There only seem to be ~4100 items, making the collecting phase only last around 7 minutes.
Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up
@@ -36,3 +36,28 @@ Swapping dbs is replaced with swapping tables, seems to work just as fine, yet m
TODO: Add a caching state, and start filling up each unique combination with the items from the db TODO: Add a caching state, and start filling up each unique combination with the items from the db
This should lead to performance gains This should lead to performance gains
## Possible future alternative
In the current state (when the cache is done), the backend might hold data 3 times (Cache, active DB table, and background DB table during the night)
### Purely Cached solution
- Every night
- Check for each category+difficulty combination how many questions there are (API has an endpoint for this)
- Track which one you are on, and how many you have
- Take blocks of 50, until the remaining items are less than 50, then get that exact amount
- When they are gathered, either
1. Divide them between 3 cache items: cat:dif:any/multiple/boolean
- (Manually filter on the type, since the API sends questions with either option together), or
2. In one like cat:dif:any and then filter it by hand when a user asks for it
- (This will replace the collection put there the night before)
- Then move to the next one
- Stop when the last combination has been collected
#### When a user requests data:
- Get the items from the cache
- Less items than requested: Inform the user to try a different combination
- More items than requested: Randomize and select a subset
### Tradeoffs
- Advantage: Less total data held on the server
- Downside: Slower gathering, since there are 25+3+4=300 unique combinations (each of which will end with a request of less than 50 items), it will require a decent number requests more than the bulk collecting that the DB does.