Merge pull request #3 from Ninjaws/develop

[Updated] Reflection

Reflection.md

@@ -1,20 +1,20 @@
-# Issues
+# Reflection
 
-## Issues during testing
+## Issues during testing (October 30th)
 ### Scheduler
 I don't know if it was the scheduler specifically, but whenever I try to test it, both the cache and the queue seem to become empty after the first function is completed.
 I don't know how to solve this yet
 
 
-## Caching solution
+## Caching solution (October 30th)
 ### Lack of data
-Not being able to gather blocks of 50 is a huge issue. The idea of reusing blocks becomes less viable if I can just get a big chunk every time for others to use
+Not being able to gather blocks of 50 is a big issue. The idea of reusing blocks becomes less viable if I can't just get a big chunk every time for others to use.
 I can't force a big chunk, because it might just return a status 1, and then the user gets nothing.
 
 Perhaps I can separate the questions myself and make a collection, that i can build the requests from. However, for every quest tracking if someone has already seen is a huge pain.
 
 
-## Possible solution
+## Possible solution (October 31st)
 Cache all data, 50 at a time
 Store it in a database
 When its done, let the db make blocks for quick cache access
@@ -28,7 +28,7 @@ Then start replacing the cache items one at a time.
 
 Users shouldn't feel this at all, because the active sessions are stored in their own cache items
 
-# Progress so far
+# Progress so far (November 3rd)
 There only seem to be ~4100 items, making the collecting phase only last around 7 minutes.
 
 Swapping dbs is replaced with swapping tables, seems to work just as fine, yet much simpler to set up
@@ -36,3 +36,28 @@ Swapping dbs is replaced with swapping tables, seems to work just as fine, yet m
 TODO: Add a caching state, and start filling up each unique combination with the items from the db
 This should lead to performance gains
 
+## Possible future alternative
+In the current state (when the cache is done), the backend might hold data 3 times (Cache, active DB table, and background DB table during the night)
+
+### Purely Cached solution
+- Every night
+- Check for each category+difficulty combination how many questions there are (API has an endpoint for this)
+- Track which one you are on, and how many you have
+- Take blocks of 50, until the remaining items are less than 50, then get that exact amount
+- When they are gathered, either 
+    1. Divide them between 3 cache items: cat:dif:any/multiple/boolean 
+    - (Manually filter on the type, since the API sends questions with either option together), or 
+    2. In one like cat:dif:any and then filter it by hand when a user asks for it
+    - (This will replace the collection put there the night before)
+- Then move to the next one
+- Stop when the last combination has been collected
+
+#### When a user requests data: 
+- Get the items from the cache
+- Less items than requested: Inform the user to try a different combination
+- More items than requested: Randomize and select a subset
+
+### Tradeoffs
+- Advantage: Less total data held on the server
+
+- Downside: Slower gathering, since there are 25+3+4=300 unique combinations (each of which will end with a request of less than 50 items), it will require a decent number requests more than the bulk collecting that the DB does.